receive unknown length frames with UART

[Adam98]

Hello everyone,

I’m working on an embedded project where I need to receive a variable-length stream of bytes via UART on an MCU  STM32H750. The key points are:

• All bytes are payload, no start or stop bytes, no protocol framing.

• The number of bytes is undefined and variable.

• The byte stream starts when the MCU is ready.

• I need to calculate CRC (Ethernet CRC32) over all received bytes.

• After receiving all data, I must output the CRC via UART.

• The UART baud rate is very high (greater than 900,000 baud).

My current approach:

• I use UART interrupts to receive bytes one by one.

• I update the CRC continuously with every received byte inside the interrupt handler.

• This works fine for small data, and CRC calculation seems correct.

• However, when I try to receive a large file or big data stream, the CRC isnt correct, the crc will be outputed before the whole data recieved.

I suspect the problem is related to handling the continuous stream without any start/stop or length information, and possibly buffer management or interrupt handling at such a high baud rate.

My questions:

1. How can I reliably receive an undefined-length UART stream without framing or start/stop bytes?

2. What techniques can I use to detect the end of transmission if there is no protocol?

3. Are there best practices for handling UART reception and CRC calculation in an interrupt-driven system at very high baud rates?

4. How can I avoid losing bytes or corruption when the incoming data size is large?

5. Any example code snippets or references for a similar use case would be great!

6. I try to use dma (ping pong buffer and update crc but it dosent work because i cant find the best time and place to calculate crc...)

Thanks a lot for your help!

[TDK · Accepted solution]

Use HAL_UARTEx_ReceiveToIdle_DMA and use the IDLE event to know when a frame stops.

STM32CubeF4/Projects/STM32446E-Nucleo/Examples/UART/UART_ReceptionToIdle_CircularDMA/Src/main.c at 0c04cc8f2360e54ced149afb43195a03f529a1c5 · STMicroelectronics/STM32CubeF4

[Pavel A.]

> After receiving all data, I must output the CRC via UART.

How do you decide that all data is received?  

Do you need to calculate the CRC over *all* data or over each "stream" or how you define the "units" of the data?

STM32H7 can handle continuous UART RX interrupts at ~ 100 per ms, with properly written code (not the "HAL" library).  Enabling the UART FIFO can help as well. Using the CRC accelerator of STM32 can help, if the CRC algorithm is suitable for you.  DMA [with timeout/idle detection] seems more attractive, but more complicated.

> How can I avoid losing bytes or corruption when the incoming data size is large?

An excellent question. The "streams" should not be too long, exactly with purpose of lessening chance of loss or corruption within single "stream" and minimize retransmissions.  This cannot be avoided and should be handled properly.

[Karl Yamashita]

To get a better understanding of what is going on...

• If the packet has no protocol or some kind of identifier, how do you know which packet is what?
• What are the purpose of these variable length packets after you calculate the CRC and send it out over UART?
• Is it possible that the sending device may pause in the middle of sending the large packet?
• How often are the packets being sent?
• What are some examples size of these large packets?

My questions:

1. How can I reliably receive an undefined-length UART stream without framing or start/stop bytes? Use DMA

2. What techniques can I use to detect the end of transmission if there is no protocol? Use DMA with idle.

3. Are there best practices for handling UART reception and CRC calculation in an interrupt-driven system at very high baud rates? Don't calculate in the interrupt/callback

4. How can I avoid losing bytes or corruption when the incoming data size is large? Use DMA

5. Any example code snippets or references for a similar use case would be great! 

6. I try to use dma (ping pong buffer and update crc but it dosent work because i cant find the best time and place to calculate crc...) You need to save the DMA data to a queue buffer as the TC and HT occur. When you get an idle interrupt, you can increment the queue pointer. So as you're calculating the CRC on the 1st queue in the main while loop, any new DMA data will be saved to the next queue. That way you don't miss any incoming data as you're incrementing through the current queue and calculating the CRC

---

[Adam98]

thank you alll

The Programm is now working using HAL_UARTEx_ReceiveToIdle_DMA and use the IDLE event to know when a frame stops.

but another problem that my programm works only with baud rates until 460800. if i want to use 921600 baud rates it dosent work properly and the value of the CRC is wrong!!!

[Tesla DeLorean]

Perhaps look at WHY the CRC is wrong..

Which data bytes did you receive that were incorrect or missing?

What's the aversion to any kind of protocol or framing? Something like SLIP is a very simple method to identify beginning and ending frames, whilst still permitting any 8-bit byte value to get from one end to the other.

Other protocols, perhaps use preambles and lengths. What about multi-controller / 9-bit methods for identifying data vs control bytes?

Using inter-symbol or inter-packet gaps that you can generate and detect effectively?

[Karl Yamashita]

How many bytes are you talking about?

[Adam98]

Data can be in KB or MB , the goal here that my application could read data streams/ files from Internet in KB or also MB… and calculate the CRC32 ccording to Ethernet and send it back with UART. My code works well using 115200 Baudrates but using 921600 data will be losts because maybe it is too fast to recieve this data

[Karl Yamashita]

Ahh, a file makes sense now. Binary and/or text file? What would be considered a stream?

For baud 921600, It depends on the % error for either devices connected. You might want to check the data sheet to see if there is a formula to calculate a % error. Clock frequency for USART1 will play part in the % error. 

It looks like the H750 only has 1024KB of ram. So if you're receiving packets in the MB, it would mean you're not storing the data for later use? Instead, you only want to calculate the checksum as data is being received and discard the data, right?  

[Adam98]

Yes I send File with Tera Term using COM PORT (BINARY !)

I didn't understand what you meant with the error?

Yes I just use RX callback functions to copy the update the actual CRC and in the main I output this actual value until the end of the file, so the last value is my Final CRC. I will send here my solution in text file. I really need help for this problem.