Receiving big data over UART, sending it through USB CDC - missing data.

[bmak]

Hello.

So, my application contains of STM32F407VG and another CPU (OpenMV H7 if you are curious, but it doesn't matter) that is sending frame buffer from camera over UART. It isn't crucial in my application, but I just wanted to try doing that.

So, frame buffer is about 12 - 13 kB in size. I tried to find the best solution to transmit and receive this data. Here is what I did:

• OpenMV sends 1000 bytes (one byte from one write operation), then waits (about 500ms), then sends another 1000 bytes etc.
• STM32 works in DMA circural receive mode, interrupt occures when 2 bytes are ready to read. So I read 2 bytes, write it to my array, until I fill it with 1000 bytes. Then, when OpenMV is waiting, I write it to SD card (I don't do that in interrupt! I set a flag and do that in while loop!), clear array, receive next 1000 bytes.

This approach works well, written file is fine, none of the bytes are missing. If you have any suggestion on how receiving from UART could be done easier or better - feel free to tell me :).

My problems start when I connect my STM to computer throughout USB (which works in CDC and MSC mode).

My first test was to do the same, write the same file of received bytes, but when being plugged in. Sometimes it works, sometimes it don't. I have a feeling that it's the problem of timing, maybe my application works slower when being plugged, I don't know. When I look at written file I can see that sometimes it misses a few bytes - generally there is no rule, just random number of bytes at random parts of time.

My second test was not to write received 1000 bytes to SD card, but send it throughout USB CDC to computer. One time I managed to receive "something", because it wasn't a whole frame buffer, but just random pieces of it. But, like I said - it was just one time, the rest of my attempt didn't work at all, and I didn't receive anything.

Here is my UART config:

// main.c
static void MX_USART1_UART_Init(void)
{
  huart1.Instance = USART1;
  huart1.Init.BaudRate = 115200;
  huart1.Init.WordLength = UART_WORDLENGTH_8B;
  huart1.Init.StopBits = UART_STOPBITS_1;
  huart1.Init.Parity = UART_PARITY_NONE;
  huart1.Init.Mode = UART_MODE_TX_RX;
  huart1.Init.HwFlowCtl = UART_HWCONTROL_NONE;
  huart1.Init.OverSampling = UART_OVERSAMPLING_16;
  if (HAL_UART_Init(&huart1) != HAL_OK)
  {
    Error_Handler();
  }
}
 
//stm32f4xx_hal_msp.c
void HAL_UART_MspInit(UART_HandleTypeDef* huart)
{
  GPIO_InitTypeDef GPIO_InitStruct = {0};
  if(huart->Instance==USART1)
  {
    GPIO_InitStruct.Pin = GPIO_PIN_6|GPIO_PIN_7;
    GPIO_InitStruct.Mode = GPIO_MODE_AF_PP;
    GPIO_InitStruct.Pull = GPIO_NOPULL;
    GPIO_InitStruct.Speed = GPIO_SPEED_FREQ_VERY_HIGH;
    GPIO_InitStruct.Alternate = GPIO_AF7_USART1;
    HAL_GPIO_Init(GPIOB, &GPIO_InitStruct);
 
   hdma_usart1_rx.Instance = DMA2_Stream2;
   hdma_usart1_rx.Init.Channel = DMA_CHANNEL_4;
   hdma_usart1_rx.Init.Direction = DMA_PERIPH_TO_MEMORY;
   hdma_usart1_rx.Init.PeriphInc = DMA_PINC_DISABLE;
   hdma_usart1_rx.Init.MemInc = DMA_MINC_ENABLE;
   hdma_usart1_rx.Init.PeriphDataAlignment = DMA_PDATAALIGN_BYTE;
   hdma_usart1_rx.Init.MemDataAlignment = DMA_MDATAALIGN_BYTE;
   hdma_usart1_rx.Init.Mode = DMA_CIRCULAR;
   hdma_usart1_rx.Init.Priority = DMA_PRIORITY_LOW;
   hdma_usart1_rx.Init.FIFOMode = DMA_FIFOMODE_DISABLE;
   if (HAL_DMA_Init(&hdma_usart1_rx) != HAL_OK)
   {
	 Error_Handler();
   }
 
   __HAL_LINKDMA(huart,hdmarx,hdma_usart1_rx);
 
   hdma_usart1_tx.Instance = DMA2_Stream7;
   hdma_usart1_tx.Init.Channel = DMA_CHANNEL_4;
   hdma_usart1_tx.Init.Direction = DMA_MEMORY_TO_PERIPH;
   hdma_usart1_tx.Init.PeriphInc = DMA_PINC_DISABLE;
   hdma_usart1_tx.Init.MemInc = DMA_MINC_ENABLE;
   hdma_usart1_tx.Init.PeriphDataAlignment = DMA_PDATAALIGN_BYTE;
   hdma_usart1_tx.Init.MemDataAlignment = DMA_MDATAALIGN_BYTE;
   hdma_usart1_tx.Init.Mode = DMA_NORMAL;
   hdma_usart1_tx.Init.Priority = DMA_PRIORITY_LOW;
   hdma_usart1_tx.Init.FIFOMode = DMA_FIFOMODE_DISABLE;
   if (HAL_DMA_Init(&hdma_usart1_tx) != HAL_OK)
   {
	 Error_Handler();
   }
 
   __HAL_LINKDMA(huart,hdmatx,hdma_usart1_tx);
 
    HAL_NVIC_SetPriority(USART1_IRQn, 0, 0);
    HAL_NVIC_EnableIRQ(USART1_IRQn);
  }
 
}

Here is how I write to file or send through CDC in while loop:

if(write_photo){
         //write to file
	 f_write(&fil_log, (uint8_t*)photo, photo_indx + 2, &bw);
	 f_sync(&fil_log);
 
        //send to PC
        //CDC_Transmit_FS(uint8_t*)photo, photo_indx + 2);
	 
         write_photo = false;
	 memset(photo, 0, sizeof photo);
	photo_indx = 0;
 }

Here is interrupt routine for UART:

void HAL_UART_RxCpltCallback(UART_HandleTypeDef *huart){
	if(strcmp(mv_received, "ph") == 0){ //start of frame buffer
		photo_coming = true;
		size_coming = false;
		photo_indx = 0;
	}else if(strcmp(mv_received, "sp") == 0){ //end of frame buffer
		write_photo = true;
		photo_coming = false;
	}else{
		if(photo_coming){
			photo[photo_indx] = mv_received[0];
			photo[photo_indx + 1] = mv_received[1];
 
			if(photo_indx + 1 == 999){ // 1000s byte
				write_photo = true;
			}else{
				photo_indx += 2;
			}
		}
	}
 
	HAL_UART_Receive_DMA(&huart1, mv_received, 2); //listen for the next 2 bytes
 
}

So, did any of you also tried to send big data through USB CDC? Or receive big data through UART?

If you don't understand me or something isn't explained enough - feel free to ask. Also If you want to see any more of my code, please tell too - I'm not really sure of what you want to see.

Have a nice day! 🙂

[gbm]

Just one remark: CDC_Transmit_FS() must be called from the interrupt service routine of the same priority as USB interrupt. If you call it from any other context (like loop in main) it will fail sooner or later.

[Bob S]

Huh? What makes you say that? I haven't run USB on an F4xx in quite a while, but on the G4xx and my ancient F4xx code that is certainly not the case.

[gbm]

What makes me say that? The experience and detailed analysis of HAL code. Maybe the problem was corrected, but at least that's what it looked like a year ago. This is(was?) something very general for any USB device code generated by Cube - CDC, HID and others. USB HAL routines are/were not reentrant and the USB interrupt handler was of course using them, so attempting to do anything with USB in a code running at a lower priority and hence interrupted by USB interrupt sooner or later led to USB stack hangup or errors.

[Harvey White]

Just some general observations:

First: if you're using an RTOS, you must have semaphores to access things like uarts, because otherwise, in a general situation, you can have contention for the resource.

Secondly, you have a potential problem in synchronization. What I find best is to send a packet of information, then wait for some kind of ACK to be transmitted back. This synchronizes the packets on a software level. How do you know that the system is ready to listen. Note that this is a system/software thing.

Thirdly, if you can use RTS and CTS, the USART/UARTS finally have a FIFO, so you hardware wise sent bytes until the fifo is (half)full, then stop. If the other chip is slow, you want to use half full. This handles the hardware handshake. I think you need both.

As an example, if you're sending data to a UART, and the receiving system is out to lunch, then you lose data. Your single character ack/nack code can request a block of data where the checksum didn't match. You are using checksums?

Perhaps you just need to make the communications software a bit more robust?

[bmak]

Well, I used CDC_Transmit_FS() in while loop, it always worked. But, maybe it can be a thing, could you please send me some kind of source of this information so I could read about it more? Thanks!

[bmak]

I'm not using RTOS.

I will try to use FIFO and checksums, didn't think about that one.

Thanks!

[Harvey White]

using an RTOS just complicates the process, although once you figure in the required steps, it becomes simple enough.

The FIFO should be built into the UART/USART and should be enabled. I'd set the threshold to 1/2. But you really need flow control, either hardware if possible, or software, if supported. The need for the FIFO is that the action of the RTS/CTS or software control may not be instantaneous, and may fall in the middle of a transmission. Having the fifo send out the "stop, no more for now" signals says that there will always be a space in the FIFO for that data.

Since you're not using an RTOS, then the interrupts ought to be always on. In that case, you could try the UART/USART with interrupts and DMA building up a circular (if you want) buffer.

[gbm]

Well, you have all the source (code). Some low level USB routines perform single bit modifications of USB peripheral registers, implemented with logic operations on USB registers. These routines are called by Transmit and by USB ISR. It's easy to imagine what happens if the interrupt is serviced in the middle of such bit modification called from Transmit.