Fast data logging with STM32F4 using USB_OTG_FS and FatFS

[GDall.387]

Hi everyone,

I'm trying to log data from different sensor into a USB flash drive with a 500Hz sampling frequency.

I set up a timer to define the sampling frequency for data logging and everything seems to work fine.

The problems comes when i try to write the data to a log file (I've tryed txt, csv and bin, but it doesn't seems to make a difference for FatFs) beacuse the writing instruction introduces a irregoular delay.

So far i've tried:

• buffering data in different way to match sector size
• formatting usb disk in different way (FAT32, FAT16, exFAT)
• cleaning code as much as possible to just do the writing operation while logging

Does anyone know a method to speed up writing operation?

Thank you

Here's the main:

 if(eff==1){
 
        	if(fs==1){
       	          HAL_GPIO_WritePin(LED2_GPIO_Port, LED2_Pin, RESET);
       	          myData = LIS3DSH_GetDataScaled();
       	       	  x=(int)(myData.x*9806.65);
       	       	  acc_raw[k]=x;
       	       	  k++;
       	       	  y=(int)(myData.y*9806.65);
       	       	  acc_raw[k]=y;
       	          k++;
       	       	  z=(int)(myData.z*9806.65);
       	       	  acc_raw[k]=z;
       	       	  k++;
       	       	  HAL_GPIO_WritePin(LED2_GPIO_Port, LED2_Pin,  SET);
       	          fs=0;
       	          }
  
        		if(k==99){
        			s++;
        			for(int g=0; g<98; g=g+3){
 
        			f_printf(&myFilea, "%d : %d, %d, %d\n", s, acc_raw[g], acc_raw[g+1], acc_raw[g+2]  );
        				
        			}
       			k=0;
        		bufclear1();
 
        		}
 
 
 if(HAL_GPIO_ReadPin(GPIOA, GPIO_PIN_0) == GPIO_PIN_SET && write_var==0){
 
                         	 i=0;
                         	write_var=1;
                          	 while(deb<250){
 
                          	 	 }
 
                          	 eff=1;
                           	  }
 
  if(one_sec==500 && write_var==1){
          	  	  	  	 for(int g=0; g<98; g=g+3){
 
          f_printf(&myFilea, "%d : %d, %d, %d\n", s+1, acc_raw[g], acc_raw[g+1], acc_raw[g+2]  );
          	          				//g=g+2;
          	          			}
                      	 HAL_GPIO_WritePin(LED2_GPIO_Port, LED2_Pin, GPIO_PIN_RESET);
                      	 HAL_GPIO_WritePin(LED1_GPIO_Port, LED1_Pin, GPIO_PIN_RESET);
                      	 HAL_GPIO_WritePin(LED4_GPIO_Port, LED4_Pin, GPIO_PIN_RESET);
                      	 f_close(&myFilea);
                      	 f_close(&myFilet);
                      	 write_var=0;
                      	 s=0;
                      	 deb=0;
                      	 eff=0;
                      	 open=0;
                      	 one_sec=0;
                      	 acqu_count++;
                      	 HAL_Delay(500);
                       	  }
       if(Appli_state == APPLICATION_DISCONNECT){
            	  HAL_GPIO_WritePin(LED1_GPIO_Port, LED1_Pin, GPIO_PIN_RESET);
            	  HAL_GPIO_WritePin(LED2_GPIO_Port, LED2_Pin, GPIO_PIN_RESET);
            	  open=0;
              }
      	 	
       if(open==0){
      	 	    switch(Appli_state){
      	 	    case APPLICATION_IDLE:
      	 		break;
      	 	    case APPLICATION_START:
      	 	    	res=f_mount(&myUsbFatFS, (TCHAR const*)USBHPath, 0);
      	 	    	if( res != FR_OK)
      	 	    					{
      	 	    						/* FatFs Initialization Error */
      	 	    						Error_Handler();
      	 	    					}
      	 	    					else
      	 	    					{
      	 	    						//HAL_GPIO_WritePin(LED1_GPIO_Port, LED1_Pin, GPIO_PIN_SET);
      	 	    					}
      	 	    					break;
      	 	    case APPLICATION_READY:
      	 	    					sprintf (buffer, "A%d.bin",acqu_count);
      	 	    					res=f_open(&myFilea, buffer, FA_OPEN_APPEND | FA_WRITE );
      	 	    					bufclear();
      	 	    					open=1;
      	  	 	    				HAL_GPIO_WritePin(LED1_GPIO_Port, LED1_Pin, GPIO_PIN_SET );
 
      	 	    					//HAL_Delay(500);
      	 	    				//}
 
 
      	 	   	break;
      	 	    case APPLICATION_DISCONNECT:
      	 	    	  //tUTRN GREEN ON
      	 	    	   HAL_GPIO_WritePin(LED1_GPIO_Port, LED1_Pin, GPIO_PIN_RESET);
      	 	   	break;
      	 	    }
      	 	  }
      }
 
 

Here's the Timer interrupt handler:

void HAL_TIM_PeriodElapsedCallback(TIM_HandleTypeDef *htim)
{
 /* Prevent unused argument(s) compilation warning */
		HAL_GPIO_TogglePin(LED3_GPIO_Port, LED3_Pin);
 
		if(eff==1){
			one_sec++;
		}
 
		if(write_var==1){
		deb++;
		fs=1;
		if(temp_counter <250) temp_counter++;
		else {
			fs_temp=1;
			temp_counter=0;
		}
		//i=1-i;
		}
}

[Tesla DeLorean]

Not sure how your states advance here, or what you do to prevent multiple f_mount or f_open calls.

Using any f_printf() or f_write() with small bursts of data is going to be brutal. I personally would shoot for 8KB or 32KB

If you can't handle significant hits in your loop you're going to have to buffer more data, and separate the writing into a worker task/thread which has more immunity in the ebb/flow

[Bill Dempsey]

Here are a couple of things to try to learn your way through this issue even though it seems like you've tried some of this already. Set up a simple write and read test by writing a large amount of data made up of smaller blocks such as (512B, 1024B, 2048B, etc) and then read them back using the same block size. Do this as a fully dedicated loop and measure the time it takes to write/read all the data and you'll get an average write/read speed from your sdcard (hint: this code exists everywhere on the net...). You're going to find there is a "sweet" spot for the block size that works best for you. Then play around with the FatFS "conf" file and change when FatFS syncs (sector or cluster). Rerun the speed test and you'll find a difference in performance when you change this setting.

By writing the "petite" blocks that you are doing that don't fit a sector you are forcing a whole bunch of data to be read from the flash, altered, and then be written back. This is going to hurt the overall performance. Also, when the flash controller has to switch sectors or clusters you'll find the writes can stall for up to 10x or more normal speed sometimes depending on what is already out on disk and if the controller is attempting to erase and write all behind the scenes.

(There is also a way to disable write verification on transfer which will also speed things up...can't remember where that lives tho!)

What you're going to find is that you really need to manage your data in larger chunks. Maybe you can't take the risk of losing data between writes due to power fails so you will end up paying a penalty for writing all the time. That's a system design issue... Be sure to also examine a large amount of write times. I bet you'll find that 512B or larger takes more than 2ms (500Hz rate) so you will need to buffer data significantly. If you're using SPI you might need to crank up the clock speed to transfer to the SD internal write buffer. That is only one part of the write speed issue but it does help.

Out of curiosity, why are you doing the float multiply against a constant for the accelerometer data? Why not read the signed integer data from the LIS3DSH and store it "raw"? Then in post-processing do any conversion. It won't affect you much right now with this small code size but in a busy system it doesn't make sense to do multiplies on well-known data that can be converted later.

[GDall.387]

Thank you both for the answers.

@Community member​ What do you mean by " separating the writing into a worker thread" ? Is there a specific way to do it?

@Bill Dempsey​ Just to check if procedure is correct, to bufferize writing i set un a char array with the variable dimensions (512 up to 32k) and while reading data from the accelerometer i sprintf it to the string, until it's full of data and the i write the entire char array into SD. Is this the correct procedure to bufferize?

The fact that i'm multiplying accelerometers data was beacause of limit of f_printf in printing float, i should modify that part of the code as you say, thanks

[Bill Dempsey]

I think you have the gist of it but understand you need a data "capture" and a data "write" buffer. With two equal-sized memory arrays it's easy to swap back and forth while writing and capturing. The ISR will point to one buffer while the main loop points to the other...

Regarding the side question -- I was really talking about the *need* for the mulitply - there's no need if both sides using the data know its relative value. Sure it's convenient but if you are trying to write only "usable" data to the SD card you can just store the 16-bit data for X,Y,Z instead of multiplying it and then storing it as an ASCII converted string. I know you're nowhere near the processing limits of the micro you're using but think about that in the future when you might be hurting for cpu cycles.

[GDall.387]

@Bill Dempsey​  I managed to reach desired writing speed (and even go further) with your suggestion, but now i'm facing another problem.

Basically i'm reading two different sensor (one accelerometer and one thermocouple) and i fill up a buffer in the interrupt routine of the timer, for then copying it to another buffer when full and then write the last one to the USB stick while the first one can be filled again. Everything seems to run fine, but after about 3/4 minutes it goeas into Hard Fault handler.

After investigating a bit, it's seems like a stack overflow problem. I've tried increasing stack size, but without any results.

Maybe your experience could help me resolving even this problem.

Thank you very much

  while (1)
  {
    /* USER CODE END WHILE */
 
    MX_USB_HOST_Process();
 
 
 
 
 
    /* USER CODE BEGIN 3 */
 
 
        	if(fs==1 && eff==1){
       	          HAL_GPIO_WritePin(LED2_GPIO_Port, LED2_Pin, RESET);
       	          myData = LIS3DSH_GetDataScaled();
       	       	  HAL_GPIO_WritePin(LED2_GPIO_Port, LED2_Pin,  SET);
       	          fs=0;
       	          }
 
        //Writing to file as buffer is full
        		if(k==100){
        				k=0;
        				memcpy(USB_buff_write, USB_buff_read, sizeof(USB_buff_read));
        				clearUSB_L(USB_buff_read);
        				f_write(&myFilea, USB_buff_write, strlen(USB_buff_write), NULL);
        				//f_sync(&myFilea);
        				clearUSB_L(USB_buff_write);
 
        					}
 
        		if(t==150){
        			 t=0;
        		     memcpy(USB_buff_writet, USB_buff_readt, sizeof(USB_buff_readt));
        		     clearUSB_S(USB_buff_readt);
        		     f_write(&myFilet, USB_buff_writet, strlen(USB_buff_writet), NULL);
        		     clearUSB_S(USB_buff_writet);
 
        		        	}
 
       	if(fs_temp==1 && eff==1){
       		HAL_GPIO_TogglePin(LED4_GPIO_Port, LED4_Pin);
       		redtemp=(float)readData()*0.25;
 
       	    fs_temp=0;
       	    t++;
       		}

And the Handler routine:

void HAL_TIM_PeriodElapsedCallback(TIM_HandleTypeDef *htim)
{
 /* Prevent unused argument(s) compilation warning */
		HAL_GPIO_TogglePin(LED3_GPIO_Port, LED3_Pin);
 
		if(eff==1){
			sprintf(USB_buff_read, "%s%f, %f, %f\n",USB_buff_read, myData.x, myData.y, myData.z );
			k++;
			//one_sec++;
 
		}
 
		if(temp_counter < 250) temp_counter++;
						else {
							fs_temp=1;
							sprintf(USB_buff_readt, "%s%.2f\n",USB_buff_readt, redtemp );
							temp_counter=0;
						}
 
		fs=1;
}

Thanks

[Bill Dempsey]

Definitely don't want to do a memcpy approach. Use two buffers (for each device)

BUFFER_A

BUFFER_B

Capture to A while writing B to SDCARD

when capture A buffer full then

Capture to B while writing A to SDCARD

and then start over

We call these "ping-pong" buffers. There of course is some init logic and flags that wrap around the code so you know when the buffers are valid but no need to copy.

And if the callback is at the interrupt level, which I suspect it is, there may be an issue with sprintf in interrupts...that is something I would check with the community. I personally have been bitten by any "printf" in an ISR so I never use them.

[Tesla DeLorean]

strlen() requires buffers that are properly terminated with NUL characters.

sprintf() returns a length.

Look at what specifically is Hard Faulting (disassembled instructions and registers), try using a proper fault handler rather than a while(1) loop. Posted examples several times.

[GDall.387]

Thanks for tour time and effort trying to help me

@Bill Dempsey​ 

The aim of what i was doing was exactly a ping-pong kind of thing. Before "sprintfing" in the ISR i've tried doing the normal procedure for ping pong buffering, but i had the same problem as the f_write command when writing one buffer, ​was delaying the data acquisition in the other one in the same way as without ping pong, while i found out that in the ISR the buffer Is filled even if a write instruction is taking more time than my sampling period. I'will try again with a better code, maybe i was missing something the last time i tried.

Basically what i need to in pills is:

- Using the ISR Just ti raise the flag timing the sampling instant ​

- In the main, whenever a sampling flag Is raised, filling One buffer

-As soon as One Is full, swap the buffers and write the full one ti the USB, hoping the other one is still filled while the full one is written

Am i right? ​

​@Community member​ 

From the brief of the sprintf function It Is actually terminating the string with a null character. I'also tried strcpy() with the same result

From the disassembly i can't find the real cause of the Hard Fault, i'm gonna try Better

Thanks ​

[Bill Dempsey]

Hmmm...can you publish the write-speed findings when you did the earlier tests I suggested? That would help understand the true throughput of the system. There can't be any "hope" as part of it working.

From the code I saw, it did not appear to be "ping-pong" style without copies...perhaps that code stayed on the bench?

Remember this comment?

"you will end up paying a penalty for writing all the time. That's a system design issue... Be sure to also examine a large amount of write times. I bet you'll find that 512B or larger takes more than 2ms (500Hz rate) so you will need to buffer data significantly"

If you are pulling in more data than you are writing you are going to have a collision. You have to decouple the input from the output. Setting buffer size and determining system requirements is part of what seems to be missing. If you have the calculations you're using (bytes/sec inbound, bytes/sec outbound) and you'd like to share, that'd be a better starting point than why did my code hard-fault?

The input is coming from an ISR. You are burning around 32 bytes with your formatted string per capture. I'll assume you did a 512B buffer. That means you need to write after every 16th ISR. So in your case at 500Hz you need to write every 32ms. Well, if you go back to your write-evaluation test and really dig-in you're going to find that *on occasion* the SD card writes of 512B buffer actually take longer than 32ms as some behind the scenes FAT table updates are getting made. That then implies that a two-buffer system will NOT work. So now the problem becomes perhaps a 3-buffer solution? Or 4?

Once you can *guarantee* that the system I/O rate is balanced you don't need to hope. I, like others on here, do the same kind of system you imply you are working on and it runs "forever" (assuming I don't run out of memory on the SDCARD). I have a multi-buffer "FIFO" implemented in software that helps average my slow writes with the fast ones. I made sure that my average write speed was >> than my input read speed. The rest is easy.

One final comment: make sure that the ISR write does not over-write its buffer. I did not see where you were tracking the remaining buffer size vs the input capture size. The sprintf will blow across a memory boundary as it has no bounds-checking. If you've made sure that this can't happen then good, it's one less thing to worry about.