FatFs Writing Cmds(f_printf) Slows Down ADC

[ADynr.1]

Hello,

I'm using SDMMC to save ADC DMA data to an SD card on Nucleo H723ZG. I made some tests and got distorted results. Then I realized a problem, sampling speed is too slow when I use fatfs writing commands. What might be the problem?

I use a 64K buffer for ADC.

The code for the writing process with half and full callback:

/* USER CODE BEGIN 4 */
void HAL_ADC_ConvHalfCpltCallback(ADC_HandleTypeDef* hadc) {
		for (int i = 0; i < ADC_BUF_LEN/2; i++) {
		fres = f_printf(&fil, "%d\n", adc_buf[i]);
		}
}
void HAL_ADC_ConvCpltCallback(ADC_HandleTypeDef* hadc) {
	HAL_GPIO_TogglePin(GPIOB, GPIO_PIN_11);
	for (int i = ADC_BUF_LEN/2; i < ADC_BUF_LEN; i++) {
		fres = f_printf(&fil, "%d\n", adc_buf[i]);
	}
	if(HAL_GPIO_ReadPin(B1_GPIO_Port, B1_Pin)==GPIO_PIN_SET){  //Check if button pressed
			      f_close(&fil);
			      f_mount(NULL, "", 0);
			      myprintf("closed\r\n");
		  }
}
/* USER CODE END 4 */

The first pic is from full callback without fatfs write cmds:

 

The second pic is with fatfs write cmds.

 

Another problem is it gets much slower if f_close is not done yet.

[Tesla DeLorean]

>>What might be the problem?

Doing things that block in interrupt/callback context.

Doing thousands of small writes in human/ASCII form?

If you hope to stream, perhaps save the 32KB buffer sections, in binary, in the foreground as they fill.

If you need ASCII text, perhaps use an sprintf() and fill an entire buffer that's some multiple of the underlying sectors/clusters. Say a 16KB spill-buffer, or as big as you can afford.

[ADynr.1]

Thank you so much for your answer.

If I can convert from binary to ASCII later on PC, binary is also okay.

I was thinking starting the writing process in a callback context should work in the background until the next callback comes. In this way, ADC won't stop and data would be continuously saved.

Could you please provide some codes for that you mentioned? Because I'm very new on this board and struggling to find out which and how I should use it. How can I save in binary and foreground as they fill?

For ASCII, isn't it similar to what I used?

[Tesla DeLorean]

>>For ASCII, isn't it similar to what I used?

If you have to use ASCII, call sprintf() into a buffer 16000 times, and f_write() once, or twice.

You're calling f_printf() / f_write() 16000 times which is going to continuously churn the file system with no buffering or caching. And take a long time

>>Could you please provide some codes for that you mentioned?

Understand that I don't have hours to write code for you, jobs in this field aren't typically about copy-n-pasting solutions others have come up with.

You need to think about process and flow, and how to optimize the things which are particularly slow. And slowness that compounds with repetitive calling.

You're trying to move a large volume of data, don't use a small cup, use a large bucket, make less trips.

Callback/interrupt context functions need to be quick, otherwise you're going to block all the other interrupts with equal or lesser priority.

Flag you've got 32KB of data in the ping or pong buffer, and have your loop in main() do the saving. It could be another thread or task, it just needs to be disconnected. The time between callbacks defines your window to get the job done. FatFs / SDIO can't do concurrent operations, you need to serialize the operation. In the model you're using perhaps the main() foreground loop is the best place to do that.

sprintf() returns the length of the string printed. You can use that to advance a pointer or index, without using strlen() or guessing.

A spill-buffer is one that's slightly larger than you need, you need to be writing a large aligned block to the file system, unaligned blocks will require read/write operations to stitch in the new data. So you write 16 KB with f_write(), and as you don't know where the boundary for the last sprintf() falls you have a bigger buffer to manage the spanning case, you write the bulk portion, and then pull-down the overflow to the front of the next buffer, and continue. When done you flush any pending portion of the buffer with a final f_write() and then f_close()

[Tesla DeLorean]

>>Because I'm very new on this board and struggling to find out which and how I should use it. 

The problem is less about the board, but how you approach the task.

With block storage devices any interaction below the block/sector size is going to take a lot of time and processing to manage. Anything that's not a multiple of the block/sector size, or spans groups of them has to be managed.

[ADynr.1]

Hi @Community member​ @AScha.3​ and others,

I'm trying a new algorithm regarding your suggestions but I have a problem. I'm using this code but It saves the first 64000 samples, then it saves "0" for the next 64000. Then again it saves 64000 samples. What might be the problem? I decreased the ADC speed so much to test, but nothing changed. I guess it is not related to writing speed.

 

#define ADC_BUF_LEN 64000
#define SPILL_BUF_SIZE 64000 // 16 KB spill buffer
char spill_buf[SPILL_BUF_SIZE];
int spill_buf_idx = 0;
 
uint32_t last_write_time = 0;
uint32_t max_write_interval = 0; // Maximum time between writes (in microseconds)
uint32_t adc_buf[ADC_BUF_LEN];
 
void write_to_file(char* data, int len) {
  int written_len = 0;
  while (written_len < len) {
    int remaining_len = len - written_len;
    int copy_len = remaining_len;
    if (spill_buf_idx + copy_len > SPILL_BUF_SIZE) {
      // Spill buffer is full, write as much as possible and flush
      copy_len = SPILL_BUF_SIZE - spill_buf_idx;
      memcpy(spill_buf + spill_buf_idx, data + written_len, copy_len);
      uint32_t start_time = HAL_GetTick(); // Start measuring time for write operation
      f_write(&fil, spill_buf, SPILL_BUF_SIZE, &fres);
      uint32_t end_time = HAL_GetTick(); // End measuring time for write operation
      uint32_t write_interval = (end_time - start_time) * 1000; // Convert time to microseconds
      if (write_interval > max_write_interval) {
        max_write_interval = write_interval;
      }
      if (fres != FR_OK) {
        // Handle error
      }
      spill_buf_idx = 0;
      last_write_time = end_time;
    } else {
      // Copy data into spill buffer
      memcpy(spill_buf + spill_buf_idx, data + written_len, copy_len);
      spill_buf_idx += copy_len;
    }
    written_len += copy_len;
  }
  // If the time between writes is too long, notify over the serial port
  if (HAL_GetTick() - last_write_time > max_write_interval * ADC_BUF_LEN) {
    myprintf("Warning: Write speed may not be sufficient for the ADC sampling rate.\r\n");
  }
}
 
//void HAL_ADC_ConvHalfCpltCallback(ADC_HandleTypeDef* hadc) {
	// Write the first half of the buffer to the spill buffer
	//	write_to_file((char*) adc_buf, ADC_BUF_LEN/2 * sizeof(int));
//}
// In the ADC conversion complete callback:
void HAL_ADC_ConvCpltCallback(ADC_HandleTypeDef* hadc) {
	// Write the first half of the buffer to the spill buffer
	write_to_file((char*) adc_buf, ADC_BUF_LEN/2 * sizeof(int));
	// Write the second half of the buffer to the spill buffer
	write_to_file((char*) adc_buf + ADC_BUF_LEN/2 * sizeof(int), ADC_BUF_LEN/2 * sizeof(int));
	if(HAL_GPIO_ReadPin(B1_GPIO_Port, B1_Pin) == GPIO_PIN_SET) {
		// Button is pressed, flush the remaining data and close the file
		if (spill_buf_idx > 0) {
			f_write(&fil, spill_buf, spill_buf_idx, &fres);
			if (fres != FR_OK) {
				// Handle error
				}
	}
		f_close(&fil);
		f_mount(NULL, "", 0);
		myprintf("done\r\n");
	}
}

[AScha.3]

>I guess it is not related to writing speed.

me too. 🙂

pointer wrong...or ?? seems 1x 64KB ok, then 64KB zero.

what speed you can get and write (in 64 KB with data)?

[ADynr.1]

Unfortunately, the sd card can not work over 32Mhz SDMMMC, but output seems fine in 14bits while 16bits results are distorted. (in saved 64k samples, there are still 0s)

Do you mean "\r\n" by pointers? It's saving in binary, I don't know how to add pointers to that?

Also, I don't know how to measure speed of writing. Is there a way to that? I wanted to that, but couldn't find an understandable example for me. Should I use from beginning to fclose? Then reading the timer value from serial port, and comparing total sample numbers in sd card?

[ADynr.1]

Thank you for the answer.

I have found that writing data to sd card stage was slowing down the ADC.

I​ tried smaller buffer sizes too, but the current problem(first 64k ok, then 0s)still exists.

I am using DMA ADC.

I am using fwrite to get max speed now, do you think that fprintf would be faster?

I will try to use interrupts. Do I really need that? Because DMA ADC fills the buffer, the cpu just needs to write data to sd card.

[AScha.3]

pointers > in : write_to_file((char*) adc_buf, .../ + save binary is correct.

speed test: you make 32000 samples at 1Ms , so this should need 32ms . ->

so have 32ms to write to sdcard. (but should be shorter, to have some safety margin.)-> must have more than 2MB/s , that should be possible at 32MHz + 4 bit mode .

for testing, set a var with systick, and read in debug mode at breakpoint after the save action.; or (i do most time) set a port pin hi at begin and lo at end of "action to test" and look with DSO at the pin.

but i see no circular dma with 2 blocks in use...??

try:

• set dma circular, half-word(16bit) ;
• use/set callbacks for this, in Cube code gen. -> advanced set.
• define 64KB, uint16_t adc_buf[1024*32]
• use half of buf for continuous transfer, one half is adc-writing, other saved at same time to SD in callback
• set low priority for the int -> callbacks (high number in nvic , maybe 😎
• use: HAL_ADC_ConvCpltCallback(ADC_HandleTypeDef *hadc)...
• and: HAL_ADC_ConvHalfCpltCallback(ADC_HandleTypeDef *hadc)
• to save 1/2 of buf at right time 🙂
• try: f_write (&myfile, adc_buf, sizeof(adc_buf)/2 , &readed)
• +CpltCallback: f_write (&myfile, &adc_buf[sizeof(adc_buf)/2], sizeof(adc_buf)/2 , &readed)