How to write data efficiently to an SD card ?

[johngj]

I am writing data to an SD card for a data logger application, but I have very little experience with writing to SD cards so I want to ensure I am writing the data efficiently.

My understanding is that data is written as a sector of 512 bytes.  So even if I write a single byte, it will need to write all 512 bytes in a sector to do so ?

I am using the FatFs driver with an SPI interface running on the NUCLEO-L433RC-P dev board.

At the moment I create a binary file and start the file with the UNIX time, followed by samples (each 14 bytes in length) until the data log is completed. 

Each sample consists of a 4-byte tick time and 10-bytes of serial data received via the UART (at a 100ms rate):

• Tick time from HAL_GetTick()  (32 bits)
• Temperature (8 bits)
• Voltage high byte (8 bits)
• Voltage low byte (8 bits)
• Current high byte (8 bits)
• Current low byte (8 bits)
• Consumption high byte (8 bits)
• Consumption low byte (8 bits)
• Speed high byte (8 bits)
• Speed low byte (8 bits)
• CRC (8 bits)

The diagram below shows the structure of the binary file, with just 4 samples for simplicity.  The number of samples for a real data log would obviously be much higher than could fill the entire SD card.

I currently write to the SD card (using f_write) each time the serial data is received (~100ms), but I believe this is inefficient because it writes a sector of 512 bytes a time and it takes a certain amount of time to write the data.  I also think this technique might cause additional wear on the SD card ?

So how should I write this data efficiently ?

Should I wait until I have enough data to fit into a single sector before performing the write ?  If so, I could only fit 36 samples in a sector as this would be 508 bytes...

• 36 samples would be 508 bytes i.e. 4 bytes (UNIX time) + 14 bytes (data packet) * 36 = 508 bytes

But this would leave 4 spare bytes in each sector, in which case if these bytes are unused it would be inefficient in terms of memory space.  Or are the unused spare bytes somehow used when the next write occurs ?

Is the sector size for all SD cards always 512 bytes, or does the sector size depend on how the SD card is formatted ?

I notice that when I format the SD card, there is an allocation unit size.  Is this anything to do with the sector size ?

Eventually I want to use larger SD cards so will need to use the exFAT format.

My code is attached for reference

[TDK]

In broad terms:

• Buffer your data, write in chunks of 512 bytes at a time. 4K or 8K is a good size.
• Make your code robust to write delays of up to 250 ms or so.
• Most cards have wear leveling, don't try to do this for the card.
• The filesystem adds an additional complication to things, but allocation size should be aligned to sector size and the above performance guidelines are valid.

[Tesla DeLorean]

Large aligned blocks, minimally on sector boundaries, cluster would be better

Sweet spot probably around 32 KB.

Would probably use a spill-buffer, ie 32KB + maximal message length, keep adding data until you hit 32 KB, write that, and pull down any residual, rinse, repeat. Flush out remaining data, and close file when done.

Have the buffer management / writing operating independently of data collection, perhaps in main loop. FATFS / SDIO aren't going to permit concurrent operation.

Enough prebuffering to cover write, and file system maintenance / update, ie fat tables, deblocking. Doing f_sync() or f_open()/f_close() often / repetitively will drag significantly.

[tjaekel]

TDK is a great guy, always with great hints.

If you want to use FatFS, so that you can take the SD card to another PC - you have to live with the limitation of the "speed degradation" due to using a file system.

If you want to use the SD card just as a (fast) storage media (like external memory) - where you read back later the data on the same MCU: no need to use a FAT file system: you can handle SD card like a memory with 512 bytes sectors (like a NAND or NOR flash).

As TDK has mentioned: buffering a larger amount of data helps a lot. Make it a multiple of a sector size (N * 512 bytes).

The drawback: you need larger memory (RAM) on MCU and the data sits still just in memory until you write it back (you flush the temporary buffer to the SD card).
It just risky in terms: you could lose data, e.g. you reset the MCU, you have a FW bug... - the piece of data is lost.

And you cannot power down the MCU: you have to use a "soft power down" approach which gives you a chance to save the outstanding buffers to the SD card (via the FatFS) before you really switch off (flush all open files, tear down the file system...).

But, there can be a delicate issue:

If you want to write AND read back at the same time, e.g. you write N * 512 bytes, but later you want to read back - and all this data sits still in memory (not yet exceeding the limit to flush) - it is a bit more work to.
Usually, you would assume to write into a file and later you set the file seek pointer and you read back. But when hold in temporary buffer - it is not a file: you have to implement a "read back from cache" (the temporary buffer).

The drawback with FatFS (any file system) is:

a single byte written allocates a (potentially) new sector, or: an existing sector (512bytes) has to be read, add this byte and write the sector back (512bytes). In addition: FatFS will update the FAT, the file size, the date, ... (another read-modify-write of another sector).

One single byte written into a file is a huge overhead (assume a read-modify-write of 512byte sectors done TWICE).
Therefore: buffering your writes and write in larger chunks (N *512) is a really great advice.

BTW: potentially, your FatFS will already buffer for 512byte sectors. It might have already a "data cache", so that a single byte write might be OK.
--> but see the issue with powering down and flushing the buffers for a file, for the file system before you really switch off.

[johngj]

Im confused how the f_write function works.

The "File object structure" (FIL) has an element called buf which I currently have set to a size of 4092.

During the f_write function there are single and multiple block write functions which is when the data is actually written to the SD card.

Single block write data from fp->buf:

disk_write(fs->drv, fp->buf, fp->sect, 1)

Multiple block write data from wbuff:

disk_write(fs->drv, wbuff, sect, cc)

But there is also a memory copy from fp->buf to wbuff (data is not actually written to the SD card):

mem_cpy(fp->buf, wbuff + ((fp->sect - sect) * SS(fs)), SS(fs));

Is the idea to setup an initial buffer, which f_write then copies into the "File object structure" (FIL) buffer ?  The f-write will copy the "File object structure" (FIL) buffer to the SD card (single or multiple block write) ?

But then there are two buffers, the initial buffer and the "File object structure" (FIL) buffer.  Isnt it better to write the data directly to the "File object structure" (FIL) buffer ?

I am trying to write 8192 bytes at a time, so I have a buffer sd_data [8192].

When 8192 bytes have been received, I then call f_write and point to that buffer.

But f_write then just copies the data into  the "File object structure" (FIL) buffer

I've got a stream of data continuously being received by the UART which I am trying to write to the SD card, but I'm struggling to implement this and don't understand all these buffers.

Is there any examples or documentation ?

[tjaekel]

I think, the buffering is done already properly in your File System:
Even you write just one byte - the buffer is just flushed to SD card when the sector (512Bytes per sector or even a cluster = N x sectors) is full.
But it means also: never remove the SD card unless you have released/closed all. There is a function to shut down the File System, a function to flush any outstanding data (sitting in "cache) when you have written a file.

f_close() should actually flush. But if you are not sure, you can f_flush() anytime.

Never remove SD card when the File System is still up and running, it was not shut down (dis-mounted), even nothing goes on anymore in your FW (no writes anymore, "all done"). There can be still "updates" sitting in memory (caches) never written to SD card. The SD card can be corrupted otherwise.

OK, the FIL structure:
Actually, it is a "private" structure! Even it has a buffer inside, potentially acting as a "cache" (see before), you are never "allowed" to manipulate or write into the members of this structure yourself.

If you would do so - you would "confuse" the File System, even you could corrupt data still to write/flush to SD card.

The "only" interface for you is the File System API, e.g. functions like "f_write()", f_read()", f_seek". But the FIL, as a structure, is just a handle. Never touch anything inside. It is used by the File System, "just" to remember what the opened files is, what the status is, how much "cache" was already written to SD card etc. As a user - it is just an address (handle) which you have to provide when relating to actions on a particular file (referenced via the handle, as "synonym" for file name after opened).

This handle uses just "your" memory space, with all the buffers needed for "caching" the data, but you should never corrupt, modify, change such a handle (the data "inside" is "private" for the File System only).

BTW:
You can implement your own "caching", e.g.: collect up to 4KByte of data to write and then you fire f_write() - with 4KByte buffer.
But remains true: never forget to close and flush files and the file system.

I think: a good file system (like FatFS, AZURE RTOS file system) - they do already the "caching" for you. no worries if you just write single bytes. I do not see a need to handle a "buffering" (caching) in your application code. I assume, it will not really improve any throughput (which depends at the end anyway what the HW interface to SD card is, e.g. simple SPI, QSPI, 4-bit SDIO, ...). And the buffer (sector and cluster handling) should be implemented in the File System (I am sure, it is).

[johngj]

Thanks tjaekel

It sounds like my approach was correct in the first place, but I'm having a few issues.

I understand how f_sync and f_close are required before removing the SD card or powering down etc.

For now I am using the push button switch (as an external interrupt) to start and stop the data log.  This indicates when f_open needs to create a new file and f_close needs to close the file.

However, with my current implementation I am seeing incorrect data values in the file and loosing the last part of data because I'm not sure how to force the data to be written when I press the button to stop the data log.

Below are details about my implementation and the issues it has...

I am sending UART data (using Realterm) which will be written as a binary file to the SD card.

The file structure is as follows:

• Initial four bytes is the UNIX time:  For debugging purposes this is a fixed value of 1708187420 (0x65D0DF1C)
• Next 14 bytes is a "sample" which contains…
  • Four bytes for the time stamp: For debugging purposes this starts at 1 and increments by 1
  • 10 bytes of telemetry data:  For debugging purposes the values are 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x0A.  

I send 585 'samples' using Realterm which is 8190 bytes for each send (i.e. 585 x 14). 

Note: In the real application, the 10 bytes of UART data is transmitted at a 100ms rate.  But for debugging purposes I use Realterm to send all 8190 bytes transmitted in one go.

The UART interrupt is configured to receive 10 bytes, during this interrupt the time 'stamp' is incremented and a data received flag is set...

void HAL_UARTEx_RxEventCallback(UART_HandleTypeDef *huart, uint16_t Size)
{
	if (huart->Instance == LPUART1)
	{
		HAL_UARTEx_ReceiveToIdle_IT(&hlpuart1, rx_buff, 10);
		stamp++;
		if (log_state == LOG_STATE_RUNNING)
		{
			data_received = 1;
		}
	}
}

In the main loop there is a state machine for different logging states (these states are controlled using the push button switch).  When the data log is running, the 'write_buffer' function is called if the data received flag is set....

	while (1)
	{
		switch (log_state)
		{
		case LOG_STATE_IDLE:  // not logging
			break;

		case LOG_STATE_START:	// start logging
			unix_timestamp();
			get_fattime();
			fresult = f_open(&fil,UnixToD.ASCII, FA_OPEN_ALWAYS | FA_READ | FA_WRITE);
			newLog = 1;
			log_state = 3;
			break;

		case LOG_STATE_STOP: // stop logging
			write_buffer(&sd_buffer);
			data_received = 0;
			fresult = f_close(&fil);
			log_state = 0;
			break;

		case LOG_STATE_RUNNING: // logging
		if(data_received)
		{
			write_buffer(&sd_buffer);
			data_received = 0;
		}
			break;

		default:
			break;
		}
	}

As mentioned, the log state is controlled by the push button switch using the external interrupt...

void HAL_GPIO_EXTI_Callback (uint16_t GPIO_Pin)
{
	if(GPIO_Pin == GPIO_PIN_13)
	{
		if (log_state == LOG_STATE_IDLE)
		{
			log_state = LOG_STATE_START;
		}
		if (log_state == LOG_STATE_RUNNING)
		{
			log_state = LOG_STATE_STOP;
		}
	}
}

The UART data and time stamp is written to the buffer when the 'write_buffer' function is called (BYTES_TO_WRITE is defined as 8192)...

void write_buffer (uint8_t *buff_ptr)
{
	static uint8_t rx_idx;
	static uint16_t sd_idx;
	static uint16_t sd_idx_end;
	static uint32_t stamp_mask;
	static uint8_t stamp_shift;

	if (newLog)	// write UNIX time (first 4 bytes of log file)
	{
		*buff_ptr = UnixToD.UnixToD & 0x000000FF;
		*(buff_ptr+1) = (UnixToD.UnixToD & 0x0000FF00) >> 8;
		*(buff_ptr+2) = (UnixToD.UnixToD & 0x00FF0000) >> 16;
		*(buff_ptr+3) = (UnixToD.UnixToD & 0xFF000000) >> 24;
		sd_idx = 4;
		newLog = 0;
	}

	stamp_mask = 0x000000FF;
	stamp_shift = 0;
	sd_idx_end = (sd_idx + 4) & 0x3FFF;

	do
	{
		*(buff_ptr+sd_idx) = (stamp & stamp_mask) >> stamp_shift;
		stamp_mask = stamp_mask << 8;
		stamp_shift = stamp_shift + 8;

		if (sd_idx == (BYTES_TO_WRITE -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_a, BYTES_TO_WRITE , &bw);
		}
		if (sd_idx == ((BYTES_TO_WRITE * 2) -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_b, BYTES_TO_WRITE , &bw);
		}
		sd_idx = (sd_idx + 1) & 0x3FFF;
	}
	while (sd_idx != sd_idx_end);

	sd_idx_end = (sd_idx + 10) & 0x3FFF;
	rx_idx=0;
	do
	{
		*(buff_ptr+sd_idx) = rx_buff[rx_idx];
		rx_idx++;
		if (sd_idx == (BYTES_TO_WRITE -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_a, BYTES_TO_WRITE , &bw);
		}
		if (sd_idx == ((BYTES_TO_WRITE * 2) -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_b, BYTES_TO_WRITE , &bw);
		}
		sd_idx = (sd_idx + 1) & 0x3FFF;
	}
	while (sd_idx != sd_idx_end);

	if (log_state == LOG_STATE_STOP)
	{
		f_sync(&fil);
	}
}

The variable 'sd_buffer' is defined as a structure...

Buffer_t sd_buffer;

typedef struct T_buffer
{
	uint8_t buffer_a[BYTES_TO_WRITE];
	uint8_t buffer_b[BYTES_TO_WRITE];
}Buffer_t;

I start seeing incorrect data values in the file after sending more than 3 x 585 'samples' using Realterm.  For the moment I've only looked at the time 'stamp' values, which should start at 1 and increment by 1.

• For the 1st 585 samples sent by Realterm, the time 'stamp' starts at 1 and ends at 585 (correct).
• For the 2nd 585 samples sent by Realterm, the time 'stamp' starts at 586 and ends at 1170 (correct).
• For the 3rd 585 samples sent by Realterm, the time 'stamp' starts at 1171 and ends at 1755 (correct).
• For the 4th 585 samples sent by Realterm, the initial time 'stamp' starts at 1756 (correct) but the next time 'stamp' is 1844 (incorrect - it should be 1757).  The 'stamp' continues to increment but the values are wrong.

So something is going wrong during the write_buffer function but I can't see why.  Maybe my implementation is poor?

The next issue is how to force the data to be written when I press the button to stop the data log. 

Again, this might be because my implementation is poor, because 'f_write' is only called when the number of bytes to write is 8191 (which uses buffer a data) or 16,383 (which uses buffer b data).

So if I stop the data log when the bytes to write is neither of these values, then the data since the last f_write will be lost.

This is why I am questions whether my implementation is correct.  The idea of having buffers a and b (each 8192 bytes in size) was so that UART data could be updated in one buffer whilst the data in the other buffer is written to the SD card.

I am just trying to write 8192 bytes at a time when enough data has been received, so that the SD card writes are efficient and as quick as possible.

Project is attached for reference (sdcard.7z)  

[johngj]

Thanks tjaekel

It sounds like my approach was correct in the first place, but I'm having a few issues.

I understand how f_sync and f_close are required before removing the SD card or powering down etc.

For now I am using the push button switch (as an external interrupt) to start and stop the data log.  This indicates when f_open needs to create a new file and f_close needs to close the file.

However, with my current implementation I am seeing incorrect data values in the file and loosing the last part of data because I'm not sure how to force the data to be written when I press the button to stop the data log.

Below are details about my implementation and the issues it has...

I am sending UART data (using Realterm) which will be written as a binary file to the SD card.

The file structure is as follows:

• Initial four bytes is the UNIX time:  For debugging purposes this is a fixed value of 1708187420 (0x65D0DF1C)
• Next 14 bytes is a "sample" which contains…
• Four bytes for the time stamp: For debugging purposes this starts at 1 and increments by 1
• 10 bytes of telemetry data:  For debugging purposes the values are 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x0A.  

I send 585 'samples' using Realterm which is 8190 bytes for each send (i.e. 585 x 14). 

Note: In the real application, the 10 bytes of UART data is transmitted at a 100ms rate.  But for debugging purposes I use Realterm to send all 8190 bytes transmitted in one go.

The UART interrupt is configured to receive 10 bytes, during this interrupt the time 'stamp' is incremented and a data received flag is set...

void HAL_UARTEx_RxEventCallback(UART_HandleTypeDef *huart, uint16_t Size)
{
	if (huart->Instance == LPUART1)
	{
		HAL_UARTEx_ReceiveToIdle_IT(&hlpuart1, rx_buff, 10);
		//stamp = HAL_GetTick();
		stamp++;
		if (log_state == LOG_STATE_RUNNING)
		{
			data_received = 1;
		}
	}
}

In the main loop there is a state machine for different logging states (these states are controlled using the push button switch).  When the data log is running, the 'write_buffer' function is called if the data received flag is set....

while (1)
	{
		if(data_received)
		{
			write_buffer(&sd_buffer);
			data_received = 0;
		}

		switch (log_state)
		{
		case LOG_STATE_IDLE:  // not logging
			break;

		case LOG_STATE_START:	// start logging
			unix_timestamp();
			get_fattime();
			fresult = f_open(&fil,UnixToD.ASCII, FA_OPEN_ALWAYS | FA_READ | FA_WRITE);
			newLog = 1;
			log_state = 3;
			break;

		case LOG_STATE_STOP: // stop logging
			write_buffer(&sd_buffer);
			data_received = 0;
			fresult = f_close(&fil);
			log_state = 0;
			break;

		case LOG_STATE_RUNNING: // logging
			break;

		default:
			break;
		}
	}
}

As mentioned, the log state is controlled by the push button switch using the external interrupt...

void HAL_GPIO_EXTI_Callback (uint16_t GPIO_Pin)
{
	if(GPIO_Pin == GPIO_PIN_13)
	{
		if (log_state == LOG_STATE_IDLE)
		{
			log_state = LOG_STATE_START;
		}
		if (log_state == LOG_STATE_RUNNING)
		{
			log_state = LOG_STATE_STOP;
		}

	}
}

The UART data and time stamp is written to the buffer when the 'write_buffer' function is called (BYTES_TO_WRITE is defined as 8192)...

void write_buffer (uint8_t *buff_ptr)
{
	static uint8_t rx_idx;
	static uint16_t sd_idx;
	static uint16_t sd_idx_end;
	static uint32_t stamp_mask;
	static uint8_t stamp_shift;

	if (newLog)	// write UNIX time (first 4 bytes of log file)
	{
		*buff_ptr = UnixToD.UnixToD & 0x000000FF;
		*(buff_ptr+1) = (UnixToD.UnixToD & 0x0000FF00) >> 8;
		*(buff_ptr+2) = (UnixToD.UnixToD & 0x00FF0000) >> 16;
		*(buff_ptr+3) = (UnixToD.UnixToD & 0xFF000000) >> 24;
		sd_idx = 4;
		newLog = 0;
	}

	stamp_mask = 0x000000FF;
	stamp_shift = 0;
	sd_idx_end = (sd_idx + 4) & 0x3FFF;

	do
	{
		*(buff_ptr+sd_idx) = (stamp & stamp_mask) >> stamp_shift;
		stamp_mask = stamp_mask << 8;
		stamp_shift = stamp_shift + 8;

		if (sd_idx == (BYTES_TO_WRITE -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_a, BYTES_TO_WRITE , &bw);
		}
		if (sd_idx == ((BYTES_TO_WRITE * 2) -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_b, BYTES_TO_WRITE , &bw);
		}
		sd_idx = (sd_idx + 1) & 0x3FFF;
	}
	while (sd_idx != sd_idx_end);

	sd_idx_end = (sd_idx + 10) & 0x3FFF;
	rx_idx=0;
	do
	{
		*(buff_ptr+sd_idx) = rx_buff[rx_idx];
		rx_idx++;
		if (sd_idx == (BYTES_TO_WRITE -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_a, BYTES_TO_WRITE , &bw);
		}
		if (sd_idx == ((BYTES_TO_WRITE * 2) -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_b, BYTES_TO_WRITE , &bw);
		}
		sd_idx = (sd_idx + 1) & 0x3FFF;
	}
	while (sd_idx != sd_idx_end);

	if (log_state == LOG_STATE_STOP)
	{
		f_sync(&fil);
	}
}

The variable 'sd_buffer' is defined as a structure...

typedef struct T_buffer
{
	uint8_t buffer_a[BYTES_TO_WRITE];
	uint8_t buffer_b[BYTES_TO_WRITE];
}Buffer_t;

I start seeing incorrect data values in the file after sending more than 3 x 585 'samples' using Realterm.  For the moment I've only looked at the time 'stamp' values, which should start at 1 and increment by 1.

• For the 1st 585 samples sent by Realterm, the time 'stamp' starts at 1 and ends at 585 (correct).
• For the 2nd 585 samples sent by Realterm, the time 'stamp' starts at 586 and ends at 1170 (correct).
• For the 3rd 585 samples sent by Realterm, the time 'stamp' starts at 1171 and ends at 1755 (correct).
• For the 4th 585 samples sent by Realterm, the initial time 'stamp' starts at 1756 (correct) but the next time 'stamp' is 1844 (incorrect - it should be 1757).  The 'stamp' continues to increment but the values are wrong.

So something is going wrong during the write_buffer function but I can't see why.  Maybe my implementation is poor?

The next issue is how to force the data to be written when I press the button to stop the data log. 

Again, this might be because my implementation is poor, because 'f_write' is only called when the number of bytes to write is 8191 (which uses buffer a data) or 16,383 (which uses buffer b data).

So if I stop the data log when the bytes to write is neither of these values, then the data since the last f_write will be lost.

This is why I am questions whether my implementation is correct.  The idea of having buffers a and b (each 8192 bytes in size) was so that UART data could be updated in one buffer whilst the data in the other buffer is written to the SD card.

I am just trying to write 8192 bytes at a time when enough data has been received, so that the SD card writes are efficient and as quick as possible.

Project attached for reference (see sdcard.7z)  

[johngj]

Thanks tjaekel

It sounds like my approach was correct in the first place, but I'm having a few issues.

I understand how f_sync and f_close are required before removing the SD card or powering down etc.

For now I am using the push button switch (as an external interrupt) to start and stop the data log.  This indicates when f_open needs to create a new file and f_close needs to close the file.

However, with my current implementation I am seeing incorrect data values in the file and loosing the last part of data because I'm not sure how to force the data to be written when I press the button to stop the data log.

Below are details about my implementation and the issues it has...

I am sending UART data (using Realterm) which will be written as a binary file to the SD card.

The file structure is as follows:

• Initial four bytes is the UNIX time:  For debugging purposes this is a fixed value of 1708187420 (0x65D0DF1C)
• Next 14 bytes is a "sample" which contains…
• Four bytes for the time stamp: For debugging purposes this starts at 1 and increments by 1
• 10 bytes of telemetry data:  For debugging purposes the values are 0x01 0x02 0x03 0x04 0x05 0x06 0x07 0x08 0x09 0x0A.  

I send 585 'samples' using Realterm which is 8190 bytes for each send (i.e. 585 x 14). 

Note: In the real application, the 10 bytes of UART data is transmitted at a 100ms rate.  But for debugging purposes I use Realterm to send all 8190 bytes transmitted in one go.

The UART interrupt is configured to receive 10 bytes, during this interrupt the time 'stamp' is incremented and a data received flag is set...

void HAL_UARTEx_RxEventCallback(UART_HandleTypeDef *huart, uint16_t Size)
{
	if (huart->Instance == LPUART1)
	{
		HAL_UARTEx_ReceiveToIdle_IT(&hlpuart1, rx_buff, 10);
		//write_buffer(&sd_buffer);
		//stamp = HAL_GetTick();
		stamp++;
		if (log_state == LOG_STATE_RUNNING)
		{
			data_received = 1;
		}
	}
}

In the main loop there is a state machine for different logging states (these states are controlled using the push button switch).  When the data log is running, the 'write_buffer' function is called if the data received flag is set....

while (1)
	{
		if(data_received)
		{
			write_buffer(&sd_buffer);
			data_received = 0;
		}

		switch (log_state)
		{
		case LOG_STATE_IDLE:  // not logging
			break;

		case LOG_STATE_START:	// start logging
			unix_timestamp();
			get_fattime();
			fresult = f_open(&fil,UnixToD.ASCII, FA_OPEN_ALWAYS | FA_READ | FA_WRITE);
			newLog = 1;
			log_state = 3;
			break;

		case LOG_STATE_STOP: // stop logging
			write_buffer(&sd_buffer);
			data_received = 0;
			fresult = f_close(&fil);
			log_state = 0;
			break;

		case LOG_STATE_RUNNING: // logging
			break;

		default:
			break;
		}
	}

As mentioned, the log state is controlled by the push button switch using the external interrupt...

void HAL_GPIO_EXTI_Callback (uint16_t GPIO_Pin)
{
	if(GPIO_Pin == GPIO_PIN_13)
	{
		if (log_state == LOG_STATE_IDLE)
		{
			log_state = LOG_STATE_START;
		}
		if (log_state == LOG_STATE_RUNNING)
		{
			log_state = LOG_STATE_STOP;
		}

	}
}

The UART data and time stamp is written to the buffer when the 'write_buffer' function is called (BYTES_TO_WRITE is defined as 8192)...

void write_buffer (uint8_t *buff_ptr)
{
	static uint8_t rx_idx;
	static uint16_t sd_idx;
	static uint16_t sd_idx_end;
	static uint32_t stamp_mask;
	static uint8_t stamp_shift;

	if (newLog)	// write UNIX time (first 4 bytes of log file)
	{
		*buff_ptr = UnixToD.UnixToD & 0x000000FF;
		*(buff_ptr+1) = (UnixToD.UnixToD & 0x0000FF00) >> 8;
		*(buff_ptr+2) = (UnixToD.UnixToD & 0x00FF0000) >> 16;
		*(buff_ptr+3) = (UnixToD.UnixToD & 0xFF000000) >> 24;
		sd_idx = 4;
		newLog = 0;
	}

	stamp_mask = 0x000000FF;
	stamp_shift = 0;
	sd_idx_end = (sd_idx + 4) & 0x3FFF;

	do
	{
		*(buff_ptr+sd_idx) = (stamp & stamp_mask) >> stamp_shift;
		stamp_mask = stamp_mask << 8;
		stamp_shift = stamp_shift + 8;

		if (sd_idx == (BYTES_TO_WRITE -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_a, BYTES_TO_WRITE , &bw);
		}
		if (sd_idx == ((BYTES_TO_WRITE * 2) -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_b, BYTES_TO_WRITE , &bw);
		}
		sd_idx = (sd_idx + 1) & 0x3FFF;
	}
	while (sd_idx != sd_idx_end);

	sd_idx_end = (sd_idx + 10) & 0x3FFF;
	rx_idx=0;
	do
	{
		*(buff_ptr+sd_idx) = rx_buff[rx_idx];
		rx_idx++;
		if (sd_idx == (BYTES_TO_WRITE -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_a, BYTES_TO_WRITE , &bw);
		}
		if (sd_idx == ((BYTES_TO_WRITE * 2) -1))
		{
			fresult = f_write(&fil, &sd_buffer.buffer_b, BYTES_TO_WRITE , &bw);
		}
		sd_idx = (sd_idx + 1) & 0x3FFF;
	}
	while (sd_idx != sd_idx_end);

	if (log_state == LOG_STATE_STOP)
	{
		f_sync(&fil);
	}
}

The variable 'sd_buffer' is defined as a structure...

typedef struct T_buffer
{
	uint8_t buffer_a[BYTES_TO_WRITE];
	uint8_t buffer_b[BYTES_TO_WRITE];
}Buffer_t;

I start seeing incorrect data values in the file after sending more than 3 x 585 'samples' using Realterm.  For the moment I've only looked at the time 'stamp' values, which should start at 1 and increment by 1.

• For the 1st 585 samples sent by Realterm, the time 'stamp' starts at 1 and ends at 585 (correct).
• For the 2nd 585 samples sent by Realterm, the time 'stamp' starts at 586 and ends at 1170 (correct).
• For the 3rd 585 samples sent by Realterm, the time 'stamp' starts at 1171 and ends at 1755 (correct).
• For the 4th 585 samples sent by Realterm, the initial time 'stamp' starts at 1756 (correct) but the next time 'stamp' is 1844 (incorrect - it should be 1757).  The 'stamp' continues to increment but the values are wrong.

So something is going wrong during the write_buffer function but I can't see why.  Maybe my implementation is poor?

The next issue is how to force the data to be written when I press the button to stop the data log. 

Again, this might be because my implementation is poor, because 'f_write' is only called when the number of bytes to write is 8191 (which uses buffer a data) or 16,383 (which uses buffer b data).

So if I stop the data log when the bytes to write is neither of these values, then the data since the last f_write will be lost.

This is why I am questions whether my implementation is correct.  The idea of having buffers a and b (each 8192 bytes in size) was so that UART data could be updated in one buffer whilst the data in the other buffer is written to the SD card.

I am just trying to write 8192 bytes at a time when enough data has been received, so that the SD card writes are efficient and as quick as possible.

Project is attached for reference.  

[johngj]

PS.  Apologies for the last 3 posts being the same.  They were not appearing after posting, so I repeated and then all 3 showed up at once !!!