How to improve random long write times using USB flash drive and fatfs

[MStea.1]

I'm using a nucleo-f7 development board as the basis for a data logger. The logger is using three spi slaves with DMA to grab data streaming from a sensor suite and write it to a USB flash drive with occasional data headers. The problem I'm having is that I'm getting an occasional very long write time.

I'm writing data from three separate A2Ds to three separate files. Normally the time it takes to f_write data to the USB flash drive is around 600 ms, but occasionally this will jump to 1800 ms. Currently, the system uses DMA TC interrupt that writes data from the smaller 4096 byte DMA buffer to a significantly larger 90 kilobyte circular "data buffer". I've made these data buffers as large as I can to help account for random write delays, but this occasional 1200 ms delay for a single modality is too much and will cause the system to overrun each data buffer.

Here are the main functions that control the file system and controls the DMA buffer transfers. I'm hoping for some advice on limiting this random additional time lag and or other concerns with how I'm handling the data write.

//Writes Data From the DMA Buffer to the Big Modality Data Buffer

//Happens only during DMA ISR

void dma2Buffer(data* modality)

{

uint8_t *dataBuffer = modality->dataBuffer;

uint8_t *dmaBuffer = &modality->dmaBuffer[ (modality->dmaFlag * DMA_WRITE_SIZE) ]; //It's Either 0 or 4096 since the Buffer is 8192 bytes this means the first or second half

uint32_t dmaPtr   = modality->dmaPtr;

//Copies Data from the DMA Buffer to the Big Data Buffer

memcpy(&dataBuffer[dmaPtr], dmaBuffer, DMA_WRITE_SIZE);

//Swaps the DMA Buffer Flag

modality->dmaPtr   = (modality->dmaPtr + DMA_WRITE_SIZE) % modality->maxBufferSize;

modality->bufferSize += DMA_WRITE_SIZE;

modality->dmaFlag  = !modality->dmaFlag;

if(modality->bufferSize > modality->maxBufferSize)

modality->overrunFlag = TRUE;

}

//Writes Data from the big Modality Data Buffer to the USB flash drive file

//Happens in the dmaStateMachine

void buffer2Usb(data* modality, _Bool isTimeBlock)

{

//Corrects overrun Condition

if(modality->overrunFlag)

{

correctOverrun(modality);

}

//Flag checked if it's time to write the data header

if(modality->hdrFlag)

{

usbWriteHdr(modality, isTimeBlock);

}

//Writes Data From the Data Buffer to the USB File

if(isTimeBlock)

{

writeData(modality, TIME_MSG_SIZE, isTimeBlock);

}

else if( modality->bufferSize < (modality->dataBlockSize - modality->dataTilNextBlock) ) //writes the contents of the dataBuffer to the flash drive until the start of the next data block

{

writeData(modality, modality->bufferSize, isTimeBlock);

}

else //writes entirety of the dataBuffer to the USB flash drive

{

writeData(modality, (modality->dataBlockSize - modality->dataTilNextBlock), isTimeBlock );

modality->hdrFlag = TRUE;

}

//Sets dataRdy Flag to False if there's no data in the buffer

if(modality->bufferSize == 0)

{

modality->dataRdyFlag = FALSE;

}

}

//Writes the number of bytes specified in writeSize to the modality file

static void writeData(data* modality, int writeSize, _Bool isTimeMsg)

{

int bytesWritten = 0;

//Checks if there is no data in the buffer, and returns if this is the case

if(modality->bufferSize <= 0)

return;

if( modality->usbPtr + writeSize >= modality->maxBufferSize) //If the buffer loops.

{

bytesWritten = usbWrite(modality, (modality->maxBufferSize - modality->usbPtr), isTimeMsg );

modality->bufferSize -= bytesWritten;

modality->dataTilNextBlock += bytesWritten;

writeSize -= bytesWritten;

modality->usbPtr = 0; //Set back to the beginning of the buffer

bytesWritten = usbWrite(modality, writeSize, isTimeMsg);

modality->bufferSize -= bytesWritten;

modality->dataTilNextBlock += bytesWritten;

modality->usbPtr = (modality->usbPtr + bytesWritten) % modality->maxBufferSize;

}

else

{

bytesWritten = usbWrite(modality, writeSize, isTimeMsg);

modality->bufferSize -= bytesWritten;

modality->dataTilNextBlock += bytesWritten;

modality->usbPtr = (modality->usbPtr + bytesWritten) % modality->maxBufferSize;

}

}

//File System Functions

//Opens and individual file for a modality

void newFile(int id)

{

//Opens the individual Modality File

fileNameGen(id);

checkForErrors(id, f_open( &usbFiles[id], FileName, FILEMODE), 3);

//Expands the File to pre-allocate space on the flash drive

f_expand(&usbFiles[id], (FSIZE_t)(dataBlockSize[id] * 3600), 1);

//Resets the File Bytes Written Counter

curFileSize[id] = 0;

//Maximum supported number of files is 24, or 8 * 3 files. This creates a new directory after it reaches that number which is the maximum supported per directory

if((FileCount%24) == 0)

genDirectory();

}

//Interface for the file system that writes the data to the correct modality file

int usbWrite(data* modality, int writeSize, _Bool isTimeBlock)

{

int bytesWritten = 0;

int id = modality->id;

uint8_t *buffer = &(modality->dataBuffer[modality->usbPtr]);

if(isTimeBlock)

{

int offset = findTimeMsg(buffer, modality->id);

if(offset != 0)

{

buffer = &(modality->dataBuffer[modality->usbPtr + offset]); //Adjusts location of the Time block in relation to the rest of the data if necessary

}

}

checkForErrors(id, f_write(&usbFiles[id], buffer, writeSize, (void *)&bytesWritten), 1);

curFileSize[modality->id] += bytesWritten;

return bytesWritten;

}

//Syncs The Files Every 60 Seconds

void syncFiles(void)

{

checkForErrors(MAG, f_sync(&usbFiles[MAG]), 0);

checkForErrors(ACS, f_sync(&usbFiles[ACS]), 0);

checkForErrors(SES, f_sync(&usbFiles[SES]), 0);

}

[waclawek.jan · Accepted solution]

USB FLASH - similarly to SD, eMMC, CF or any other managed FLASH - has its own controller, which handles everythinig needed in he FLASH memory itself, including ECC management, erasing and moving blocks around because of wear leveling. In other words, you cannot influence how long these operations will take, and there is NO guarantee for timing of these operations whatsoever (there are relatively rare devices with *some* guarantees, mainly among Compact Flash, aimed at the high-end professional photography/audiovisual market; there may be some amongst CF/USB, I don't follow this niche).

If you want guaranteed latencies, use NOR FLASH. Yes, it's expensive for the megabit, that's the price of the guaranteed access time (and endurance and retention, which tend to be neglected in the managed FLASHes too).

JW