FAT_FS f_write: Writes wrong data to file when writing odd number of bytes

GS1 · ‎2019-08-19

Hi all,

In my project with STM32H743 (created with CubeMX version 5.0.1) I have a strange behaviour of f_write when writing data to the file and the amount is not a number which can be devided by 32.

In my test I wrote 256, 512 or 4096 Bytes at a time. Everything perfect. Data in File as expected.

BUT: When writing e.g. 590 Bytes (which is not dividable by 32) -> Errors appear.

I create a test file with following text pattern (590 Bytes)

0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklm

0123456789##

This pattern is written to a file 100 times appending to an open file.

Now the result shows wrong blocks after multiple writes. Here is the wrong part:

0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklm

0123456789##

012345678989:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklm

0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklm

01234567:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklm

0123456789##

0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklm

In one block 2 characters are moved around to the wrong place!!

It seams that this issue only appears when writing a number of Bytes which is not dividable by 32 at a time, 256 is ok, 512 is ok, 4096 is ok, but: 590 is not ok! (This is just an assumption on the current state of testing)

Can anyone help?

ff.c Revision:

#if _FATFS != 68300 /* Revision ID */

Tesla DeLorean · ‎2019-08-19

Caching is done on 32-byte boundaries, you need to make sure any coherency code handles this correctly.

DMA will also have alignment requirements.

Make sure your write routines clean the cache, and perhaps handle unaligned cases.

Tips, Buy me a coffee, or three.. PayPal Venmo
Up vote any posts that you find helpful, it shows what's working..

GS1 · ‎2019-08-19

Cache is actually already handled in the routines. The wrong data even appears when DCache is disabled totally.

The mistake appears after a repeated and regular amount of block writes of 590 Bytes in this example:

13 ok blocks

-- wrong block

5 ok blocks

-- wrong block

19 ok blocks

-- wrong block

5 ok blocks

-- wrong block

13 ok blocks

-- wrong block

5 ok blocks

-- wrong block

19 ok blocks

and so on.

There is strange repeated pattern behind it.

Tesla DeLorean · ‎2019-08-19

You might try a newer version of FatFS that isn't 2+ years old, in case such an error has been fixed.

You could try building without optimization, and see if there is a tool-chain related issue. A software issue (logic/compliation) should be platform agnostic.

I try to avoid using f_write in this fashion because it is super inefficient and requires multiple reads/writes to address sector spanning issue.

Test case seems reasonably scoped, might try when I find some time.

Tips, Buy me a coffee, or three.. PayPal Venmo
Up vote any posts that you find helpful, it shows what's working..

Tesla DeLorean · ‎2019-08-19

//****************************************************************************
 
uint8_t testpattern[] =
"0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklm"
"0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklm"
"0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklm"
"0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklm"
"0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklm"
"0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklm"
"0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklm"
"0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklm"
"0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklm"
"0123456789##";
 
//****************************************************************************
 
void gsatstest(char *Filename)
{
  FRESULT res;
  FIL fil;
 
  if (f_open(&fil, Filename, FA_OPEN_ALWAYS | FA_WRITE) == FR_OK)
  {
    UINT BytesWritten;
 
    res = f_lseek(&fil, f_size(&fil));
 
    if (res == FR_OK)
      res = f_write(&fil, testpattern, sizeof(testpattern), &BytesWritten);
 
    f_close(&fil);
 
    if (res != FR_OK)
      puts("Some amount of fail detected");
  }
  else
    puts("Failed to open");
}
 
//****************************************************************************
 
void gsatsloop(void)
{
  char Filename[] = "TEST.TXT";
  int i;
  uint32_t Crc = 0xFFFFFFFF;
 
  printf("pattern size %d\n", sizeof(testpattern));
  for(i=0; i<100; i++)
    Crc = CRC32(Crc, sizeof(testpattern), testpattern);
 
  printf("CRC32:%08X Memory\n", Crc);
 
  for(i=0; i<100; i++)
    gsatstest(Filename);
 
  Crc = CrcFile(Filename, 65536);
 
  printf("CRC32:%08X File\n", Crc);
}
 
//****************************************************************************

pattern size 571

CRC32:743062C1 Memory

CRC32:743062C1 File

Tips, Buy me a coffee, or three.. PayPal Venmo
Up vote any posts that you find helpful, it shows what's working..

Tesla DeLorean · ‎2019-08-19

FatFs Testing (68300)

SystemCoreClock: 400000000

Tips, Buy me a coffee, or three.. PayPal Venmo
Up vote any posts that you find helpful, it shows what's working..

GS1 · ‎2019-08-20

Hi Clive,

first of all thank you for your efforts to help me!

There are some comments on your test and remarks:

1) You mentioned that you do not use f_write: So can you please tell me, how you would write huge amounts of data (data logging with 100 Hz) to a file on SD card which has to be readable on PCs?

2) Updating to the latest release of CubeMX was not possible due to some other issues arising on USB and SD card access after the migration. :-//

I compared the ff.c file and there are no changes in the f_write routine anyway. But the HAL library was changed too much so I will in the first place stay with the old version. My project is working - except for the file write issue - currently.

3) I think simply checking the CRC is probably not the right means to prove that bytes are not shifted around in the file, because that is what actually happens. The amount written is correct, but at some place 2 or sometimes 3 bytes are shifted around.

4) In my test I do not close and reopen the file to append a new block. I open the file, write the block 100 times and then close the file.

5) Your pattern does not have the CR-LF which I introduce after one line. This however allows to open the file in notepad and have a look at the lines. Only then you can see the effect

Here is my test code:

register int x, i, m;

INT BytesWritten;

FRESULT res;

static uint8_t ucFileBuffer[2000];

uint8_t ch;

// Fill Buffer with patterns:

x = 0;

for (i = 0; i < 20; i++) // 20 Rows with 64 Characters

{

ch = '0';

for (m = 0; m < 62; m++)

{

ucFileBuffer[x++] = ch++;

}

ucFileBuffer[x++] = CHAR_CR; // 0x0D

ucFileBuffer[x++] = CHAR_LF; // 0x0A

}

// Test file writes:

res = f_open(&MyFile, "Test.TXT", FA_CREATE_ALWAYS | FA_WRITE);

if (res == FR_OK)

{

for (m = 0; m < 100; m++)

{

// Overwrite the Test pattern at a random number and set 2 '#' characters so I see where the block ends

ucFileBuffer[586] = '#';

ucFileBuffer[587] = '#';

ucFileBuffer[588] = CHAR_CR;

ucFileBuffer[589] = CHAR_LF;

ucFileBuffer[590] = 0;

res = f_write(&MyFile, ucFileBuffer, 590, &BytesWritten); // Result NOT OK!!!

}

res = f_close(&MyFile);

It seems that when writing more than 512 characters and the number is not even dividable by 32 then the byte shifting happens.

Yesterday I tried the following and succeeded:

res = f_write(&MyFile, &ucFileBuffer[0], 512, &BytesWritten); // thats how it WORKS !!!

res = f_write(&MyFile, &ucFileBuffer[512], 590-512, &BytesWritten);

Thanks again for your comments! They are very much appreciated.

BR

GS1 · ‎2019-08-20

Hi Clive,

I found a workaround solution for my data acquisition application:

I increase the amount of data so that the next 32 bytes boundary is hit. These additional bytes are set to 0 and will be ignored when reading the file in my Windows data analysis software.

This is not a clean solution, but at least I now create correct data files and I can supply this firmware release to my customers.

Hopefully I have the time to investigate, what is actually going wrong in the f_write handling in my STM32H7 project. I assume it's still a DMA or Cache handling issue which is created by ST's HAL library.

However I am wondering:

Did nobody ever run into the same problem?

Am I the only one to use FAT-FS to write files to an SD Card on the H7 and have this problem?

Tesla DeLorean · ‎2019-08-20

When logging a stream of data I use a spill buffer, holding my desired write size, and my largest stream output, ie 32768 + 590, when I meet/exceed my write size, I write the 32K and pull-down any overflowing data to the front of the next buffer frame, when I close I flushing any remaining data.

In most PC C implementation, fread/fwrite hold a buffer you don't see, FatFS doesn't do this, nor do most DISKIO implementations cache or lazy-write. Thing with embedded is that you either have to implement the buffering in one of the layers, hiding it from hard choices on the application side, or you code with some understanding of where the performance drags are, and architect your algorithm/implementation about how the sub-systems perform optimally. You want to avoid read/write to the media to deblock the data, you want to avoid unnecessary copying or duplication of data. ie sprintf() directly to the buffer

The CRC should be quite effective at determining difference in intent vs result. If you have a memory sub-system failure, that's a whole different class of problem. If one has a CRC32 for the "correct" file it will be pretty evident when you've failed.

I used a best guess at what you were doing from the narrative. The 571 number I ended up with was sufficiently prime that I would see gross failure.

>>However I am wondering: Did nobody ever run into the same problem? Am I the only one to use FAT-FS to write files to an SD Card on the H7 and have this problem?

Hospitals are where all the sick people are.

Forums can be like empty rooms, and generally when occupied everyone has some kind of issue with something. And then everything looks like their problem, however tenuously, or their problem is unique/special defying logical problem solving and accepted science. I swing at the pitches I can hit.

CubeMX can replicate broken ideas/concepts very quickly, if the underlying boilerplate is flawed, everything it generates will fail, examples with bugs get copied into everything. Nothing is tested properly.

Tips, Buy me a coffee, or three.. PayPal Venmo
Up vote any posts that you find helpful, it shows what's working..

GS1 · ‎2019-08-20

Ok, understood!

Concerning writing to a buffer, etc.: Thats almost about how I do it.

I collect the incoming data internally until my reserved buffer (currently only appx. 6000 Bytes as I have to store 4 different files for different scan rates at a time, etc.) is filled almost to the available end and then I write the whole buffer to the card. But by doing it this way the number which has to be written to the file is not dividable by 32 in most cases.

Copying a remainder of %32 (modulo) to the beginning of the next buffer is a better solution than my current one and I will consider this in the next release so the file is not needlessly filled with zeroes at the end of the written buffer.

Thanks again for your support!