FatFs unrealiable under high load on STM32H743

[JP_ama]

Hello,

I am using a STM32H743 with a W25Q128 external flash memory running FreeRTOS and FatFs. USB is configured for mass storage, to copy files from a computer to the flash memory. The following function is used to list all the files and directories in a directory and put them in a list:

void IndexFilesInDir(const char *path, SFileMgrList* psReturnList)
{
    FRESULT res;
    DIR dir;
    FILINFO fno;
    uint8_t u8Count = 0;

    res = f_opendir(&dir, path); /* Open the directory */
    if(strlen(path) != 0)
    {
    	strcpy(&psReturnList->cFilename[0][0], "...");
    	psReturnList->bIsDir[0] = true;
    	psReturnList->bHasParentDirectory = true;
    	u8Count++;
    }
    else
    {
    	psReturnList->bHasParentDirectory = false;
    }
    if (res == FR_OK) {
        for (;;) {
            res = f_readdir(&dir, &fno);                   /* Read a directory item */
            if (res != FR_OK || fno.fname[0] == 0 || u8Count >= 128) break;  /* Error or end of dir */
               	if (fno.fattrib & AM_DIR && strstr(fno.fname, "System Volume Information") == NULL)
               	{
               		psReturnList->bIsDir[u8Count] = true;
               		strcpy(&psReturnList->cFilename[u8Count][0], fno.fname);
                   	u8Count++;
               	}
               	else
               	{
               		char *dot = strrchr(fno.fname, '.');
               		// Is this a .wav file?
               		if (dot != NULL && strcmp(dot, ".wav") == 0)
               		{
               			psReturnList->bIsDir[u8Count] = false;
               			strcpy(&psReturnList->cFilename[u8Count][0], fno.fname);
               			u8Count++;
               		}
               	}
        }
        psReturnList->endIndex = u8Count;
        f_closedir(&dir);
    } else {
        printf("Failed to open \"%s\". (%u)\n", path, res);
    }
}

This is based on the example from the FatFs website. Everything works fine until I start processing some audio data (= lots of SAI DMA interrupts). Then f_readdir returns random garbage or does not all the files or it lists files from another directory from time to time. Here's what I've tried and observed:

- Disabling the caches doesn't solve or change the problem.

- Reading and writing files via USB always works perfectly fine, even under high load conditions. I conclude from this that my low level SPI driver for the W25Q128 works fine.

- FatFs is only used in one Task.

- The microcontroller is not overloaded, other RTOS Tasks still work fine. Temporarily removing all the other tasks does not help or change anything.

Does anyone have any idea what could be causing this behavior?

Thanks!

[FBL]

Hello @JP_ama 

For instance, the issue is unclear. 

Then f_readdir returns random garbage or does not all the files or it lists files from another directory from time to time. 

It seems to be independent of the platform and possibly linked to the implementation. A possible way to try to disable SAI interrupts in file system task.

[JP_ama]

Thank you @FBL 

So, do I understand you correctly that this is a known issue?

Deactivating SAI interrupts in the file system task is not really an option, as this would lead to dropouts in the audio stream.

Thanks!

[AScha.3]

>under high load

can you tell - 

- data rate on SAI (in or out) + mode (INT , DMA (+ blocksize then))

- data rate on fatfs -> USB stick (i assume)

- int priorities you set 

[JP_ama]

Hi @AScha.3 

SAI is running at 48 kHz or (worst case) 96 kHz with 24 bit. That means 4 bytes per channel = 8 bytes per sample = 768 kByte/s. Mode is DMA and the interrupt is triggered for every sample, because I am doing some zero latency FIR filtering.

Data rate for the USB Mass Storage is around 70 kByte/s write (the flash memory is quite slow) and up to 250 kByte/s read. These data rates still work for the audio running at 48 kHz. At 96 kHz it drops down to 30 kByte/s write for example. But it still is 100% reliable. I never get any read or write errors via USB with audio processing running at the same time.

However, when I am just listing the files (not even reading or loading files) using FatFs on the STM32 while audio processing is running I get random garbage every second to fifth time I read a directory. So, the data rate is really low. When I stop the SAI DMA it's 100% reliable as well.

And these are my NVIC settings:

Thanks!

[Tesla DeLorean]

I wouldn't assume DISKIO is bullet-proof from casual observation.

You're using 4KB blocks on the QSPI?

Would perhaps worry about thread-safe and concurrent operation of FATFS and DISKIO. I'd definitely recommend serializing access to SPI and QSPI FLASH resource. You say it's in one task, which presumably should achieve that.

Would instrument DISKIO to better understand the interactions and failure.

Would get a current version of FATFS, not 2017 version ST ships.

[JP_ama]

Hi @Tesla DeLorean 

Yes, I am using 4 kByte blocks. The W25Q128 is running in standard SPI configuration, not Quad SPI. It's fast enough for my application. The filesystem task is not accessing the flash memory via FatFs when USB is connected (and the other way around).

---

@Tesla DeLorean wrote:

Would instrument DISKIO to better understand the interactions and failure.

How could I do that?

I will try and replace FatFs with the latest version.

Thank you!

[AScha.3]

Hi, 

its late now...but in brief : your problem is int priority.

ie. i have on H743 (and now on H563 beginning) an audio player running. 🙂

read from SDcard or USB stick, data out with SAI /DMA circular (same you, almost); with FatFS.

when playing cd audio 44/16 , cpu load is about 3% ; with FLAC file at 96k and 32bit also no problem, cpu still under 40% load; and decoder + 2x biquad filter (double precision) needing most time here, cpu at 200MHz clock, to run cool.

so "play" with your int priorities ! all on same level is nonsense.

try: usb 4 , DMA 9+10, tim so important? if not, 12.

[JP_ama]

@jamesoleg2 

The SAI DMA interrupt is the only relevant interrupt. I used one timer to benchmark execution times. I disabled it just to make sure, no difference. USB completely disabled doesn't change anything either.

The file manager task has the lowest FreeRTOS priority. There are two other DSP tasks which obviously need a higher priority as these are critical for the audio processing. Those tasks receive notifications from the DMA ISR when a new sample block has been acquired. I've had the impression that disabling these two tasks makes the problem appear a little less frequently, but listing directory contents is still unreliable.

I am using this code for the W25Q128 SPI: https://github.com/nimaltd/w25qxx 

I don't see any resource conflicts there. I ensured task stacks are big enough. I double checked for buffer overflows, no issues.

As mentioned before, without the SAI DMA running no problems. But that also means the microcontroller is essentially idle. It's hard for me to debug this as this is not happening all the time. Any tips or tricks there? Also, when stepping through the code it happens less frequently. All I can say so far is that f_readdir is not returning the correct data. It's not completely random garbage like it would be the case with memory corruption. Usually it's simply not listing all the files, or it's listing a bunch of files mixed with files from another directory. Listing a bunch of folders without a name is common, too.

@AScha.3 

I don't understand how interrupt priorities could be the issue here. But I tried what you suggested and also tried different combinations. No success. SPI communication with the W25Q128 is not in interrupt mode. I disabled the timer, it's not needed anyways. USB enabled or completely removed makes no difference either. Also, since a lower number means higher priority, wouldn't it make more sense to keep the DMA interrupts low around 2 (= high priority) as these are the most important and time critical thing? And set the USB to a high number (= very low priority) since it is not time critical at all? 

I just wanted to illustrate that the USB Mass Storage data transfer works 100% reliable, even under high load despite having a very low priority. And it shares almost identical code for the diskio. Whereas simply listing filenames (not a lot of data to read, only takes a very short amount of time) from the FreeRTOS task fails frequently.

Thanks for all your suggestions and your help guys!

[AScha.3]

@JP_ama  , ok . I just believed, it could be related to the int prio. , because this was very important in my setup.

DMA is the most important thing in runniing continuous data stream - right. but if buffers are big enough, i am using 8K int32 , every callback has up to 40ms time for sevice; so INT prio can be very low, on the other hand in this time gap data needs to be read (or write) from USB or SD, so this has to be as fast as possible - thats why USB or SD needs higher prio. (completely different than first imagined - but now obvious ).

According to your new tests i have no idea whats the root cause.

Just - maybe, you should try a more actual version of fatfs (if you use the version by STM - its 6 y old!)

http://elm-chan.org/fsw/ff/00index_e.html