STM32H742 intermittent SD card lockup

[embeddedvoyager1nova]

Hello,

I am working on an application that uses the SDMMC peripheral to read the contents alternating audio files every 150ms. The buffers i have to store the data are large enough to store the entire file in a single burst within RAM. The problem I am running into is that intermittently, the sound "loops" indicating a lockup on the sound subsytem. I've added a small timer in SysTick in order to quantify how long and often this lockup occurs as the logic is running in a lower priority interrupt. I find that the system locks up between 300ms-1200ms. When I listen for this audibly and stop the code, I find that the code is stuck in:

if(BSP_SD_ReadBlocks_DMA((uint32_t*)buff,
                             (uint32_t) (sector),
                             count) == MSD_OK)
    {
      ReadStatus = 0;
      /* Wait that the reading process is completed or a timeout occurs */
      timeout = HAL_GetTick();
      while((ReadStatus == 0) && ((HAL_GetTick() - timeout) < SD_TIMEOUT))
      {
      }
      /* in case of a timeout return error */
      if (ReadStatus == 0)
      {
        res = RES_ERROR;
      }

in sd_diskio.c. The project is generated using STM32CubeMX on v1.13 of the processor's firmware. I have tried various methods of troubleshooting this including:
both polling and DMA implementations generated by Cube
increasing and slowing the clock divider for the SDMMC peripheral

enabling and disabling DMA cache maintenance and scratchpad buffers

forcing ISB and DSB while waiting on ReadStatus

defining my buffers in RAM_D1 and not DTCM while also making sure they're on 32 bit boundaries

debugging and putting breakpoints on failure conditions in HAL_SD_IRQHandler()

changing SD_Timeout in sd_diskio.c to see if the hang up time changes

verifying that the data passed to the IDMA is correct in the DMA template generated by Cube

Changing my clock source from PLL1Q running at 64MHz to PLL2R running at 48MHz

enabling and disabling DCache
internal pull ups on the SDMMC lines even though the lines are pulled by external hardware already
reinitializing the SD card

different PCBs/SD cards

changing the maximum output speed on the GPIO peripheral to Very High for the SDMMC pins

disabling LFN functionality to simplify the code execution

verified interrupt priorities

Nothing I seem to change changes the frequency of this problem, nor helps mask it

My interrupt priorities are as follows:

	NVIC_SetPriority(SysTick_IRQn, 0);
	NVIC_SetPriority(SDMMC1_IRQn, 1);
	NVIC_SetPriority(OTG_FS_IRQn, 2);
	NVIC_SetPriority(TIM7_IRQn, 3);
	NVIC_SetPriority(USART3_IRQn, 4);
	NVIC_SetPriority(UART8_IRQn, 5);
	NVIC_SetPriority(UART7_IRQn, 5);
	NVIC_SetPriority(USART6_IRQn, 5);
	NVIC_SetPriority(USART2_IRQn, 5);
	NVIC_SetPriority(USART1_IRQn, 5);
	NVIC_SetPriority(TIM8_UP_TIM13_IRQn, 6);

All of the USART interrupts in my current test are not running, and user code is being executed in TIM8 that is calling SD car functions. USB is also not initialized. TIM7 is a timer for a DAC to play audio from my buffers

Cube SDMMC settings, with SDMMC1 global interrupt enabled

FATFS settings with DMA template enabled in advanced settings

I have spent weeks trying to troubleshoot this issue across multiple projects using the same processor or one in the family, but it keeps peaking it's little head. Any idea on what I could be missing?

Any help would be appreciated.

[embeddedvoyager1nova]

Small file plays most times fine. Big file is playing in background on a different DAC channel. So 2 files playing at the same time. Small file being played often occasionally triggers system lockup, likely due to the frequent f_open, f_read, f_close every 150ms. The looping is heard from the big file because the big file is long and held in the 32K buffer before new data is put into the buffer when SD card responds.

I currently only have access to ADATA brand cards.

The ~100ms delay is fine and accounted for due to the size of the audio buffer. I'm having a problem where most of the time, the delay is upwards of 700ms-1000ms

[AScha.3]

So 150ms /22k -> 3K samples, 16b -> about 6KB ; just read once and then play from RAM .

H742 has > 400KB RAM, so no problem. And no overlap on file access possible.

Try this ?

[embeddedvoyager1nova]

Reading the file and then resetting the the pointer to just constantly play from RAM would be a patch. I could do the same thing by storing that specific audio file in FLASH. I am trying to find a solution to the issue instead of patching out that specific instance. I have this issue intermittently across multiple different projects on different custom PCBs that use the same processor and would like to try to find the underlying cause. Happening frequently on small audio files is just happenstance as I know it also happens on larger files occasionally. 

The time that it takes to load audio most times is ~10ms as indicated by "maxDifference". The other variables are as follows:

loadFails: file successfully opened, but initial f_read fails

openFails: the file does not open properly

closeFails: f_close fails

reopens: the file was opened, loaded, and subsequent f_reads fail when playing longer audio files

In the time it took from the first screenshot to now:

The code has reported that it took 733ms to return from a singular SD card function with no reports on failure. I just do not understand where this hang up is occurring. In this case, it is "just" audio, but data integrity and SD card responsiveness matter when it comes to things like customer bootloaders for in the field upgrades/updates. 

[AScha.3]

Ok, but you can never trust in these cards 100% , they are complex and do internal copy, refresh, wear levelling....and will fail sooner or later. Cheap cards probably more early , even on read access only !

I have such a card, 64GB, perfect at first, was reading /playing some months, then problems with click or chip noise randomly coming, then even data errors, that made strong noise.

So i was searching, reading about the problems...limited write times, ok, but also some degradation just from reading !

So buffering short files to RAM is no patch - is needed ! If it should work more than some weeks...

 And use only top brands...SanDisk is best, Samsung also good; all others....lucky, if they work long time.

[embeddedvoyager1nova]

I understand degradation in the long term. We have found these ADATA cards to be most reliable in our applications thus far, doing testing across multiple brands for months in different applications. 

I am saying it is a patch in the sense that I know this issue happens on larger files, just less frequently. I have the time now to essentially force it to happen more frequently and try to find the root cause, but I cannot think of any other causes. Even if i set the SD card timeout to 50ms, I do not see any reported bad return values. 

[AScha.3]

I read about the ADATA cards, rating is not top, but good; so should be ok here.

To exclude the hardware/speed problem, do the mods i recommended:

- add cap at card

- set pin speed medium or med/hi + pullups ON , all lines (also clk )

-- because the >>returns inconsistent FR_DISK_ERR on f_open << is typical for hardware problem