STM32H742 intermittent SD card lockup

[embeddedvoyager1nova]

Hello,

I am working on an application that uses the SDMMC peripheral to read the contents alternating audio files every 150ms. The buffers i have to store the data are large enough to store the entire file in a single burst within RAM. The problem I am running into is that intermittently, the sound "loops" indicating a lockup on the sound subsytem. I've added a small timer in SysTick in order to quantify how long and often this lockup occurs as the logic is running in a lower priority interrupt. I find that the system locks up between 300ms-1200ms. When I listen for this audibly and stop the code, I find that the code is stuck in:

if(BSP_SD_ReadBlocks_DMA((uint32_t*)buff,
                             (uint32_t) (sector),
                             count) == MSD_OK)
    {
      ReadStatus = 0;
      /* Wait that the reading process is completed or a timeout occurs */
      timeout = HAL_GetTick();
      while((ReadStatus == 0) && ((HAL_GetTick() - timeout) < SD_TIMEOUT))
      {
      }
      /* in case of a timeout return error */
      if (ReadStatus == 0)
      {
        res = RES_ERROR;
      }

in sd_diskio.c. The project is generated using STM32CubeMX on v1.13 of the processor's firmware. I have tried various methods of troubleshooting this including:
both polling and DMA implementations generated by Cube
increasing and slowing the clock divider for the SDMMC peripheral

enabling and disabling DMA cache maintenance and scratchpad buffers

forcing ISB and DSB while waiting on ReadStatus

defining my buffers in RAM_D1 and not DTCM while also making sure they're on 32 bit boundaries

debugging and putting breakpoints on failure conditions in HAL_SD_IRQHandler()

changing SD_Timeout in sd_diskio.c to see if the hang up time changes

verifying that the data passed to the IDMA is correct in the DMA template generated by Cube

Changing my clock source from PLL1Q running at 64MHz to PLL2R running at 48MHz

enabling and disabling DCache
internal pull ups on the SDMMC lines even though the lines are pulled by external hardware already
reinitializing the SD card

different PCBs/SD cards

changing the maximum output speed on the GPIO peripheral to Very High for the SDMMC pins

disabling LFN functionality to simplify the code execution

verified interrupt priorities

Nothing I seem to change changes the frequency of this problem, nor helps mask it

My interrupt priorities are as follows:

	NVIC_SetPriority(SysTick_IRQn, 0);
	NVIC_SetPriority(SDMMC1_IRQn, 1);
	NVIC_SetPriority(OTG_FS_IRQn, 2);
	NVIC_SetPriority(TIM7_IRQn, 3);
	NVIC_SetPriority(USART3_IRQn, 4);
	NVIC_SetPriority(UART8_IRQn, 5);
	NVIC_SetPriority(UART7_IRQn, 5);
	NVIC_SetPriority(USART6_IRQn, 5);
	NVIC_SetPriority(USART2_IRQn, 5);
	NVIC_SetPriority(USART1_IRQn, 5);
	NVIC_SetPriority(TIM8_UP_TIM13_IRQn, 6);

All of the USART interrupts in my current test are not running, and user code is being executed in TIM8 that is calling SD car functions. USB is also not initialized. TIM7 is a timer for a DAC to play audio from my buffers

Cube SDMMC settings, with SDMMC1 global interrupt enabled

FATFS settings with DMA template enabled in advanced settings

I have spent weeks trying to troubleshoot this issue across multiple projects using the same processor or one in the family, but it keeps peaking it's little head. Any idea on what I could be missing?

Any help would be appreciated.

[embeddedvoyager1nova]

Wouldn't a hardware problem normally cause an error flag in code in either HAL_SD_IRQHandler, SD_Read, or any of the other catches? I am not hitting breakpoints on any of the failure states of the generated code. Data timeouts, overruns, etc. Even with an SD_Timeout of 50ms, I randomly get stuck in interrupt for 300-1200ms 

[AScha.3]

Maybe , or not. dont know.

Just if signals are fine, no errors in transmitted data - and it just works 100% .

I made an audio player, with H743 , (hobby) and this running without any hardware-problem , more than a year now.

But still other problems...actually i get "hang up" on playing flac files without gap between tracks (for live cd , no pause between tracks); i dont know why, because on most tracks/CD it working perfect, only on some it loops in the decoder then....but starting this track separate, with pause, plays perfect. ...difficult...even no debug, because if i stop in debug on the gapless play mode, everything looks ok, running the prog then...plays. no hang up. 

But these are problems on "higher level" , i suppose something with the timing, when buffers are filled, but for now no idea, why.

[embeddedvoyager1nova]

This sounds potentially similar. All of the other audio that I play in other states doesn't appear to have this issue. It only seems to persist when there is a very small gap between opening, playing, and closing files. In this case, the gap in <100 ms. In other cases, it's many seconds.

[AScha.3]

So maybe you should tell me, what you do there...playing ...what ?

+

Now, from what you tell, seems no hardware problem...but i have to know, how you " play  ".

Basically i use DMA->SAI->DAC .

+

When exactly the problem happens....repeatable , exact ?

[embeddedvoyager1nova]

I am playing .wav files read off of ana SD card with SDMMC. The one that I am having problems with particularly is a "tick" sound that is 50ms long, and is playing every 150ms. 

I am opening a file handle with f_open, f_reading the data into a buffer with SDMMC using the "generic DMA template" generated by Cube(which uses the SDMCC IDMA), and then setting "sound loaded" flag to tell TIM7 running at 22050Hz that a sound is ready, in which the buffer is iterated through to give data to the DAC.

so it's basically SDMMC -> IDMA -> TIM7 -> DAC

dacLeft = (LeftAudioData+LeftAudioData2)>>5;
		
		
HAL_DAC_SetValue(&hdac1,DAC1_CHANNEL_1,DAC_ALIGN_12B_R,dacLeft&0xFFF);

In this case, I have 2 "left" and 2 "right" channels where I'm averaging the 2 digital channels together to "overlay" them, while shifting 16 bit data down to 12 bit, hence the >>5

[AScha.3]

Ok, ---- strange.

I read sd-card, wav /flac/mp3 , to a 128K buffer (async.) ;

circular DMA sending SAI -> DACs 32b 2ch data, + 2x 16b data to 16b DACs; 32b goes to ES9038 , which is master.

In callbacks data is ( decoded or not ) copied to the play-buffers .

Difficult is the file end: read has to stop, data buffer missing data is filled with 0x00 ;

then next file is searched/opened , read needed mode (44k...96k, 16...32b, etc), set mode and start play.

My actual "problem" comes, when i try to play uninterrupted to next file.

But only at some files, repeatable , all other playing always fine.

+

in your case...no circular DMA used ?  

+ you play to internal 12b dac , right ? why at half rate ?

[embeddedvoyager1nova]

In my case, the file is not closed until the audio is finished playing, so there is inherent delay between file end and needing to close the file 

No circular DMA is used because files tend to only need to be played once, or an occasional loop. My buffers are 32.7K. I read an entire buffer's worth of data on the initial open, and the overwrite half of the buffer back and forth when that audio has already been played. In our tests, 22k as not been fast enough to need to use DMA to pass data to a DAC. simply running a 22k timer suffices.

On the speakers we use, we find there is no discernable difference between 44k and 22k audio, so to save on time waiting for SD card functionality, overall space necessary in RAM to store the data, as well as execution time to do other things the application is actually meant to do, we export all of our audio as 22k 16 bit. The audio buffers are normally "big enough" that if there is some sort of small delay caused by some lower priority interrupt, the audio is uninterrupted due to the buffer being large enough, and the timer being a higher priority.

[AScha.3]

Ok, but are your files bigger than the 32K buffer ? then need reload, and this will give "noise", because you never will "hit" it in time.

So what happens : sound loops...or a tick noise...or both ?

[embeddedvoyager1nova]

The smallest file is 4KB. This is the "tick" sound file. I have files that range up to 1MB as well. 

When this error occurs, there is stale data in the audio buffer because the SD does not seem to be returning fresh data while also not giving a failure condition(FR_NOT_READY, etc). Because there is stale, "played" data, the audio loops(think portions of background music playing over and over again) until the SD cards finally gives it new data for whichever file it's trying to play at the time. This problem is shown more when im playing the "tick" sound because it is played so frequently. 

There is no "noise" when needing to load/reload files when one is being played, because the half of the buffer that has yet to be played masks the f_open and f_read time of new data when necessary.

[AScha.3]

Not sure, i understand it all...

but seems: small file playing is fine ?

The looping only sometimes at big files - right ?

Did you check with different sd-cards ? (SanDisk is top.)

SD-cards can have delays of 100ms and more - sometimes - , this in the spec of these cards, but usually respond in 1...2ms.