Gaps when playing audio file with DAC

I'm trying to play a sound file with raw PCM data using the DAC:

uint8_t buffer[16384]; // 4-aligned
uint8_t *bufptr[2] = {buffer, buffer + sizeof(buffer) / 2};
int curr_buffer = 0;
volatile int audio_dma_half_complete = 0;
 
f_read(fh, buffer, sizeof(buffer), &br);
 
HAL_TIM_Base_Start(&htim6);
 
sConfig.DAC_Trigger = DAC_TRIGGER_T6_TRGO;
sConfig.DAC_OutputBuffer = DAC_OUTPUTBUFFER_ENABLE;
if (HAL_DAC_ConfigChannel(&hdac, &sConfig, DAC_CHANNEL_2) != HAL_OK)
 return 1;
 
if (HAL_DAC_Start_DMA(&hdac, DAC_CHANNEL_2, (uint32_t *)buffer, file_size,
 DAC_ALIGN_8B_R) != HAL_OK)
 return 2;
 
while (1) {
 while (!audio_dma_half_complete);
 f_read(fh, bufptr[curr_buffer], sizeof(buffer) / 2, &br);
 if (br == 0)
 break;
 audio_dma_half_complete = 0;
 curr_buffer = !curr_buffer; // toggle buffer half
}
 
HAL_TIM_Base_Stop(&htim6);
HAL_DAC_Stop_DMA(&hdac, DAC_CHANNEL_2);

Note that the file contains raw PCM data that as already been reduced to 8 or 12 bits, resp.

This kind of works, but there are regular gaps of about 600 ms of silence, followed by 2-3 s of the signal.

While the signal itself makes me think I have an endianness problem, the main issue are the gaps. (This diagram shows a gap for 12R data; for byte data, the gaps are more than 1 second wide.)

My first explanation were underruns, but doubling the SDMMC speed had absolutely no effect.

The second explanation were external interrupts, which in theory should be regular but run very shortly. But disabling all EXTI interrupts had again no effect.

What else might cause these gaps? Is it a priority problem between different peripherals?