Cummulative data loss in SD write

[rumlyen]

Posted on March 30, 2016 at 05:30

Hello all,

I am facing some data loss while SD card writing. My application needs to save timer count and accerelometer data in a 8KB buffer in every 10ms (Tim_9 interrupt) and when the buffer is full, it'll write to SD card. Tim_9 interrupt has higher priority (0) than SDIO (1). It seems to loose 10~12 samples in every ~3s; What might be the reason? Is it missing data when processor executes SD write command ? I've tried double buffer: to store samples in another buffer when SD write executes. No improvement. Is my approach wrong? I request your comment to fix it. Thank you. int WriteIndex= 0; char BufferSting[2][8192]; int main(void) {     ....... memset(&fs32, 0, sizeof(FATFS));   // SD card initialization res = f_mount(0, &fs32); memset(&fil, 0, sizeof(FIL)); res=f_open(&fil, filename, FA_OPEN_ALWAYS | FA_WRITE); ..... while(1) { PWR_EnterSleepMode(PWR_Regulator_ON, PWR_SLEEPEntry_WFI); if(sample_flag == 1)  // 10ms interrupt from Timer 9 { Acc_GetXYZ_Data16(&x,&y,&z); // Accerelometer data. if(WriteIndex >= 8192)  // Buffer full {         f_lseek(&fil, fil.fsize);     f_write(&fil, BufferSting[bf], strlen(BufferSting[bf]), &BytesWritten);  // SD card writing     f_sync(&fil);   WriteIndex=0;    if (bf==0) bf=1;   // alternative buffer selection else bf=0; WriteIndex += sprintf(&BufferSting[bf][WriteIndex],''%d\t%d\t%d\t%d\r\n'',CNT32,x,y,z); }          else { WriteIndex += sprintf(&BufferSting[bf][WriteIndex],''%d\t%d\t%d\t%d\r\n'',CNT32,x,y,z); } sample_flag = 0; } }     void TIM9_IRQHandler(void) { if (TIM_GetITStatus(TIM9, TIM_IT_Update) != RESET) { CNT32= TIM5->CNT;  // 32-bit count TIM5->CNT = 0; TIM_ClearITPendingBit(TIM9, TIM_IT_Update);         sample_flag = 1; } }  

[TDK]

Posted on March 30, 2016 at 06:46

How are you verifying that you're losing data?  What specifically is making you think you're missing data?

I would expect f_write to take more than 10ms, which means your code that messes with sample_flag will likely be unhandled during that time.  Why not put the data into a circular buffer in the interrupt instead?  Write the first half into a file when it fills up, then write the second half when that fills up.

[Tesla DeLorean]

Posted on March 30, 2016 at 07:30

Your code does not handle the spanning cases at all well. You use strlen() when you already know the length, and it is more than likely the data doesn't fit cleanly into 8192 bytes.

I'd move the xyz and sprintf stuff in the IRQ presuming they can be done in the 10ms window, have that dispatch whole 8192 blocks to the foreground writer, and have the memory array slightly larger than 8192 to manage the spanning. Ie copy the remaing overflow to the front of the next buffer frame.

[rumlyen]

Posted on March 30, 2016 at 07:32

Thank you for your reply.

I counted out the samples through MATLAB. Timer 5 Count here is a external pulse count and some experiments indicate certain data loss. Using same code, I added time-stamp value to further check number of samples in second. :( Can you please refer any sample code for circular buffer? I primarily thought SDIO DMA operation might not interfere sample saving in alternative buffer. I saved the data in string buffer(s) to write in a text file to ease out further process. I guess this doesn't slow down the process. I request your kind comment.

[rumlyen]

Posted on March 30, 2016 at 07:47

Thank you Clive. I got that except last point, how to copy the remaining overflow to the front of next buffer frame ?

Do I need another buffer or one buffer is enough to do that? I am confused: can it really save sensor data parallel to SD write (DMA) operation. Or there would always a few loss. Thank you for your time spending in this topic.

[Tesla DeLorean]

Posted on March 30, 2016 at 08:16

I know this can be done without data loss, it is your method which is flawed.

You need the buffers to be larger to accommodate the overflow, without running the numbers I suspect 32 bytes would sufffice. You always dispatch a 8192 byte write, and copy the x-8192 bytes from the overflow tail to the head of the new buffer and advance the pointer. I would have the writer in the foreground, and everythings else in the background, so the blocking via fatfs and sdio doesn't choke the data accumulation.

[AvaTar]

Posted on March 30, 2016 at 10:43

> You need the buffers to be larger to accommodate the overflow, without running the numbers I suspect 32 bytes would sufffice. You always dispatch a 8192 byte write, and copy the x-8192 bytes from the overflow tail to the head of the new buffer and advance the pointer.

Alternatively, you can stuff your entries up with spaces to a fixed 2^n size, say 16, 32, or 64 byte, so that they integrally add up to your buffer size. That way, you avoid a complex double-buffering by sacrificing some disk space and perhaps some write performance. Most probably these are negligible, compared to the nonlinearities introduced by the SD card manager block sizes.

[rumlyen]

Posted on March 31, 2016 at 05:32

Thank you.

I've pushed data saving in Timer 9 int handler and SD writing in foreground. It's now accurately saving data. I've also included RTC time stamp to see total no of samples per second. I wonder would it work accurately with 1ms Timer 9 interrupt or a system with multiple interrupts? Honestly, I've not understood well how to put overflow portions in the start of next buffer. Hence I kept 8192 KB double-buffer and saving up-to 8120+ Bytes. My code stands like below: int WriteIndex= 0; char BufferSting[2][8192]; int bf  = 0; // Buffer pointer to store samples int bf1 = 1; // Buffer pointer to write in SD card int main(void) {      ....... memset(&fs32, 0, sizeof(FATFS));   // SD card initialization res = f_mount(0, &fs32); memset(&fil, 0, sizeof(FIL)); res=f_open(&fil, filename, FA_OPEN_ALWAYS | FA_WRITE); ..... while(1) { PWR_EnterSleepMode(PWR_Regulator_ON, PWR_SLEEPEntry_WFI);    if(SD_write ==1)  // SD card write flag when buffer full     {     f_lseek(&fil, fil.fsize);     f_write(&fil, BufferSting[bf1], strlen(BufferSting[bf1]), &BytesWritten);  // bf1= buffer index to write in SD card     f_sync(&fil);     SD_write =0;    // reset SD card write flag    } }     void TIM9_IRQHandler(void) { if (TIM_GetITStatus(TIM9, TIM_IT_Update) != RESET) { CNT32= TIM5->CNT;  // 32-bit count TIM5->CNT = 0; Acc_GetXYZ_Data16(&x,&y,&z); // Accerelometer data. RTC_GetTime(RTC_Format_BIN, &RTC_TimeStructure); //Time Stamp if(WriteIndex >= 8120)  // Buffer full, leaving some space (8192-8120) for overflow {    WriteIndex=0;    if (bf==0) { bf=1;    //  alternative buffer selection to store bf1=0;   // buffer to write in SD card } else { bf=0;    // alternative buffer selection to store bf1=1;   // buffer to write in SD card } SD_write = 1; // SD card write flag //WriteIndex += sprintf(&BufferSting[bf][WriteIndex],''%d\t%d\t%d\t%d\r\n'',CNT32,x,y,z); WriteIndex += sprintf(&BufferSting[bf][WriteIndex],''%0.2d\t%0.2d\t%0.2d\t%d\t%d\t%d\t%d\r\n'',RTC_TimeStructure.RTC_Hours, RTC_TimeStructure.RTC_Minutes,RTC_TimeStructure.RTC_Seconds,CNT32,x,y,z); }          else { //WriteIndex += sprintf(&BufferSting[bf][WriteIndex],''%d\t%d\t%d\t%d\r\n'',CNT32,x,y,z); WriteIndex += sprintf(&BufferSting[bf][WriteIndex],''%0.2d\t%0.2d\t%0.2d\t%d\t%d\t%d\t%d\r\n'',RTC_TimeStructure.RTC_Hours, RTC_TimeStructure.RTC_Minutes,RTC_TimeStructure.RTC_Seconds,CNT32,x,y,z); } TIM_ClearITPendingBit(TIM9, TIM_IT_Update);         } }

[rumlyen]

Posted on April 05, 2016 at 08:56

Hello again,

I'm facing another issue here. I'm having right amount of data now, but, periodic spikes in pulse count (probably for a delay introduced by SD write). To figure out the reason, I've tried fixed length double buffer with only Pulse count and Time-stamp. It indicates when I employ following 'sleep' instruction, some delay appears in pulse count when SD write starts (hence, a spike; say count value turns ~110000 instead of ~90000 count for 1024, 2048,..th sample for the following code). It works well without sleep instruction. What should I do to solve this issue. Need to employ circular buffer or sleep instruction hasn't been configured here correctly. I need sleep mode for highest possible power saving. I sincerely request your kind comment. I'm stuck with the code: struct storage{           // 1024 *8 Byte= 8192 Byte     uint8_t Hour[1024];     uint8_t Min[1024];     uint8_t Sec[1024];     uint8_t SSec[1024];     uint32_t Pulse[1024];    }; struct storage Sbuffer[2]; uint8_t s_flag = 0 ; // flag for Data Storage, 0=buffer1, 1= buffer2 uint8_t w_flag = 1 ; // flag for SD Buffer write, 0=buffer1, 1= buffer2 uint16_t Bf_cnt = 0; // Buffer index to store samples    void TIM9_IRQHandler(void) {      if (TIM_GetITStatus(TIM9, TIM_IT_Update) != RESET)     {         Sbuffer[s_flag].Pulse[Bf_cnt]=TIM5->CNT;           TIM5->CNT = 0;             Sbuffer[s_flag].SSec[Bf_cnt]= (uint8_t)(RTC->SSR);    //  Timer_temp = (uint32_t)(RTC->TR & 0x007F7F7F); // Sbuffer[s_flag].Hour[Bf_cnt] = (uint8_t)((Timer_temp & (RTC_TR_HT | RTC_TR_HU)) >> 16);  // Sbuffer[s_flag].Min[Bf_cnt] = (uint8_t)((Timer_temp  & (RTC_TR_MNT | RTC_TR_MNU)) >>8); //  Sbuffer[s_flag].Sec[Bf_cnt] = (uint8_t)(Timer_temp  & (RTC_TR_ST | RTC_TR_SU));       //Buffer Switching      if(Bf_cnt==1023)     {    Bf_cnt=0;         SD_write = 1;                  if (s_flag==0)  // Setting Next Storage Flag and SD write Flag         {         w_flag=0;         s_flag=1;         }         else         {         w_flag=1;         s_flag=0;         }         }     else     {             Bf_cnt++;     }              TIM_ClearITPendingBit(TIM9, TIM_IT_Update);    // Clear Update Flag }     } void Interrupt_Conf(void) {   NVIC_InitTypeDef NVIC_InitStructure;   NVIC_PriorityGroupConfig(NVIC_PriorityGroup_2);   NVIC_InitStructure.NVIC_IRQChannel = USB_LP_IRQn;    // USB Mass Storage   NVIC_InitStructure.NVIC_IRQChannelPreemptionPriority = 2;   NVIC_InitStructure.NVIC_IRQChannelSubPriority = 0;   NVIC_InitStructure.NVIC_IRQChannelCmd = ENABLE;   NVIC_Init(&NVIC_InitStructure);        NVIC_InitStructure.NVIC_IRQChannel = SDIO_IRQn;   NVIC_InitStructure.NVIC_IRQChannelPreemptionPriority = 1;   NVIC_InitStructure.NVIC_IRQChannelSubPriority = 0;   NVIC_Init(&NVIC_InitStructure);       NVIC_InitStructure.NVIC_IRQChannel = TIM9_IRQn;   NVIC_InitStructure.NVIC_IRQChannelPreemptionPriority = 0;   NVIC_InitStructure.NVIC_IRQChannelSubPriority = 0;   NVIC_InitStructure.NVIC_IRQChannelCmd = ENABLE;   NVIC_Init(&NVIC_InitStructure);    } int main(void) {      Interrupt_Conf(); // Function Description Below ...... memset(&fs32, 0, sizeof(FATFS)); res = f_mount(0, &fs32); memset(&fil, 0, sizeof(FIL)); sprintf(filename, ''%03d.BIN'', GetNextIndex('''') ); res=f_open(&fil, filename, FA_OPEN_ALWAYS | FA_WRITE);   while (1) { PWR_EnterSleepMode(PWR_Regulator_ON, PWR_SLEEPEntry_WFI);   // Without Changing Sytem clock 32MHz (PLL Driven HSE) if(SD_write ==1) { f_lseek(&fil, fil.fsize); f_write(&fil, &Sbuffer[w_flag], 8192, &BytesWritten); f_sync(&fil);    SD_write =0; }     }