We have recently been seeing some GUI freezes happening in one of our products. We had a process that creates and saves logs to a CSV file stored on a SD card in our unit. During this save process, we have a handful of task switching events going from our Frontend to our backend. The front end prompts a nondismissible modal to indicate a log is being saved, while the backend handles the actual saving and logging to the SD card. It also handles a large number of calculations we’re doing during the save process.
We don’t have a surefire way to recreate the freeze, instead we have some automation setup within our Model.tick() to “click” on spots of our screen depending on a handful of flags that get set for the currently shown screen. This automation eventually manages to hit the freeze, it varies how long it takes. Sometimes it happens within the first 10 or so iterations, and other times we could go through the entire process 100 times before seeing the freeze. We have also hooked up our automation for a handful of similar screens that look similar to the process that is causing our freeze. This has so far not resulted in a freeze. UI wise, there isnt a whole lot of difference between these screens and the ones that cause the freeze. There is a good amount of difference however in the backend with the type of calculations we’re doing. They are less intensive compared to the process that ends up freezing.
We have explored several avenues that seemed similar here in the ST forums as well as elsewhere on the internet, but none of the conclusions in those threads have helped us. Our closest solution is this exploration. A coworker of mine recently posted in that thread, with the hopes of someone seeing it and giving us some insight on a potential solution since a lot of the communication seemed to have happened over private messages.
We have setup the FreeRTOS Hard Fault Handler, including writing the registers to our EEPROM that we can read back later after recovering from the frozen state. We have also implemented a similar setup for the Bus Fault and Usage Fault handlers. All three handlers were not touched during any of the freezes we have experienced.
We have four total tasks that we swap between. Our backend task, a frontend gui task, a USB task, and a idle task. We start the backend task as well as the gui task within our main, backend having an above normal priority and the guiTask having a below normal priority. The USBTask is started within the backend Task initialization, but it is being handled by USB_HOST and USB_DEVICE middleware generated by CubeMX. The USB task is a higher priority than our gui task, but lower than the backend task, so we can try to guarantee that when a log is saved, it's saved before the GUI transitions to a new screen. At one point we disabled the USB task, this has been the “best” solution we’ve found. The freeze continues to happen, but we only experienced it twice on two out of ten units over a period of 100+ hours. While this is promising, we are not confident enough in the “solution” to move forward with it. Nor do we know why exactly this change has increased the time before a freeze, with limited performance slowdown.
We have several timer interrupts that handle peripherals from updating the time on the device, monitor a sleep timer for when the device isnt in use, as well as monitoring some voltage monitoring for our batteries as well as checking if we’re using AC power instead. These have all been non-essential for the process that is ongoing when a freeze happens, and the freezes continue with them removed.
We also explored disabling DMA2D from our device, but this did not seem to be useful; the freeze continued. If anything, once we disabled DMA2D, the freeze started to happen less than what we have experienced to that point. During this time, we also disabled the ART ACCELERATOR within CubeMX, but it also seemed to have helped anything since we eventually got a freeze on the device.
Tl;dr A device of ours is experiencing an anomaly that is causing the GUI to freeze, preventing the user from continuing, prompting a full power restart. After exploring several possible solutions, disabling nonessential interruptions, graphical enhancements, verbose fault handling, as well task manipulation, none of which have seemed to help. We are still stumped on what is causing our freeze and are looking for guidance.
To expand on our setup:
- TGFX 4.22
- STM32CubeMX 6.7 (Built in 6.5, not merged to higher version)
- STMCubeIDE 1.6.1
- CMSIS V1.02
- FreeRTOS 10.2.1
- FATFS Version R0.12c
- STM32F746
- DMA2D LTDC FMC
- SDIO DMA
- SDMMC1
- Multiple I2C devices
