Inference on STM32N6 using FreeRTOS

[mtv]

Hello,

I am trying to transform the object detection project that can be found in ST model zoo services from bare-metal to FreeRTOS using STM32N6570-DK board.

Following https://community.st.com/t5/stm32-mcus/how-to-use-freertos-with-stm32n6/ta-p/805334

I was able to create a running project with some threads that blink the LEDs and display the camera input to the LCD.

However when I try to add a model that performs inference in one thread, the program breaks in LL_ATON_OSAL_WFE(). I followed the directions in https://stedgeai-dc.st.com/assets/embedded-docs/stneuralart_api_and_stack.html#freertos but they are not very detailed in the context of the whole project.

Is there any running example that performs inference in the NPU inside a FreeRTOS thread?

There is a object detection project in https://github.com/STMicroelectronics/x-cube-n6-ai-people-detection-tracking

that I was able to run but it does not use the CMSIS RTOS2 and the FreeRTOS is configured to support only static allocation of the threads. Is there a reason for that? 

Thanks in advance for any help.

[Julian E.]

Hello @mtv,

Concerning your question "Is there any running example that performs inference in the NPU inside a FreeRTOS thread?"

As you mentioned, yes there are some:

https://github.com/STMicroelectronics/x-cube-n6-ai-people-detection-tracking
https://github.com/STMicroelectronics/x-cube-n6-ai-hand-landmarks
https://github.com/STMicroelectronics/x-cube-n6-ai-multi-pose-estimation

They all use directly FreeRTOS and don't make use of the CMSIS RTOS v2 abstraction layer. The main reason for that is that the developers of such examples didn't go through CubeMX. They've directly used the HAL package and the ST Neural ART API for the inferences.

If you want to reproduce them using CMSIS-RTOS v2, I suggest you look at the FreeRTOS kernel configuration and try to use the same when configuring your kernel. Moreover, I also suggest you take a look on the security configurations (RIF and BSEC). They may be the reason why your program is breaking on LL_ATON_OSAL_WFE().

@SlothGrill Any hint you may have concerning the  LL_ATON_OSAL_WFE() issue?

As for the last part: is configured to support only static allocation of the threads. Is there a reason for that?

Well, I wasn't the one to develop them, but I would say it's just a matter of memory management choice and robustness. If you don't allow dynamic allocation for tasks and don't use anywhere else, you don't have to worry with the heap and you can avoid any kind of heap fragmentation problem once you'll have no heap runtime activity. Also, you can precisely set the tasks stack depth. In the people detection tracking, in the Docs folder, you can see that, for example, by choice only 20 KB were used for threads stacks.

Have a good day,

Julian

[SlothGrill]

Hello

Please also check that the clocks that should not be gated during WFE are indeed not gated (e.g. if the NPU clocks remain ON when you enter WFE, otherwise the NPU will go to sleep as soon as you execute WFE, and no inference can be done ... your system is stuck)

This is done by the lines LL_*_EnableClockLowPower() in the examples provided by @Julian E.

Let us know if this helps.

[mtv]

Thank you both for the replies.

From what I understand I have to focus on the clock and security configurations. 

Regarding the clock I have included the lines with  LL_*_EnableClockLowPower() in the end of the hardware init sequence in the same way as is in the examples by @Julian E. but the problem remains. The examples in ST zoo call this function

I noticed that it has the same description as in 

So what is the difference?

In general I think I have to understand the general clock configuration in the FreeRTOS examples. The SysTick is used by the FreeRTOS, correct? So what is the SYS time base? Because in 

https://community.st.com/t5/stm32-mcus/how-to-use-freertos-with-stm32n6/ta-p/805334

in 2.3 Configure the SYS timer it says that a TIMER must be assigned for the SYS time base.

Thanks for your help

[SlothGrill]

Hello

I'm not an expert, but this depends on your configuration, and don't know if this is the source of the issue... 

But usually with RTOS, an independent timer is used as a sys timebase (eg. TIM2). The same problem can then occur when going into LP modes, if TIM2LPEN is not set (part of the RCC_APB1LLPENR), entering in WFE will let the timer stuck.

(using the approach with LL_xxx_enableClockLowPower functions above enables all the clocks during LP modes --less error prone, but will consume more on the whole--, the __HAL_RCC approach is finer-grained, and thus you have to be careful on what you absolutely don't want to clock-gate when in LP)

again, maybe this is not your issue, but better to make sure to check that out :)

[mtv]

Thank you for the explanation.

Is there something else that may cause LL_ATON_OSAL_WFE()  to break? Something with the caches maybe?

[SlothGrill]

Hello
Not that i'm aware of... the MCU cache is not used (since the mcu sleeps) and in your case you seem to have let the NPUcache (CACHEAXI) on during wfe.

Did you try to enable all the clocks during wfe ? to check if the issue lies there or if this is totally unrrelated ?

You can also remove AI from your project and just do a "wait for event" with free rtos and check if your code breaks if you fear the culprit is in the "ai" stack. i.e. start simple, verify it works, and then add features to your code, this will ease the debug.

Tell us what you test and what are your results for the test you run...

Best regards.

[mtv]

I have been able to run a FreeRTOS project with some threads that manage the LEDs, the camera and the LCD. So if I get an event from the camera, I assume that this part is configured ok. Are you suggesting something else?

The problem starts when I add the inference part. How can I enable the clocks during the wfe? 

The inference code breaks at LL_ATON_OSAL_WFE(). Where do you suggest to add the enable?

do {

/* Execute first/next step */

ll_aton_rt_ret = LL_ATON_RT_RunEpochBlock(&NN_Instance_Default);

/* Wait for next event */

if (ll_aton_rt_ret == LL_ATON_RT_WFE)

LL_ATON_OSAL_WFE();

} while (ll_aton_rt_ret != LL_ATON_RT_DONE);

Thanks 

[SlothGrill]

Hello
Can you enable all clocks during wfe and see if this helps ? (sorry, i don't know what you tested/not tested).

As stated above, this can be done with the code you sent full of LL_xxx_enableClockLowPower functions.

Cheers.