How to use custom AI model for Stm32n6570?

[Anhem]

My question is same the title. I know that you have zoo model with custom training script to training or fine tuning based on provided model. But I have a new model, that is trained in other environment, I exported it to ONNX. How can I deploy this model to this board? Thank you.

[Julian E. · Accepted solution]

Hello @Anhem ,

The first information to have is that it is a tool called the ST Edge AI Core that convert your model form TF, ONXX or TFlite to C code for all the tools that we have. Then you have different alternatives depending on what you do:

The most straightforward is to use STM32CubeMX and X-Cube-AI. You can do multiple things and mainly validate a model on target and generate example code.

I described it here for another eval board, just select the N6 and do the same.

Solved: Re: Deplying a deep learning model on NUCLEO-H753Z... - STMicroelectronics Community

When installing X Cube AI, you will be asked to provide the path to the zip for the NPU addon of the ST Edge AI Core, you can download it here (the last one): STEdgeAI-Core - Artificial intelligence (AI) optimizer technology for STMicroelectronics products - STMicroelectronics 

I believe that the template application generated by CubeMX is a code that do an inference of your model with a random input and return via serial some metrics. I may be wrong about that.

You can also use the ST Edge AI Core in standalone to get the C model and add it manually to your project. I would suggest using the first option as it does it for you.

You can also try to use the model zoo deployment script. I think that if you have a model that has inputs and outputs similar to what the model in model zoo has, the automatic deployment should be able to create a firmware example with your model. The conversion to C code will work as it does the same thing with the ST Edge AI Core, but the STM32CubeIDE that is modified and flashed to your board might not be compatible with your model I/O.
For example, if you do image classification, your input is most likely (height,width,channels) and output a buffer of size number of classes. Then in this case it is the same as model zoo models. (They maybe use one more dimension in input, the batch, download one model from model zoo and look at it.)

You can find documentation here:

https://stedgeai-dc.st.com/assets/embedded-docs/index.html

Have a good day,

Julian

[Anhem]

Hi @Julian E. 

Thank you for your response. I will check it. If something does not work, I will create a new topic.

[Julian E.]

Hello @Anhem ,

Additionally, you can find the CubeAI documentation here: https://www.st.com/en/embedded-software/x-cube-ai.html#documentation

Have a good day,

Julian

[Anhem]

@Julian E. 

I almost got the flow to deploy custom AI model on STM board:
Trained model => .tflite or ONNX => CubeMX to generate C code => use the generated C code in CubeIDE.

As my understanding, this step CubeMX to generate C code only generate C code with random input and get output from this random input. Is that right?

It means that we need to implement preprocessing input image and post processing for output in CubeIDE. Is that right?

My board is STM32N6570-DK. In CubeIDE, how I can load static image or using captured image from built-in camera to load into model to run inference? The question is same for output. We need to do these things in CubeIDE by C/C++ source code. Is that right? If you have sample for these items (my AI model is for object detection, YOLOX), please give the link.

As my understanding, STMicroelectronics also provides model zoo and we can use it to deploy sample models on the board with Python script (not C/C++). My question is that: for my custom model, after step CubeIDE, what I need to do to deploy my model on STM32N6570-DK?

Sorry for many questions. Thank you.

[Julian E.]

Hello @Anhem ,

Taking your questions one by one:

1. As my understanding, this step CubeMX to generate C code only generate C code with random input and get output from this random input. Is that right?

Yes exactly, CubeMX uses the ST Edge AI Core in background to convert your python mode to C code and also integrate the files in an STM32CubeIDE example project that do an inference with a random input and send via serial some metrics.

 2. It means that we need to implement preprocessing input image and post processing for output in CubeIDE. Is that right?

Correct, in the template code, I believe there are function template for pre and post processing.

3. My board is STM32N6570-DK. In CubeIDE, how I can load static image or using captured image from built-in camera to load into model to run inference? The question is same for output. We need to do these things in CubeIDE by C/C++ source code. Is that right? If you have sample for these items (my AI model is for object detection, YOLOX), please give the link.

We have the STM32N6 AI Getting started that contains multiple complete applications (camera, preprocessing, postprocessing and display) like classification, object detection, segmentation etc. You can download it here and take a look at the source files and code:

STM32N6-AI - AI software ecosystem for STM32N6 with Neural-ART accelerator - STMicroelectronics

4. As my understanding, STMicroelectronics also provides model zoo and we can use it to deploy sample models on the board with Python script (not C/C++). My question is that: for my custom model, after step CubeIDE, what I need to do to deploy my model on STM32N6570-DK?

Once again, you are right (well done haha). To describe it briefly, the model zoo is a github repository that contains both model and scripts to help you to get started with embedded AI. For the script part, you can train/retrain a model, quantize it, benchmark it ... and deploy it on the N6 in your case. What the deploy script mostly do is taking the getting started I linked above and using the ST Edge AI Core to replace the base model by your own model. 

Deployment object detection for N6: stm32ai-modelzoo-services/object_detection/deployment/README_STM32N6.md at main · STMicroelectronics/stm32ai-modelzoo-services

If I resume the question 3 and 4, you can eitheir use the getting started as a standalone and read the doc to deploy it manually or use it with model zoo to use it with the scripts.

Be careful, if you are doing object detection with a yoloX for example, you have to edit some postprocessing parameters (mostly related to bounding boxes and naming). It is explained in the zip that you download in en.n6-ai-getstarted-v1.0.0\application_code\object_detection\STM32N6\Doc\posprocess_defines.md

Let me know if something is not clear :)

Have a good day,

Julian

[Anhem]

Hi @Julian E. 

Thank you for you detailed response.

Answers for questions 1, 2, 3 are cleared. About answer for question 4 let me confirm again:

- In the case, if my custom model is same one model in the ModelZoo, I can reuse ModelZoo for fine tuning and deploy directly on the board (no need to use CubeMX and CubeIDE). Is that right?

- In the case, if my custom model is NOT in the ModelZoo. I need to do following steps:

• Step 1: CubeMX to generate C code,
• Step 2: CubeIDE to add preprocessing (load image, video, resize...) and postprocessing
• Step 3: Currently I am not cleared about this step to deploy my custom model on the board after step 2. As my understanding, I can have 2 options:
  • NOT use model zoo (use CubeIDE directly): In this case, I can directly deploy my custom model on the board via CubeIDE (In the past, I worked with NXP board and I could deploy model on the board by using IDE)
  • USE model zoo: In this case, I need to custom provided Python script to fit with my custom model and deploy model on the board by using Python script (NOT via CubeIDE)

Please confirm these 2 options of my understanding (I am not clear whether they are correct). If 2 options are correct, please give me advice to choose the best one for newbie as me. My target is deploy my custom model on the board and in the future I can evaluate accuracy of models on the board (run inference with many static images on the board) and also I can run inference with captured image for camera realtime for my real application.

Thank you so much. Please let me know if my question is not clear. 

[Julian E.]

Hello @Anhem ,

- In the case, if my custom model is same one model in the ModelZoo, I can reuse ModelZoo for fine tuning and deploy directly on the board (no need to use CubeMX and CubeIDE). Is that right?

Yes, that is right. Note that you can open the source code of the application being deployed. You can the cubeIDE project in /Your_model_zoo_folder/application_code/Your_Use_Case/STM32N6/STM32CubeIDE.

You can do that to either modify the application to suit your needs or understand what has been done.

- In the case, if my custom model is NOT in the ModelZoo.....

You have 2 options:

1. If your model is in tensorflow, you can still use model zoo following this guide:

https://github.com/STMicroelectronics/stm32ai-modelzoo-services/blob/main/image_classification/deployment/doc/tuto/how_to_define_and_train_my_own_model.md

2. if your model is not in tensorflow or any other reason, you can choose not to go with model zoo. In this case, the easiest way is to open CubeMX, select the N6 as target, set all you I/O configuration, set X Cube AI with the template application and generate the code (a CubeIDE project)

With this you will end up with an example code that run an inference with a random input. You will need to add the preprocessing, postprocessing, the camera, the display on your own (you can look at the model zoo source code to see what has been done).

I think the easiest way to create the application you are describing is to use model zoo if you can and edit the basic application to suit your needs. I did not test much what I linked to you; it worked with simple convolutional networks as in the example but still should work with anything (as long as the layer you use are compatible with the ST Edge AI Core).

Have a good day,

Julian

[Anhem]

Hi @Julian E. 

Thank you for your response.

Now I could totally understand the overall flow to deploy AI model on STM32N6570-DK.

Now, Pytorch models are dominant compared to Tensorflow models and maybe I need to take more effort to deploy my Pytorch => ONNX model on the board.

Thank you again.