2022-01-18 01:55 AM
Dear support-team,
I've encountered quite some problems when loading a pre-trained Pytorch mobilenetv2 model in Xcube-AI.
I have tried the following with the app stm32ai located in the installation repository of X-Cube-AI/7.0.0.
I found that:
Looking forward to your answer,
Best regards,
Zhen
2022-01-18 02:39 AM
Dear Zhen,
thanks for your questions.
Her below my answers:
1) We have currently a limited support to ONNX quantized models in XCubeAI. We can support only static quantized models and we support QLinearConv, QLinearMatmul, QuantizeLinear and DequantizeLinear layers (as explained in the documentation). The post-training quantization is supported by XCubeAI CLI only for keras models. So, external tools should be used for quantizing ONNX models.
2) XCubeAI manages both channel last and channel first models (typically ONNX models are channel first). I would recommend you to give a look at "Channel first support for ONNX model" section in the XCubeAI documentation. Moreover Transpose layer is also supported.
If you still encounter any issues, please let us know.
It could be useful for us to see the error messages you get.
Thanks and Best Regards
Valeria
2022-01-18 06:14 AM
Dear Valeria,
Many thanks for the prompt reply. It's very helpful.
I tried to load my quantized int8 mobilenetv2 again in XcubeAI, and the error msg showed that:
NotImplementedError: Unsupported layer types: QLinearAdd, QLinearGlobalAveragePool
[8362] Failed to execute script stm32ai
which confirmed that this type of ONNX model is still not supported yet on your platform.
To find a workaround, I also tried to convert ONNX fp32 model to Keras/tensorflow and then to TFLite, as I understood that TFLite is better supported on your platform. However, despite that I have converted both models successfully, they cannot be analyzed on XcubeAI. The full error msg of loading Keras fp32 & the TFLite f32 are shown in the attachment [TF version: 2.5.0; Keras version 2.5.0].
Do you know where the problem comes from?
Looking forward to reply,
Best,
Zhen
2022-01-21 02:28 AM
If you look at the X-Cube-AI documentation in the STM32CUBEIDE there is a section on support ONNX operators (It's called ONNX toolbox support)
It is located under your home directory
~/STM32Cube/Repository/Pack/STMicroelectronics/X-CUBE-AI/7.1.0/Documentation/index.html
It can also be directly accessed from STM32CubeMX using Help->X-Cube-AI Documentation. The menu option is available once you have opened a project in STM32CubeMX with the X-Cube-AI pack enabled in the project
This should help to know what ONNX operators are supported for you.
If you haven't tried it I recommend using the application template as this will set up your inputs/outputs correctly (as I've had issues here previously). The template will also generate the code so I highly recommend you give it a go.
Hope this helps 🙂
2022-01-24 12:24 AM
Dear Zhen,
I confirm that you can find the list of the supported operators in ~/STM32Cube/Repository/Pack/STMicroelectronics/X-CUBE-AI/7.1.0/Documentation/index.html.
The QLinearAdd and QLinearGlobalAveragePool layers are not in vanilla ONNX definition. How did you quantize your model? Did you use the quantization script from onnxruntime?
The error you got with Keras model is due to channel first convolutions that are not supported in CubeAI for Keras framework.
The error you got with tflite is not totally clear. Could you make a try with the newest CubeAI 7.1.0 version?
Thanks and Best Regards
Valeria
2022-01-24 03:09 AM
Dear Valeria,
Thank you for your reply.
Regarding the quantization of ONNX: yes, I used the quantize_static function from onnxruntime/quantization/quantize module. I wrote a custom calibration_data_reader class to load/preprocess the dataset.
Regarding the error I got from converted TFLite model, I hereby confirm that the problem is already solved with the new release of CubeAI 7.1.0.
Best regards,
Zhen