BYOM on STM32N6570-DK: How to handle custom preprocessing/feature extraction with Model Zoo services?

[Sahand]

Hi all — I’ve successfully trained → quantized → deployed one of the audio models from the STM32 AI Model Zoo to an STM32N6570-DK (N6) and can see correct output over the serial port. Now I want to deploy my own model. The model was trained on custom features. I can export to ONNX with fixed input size and compatible ops and I know how to run the quantization + deployment flows from the Model Zoo service.

My questions:

1. Preprocessing location

   • The N6 audio example seems to compute only log-mel on-device. Can the Model Zoo service itself be configured to produce my custom features on the target, or is the intended approach to modify the STM32 application C code to implement my feature pipeline before inference?

2. Using ONNX with custom features

   • Anything specific I should set in the user config for ONNX quantization with a non-mel feature front-end (e.g., any flags to disable built-in mel generation assumptions)?

[Julian E.]

Hello @Sahand,

1 - I don't know exactly what you mean by custom features, but there are several cases:

• If you want log-melspectrograms but with different parameters (n_fft, hop_length, etc.), you can change them either in user_config.yaml or directly in ai_model_config.h
• If you want a magnitude spectrogram, he can modify, in ai_model_config.h:
  #define CTRL_X_CUBE_AI_MODE_OUTPUT_1 (CTRL_AI_SPECTROGRAM)
  #define CTRL_X_CUBE_AI_PREPROC (CTRL_AI_STFT)
  (Basically, it reuses the speech enhancement preproc). It's possible that some additional changes are needed, I'm not sure. If he wants other spectral features (MFCC, complex spectrogram), you can reuse the functions from the C audio preproc library (in Middlewares/ST/STM32_AI_AudioPreprocessing_Library).
• If you want something else, you will have to program it himself in the C application. We can provide more details if you explain what exactly you want.

2 - ONNX quantization uses a calibration dataset, which is composed of log-melspectrograms in the model zoo, so if you want other features, you will have to perform the quantization himself and provide your custom features in the calibration dataset.

Have a good day,

Julian

[Sahand]

Hi Julian,

thanks for your reply, I want to have other sets of features rather than
log-melspectograms. More specifically, I have trained a model using PCEN,
spectral centroid and rolloff, and harmonic mel-spectrogram. Any help would
be appreciated.

Thanks!
Sahand

[Julian E.]

HI @Sahand,

Unfortunately, none of the listed processing you listed is available on our C application, you will have to code them yourself.

Have a good day,

Julian

[Sahand]

Thanks Julian,

To apply these preprocessing steps, do I need to implement them in the C
application, or is it enough to modify the existing Python code (for
example, the scripts in the “preprocessing” folder) and update the
user_config.yaml file? Ideally, I’d like to know whether running
stm32ai_main.py —which executes the preprocessing pipeline— automatically
generates the corresponding C code.

Thanks,
Sahand

[Julian E.]

Hello @Sahand,

I think you need to implement them in C because we don't have them.

My understanding is that the model zoo team coded the librosa (python) equivalent in C for the preprocessing they use. 

So, when editing the user_config.yaml, it must copy it to the C application.

But they did not reproduce everything from librosa. So, if you use something we don't provide, you need to implement it yourself. (the python pipeline will not be able to generate automatically the corresponding C code).

Have a good day,

Julian