Spectrogram recognition with X-Cube AI on STM32F746

[ALiss]

Hello,

I am doing a project where I will be implementing a trained neural network (trained with Keras) onto a STM32F746-DISCOVERY board with X-Cube AI. The goal is to train the network to recognize audio samples converted into spectrograms. This would mean that on the microcontroller, I would need to convert the audio input into spectrogram images, and then input that into the neural network for recognition.

Does anyone have any good sources or similar projects regarding either creating spectrograms on a STM32 microcontroller or a good image recognition project on an STM32 MCU using x-cube AI?

Thank you!

[KnarfB]

for the signal processing part you may look at X-CUBE-DSPDEMO

[ALiss]

Thank you! I will look into it.

[Gln · Accepted solution]

​Hi @ALiss​ ,

In FP-AI-SENSING1 v3.0.0, there is an STM32_AI_AudioPreprocessing_Library Middleware library that can be used exactly for this purpose.  The library provides the building blocks for spectral analysis and feature extraction, such as:

• Spectrogram computation
• Mel-scaled and LogMel-scaled spectrogram computation
• Mel-frequency cepstral coefficients (MFCCs) computation

An example (custom) usage can be found in asc_processing.c

Best regards,

Guillaume

[ALiss]

That is great, thank you Guillaume!

[Gerardo Trotta]

The library seems to be a little bit buggy.

Expecially the asc_processing.c

Is it possible to check togheter some weird behaviour?

[Gln]

Hello @Gerardo Trotta​ ,

Sure. What is your issue?

Guillaume

[Gerardo Trotta]

Hello @Gln​ ,

first one is in function

ASC_OutputTypeDef ASC_Run(float32_t *pBuffer)
{
  ai_float dense_2_out[AI_CONTACT_OUT_1_SIZE] = {0.0, 0.0, 0.0};
 
  /* Create a Mel-scaled spectrogram column */
   MelSpectrogramColumn(&S_MelSpectr, pBuffer, aColBuffer);
  /* Reshape and copy into output spectrogram column */
  for (int i = 0; i < NMELS; i++)
  {
    aSpectrogram[i * SPECTROGRAM_COLS + SpectrColIndex] = aColBuffer[i];
  }
  SpectrColIndex++;
 
  if (SpectrColIndex == SPECTROGRAM_COLS)
  {
    SpectrColIndex = 0;
 
    /* Convert to LogMel-scaled Spectrogram */
    PowerTodB(aSpectrogram);
 
    /* Run AI Network */
    ASC_NN_Run(aSpectrogram, dense_2_out);
 
    /* AI Network post processing */
    //ClassificationCode = ASC_PostProc(dense_2_out);
 
    return ClassificationCode;
  }
  else
  {
    return ASC_UNDEFINED;
  }

The variable :

SpectrColIndex 

is not initialized. In my H7 the reshape cycle does not work, unless SpectrColIndex  is initialized to zero, befaore calling ASC_Run.

The second one is a crash during ai_run. But this may be because I have to recalculate common tables, isn'it?

Thank you

JJ

[Gln]

SpectrColIndex is global variable, part of the bss segment. It is initialized to 0 during startup.

If needed, you can explicitly initialize SpectrColIndex to 0 in ASC_Init().

The second crash during aiRun might be caused by insufficient stack space and/or I/O buffer address corruption.

Recalculating common tables is only required if you change some preprocessing parameters and you want to avoid going through the MelFilterbank_Init, and Window_Init. The lookup tables stored in common_tables.h are for a given configuration. If you are using different preprocessing parameters, these lookup tables can be created at runtime in RAM using the _Init() functions. This is not the case in FP-AI-SENSING1. The preprocessing lookup tables have been generated offline and stored in ROM Flash using common_tables.c

Regards,

Guillaume

[Gerardo Trotta]

Hello,

almost clear. Thank you.

The NN still crash in a strange point; exiting form this function :

ASC_StatusTypeDef ASC_NN_Run(float32_t *pSpectrogram, float32_t *pNetworkOut)
{
  ai_i8 AscNnOutput[AI_CONTACT_OUT_1_SIZE];
  ai_i8 AscNnInput[AI_CONTACT_IN_1_SIZE];
 
  /* Z-Score Scaling on input feature */
  for (uint32_t i = 0; i < SPECTROGRAM_ROWS * SPECTROGRAM_COLS; i++)
  {
    pSpectrogram[i] = (pSpectrogram[i] - featureScalerMean[i]) / featureScalerStd[i];
  }
 
  aiConvertInputFloat_2_Int8(AI_CONTACT_MODEL_NAME, AI_CONTACT_MODEL_CTX,pSpectrogram, AscNnInput);
  aiRun(AI_CONTACT_MODEL_NAME, AI_CONTACT_MODEL_CTX, AscNnInput,AscNnOutput);
  aiConvertOutputInt8_2_Float(AI_CONTACT_MODEL_NAME, AI_CONTACT_MODEL_CTX,AscNnOutput, pNetworkOut);
 
  return ASC_OK;
}

aiRun is excuted, but on line 16 it crash. Stack is 2000 now (before it was 400). And NN report is :

input      : input_0 [121 items, 484 B, ai_float, FLOAT32]
input (total)  : 484 B
output     : dense_3_nl [17 items, 68 B, ai_float, FLOAT32]
output (total) : 68 B
params #    : 302,689 items (1182.38 KiB)
macc      : 2,378,879
rom (ro)    : 1,210,756 (1182.38 KiB) 
ram (rw) wb+io : 31,936 + 552 (31.19 KiB + 552 B)

What is the laison between NN data and mimum stack?

Now I'm recalculating tables as you suggest.

Thank you