How to implement a neural network on LSM6DSO16IS and ISM330IS with ISPU

Denise SANFILIPPO · ‎2024-11-12

Summary

This article showcases how you can implement a neural network on LSM6DSO16IS and ISM330IS with ISPU using ST ecosystem based on boards and software tools.

1. Hardware and software prerequisites

ST offers two kinds of sensors with ISPU, one for personal electronics and consumer applications, called LSM6DSO16IS, and the other one for industrial applications, called ISM330IS.

These sensors are also available as DIL24 adapter boards which mount on top of other boards available inside ST ecosystem.

Figure 1: DIL24 adapter boards for MEMS sensors with ISPU

ST has a large portfolio of boards which are used for evaluation, development, and prototyping.

Figure 2: MEMS ecosystem for ISPU

On the left side of the Figure 2, you can see the professional MEMS tool board, recently updated with the new STEVAL-MKI109D.

On top of this board, all the MEMS DIL24 adapters can be plugged for evaluation.

Then, STM32 Nucleo boards are available along with MEMS expansion boards, such as X-NUCLEO-IKS4A1, for development.

Finally, some form-factor boards like the SensorTile.box PRO and the STWIN.box are useful for rapid prototyping. These boards have a battery inside and they can communicate through Bluetooth with the mobile application on your phone.

The boards are provided with some companion software which allows to program the ISPU. This includes ISPU-Toolchain, NanoEdge AI Studio. Furthermore, to optimize the model which is embedded on the sensor, such as STEdgeAI-Core and MEMS Studio.
Finally, there are also examples of applications both on GitHub and on the X-CUBE-ISPU package.

2. How to program the ISPU

It consists in three programming steps.
The first programming step consists in creating a neural network model (see step 1 in Figure 3) with an external tool and optimizing it. You can use MEMS Studio which is a graphical user interface or STEdgeAI-Core which is a command-line interface.

Figure 3: ISPU programming steps

With these software tools, you can convert the neural network into C code for ISPU in the second step (see step 2 in figure 3).
In step 3, the C code must be integrated into a template for ISPU and compiled with an Eclipse-based IDE like STM32CubeIDE or using the ISPU toolchain. Finally, you can get the ISPU configuration.

The generated .ucf or .h file with the configuration can load in MEMS Studio or in other development environments.

These software tools are part of the ST Edge AI Suite, a suite containing different tools starting from ideas case studies, and model zoo, such as:

Figure 4: Free tools to run edge AI with NN on ST smart sensors with ISPU

Tools for data collection like the High Speed Datalog
Tools for automated machine learning like NanoEdgeAIStudio, tools for model optimization like MEMS Studio and STEdgeAI-Core
Tools for neural network benchmarking with the ST Edge AI Developer Cloud
Tools for deployment, for which you can find some code examples to integrate and deploy the model generated

Let’s now deeply analyze the three programming steps already shown in Figure 3.

3. Step 1: Optimize

The first step for programming the ISPU consists in optimizing the neural network model, which is generated with external tools.
This step can be done with STEdgeAI-Core or MEMS Studio.

Figure 5: ISPU programming: Step 1 - Optimize

Here you can see some details about the STEdgeAI-Core technology.

Figure 6: ST Edge AI Core technology

It’s designed to optimize and deploy neural networks on any ST device, ensuring the best interaction and performance between software and hardware.
STEdgeAI-Core is available as a command-line interface and allows an automatic conversion of an app training AI algorithm, like a neural network into the equivalent C code embedded in the application.
The models are built for different targets, including sensors with ISPU.
In the STEdgeAI-Core command-line interface, there are three main commands for the ISPU: analyze, generate, and validate. Figure 7 provides further details about these three commands.

Figure 7: ST Edge AI – Commands for ISPU

In the ISPU repository on GitHub, you can find the tutorial with a complete flow for deploying a trained neural network inside a sensor embedding the ISPU using STEdgeAI-Core.

The use case consists of human activity recognition (HAR) using accelerometer data to train a small neural network (NN) to recognize four activities: stationary, walking, running, and cycling.

The steps in the tutorial can be replicated with any board supporting the ISPU.

You can find a Jupyter notebook ready to use for the training part with some available data logs. In the Python code available in the notebook you can:

Load the data from the csv logs
Segment data into two-second windows and label windows
Divide the dataset into training
Validate and test sets
Train the neural network
Quantize model weights
Save all the files in the output folder.

For more details, refer to the readme file.

Once you have the neural network model, you can either run the STEdgeAI-Core command line with the commands described previously. Alternatively, use MEMS Studio to perform the same task in a graphical user interface.

Let’s open MEMS Studio to import the model in the [ISPU Model Converter] section.
Click the [Run Generate] button to obtain a C-code implementation of the neural network model for the ISPU. You’ll be able to review the CLI logs by clicking on the [Terminal] button.

Figure 8: MEMS Studio: Load NN model / Generate tab

The [Analyze] tab shows information about the model converter. There’s a radar chart with a graphical representation of RAM usage, weights, and activation of the processed model.
Clicking on the [Inspect Report] button, you can review the generated report.

Figure 9: MEMS Studio: Analyze tab

Then you have the Validate tab where you can evaluate the model and review the summary of the validation report through a data table, graphical radar charts, and confusion matrix.

Figure 10: MEMS Studio: Validate tab

Right now, the board isn’t connected. If you have the Nucleo board connected to your PC, you can validate the target directly.

In the last tab, there’s a benchmark to compare the performance of the different models generated.

Figure 11: MEMS Studio: Benchmark tab

In all the pages of the model converter, you can review the CLI logs by clicking on the [Terminal] button.

Figure 12: MEMS Studio: review the CLI logs by clicking on the Terminal button

Once you have optimized the model, you can use the generated C-code created by MEMS Studio to obtain a configuration for ISPU embedded in the sensor.

4. Step 2: Generate

Figure 13: ISPU programming: Step 2 - Generate

Figure 14 shows the folder where the C code for the ISPU is generated.
There are three subfolders, Inc, Lib and Source, with header files and C code of the neural network.

Figure 14: MEMS Studio: ISPU Model Converter tab

The C code generated can also be integrated in the ISPU template and built using the ISPU toolchain or an Eclipse based IDE. This step is necessary to generate the configuration file (.ucf file) for the sensor, which is described in the next ISPU programming step.

5. Step 3: Integrate and compile

Figure 15: ISPU programming: Step 3 – Integrate and compile

The template to generate the .ucf file is located on the GitHub repository for ISPU or in the X-CUBE-ISPU package.

To generate the .ucf file, you need to integrate the code previously generated by MEMS Studio and to build the binary file using the ISPU toolchain.

Figure 16: ISPU development flow

Here are the details about the ISPU toolchain which include the C/C++ compiler to generate the binary and the ispu_gen tool to generate the .ucf and the .h file.

The toolchain is also available as a plug-in for Eclipse based IDEs including STM32CubeIDE.

Figure 17: ISPU Toolchain

Once you’ve copied these three folders, that is, inc, lib, src, generated by MEMS Studio in the ISPU template, you can build the project with the STM32CubeIDE.

Figure 18: Copy the three folders, i.e. inc, lib, src, generated by MEMS Studio in the ISPU template before building the project with the STM32CubeIDE.

You’ll get some warnings which is fixed by implementing the missing code in the main.c file.

The missing code has the purpose to define the inputs and outputs of the neural network.

You can find details on how to implement this part in the readme file available with the template.

In particular, the code you need to customize has the following purpose:

to reinterpret the input and output buffers as array of the expected shape for the neural network
to insert accelerometer data in the input buffer
to define when to run the network, for instance every 52 sample in this case
to assign the output as the most probable result

Figure 19: Rebuilding the project in STM32CubeIDE to get the configuration file (.ucf file) for the ISPU

When you’ve done all these modifications to the code, you can rebuild the project and get the configuration for the ISPU in term of the .ucf file.

Note that the GitHub example also provides a JSON file, containing information on how output data are structured to allow the visualization in MEMS Studio.

When you have the configuration file for ISPU, you can test it through some evaluation or development board.

For example, let’s mount the X-NUCLEO-IKS4A1 expansion on top of NUCLEO-U575ZI-Q, as shown in Figure 20.

In this case, you haven’t mounted any DIL24 adapter on top of the expansion board because the X-NUCLEO-IKS4A1 already contains the LSM6DSO16IS sensor with ISPU.

In the X-CUBE-ISPU package, you can find the software to use with not only these boards, but also other boards. In fact, X-CUBE-ISPU is an expansion package for ISPU development containing documentation examples and STM32 firmware for enabling communication with the MEMS Studio.

Figure 20: X-NUCLEO-IKS4A1 expansion mounted on top of NUCLEO-U575ZI-Q

Let’s consider the binary file for the Nucleo board with the STM32U5 microcontroller supporting the X-NUCLEO-IKS4A1 expansion board. The binary file is available at the path …\Projects\NUCLEO-U575ZI-Q\Examples\IKS4A1\LSM6DSO16IS_DataLogExtended\Binary inside the downloaded X-CUBE-ISPU software package. You can program your Nucleo board with this firmware. Once the firmware loads in the Nucleo board, you can test the configuration for ISPU by loading the configuration file previously created.

After that, open MEMS Studio and connect it to the Nucleo board. Select LSM6DSO16IS as a sensor with ISPU, then load the configuration file previously generated by STM32CubeIDE by pressing on the [Load configuration file] button (see Figure 21).

Figure 21: MEMS Studio: Load configuration file

And finally test the application including the ISPU configuration with MEMS Studio.
This example is an activity recognition, so you can see the outputs stationary, working, running, and so on, as shown in Figure 22.

Figure 22: MEMS Studio: Data Monitoring of ISPU output

Conclusion

In this article, we covered to run neural networks in ST MEMS sensor with ISPU, which are LSM6DSO16IS and ISM330IS.

However, in the ISPU you can also run other kinds of algorithms, for instance, for anomaly detection or for other machine learning classifications.
ST provides different tools depending on the use case that you want to handle, depending on the level of your expertise.
For anomaly detection algorithms, we suggest using NanoEdge AI Studio. For other machine learning algorithms, besides the tools showcased in this article, there’s also the possibility to use the tool from our partner Neuton. In fact, Neuton's tool allows embedding the models in the ISPU for different use cases, even without writing any line of code.

Here’s a summary of the available tools with respect to the applications:

Figure 23: ISPU Use cases and available tools