on - edited on by
Laurids_PETERSE
Summary
This article showcases the ISPU capabilities in gesture recognition applications. It focuses on movements performed with a magic wand.
ISPU is an ultralow power, computationally efficient, high-performance programmable core that can execute signal processing and AI algorithms in the edge.
In this example, we use ISPU to run a neural network which recognizes the I-S-P-U sequence of letters. The kit consists of LSM6DSO16IS sensor mounted on the SensorTile.box PRO board through the STEVAL-MKI229A DIL24 adapter board.
The output of this tutorial with the LSM6DSO16IS sensor can be viewed in the following video.
1. Prerequisites
- Wand to hold SensorTile.box PRO, printed with a 3D printer.
Sample files for the 3D printer are attached to this article. - SensorTile.box PRO
- STEVAL-MKI229A (LSM6DSO16IS adapter board for a standard DIL24 socket)
2. Developing an air-written letter recognition deep-learning model with ISPU
Here’s a description of how to create an air-written letter recognition deep-learning model running on the ISPU core embedded in ST MEMS sensor LSM6DSO16IS.
The model is able to detect the letters I, S, P, and U drawn as shown below:
The model is a simple and tiny 1D CNN architecture composed of three feature extraction blocks (Conv1D + BatchNorm + MaxPooling) and one simple classification stage (dense layer + softmax). It perfectly fits the ISPU characteristic in terms of model size and execution time. To visualize the following image, see the official GitHub repository for ISPU.
The model is trained on a custom dataset consisting of accelerometer data only, sampled at 52 Hz with 8 g full-scale. The data is collected using a LSM6DSO16IS smart IMU mounted with the orientation shown in the figure below.
Figure 1: Data are collected using a LSM6DSO16IS smart IMU mounted with these orientations
In terms of classification, four classes are defined. They are the four letters I-S-P-U to identify during the gesture. The software logic is based on a finite state machine with start and reset gestures.
ISPU power consumption during the inference cycle is around 50 µA during the average power consumption, since we’re just running the accelerometer.
The accelerometers average power consumption is about 180 µA @ 52 Hz ODR in high performance mode.
In terms of accuracy of the model algorithm, we’re talking about more than 97.7% and the total footprint is around 20 KB.
The benefits of running the AI algorithm at the edge, on the LSM6DSO16IS sensor are substantial. It significantly reduces power consumption, which is crucial for battery-operated devices.
Moreover, it frees up MCU bandwidth, allowing other critical tasks to perform without a hitch.
New models can be trained, validated, and prepared for deployment with the resources and info available inside the GitHub for ISPU.
The pretrained model for recognizing the letters I-S-P-U is available at models/lsm6dso16is_ISPU_reference_dataset.
In the model zoo section, you’ll find all the files you need to customize the application. For instance, adding different letters or training the model with a different person.
You’ll also find a referenced dataset with the logs from different users and the model pretrained with it. In addition, inside the GitHub repository, scripts are provided to retrain the model based on user dataset or configuration.
There are also automatic procedures to evaluate the retraining model on a PC and on the ISPU and to regenerate the ISPU project for deployment.
Conclusion
LSM6DSO16IS is a MEMS sensor, featuring the Integrated Signal Processing Unit (ISPU).
This ultralow power, high-performance programmable core is revolutionizing the way we interact with devices.
The ISPU's capabilities are vividly demonstrated through this use case.
Here, the ISPU is embedded within the MEMS sensor to perform complex signal processing and AI algorithms right at the edge. This means that the processing happens where the data is captured, leading to lightning-fast responses and interactions.
This article showcases how the ISPU can seamlessly run a neural network that recognizes a sequence of gestures. With a simple movement of a wand kit, the ISPU can identify the letters I-S-P-U with remarkable accuracy. This isn’t just a step forward in remote-control technology: it's a leap towards creating more natural, intuitive ways for users to communicate with their devices.
Feel free to ask questions and/or post your comment in the thread below. Enjoy!
