Running anomaly detection in sensors represents a major innovation in predictive maintenance for industrial applications and offers significant advantages in system-level power efficiency, downtime reduction, and maintenance costs.
The challenge for designers who want to monitor warning signs of imminent equipment failure is how to build reliable battery-operated sensor nodes for predictive maintenance applications.
Leveraging AI at the edge, you can move part of the artificial intelligence closer to the data acquisition, so the anomaly detection application can run in the sensors. Once the anomaly is detected, the sensor wakes up the host processor for further intervention. NanoEdge AI Studio offers a quick and intuitive approach to building anomaly detection applications in sensors.
In this knowledge article, you will learn how to easily implement machine learning on the ISPU embedded in ST ISM330ISN inertial sensors for anomaly detection functionality at the edge.
The hardware configuration used to achieve our results is the following:
It is possible to collect all the buffers that you want, but for a simple use case, 100 samples are enough and they represent the normal behavior of the fan
For the simplicity of the knowledge article there will be collected only data representing one mode of operation on our fan which is the max speed, but of course it is possible to gather data representing all the different states that are to be considered as the normal behavior of the fan.
It is possible, for instance to record signals from all speeds, not only one, in that case it is possible to concatenate everything and that will represent the normal regimes in one single file.
The signal screen summarizes the content of the data that were just logged.
On the right-hand side, there is a preview of the imported data with the FFT plots for all the accelerometer axes.
There is also the possibility to manually add filters to remove unwanted frequencies, by doing it, the frequency plot will change immediately according to the cut made. During this knowledge article this part is skipped.
At this point the NEAI studio will start working hard to find the best possible library for our use case given the signal we provided. The nano edge library is composed by three main blocks:
The emulator step is the step where it is possible to test the library directly from the SW to check if its to understand if what you have generated with your benchmark works properly before the final implementation.
Once again, let’s use the serial data logger to collect data for your normal condition meaning when the speed is at the max.
One thing to highlight here is the fact that I just showed you the different speed case, however, if I turn off the fan, the anomaly will be detected and the same if I tap on the fan itself.
I want to remind that the anomaly detection is a first step to the predictive maintenance which just says when something is diverging from its normal behavior.
In the last step of the NEAI studio by clicking on generate library, it is possible to download the .h file with the library found in a .zip package.
They need to be replaced in order to get the right library parameters such as the data rate, the range, the sample size per axis and the number of axes which were selected for the datalogging part.
Moreover, the ispu_neai.h is the binary configuration file which runs in the MEMS Sensor itself.
Hence clicking on the import project icon, we select existing projects into workspace.
Next it is needed to browse the project in the directory where the two files were copied:
x-cube-ispu\Projects\NUCLEO-F401RE\Examples\CUSTOM\ ISM330ISN_NEAI_AnomalyDetection
The first box must be unflagged, keeping just the second where it is written STM32CubeIDE.
There is a main consideration to be done here to understand how the code works since its aim is not to compare two different states, but to use a library like the one generated with the NEAI to make comparison between a normal state and an anomaly state. It works mainly as a state machine:
Next step is to build the project using the building button at the top left of the screen and once that is done, after connecting the hardware setup to the laptop, the debug can start.
The debug starts after clicking on the green bug and once this is done, we can check the results in a terminal.
Launching putty, the first step is to click on the serial mode and write the correct serial line and the speed. While the serial line can change, the speed is always 115200.
Some visualization parameters can be modified, for instance going on terminal, for a better visualization it is needed to flag:
Finally, by clicking on open, the terminal will display the values.
As anticipated, the first step is the learning phase to get the new normal data.
As soon as this step is completed, the message “learning phase end” appears ending the learning phase.
If the library works well, there will be no data streamed if the learning condition is kept and no data are streamed.
As soon as an anomaly is created, the data will start streaming for that specific kind of anomaly.
The data can be collected in a microcontroller to make a dataset for future classifications.
In this knowledge article it was shown and explained how to use the ISM330ISN which embeds the ISPU with the NEAI studio software tool.
The NEAI software tool allows to find the best possible library among many combinations starting from a set of good and bad signals which can be used once getting new data in the learning phase.
The X-CUBE-ISPU is a software tool which needs to be used to flash the library generated by the NEAI tool in the ISPU device.
You can check the following link for a step-by-step webinar on the same topic.
Webinar: Anomaly detection and AI: one step closer to ST intelligent sensors – STMicroelectronics
Before starting, what do you need?
Software
In this knowledge article, there will be the need to use the following software tools and packages:- NEAI which is the tool dedicated to find the best anomaly detection library
- X-CUBE-ISPU which is a software package dedicated to create the right binary file for the ISPU devices
- STM32CubeProgrammer which is the software tool used to compile the X-CUBE-ISPU package
Hardware
The hardware configuration used to achieve our results is the following:
- NUCLEO-F401RE which is the STM32 nucleo board
- X-NUCLEO-IKS01A3 which is the STM32 nucleo shield for consumer sensors
- STEVAL-SMKI230AA which is the DIL24 with the ISM330ISN mounted on it embedding ISPU capabilities
The anomaly detection will run to detect anomalies of a fan coil with three speeds.
Other boards configurations can be used to obtain the same result, it is needed to check the software compatibility with the hardware.
Other boards configurations can be used to obtain the same result, it is needed to check the software compatibility with the hardware.
How to find the best library for your ISPU device using the NEAI software
When opening the NEAI studio, the first thing is to open the datalogger where you can find many different board configurations you can use to run your anomaly detection algorithm.We select the configuration we use for this demo
And then it is needed to select the parameters for the data acquisition meaning:
- Data rate (Hz): 416 Hz
- Range (g): 2 g
- Sample size per axis: 128
- Number of axis: 3
It is then possible to click on generate datalogger and download the .bin file to be flashed in the HW board.
Once it is done it is possible to start building the anomaly detection use case with the NEAI software tool by creating a new project.
Once it is done it is possible to start building the anomaly detection use case with the NEAI software tool by creating a new project.
Now there are seven steps to be completed to go through the library generation.
1st Step: Project Setting
Here it needed to give a name to the project and to select the right device to be used for the anomaly detection exercise.
There are many options, but the one to select is the product: ISM330ISN.
For many devices, the settings of the max RAM, the max flash and the sensor type must be selected, for the ISPU device all the right parameters are already set.
There are many options, but the one to select is the product: ISM330ISN.
For many devices, the settings of the max RAM, the max flash and the sensor type must be selected, for the ISPU device all the right parameters are already set.
2nd Step: Getting the normal signal
For the second step, we need to click on add signal and here there are three different sources that you can use to import your normal signal.
We can import from 3 different sources:
- from a file in text or CSV format
- from .dat files in the SW packages
- From the serial USB
Selecting the option from a serial USB it is possible to stream and get data directly from the boards that we have selected to make the data logging.
The COM port should be selected, and the bound rate is fixed at 115200.
For this knowledge article, the normal signal is taken while the fan coil is running at its max speed.
So, after setting the coil to its max speed and clicking on START, the data starts streaming and getting recorded.
The COM port should be selected, and the bound rate is fixed at 115200.
For this knowledge article, the normal signal is taken while the fan coil is running at its max speed.
So, after setting the coil to its max speed and clicking on START, the data starts streaming and getting recorded.
It is possible to collect all the buffers that you want, but for a simple use case, 100 samples are enough and they represent the normal behavior of the fan
For the simplicity of the knowledge article there will be collected only data representing one mode of operation on our fan which is the max speed, but of course it is possible to gather data representing all the different states that are to be considered as the normal behavior of the fan.
It is possible, for instance to record signals from all speeds, not only one, in that case it is possible to concatenate everything and that will represent the normal regimes in one single file.
The signal screen summarizes the content of the data that were just logged.
On the right-hand side, there is a preview of the imported data with the FFT plots for all the accelerometer axes.
There is also the possibility to manually add filters to remove unwanted frequencies, by doing it, the frequency plot will change immediately according to the cut made. During this knowledge article this part is skipped.
3rd Step: Getting the abnormal signal
Then it is needed to import signals from the abnormal behavior, collecting data from what is believed to be unusual, for the sake of simplicity, only by changing the speed from the fastest to the slowest will get the abnormal signal behavior.
Hundreds of samples were recorded
There is a very important consideration which has to be done in order to understand how the system works.
The signal examples that it is provided at this point are not used to train the library, they are only to provide some context for the automatic search that will happen in the next step because all the learning will happen later on in the ISPU after the library has been embedded on the target machine. The examples of anomalies provided here do not need to be exhaustive. For a real use case for instance, they can be collected on a faulted machine or in conditions that you know are abnormal or depending on the case, they can be created artificially.
When testing the library, it is possible to detect anomalies that were never seen before, because embedded learning is what makes NEAI libraries, especially for Anomaly detection particularly adaptable.
Once all the data have been collected, let’s import them, checking the graphs.
There is a very important consideration which has to be done in order to understand how the system works.
The signal examples that it is provided at this point are not used to train the library, they are only to provide some context for the automatic search that will happen in the next step because all the learning will happen later on in the ISPU after the library has been embedded on the target machine. The examples of anomalies provided here do not need to be exhaustive. For a real use case for instance, they can be collected on a faulted machine or in conditions that you know are abnormal or depending on the case, they can be created artificially.
When testing the library, it is possible to detect anomalies that were never seen before, because embedded learning is what makes NEAI libraries, especially for Anomaly detection particularly adaptable.
Once all the data have been collected, let’s import them, checking the graphs.
4th Step: Benchmark
At this point the NEAI studio will start working hard to find the best possible library for our use case given the signal we provided. The nano edge library is composed by three main blocks:
- Signal preprocessing algorithm, this could be very simple such as offsetting, normalizing or more complex like FFT or PCA
- Choice of the ML model. In the pool of all the available models there are different classes of ML such as KNN, SVN, NN
- A set of hyperparameters for the ML model so that it can solve the problem as effectively and efficiently as possible.
For each library made of these 3 blocks, there are many possibilities which result in millions of combinations and millions of possible libraries.
Each library is tested using the signals imported previously. The library will learn the normal signals and then run inference on both normal and the abnormal signal. This all process happens many times for each library using random subsets from the data provided to minimize any possible bias.
Each library is tested using the signals imported previously. The library will learn the normal signals and then run inference on both normal and the abnormal signal. This all process happens many times for each library using random subsets from the data provided to minimize any possible bias.
The results are displayed in the graph above. On the X axis there is the signal number, and the Y axis is the % of similarity which is the raw output of the inference or detection functions.
The ideal situation will be to have all the blue dots which correspond to the normal signal to be as close to 100% similarity as possible and we will like all the red points which represent the abnormal signals to be close to 0%.
The most important performance indicator is the balanced accuracy which translates the ability of the library to correctly attribute the input signals to the correct class, either normal or abnormal. 100% accuracy would mean that all the blue points are above the 90% decision boundary and all the red are below.
Finally, the amount of RAM and flash memory are also optimized, so there is a comparison between the best library found with the RAM space.
The ideal situation will be to have all the blue dots which correspond to the normal signal to be as close to 100% similarity as possible and we will like all the red points which represent the abnormal signals to be close to 0%.
The most important performance indicator is the balanced accuracy which translates the ability of the library to correctly attribute the input signals to the correct class, either normal or abnormal. 100% accuracy would mean that all the blue points are above the 90% decision boundary and all the red are below.
Finally, the amount of RAM and flash memory are also optimized, so there is a comparison between the best library found with the RAM space.
5th Step: Emulator
The emulator step is the step where it is possible to test the library directly from the SW to check if its to understand if what you have generated with your benchmark works properly before the final implementation.
Once again, let’s use the serial data logger to collect data for your normal condition meaning when the speed is at the max.
Remember that anomaly detection libraries come untrained after the benchmark, that’s why we need to run an initial learning cycle first to understand which is the correct behavior.
Then we move to detection to see how the library behaves, again we will use serial data.
And here the inference result shows 100% similarity and that is expected since the fan is running normally.
Then we move to detection to see how the library behaves, again we will use serial data.
And here the inference result shows 100% similarity and that is expected since the fan is running normally.
Now if we change the speed, the similarity starts to drop immediately.
One thing to highlight here is the fact that I just showed you the different speed case, however, if I turn off the fan, the anomaly will be detected and the same if I tap on the fan itself.
I want to remind that the anomaly detection is a first step to the predictive maintenance which just says when something is diverging from its normal behavior.
6th Step: Validation
Once the emulation phase was completed, the validation step is just a summary of the algorithm created highlighting the overall score, the balanced accuracy and the algorithm flowchart.
7th Step: Deployment
In the last step of the NEAI studio by clicking on generate library, it is possible to download the .h file with the library found in a .zip package.
The final package that was downloaded is made by different blocks.
How to implement the generated .h file in the ISPU device
Now, once the best possible library has been generated, it is needed to use in the MEMS Sensor embedding the ISPU, the ISM330ISN.
X-CUBE-ISPU software package
The X-CUBE-ISPU software package has examples to implement the library generated with the NEAI tool, but also there are prebuild examples and templates to be used with custom made C codes running your algorithms.
In this knowledge article it will be explained how you can implement the best library found with the NEAI tool in your ISPU device.
In the X-CUBE-ISPU main page, it is needed to go in the directories matching with the hardware used for this exercise, hence:
x-cube-ispu\Projects\NUCLEO-F401RE\Examples\CUSTOM\ ISM330ISN_NEAI_AnomalyDetection
In this folder you can find some files already present which must be changed with the files generated from the NEAI studio.
Therefore, the first step is to get two files from the NEAI package and copy them in the X-CUBE-ISPU software package.
- ispu_neai.h
- ispu_ctrl.h
Moreover, the ispu_neai.h is the binary configuration file which runs in the MEMS Sensor itself.
Build the X-CUBE-ISPU program and flash it in the sensor
After opening the STM32Programmer, it is needed to import the project.Hence clicking on the import project icon, we select existing projects into workspace.
Next it is needed to browse the project in the directory where the two files were copied:
x-cube-ispu\Projects\NUCLEO-F401RE\Examples\CUSTOM\ ISM330ISN_NEAI_AnomalyDetection
The first box must be unflagged, keeping just the second where it is written STM32CubeIDE.
There is a main consideration to be done here to understand how the code works since its aim is not to compare two different states, but to use a library like the one generated with the NEAI to make comparison between a normal state and an anomaly state. It works mainly as a state machine:
- The first step is to go through the learning phase where a new normal signal is taken
- Once this new learning is completed, the library from the NEAI will run checking if the signals match the normal state and in this case no interrupts are raised. On the other side, if using the library some inferences are found between the optimal and the abnormal signal then the interrupt is raised
- Only if the anomaly is detected then the data corresponding to the anomaly will be printed
Next step is to build the project using the building button at the top left of the screen and once that is done, after connecting the hardware setup to the laptop, the debug can start.
The debug starts after clicking on the green bug and once this is done, we can check the results in a terminal.
Checking the results
In this knowledge article the open software tool putty will be used in order to see the results directly from the ISPU.Launching putty, the first step is to click on the serial mode and write the correct serial line and the speed. While the serial line can change, the speed is always 115200.
Some visualization parameters can be modified, for instance going on terminal, for a better visualization it is needed to flag:
- Implicit CR in every LF
- Implicit LF in every CR
Finally, by clicking on open, the terminal will display the values.
As anticipated, the first step is the learning phase to get the new normal data.
As soon as this step is completed, the message “learning phase end” appears ending the learning phase.
If the library works well, there will be no data streamed if the learning condition is kept and no data are streamed.
As soon as an anomaly is created, the data will start streaming for that specific kind of anomaly.
The data can be collected in a microcontroller to make a dataset for future classifications.
Conclusions
In this knowledge article it was shown and explained how to use the ISM330ISN which embeds the ISPU with the NEAI studio software tool.
The NEAI software tool allows to find the best possible library among many combinations starting from a set of good and bad signals which can be used once getting new data in the learning phase.
The X-CUBE-ISPU is a software tool which needs to be used to flash the library generated by the NEAI tool in the ISPU device.
You can check the following link for a step-by-step webinar on the same topic.
Webinar: Anomaly detection and AI: one step closer to ST intelligent sensors – STMicroelectronics
Hello @Denise SANFILIPPO and @Luca SEGHIZZI ,
Thank you so much for this update.
After reading your message I try to generate a project to make a datalogger in the NEAI using X-NUCLEO-IKS01A3 and NUCLEO-H743ZI2 (sorry for the mistyping at first message), but with no success.
I can't identify where in NEAI I can't determine that I use the X-NUCLEO-IKS01A3 and NUCLEO-H743ZI2 to make this datalogger running (please look the pictures in attached).
Do you have a step-by-step procedure that I can use to doing this?
Wait your return.
Thanks, in advanced!
Regards.