Anomaly detection fail on board

[HarishKumarSingh]

Board - B-L4S5I-IOT01A Board

AI application tool - NanoEdge AI Studio

CUBEMX

IDE - STM32CubeIDE 1.15.1

I made one model.

Model detect anomaly on nano studio.

I deploy model. Deployed system always show similarity a fix value. Sensor data change. Detection fail.

Sensor store value in int16 type variable. Sensor value is not float. Please tell to correct model.

Plse check program on link - https://drive.google.com/file/d/1vsZeAcgctPcmEx4skoKoW3RYAxauyKh1/view?usp=drive_link

Please see log file teraterm.log

[Julian E.]

Hello, 

Can you please attach your main.c instead of linking a drive link, I don't have access.

In the meantime:

• Make sure you use buffers of values not just samples to create a proper library
• test your library in NanoEdge emulator to see if the problem comes from the c implementation or from your library directly
• Make sure to do the the minimum number of learning recommended in NanoEdge 
• You can print the return of the function of the neai_anomalydetection_detect() to see if does return an error

Most NanoEdge AI function return the status of the library in the following enum, neai_state:

enum neai_state {
	NEAI_OK = 0,
	NEAI_INIT_FCT_NOT_CALLED = 123,
	NEAI_BOARD_ERROR, 
	NEAI_KNOWLEDGE_BUFFER_ERROR,
	NEAI_NOT_ENOUGH_CALL_TO_LEARNING,
	NEAI_MINIMAL_RECOMMENDED_LEARNING_DONE,
	NEAI_UNKNOWN_ERROR
};
#endif

Best regards,

Julian

[HarishKumarSingh]

Train fail.

[Julian E.]

Hello,

The init part is correct.

Concerning the learning, the minimum number of learning is known by the library, so you can do something like this:

do {
  filAccBuff()
  error_code = neai_anomalydetection_learn((float*)acc_buf)
  printf("%s\r\n",error_code)
}
while ( error_code != NEAI_MINIMAL_RECOMMENDED_LEARNING_DONE)

The neai_anomalydetection_learn can return:

	NEAI_NOT_ENOUGH_CALL_TO_LEARNING //if you need to do more learning
	NEAI_MINIMAL_RECOMMENDED_LEARNING_DONE //if you have done at least the minimum recommended number of learning

It is adviced to do more than the minimal number of learning, so you may want to do a loop and learn something like 100 signals (the number of learning depends on your library).

Then your detection part is correct. As long as you get a similarity between 0 and 100, it means that the function is working as expected. (You can still print the output of the function if you want to make sure that there is no errors).

If you get the same value every time, it may come from your library "being bad". Make sure to test the library directly in NanoEdge to see if you spot a different behavior.

Best regards, 

Julian

[HarishKumarSingh]

In nano studio. Emulator detect abnormal.

On board with different train count. Similarity is fix value. I use fan to collect vibration data. Any movement of fan do not change similarity. Please see log file teraterm.log.

Image 1 is emulator result for normal signal. Fan at speed one.

Image 2 is abnormal signal detect. Fan shake detect in emulator.

Image 3 is validation result.

In emulator result is near to perfect.

I did second benchmark. In train I change data with more variation. Validation accuracy for this benchmark is  also around 50 perctn. Emulator result is not detection. 

Some point of concern are float point conversion for this board, sensor value is in integer, train data input.

[Julian E.]

Hello,

Concerning the teraterm logs, everything seems good except for the similarity that stays the same.

The 126 output is when you need to do more learning and 127 correspond to the NEAI_MINIMAL_RECOMMENDED_LEARNING_DONE.

What is more concerning is your library. 

You have a good accuracy in benchmark but in Validation your model doesn't work at all. 57.22% balanced accuracy in an experiment is very bad.

You need to log new data, do a new benchmark and have at least 80% balanced accuracy when doing experiments in Validation.

When logging data, make sure not to collect data when the fan is not powered up.

Try to maintain the same conditions when logging data for the benchmark and logging data for the validation (position of the board, position of the fan etc). It is very important to stay in similar condition.

Best regards

Julian

[HarishKumarSingh]

For validation. What is learn file?Regular file I give fan speed 1 data. Abnormal file I give data with fan orientation change.

[Julian E. · Accepted solution]

Hello,

Learning files are like "training regular" files.

So, you need to choose a learning file (you can use the one from the benchmark for example), then a new dataset containing regular signals and a new dataset containing abnormal signals.

We use new dataset in Validation to make sure that the model has not overfitted on the training data (that it is also able to classify new data and not only the training data).

The learning is what you do in your main.c

You first init the model with the neai_anomalydetection_init()

Then call the neai_anomalydetection_learn() multiple time to retrain the model with regular/nominal data

Then you make detection with the neai_anomalydetection_detect() function.

More information here: https://wiki.st.com/stm32mcu/wiki/AI:NanoEdge_AI_Library_for_anomaly_detection_(AD)

Best regards,

Julian