Hardware validation of NN model

[Ritesh1]

I am validating a classification model on STM32H745-DISC0 board. 

The validation is successful but id does not show any accuracy related parameter.

It shows all the parameters except accuracy.

rmse

Mae

L2r

mean

std

Ritesh

[hamitiya]

Hello,

The tool shows accuracy (acc column) means "Classification accuracy". If it is not displayed, it means we were not able to detect your model as a classifier. You can still force this flag with '--classifier' or through the "Advanced Settings" in STM32CubeMX / X-CUBE-AI.

From the embedded documentation:

"

Classification accuracy (acc)

For classifier model type, Classification accuracy is what we usually mean, when the term accuracy is used. ACC is the ratio between of correct predictions to the total number of inputs. This indicator evaluates the performance of the classifier model, if a regressor type is passed, the ACC is NOT calculated and n.a. value is reported

"

Best regards,

Yanis

[Ritesh1]

Thanks for the reply.

I forced the model as classifier,  but still it does not show the accuracy. 

Ritesh

[hamitiya · Accepted solution]

Hello,

I assume here you are refering to the validation result in X-CUBE-AI interface in STM32CubeMX.

While doing the validation, could you verify if you have something like these rows in the textual output:

Computing the metrics...
 Cross accuracy report #1 (reference vs C-model)
 ----------------------------------------------------------------------------------------------------
 notes: - the output of the reference model is used as ground truth/reference value
        - 10 samples (5 items per sample)
  acc=100.00%, rmse=0.000000153, mae=0.000000065, l2r=0.000000396, nse=1.000, cos=1.000
  5 classes (10 samples)
  ---------------------------------
  C0        0    .    .    .    .
  C1        .    0    .    .    .
  C2        .    .    0    .    .
  C3        .    .    .    8    .
  C4        .    .    .    .    2
 Evaluation report (summary)
 -------------------------------------------------------------------------------------------------------------------------------------------
 Output       acc       rmse        mae         l2r         mean         std         nse         cos         tensor
 -------------------------------------------------------------------------------------------------------------------------------------------
 X-cross #1   100.00%   0.0000002   0.0000001   0.0000004   -0.0000000   0.0000002   1.0000000   1.0000000   activation_3, (5,), m_id=[5]
 -------------------------------------------------------------------------------------------------------------------------------------------
  acc  : Classification accuracy (all classes)
  rmse : Root Mean Squared Error
  mae  : Mean Absolute Error
  l2r  : L2 relative error
  nse  : Nash-Sutcliffe efficiency criteria, bigger is better, best=1, range=(-inf, 1]
  cos  : COsine Similarity, bigger is better, best=1, range=(0, 1]

You should have at least one row (X-cross #<n> where n is the output index).

Best regards,

Yanis

[Ritesh1]

Except accuracy it shows all the parameters. I will try with a different classification model.

Thanks

Ritesh 

[Ritesh1]

Could you provide the NN model that you are using for the test  purpose as I used the 3-4 different classification models and with none of them I got the accuracy during validation.

Ritesh

[hamitiya]

Hello,

For my example I use a simple model but I can reproduce this accuracy using models from our STM32 Model Zoo: https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/image_classification/pretrained_models/fdmobilenet/ST_pretrainedmodel_public_dataset

Evaluation report (summary) 
 -------------------------------------------------------------------------------------------------------------------------------------------- 
 Output       acc       rmse        mae         l2r         mean        std         nse         cos         tensor 
 -------------------------------------------------------------------------------------------------------------------------------------------- 
 X-cross #1   100.00%   0.0418388   0.0187500   0.1083516   0.0003125   0.0422624   0.9842321   0.9941229   conversion_35, (5,), m_id=[35] 
 --------------------------------------------------------------------------------------------------------------------------------------------

Is it possible to also share your model to see if I can reproduce it ?

Best regards,

Yanis

[Ritesh1]

How to provide images as a validation input.