Issues Encountered with STM32Cube AI Cloud Usage

[nour1]

I have built and trained a recursive model with my data and generated the `.h5` and `.tflite` files. While I have been able to optimize the `.h5` file successfully, I am facing difficulties quantizing it. Below is the error I receive when attempting quantization:

////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////Executing with:  {'model': '/tmp/quantization-service/24d7cfa5-3e25-4be7-b847-3fc7bf2b6d81/RNN_16_model.h5', 'data': None, 'input_type': tf.float32, 'output_type': tf.float32, 'optimization': <Optimize.DEFAULT: 'DEFAULT'>, 'output': '/tmp/quantization-service/24d7cfa5-3e25-4be7-b847-3fc7bf2b6d81', 'disable_per_channel': False}

No data specified, enabling fake quantization
Converting original model to TFLite...
/app/quantizer/cli.py:53:1: error: 'tf.TensorListReserve' op requires element_shape to be static during TF Lite transformation pass
    res = quantize_from_local_file(
^
<unknown>:0: note: loc(fused["StatefulPartitionedCall:", "StatefulPartitionedCall"]): called from
/app/quantizer/cli.py:53:1: error: failed to legalize operation 'tf.TensorListReserve' that was explicitly marked illegal
    res = quantize_from_local_file(
^
<unknown>:0: note: loc(fused["StatefulPartitionedCall:", "StatefulPartitionedCall"]): called from
<unknown>:0: error: Lowering tensor list ops is failed. Please consider using Select TF ops and disabling `_experimental_lower_tensor_list_ops` flag in the TFLite converter object. For example, converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]\n converter._experimental_lower_tensor_list_ops = False


2024-06-11 11:33:30.384275: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-06-11 11:33:30.412400: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-06-11 11:33:32.873971: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:378] Ignored output_format.
2024-06-11 11:33:32.874007: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:381] Ignored drop_control_dependency.
2024-06-11 11:33:32.874486: I tensorflow/cc/saved_model/reader.cc:83] Reading SavedModel from: /tmp/tmpkfrpccum
2024-06-11 11:33:32.876786: I tensorflow/cc/saved_model/reader.cc:51] Reading meta graph with tags { serve }
2024-06-11 11:33:32.876809: I tensorflow/cc/saved_model/reader.cc:146] Reading SavedModel debug info (if present) from: /tmp/tmpkfrpccum
2024-06-11 11:33:32.882099: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:388] MLIR V1 optimization pass is not enabled
2024-06-11 11:33:32.883401: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle.
2024-06-11 11:33:32.941507: I tensorflow/cc/saved_model/loader.cc:217] Running initialization op on SavedModel bundle at path: /tmp/tmpkfrpccum
2024-06-11 11:33:32.957142: I tensorflow/cc/saved_model/loader.cc:316] SavedModel load for tags { serve }; Status: success: OK. Took 82657 microseconds.
2024-06-11 11:33:32.990057: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
loc(callsite(callsite(fused["TensorListReserve:", callsite("sequential/simple_rnn/TensorArrayV2_1@__inference__wrapped_model_444"("/app/quantizer/cli.py":53:1) at callsite("/app/quantizer/quantize.py":63:1 at callsite("/app/quantizer/quantize.py":114:1 at callsite("/app/quantizer/helpers.py":47:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1139:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1093:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1601:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1579:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/convert_phase.py":205:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1502:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/functional.py":460:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/training.py":4031:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/base_layer.py":3470:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saved_model/base_serialization.py":61:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saved_model/layer_serialization.py":79:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saved_model/layer_serialization.py":106:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saved_model/model_serialization.py":53:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saved_model/save_impl.py":234:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saving_utils.py":158:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saving_utils.py":147:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py":65:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/training.py":590:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py":65:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/base_layer.py":1149:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py":96:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/sequential.py":398:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/functional.py":515:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/functional.py":672:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/layers/rnn/base_rnn.py":556:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py":65:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/base_layer.py":1149:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py":96:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/layers/rnn/simple_rnn.py":411:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/layers/rnn/base_rnn.py":722:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/backend.py":4988:1 at "/usr/local/lib/python3.9/site-packages/keras/src/backend.py":4989:1)))))))))))))))))))))))))))))))))))] at fused["StatefulPartitionedCall:", callsite("StatefulPartitionedCall@__inference_signature_wrapper_1132"("/app/quantizer/cli.py":53:1) at callsite("/app/quantizer/quantize.py":63:1 at callsite("/app/quantizer/quantize.py":114:1 at callsite("/app/quantizer/helpers.py":47:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1139:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1093:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1601:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1579:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/convert_phase.py":205:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1502:1 at "/usr/local/lib/python3.9/site-packages/tensorflow/python/saved_model/signature_serialization.py":168:1))))))))))]) at fused["StatefulPartitionedCall:", "StatefulPartitionedCall"])): error: 'tf.TensorListReserve' op requires element_shape to be static during TF Lite transformation pass
loc(callsite(callsite(fused["TensorListReserve:", callsite("sequential/simple_rnn/TensorArrayV2_1@__inference__wrapped_model_444"("/app/quantizer/cli.py":53:1) at callsite("/app/quantizer/quantize.py":63:1 at callsite("/app/quantizer/quantize.py":114:1 at callsite("/app/quantizer/helpers.py":47:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1139:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1093:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1601:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1579:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/convert_phase.py":205:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1502:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/functional.py":460:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/training.py":4031:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/base_layer.py":3470:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saved_model/base_serialization.py":61:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saved_model/layer_serialization.py":79:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saved_model/layer_serialization.py":106:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saved_model/model_serialization.py":53:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saved_model/save_impl.py":234:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saving_utils.py":158:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/saving/legacy/saving_utils.py":147:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py":65:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/training.py":590:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py":65:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/base_layer.py":1149:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py":96:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/sequential.py":398:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/functional.py":515:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/functional.py":672:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/layers/rnn/base_rnn.py":556:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py":65:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/engine/base_layer.py":1149:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py":96:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/layers/rnn/simple_rnn.py":411:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/layers/rnn/base_rnn.py":722:1 at callsite("/usr/local/lib/python3.9/site-packages/keras/src/backend.py":4988:1 at "/usr/local/lib/python3.9/site-packages/keras/src/backend.py":4989:1)))))))))))))))))))))))))))))))))))] at fused["StatefulPartitionedCall:", callsite("StatefulPartitionedCall@__inference_signature_wrapper_1132"("/app/quantizer/cli.py":53:1) at callsite("/app/quantizer/quantize.py":63:1 at callsite("/app/quantizer/quantize.py":114:1 at callsite("/app/quantizer/helpers.py":47:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1139:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1093:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1601:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1579:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/convert_phase.py":205:1 at callsite("/usr/local/lib/python3.9/site-packages/tensorflow/lite/python/lite.py":1502:1 at "/usr/local/lib/python3.9/site-packages/tensorflow/python/saved_model/signature_serialization.py":168:1))))))))))]) at fused["StatefulPartitionedCall:", "StatefulPartitionedCall"])): error: failed to legalize operation 'tf.TensorListReserve' that was explicitly marked illegal
error: Lowering tensor list ops is failed. Please consider using Select TF ops and disabling `_experimental_lower_tensor_list_ops` flag in the TFLite converter object. For example, converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]\n converter._experimental_lower_tensor_list_ops = False

/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

Additionally, with the `.tflite` file, I'm unable to either quantize or optimize it.

Just for information, I selected STM32 MCUs in the beginning.

[hamitiya]

Hello @nour1 ,

Is it possible to share your model in order to debug it ? I can try on my side with different version of TensorFlow Lite.

Our quantization service may rely on an older version and possible require a set of flags to be set. These flags cannot be provided as an end-user.

Since post-training quantization with fake data is only for benchmark purpose, we recommend to do it on your own using your own data (see: Post-training quantization  |  TensorFlow Lite )

Best regards,

Yanis

[nour1]

hello @hamitiya , 

Thank your for your response.

My model is this :

I will have a look on post-training quantization, Thank you.

Best regards,

Nour

[hamitiya]

Hello @nour1 

Sorry for late reply.

It looks like your model, to be quantized, requires a set of operators that are currently not native in Tensorflow Lite Converter:

- TensorListReserve, TensorListSetItem, TensorListStack

Doing local quantization and adding "tf.lite.OpsSet.SELECT_TF_OPS" in order to enable TF Kernels fallback, I am able to generate a quantized model that I can provide to you.

However, this output file will not be usable in ST Edge AI Developer Cloud since our tool does not support these layers, which are added in quantized file:

- WHILE, FlexTensorListStack, FlexTensorListReserve.

You can find more information to quantize for X-CUBE-AI here:

AI:X-CUBE-AI support of ONNX and TensorFlow quantized models - stm32mcu

Best regards,

Yanis

[hamitiya]

Hello @nour1 

Does this answer permit you to go forward with your project ?

Best regards,

Yanis