camera doesn't detect- Deploying custom object detection model on stm32n6 nucleo board

[faiq]

Hi, I am trying to deploy a custom object detection model on the STM32N6-Nucleo board.
I started with this:
https://github.com/STMicroelectronics/stm32ai-modelzoo-services/tree/main/object_detection
The training gets done, I get the performance metrics,
python stm32ai_main.py
The deployment code works fine, and I get the deployment complete message. But when I change from development mode to flash boot mode. It doesn't detect the camera.

When I program the demo camera.hex file
https://github.com/STMicroelectronics/x-cube-n6-camera-capture/tree/main/Binary/NUCLEO-N657X0-Q
or object detection .hex file,
https://github.com/STMicroelectronics/STM32N6-GettingStarted-ObjectDetection/tree/main/Binary/NUCLEO-N657X0-Q/USB-UVC-Display
the camera works perfectly.

Below are the training and deployment configuration files. These are the only files I changed. I would really appreciate any help in this regard. Thank you,

# training configure.yaml file:
general:
project_name: custom_food_yoloxn_256
logs_dir: logs
saved_models_dir: saved_models
display_figures: false
gpu_memory_limit: 16
num_threads_tflite: 12
global_seed: 127

operation_mode: chain_tqe

model:
model_type: st_yoloxn
model_name: st_yoloxn_d033_w025
input_shape: (256,256,3)

dataset:
format: darknet_yolo
dataset_name: darknet_yolo
class_names: [beverage, food]
exclude_unlabeled: true
download_data: false
max_detections: 50
train_images_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/images/train
train_annotations_path: path/r/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/labels/train
val_images_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/images/val
val_annotations_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/labels/val
test_images_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/images/test
test_annotations_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/labels/test
test_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/images/tfs_labels/test
quantization_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/images/test

preprocessing:
rescaling:
scale: 1/255
offset: 0
resizing:
aspect_ratio: fit
interpolation: nearest
color_mode: rgb

data_augmentation:
random_contrast:
factor: 0.4
random_brightness:
factor: 0.3
random_flip:
mode: horizontal
random_translation:
width_factor: 0.15
height_factor: 0.15
fill_mode: reflect
interpolation: nearest
random_rotation:
factor: 0.02
fill_mode: reflect
interpolation: nearest
random_crop:
crop_center_x: (0.25, 0.75)
crop_center_y: (0.25, 0.75)
crop_width: (0.6, 0.9)
crop_height: (0.6, 0.9)
change_rate: 0.9

training:
dropout: null
batch_size: 64
epochs: 100
optimizer:
Adam:
learning_rate: 0.0025
callbacks:
LRWarmupCosineDecay:
initial_lr: 1.0e-05
warmup_steps: 20
max_lr: 0.00125
hold_steps: 20
decay_steps: 300
end_lr: 1.0e-06
EarlyStopping:
monitor: val_loss
patience: 60
restore_best_weights: true
verbose: 1
quantization:
quantizer: TFLite_converter
quantization_type: PTQ
quantization_input_type: uint8
quantization_output_type: int8
export_dir: quantized_models
postprocessing:
confidence_thresh: 0.300
NMS_thresh: 0.5
IoU_eval_thresh: 0.5
plot_metrics: true
max_detection_boxes: 100

mlflow:
uri: ./tf/src/experiments_outputs/mlruns

hydra:
run:
dir: ./tf/src/experiments_outputs/custom_food_yoloxn_256

# deploying custom model configure file
general:
project_name: custom_food_yoloxn_256
logs_dir: logs
saved_models_dir: saved_models
display_figures: false
gpu_memory_limit: 16 #24
num_threads_tflite: 12
global_seed: 127

operation_mode: deployment

model:
model_type: st_yoloxn
model_path: C:/stm32zoo/stm32ai-modelzoo-services/object_detection/tf/src/experiments_outputs/custom_food_yoloxn_256/quantized_models/quantized_model.tflite

dataset:
format: darknet_yolo
dataset_name: darknet_yolo
class_names: [beverage, food]
exclude_unlabeled: true
download_data: false
max_detections: 50
train_images_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/images/train
train_annotations_path: path/r/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/labels/train
val_images_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/images/val
val_annotations_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/labels/val
test_images_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/images/test
test_annotations_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/labels/test
test_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/images/tfs_labels/test
quantization_path: path/yolov5/yolo_dataset_prep/yolo_ready_dataset_416/images/test

preprocessing:
rescaling:
scale: 1/255
offset: 0
resizing:
aspect_ratio: fit
interpolation: nearest
color_mode: rgb

data_augmentation:
random_contrast:
factor: 0.4
random_brightness:
factor: 0.3
random_flip:
mode: horizontal
random_translation:
width_factor: 0.15
height_factor: 0.15
fill_mode: reflect
interpolation: nearest
random_rotation:
factor: 0.02
fill_mode: reflect
interpolation: nearest
random_crop:
crop_center_x: (0.25, 0.75)
crop_center_y: (0.25, 0.75)
crop_width: (0.6, 0.9)
crop_height: (0.6, 0.9)
change_rate: 0.9

training:
dropout: null
batch_size: 64
epochs: 100
optimizer:
Adam:
learning_rate: 0.0025
callbacks:
LRWarmupCosineDecay:
initial_lr: 1.0e-05
warmup_steps: 20
max_lr: 0.00125
hold_steps: 20
decay_steps: 300
end_lr: 1.0e-06
EarlyStopping:
monitor: val_loss
patience: 60
restore_best_weights: true
verbose: 1
quantization:
quantizer: TFLite_converter
quantization_type: PTQ
quantization_input_type: uint8
quantization_output_type: int8
export_dir: quantized_models
postprocessing:
confidence_thresh: 0.300
NMS_thresh: 0.5
IoU_eval_thresh: 0.5
plot_metrics: true
max_detection_boxes: 10

tools:
stedgeai:
optimization: balanced
on_cloud: False
path_to_stedgeai: C:/ST/STEdgeAI/4.0/Utilities/windows/stedgeai.exe
path_to_cubeIDE: C:/ST/STM32CubeIDE_2.0.0/STM32CubeIDE/stm32cubeide.exe

deployment:
c_project_path: ../application_code/object_detection/STM32N6/
IDE: GCC
verbosity: 1
hardware_setup:
serie: STM32N6
board: NUCLEO-N657X0-Q
output: "UVCL"

mlflow:
uri: ./tf/src/experiments_outputs/mlruns

hydra:
run:
dir: ./tf/src/experiments_outputs/custom_food_yoloxn_256

[Julian E.]

Hi @faiq,

Do you specifically need yolox? 

I saw a lot of questions and issues regarding yolo deployment and I specifically asked for the tutorial to deploy ultralytics models to be updated.

If you think yolov8n, yolov11n and yolo26 are suitable for you, I suggest following the updated tutorial here:

ultralytics/examples/YOLOv8-STEdgeAI at main · stm32-hotspot/ultralytics

Have a good day,

Julian

[faiq]

Hi @Julian E. Thank you for your comment.
Actually, the issue seems to be with the hardware setup somehow. because I posted same question on stm32-modelzoo (https://github.com/STMicroelectronics/stm32ai-modelzoo-services/issues/46#issuecomment-4327097772)
and the person tested my quantized model and shared the results. I explained my hardware setup and he agreed everything I was doing was fine. Maybe, I need to clone the repo again and do the deploymnet again or something. You can check our conversation on that issue (https://github.com/STMicroelectronics/stm32ai-modelzoo-services/issues/46#issuecomment-4327097772). 

[Julian E.]

Hi @faiq,

Thanks for the update.

Indeed, these kinds of error are very strange because most of the time, we cannot reproduce them..

You can try to update model zoo and try again.

Please use the recommended version of cubeIDE, I know for example that the audio event detection is working for cubeIDE 1.17 only (I don't know why).

Please look at the readme of the application (in /application_code/)

Let us know if you find the cause.

Have a good day,

Julian

[faiq]

@Julian E. , you were right, for object detection also CubeIDE v1.17.0 works. Thank you so much