VL53L7CX gesture detection

[neil-viven]

Hi, I am trying to use the VL53L7CX to do gesture detection for a household product. A few questions:

Do these software downloads only run on a Windows PC? https://www.st.com/en/embedded-software/stsw-img035.html ? I am trying to do embedded development and gesture recognition on a microcontroller.

Are the "gestures" recognized only static signs, like a thumbs up, heart, etc? Or does ST have software or firmware written to recognize dynamic gestures, like a wave left, wave right, etc?

I am trying to write my own gesture recognition code to run directly on an embedded microcontroller. There seems to be a lot of spurious or incorrect distance data. Currently I am filtering for 1) distances not too close or too far 2) signal_per_spad must be sufficiently high and range_sigma_mm must be sufficiently low and 3) target_status = 5. Still I get bad or fake distance data.

What environmental variables could cause spurious or fake distances measurements? Reflected or ambient light?

I have tried using the motion_indicator library provided by ST, but without much luck. What exactly does it tell you? The amount of motion within each of the 64 zones? Can it be used to detect overall motion across the whole grid, like a wave? How is it sized relative to the 4x4 or 8x8 grid?

Anything else I should know about using the sensor? Am happy to provide more info if you have questions on the application.

Thanks very much!

[Anne BIGOT]

Hello 

ST published a dedicated software package to hand posture and gesture recognition since the STSW-IMG035.

Please have a look to the STSW-IMG050. 

There is also a dedicated page : ST gesture & hand posture recognition - STMicroelectronics

Regards

[neil-viven]

Hi @Anne BIGOT thank you for your fast response! I see there is a presentation inside the STSW-IMG050 download that advises to start with gestures_example.c in the STSW-IMG035 package. That is very helpful, thanks.

Does that mean that the gesture and hand posture recognition can be done completely on an embedded device (microcontroller) using the .c and .h files in the STSW-IMG035 and STSW-IMG050 packages? And the Windows executables are only there for a convenient application and nice GUI? Or does any of the gesture and hand posture detection get done using a PC?

[Zhiyuan.Han]

Hi Neil

Inside the STSW-IMG035 package, there are 4 sub folder, the 1st one is the GUI, which is used for quick evaluation.

The 2nd folder end with F401, this is the example based on STM32F401, this is ST 32-bit MCU, which is the solution match your request. you can refer this one as example for microcontroller.

One more thing to share, the gesture code was compiled to lib instead of source code, you can find the libs in 3rd folder. 

Br

Zhiyuan.Han

[neil-viven]

@Zhiyuan.Han thank you very much

[neil-viven]

Thanks @Zhiyuan.Han  I am working with these files in the STSW-IMG035 package and hope to get them working soon.

In the STSW-IMG050 package or Combined Usecases package, are the files that implement hand posture detection open source? Can I see how they work, or are only .h files and bins available, not all the .c files?

In Combined_Usecases_kit_1.0.0 I see the EVK which seems to run on on desktop, and for the microcontroller I see Drivers (typical STM32 files, vl53LMZ API), Middlewares, and Combined_Usecases. 

Is Middlewares the place to look for hand posture recognition? I see a lot of .h files and a couple .a files, but not very many .c files.

In Combined_Usecases_v1.0.0/Combined_Usecases, I see .h and .c files, but they don't seem to implement hand posture detection.

In STSW-IMG050, a similar situation: EVK folder with desktop files. Then in STSW-IMG050_F401_v1.1.0, I see Application, Middlewares, and Drivers (typical STM32 drivers). From what I can tell, the files in Application/Network are calling some AI functions, but I don't think I see the hand posture detection there. Application/STSW-IMG050_F401_v1.1.0 has a hand posture binary, but not .c files, and an aimodel.h with no .c file. Middlewares has .h files but not .c files.

If possible, it would be really great to see the source code used to do hand posture! Am I missing it in one of these packages, or is it hidden? Any way we can see it?

[Zhiyuan.Han]

Hi 

About the hand posture, it's based on AI solution, so there is no open source for the hand posture recognition.

if you have interesting on AI method, please access below link for more detail.  

How to do camera-free hand posture recognition with stm32cube.AI - STMicroelectronics

Br

Zhiyuan.Han

[neil-viven]

Thanks @Zhiyuan.Han, and for STSW-IMG035_F401_v1.5.2\GesturesMZ is it the same? No open source for Gesture Recognition?
The open source would really help us for both Gesture Recognition and Hand Posture. Is there anyway ST would be willing to provide this? It would really help for the product we are developing.

[labussiy · Accepted solution]

Hi @neil-viven ,

We have developed three solutions based on VL53L7/8:

• STSW-IMG035: Motion Gesture Detection (Hand Tracking X/Y/Z, Swipes, Tap, etc.)
• STSW-IMG050: Hand Posture (Static Gesture)
• STSW-IMG048: Smart Presence Detection

For each of these solutions, you can download an all-in-one package from ST.com containing:

• C project example for STM32F401 (application C code + compiled library)
• Algorithm compiled libraries for different ARM cores (M4, M3, M0+, M7, M33)
• A graphical software to display the solution outputs (fully running on the STM32 MCU)

The package includes a detailed User Guide, which I strongly recommend reviewing. It explains the package contents and how to use them.

Regarding the Hand Posture solution, it is AI-based. Everything is available: datasets, training scripts, C files, etc. You can find it on GitHub ST ModelZOO: GitHub - STMicroelectronics/stm32ai-modelzoo-services

If you are not familiar with Python and AI, we developed a graphical interface for AI model training and solution customization: Edge AI tools - STMicroelectronics

Finally, the Combined project available on ST.com shows how to integrate all three solutions together.

It's a lot of information, but with the User Guide and the links I shared, I believe you can start exploring the solutions :beaming_face_with_smiling_eyes:.

Best regards,
Yann