STSW-IMG501 Programmed Frame Rate Doesn't Match Software Output

astral256 · ‎2024-09-26

I am using the STSW-IMG501 software with the STEVAL-EVK-U0I1 evaluation kit and the STEVAL-56G3MAI1 evaluation board for the VD56G3 image sensor. I am trying to capture videos of my target application at high frame rates to assess the sensor properties, but I am unable to get the STSW-IMG501 to capture at frame rates equivalent to what is programmed in the VD56G3 image sensor.

A short description of applicable frame rate parameters:

Line Length - 1236 (0x0300, 0xd4, 0x04)
Pixel Clock - 160.8 MHz
Line Time - 7.69 us (Line Length / Pixel Clock)
Subsampling/Binning Factor - 1 (no subsampling or binning)
Ystart - 442 (0x045a, 0xba, 0x01)
Yend - 921 (0x045c, 0x99, 0x03)
Frame Length Offset - 69
Minimum Frame Length - 549 ((Yend - Ystart+1)/(subsampling factor) + frame length offset)
Target Frame Rate - 200 Hz
Programmed Frame Length - 650 (0x0458, 0x8a, 0x02)
Target Frame Duration - 5 ms
Target Exposure Duration - 3.5 ms
Coarse Exposure - 455 (0x044e, 0xc7, 0x01)

In order to verify my configuration of the frame rate, I configured GPIO0 to be FSYNC_OUT (0x0467, 0x00). To control the 74LVC1T45GF translating transceiver on the STEVAL-56G3MAI1, I set GPIO1 to be GPIO_OUT, High (0x0468, 0x15). I then soldered a wire onto the 0 Ohm jumper R55 on STEVAL-EVK-U0I1 and was able to measure a pulse waveform with frequency 200 Hz, so I know for certain that the VD56G3 is programmed to run at 200 Hz and is running at 200 Hz.

However, when running STSW-IMG501, the "execution rate" field in the upper left corner of the UI only jumps around ~60 FPS. When counting on fingers in front of the video, I can verify that a capture image sequence runs at ~70 FPS.

I did some research into the camera controller microcontroller used on the STEVAL-EVK-U0I1 (Infineon CX3) and it should be able to support 2.4 Gbps. Using similar back of paper calculations as seen on the Infineon dev community boards, I should be able to reach ~180 fps using the CX3, but I am still not seeing close to those rates (https://community.infineon.com/t5/USB-superspeed-peripherals/CX3-MIPI-Configure-for-1600x1200-resolution/td-p/706304#.).

That is all to say, what is the maximum frame rate capable of recording images from the STSW-IMG501? Are we hard limited to the ~60 Hz range, or is there additional parameterization I can perform to get it higher? Can we assume a uniform frame delay between frames? How can we measure the frame rate of image sequences captured in the STSW-IMG501? Some additional documentation in this area would be greatly appreciated.

astral256 · ‎2024-10-30

I see. I ran the sensor at a resolution 64x48 (minimizing how much the bandwidth will affect the software speed) and 88 FPS (the maximum limit for the GUI). I saved an image sequence of 10 frames as a .pgm file type and monitored the FRAME_COUNTER variable in the metadata. Across the 10 images, the frame counter was reported as:

14
15
16
18
19
28
29
30
31
32

It appears as though the host PC itself plays a very large role in the speed of transmission. Almost 10 frames were lost during a 100 ms delay between frames 5 and 6. The STSW-IMG501 software cannot be relied on to produce image data with consistent timing at high frame rates. I will look into using a different SoC to control the STEVAL-56G3MAI1 in the interim. I hope the new SDK you are working on corrects this issue.

View solution in original post

MeganeG · ‎2024-10-02

Hello,

Indeed, the EVK GUI has some limitations due to post-processing in the software. Quantifying the maximum frame rate is difficult as it depends on the PC used and the system bandwidth. At the sensor level, we can assume a uniform frame delay between frames, but this is not the case at the GUI level since it is system-dependent.

An indirect way to measure the frame rate is by using the frame counter in the metadata of the captured images.

In summary, the EVK GUI is not ideal for achieving high frame rates. We will soon release a new solution (SDK) on st.com that will address this issue. We will keep you informed.

Regards, Megane

Our community relies on fruitful exchanges and good quality content. You can thank and reward helpful and positive contributions by marking them as 'Accept as Solution'. When marking a solution, make sure it answers your original question or issue that you raised.

ST Employees that act as moderators have the right to accept the solution, judging by their expertise. This helps other community members identify useful discussions and refrain from raising the same question. If you notice any false behavior or abuse of the action, do not hesitate to 'Report Inappropriate Content'

astral256 · ‎2024-10-23

Megane,

Thank you for your response, this confirms my observations as well. I look forward to the release of your new SDK.

Could you elaborate what you mean by using the frame counter in the metadata of the captured images? I tried looking at the Exif metadata after saving the image sequence as a PNG but was unable to find any stored metadata. Are there other methods to check the metadata?

Cheers, Aster

MeganeG · ‎2024-10-23

Hi Aster,

Sure. I am not certain that we can read the metadata from PNG images, or at least not without a specific tool. You can also save the images as PGM files and use Notepad to read the metadata easily.

Regards, Megane

Our community relies on fruitful exchanges and good quality content. You can thank and reward helpful and positive contributions by marking them as 'Accept as Solution'. When marking a solution, make sure it answers your original question or issue that you raised.

ST Employees that act as moderators have the right to accept the solution, judging by their expertise. This helps other community members identify useful discussions and refrain from raising the same question. If you notice any false behavior or abuse of the action, do not hesitate to 'Report Inappropriate Content'

astral256 · ‎2024-10-30

I see. I ran the sensor at a resolution 64x48 (minimizing how much the bandwidth will affect the software speed) and 88 FPS (the maximum limit for the GUI). I saved an image sequence of 10 frames as a .pgm file type and monitored the FRAME_COUNTER variable in the metadata. Across the 10 images, the frame counter was reported as:

14
15
16
18
19
28
29
30
31
32

It appears as though the host PC itself plays a very large role in the speed of transmission. Almost 10 frames were lost during a 100 ms delay between frames 5 and 6. The STSW-IMG501 software cannot be relied on to produce image data with consistent timing at high frame rates. I will look into using a different SoC to control the STEVAL-56G3MAI1 in the interim. I hope the new SDK you are working on corrects this issue.