VD55G0 and STEVAL-EVK-U0I Python SDK: High framerate problem

[mlewandowski]

Hi,

I am using the STEVAL-EVK-U0I kit with the VD55G0 sensor and the provided Python SDK. Firstly, I wanted to say I really love the capabilities of this sensor and it looks extremely useful for my work. 

My goal is to capture high frame rates (200-500 FPS) using subsampling. I have configured the sensor for 4x4 Subsampling and a short frame length. Based on the VD55G0_REG_STATUS_FRAME_COUNTER and VD55G0_REG_STATUS_FRAME_RATE, the sensor is successfully generating ~530 FPS.

My problem is the following: While the sensors seem to be running at 530 FPS, the Python get_raw_image() function appears to be retrieving frames at 80-110 FPS. I am also getting a heap corruption error (perhaps the SDK cannot keep up with this high framerate? I am not sure).

My questions :

- Is this Python SDK capable of handling that high framerate, or am I doing something wrong?

- How can I capture these high framerates? 

- Is there any other API (C/C++?) which could be used to achieve this high framerate ? 

Code used for testing : 

import os
from vd55g0_constants import *
from image_sensor_python_sdk import *
from smart_imshow import SmartImshow
import time
from vd55g0_frame_decoding import decode_frame



VD55G0_REG_DPHYTX_CTRL = 0x093A
VD55G0_REG_MAX_EXPOSURE_LINES = 0x096c
VD55G0_REG_LOW_POWER_MODE = 0x0957
bits_per_pixel = BITS_10


# Init the board
sensor_dll = SensorDll(LIB_NAME, decode_frame)
sensor_dll.init_board()

Frame_length = 250
x_start = 2
x_end = 641
y_start = 2
y_end = 601

NO_BIN_SUB = 0
BIN_2x2 = 1
BIN_4x4 = 2
SUB_2x2 = 3
SUB_4x4 = 4
EXP_AUTOMATIC_MODE = 0
EXP_AE_FREEZE = 1
EXP_MANUAL_MODE = 2


sensor_dll.Write16(VD55G0_REG_CONTEXTS_OUT_ROI_X_START, x_start)
sensor_dll.Write16(VD55G0_REG_CONTEXTS_OUT_ROI_X_END, x_end)
sensor_dll.Write16(VD55G0_REG_CONTEXTS_OUT_ROI_Y_START, y_start)
sensor_dll.Write16(VD55G0_REG_CONTEXTS_OUT_ROI_Y_END, y_end)
sensor_dll.Write16(VD55G0_REG_STATICS_FORMAT_CTRL, bits_per_pixel)
sensor_dll.Write8(VD55G0_REG_CONTEXTS_READOUT_CTRL, SUB_4x4)
sensor_dll.Write16(VD55G0_REG_STATICS_ORIENTATION, 0x1)
sensor_dll.Write32(VD55G0_REG_SENSOR_SETTINGS_EXT_CLOCK, 12000000)
sensor_dll.Write32(VD55G0_REG_SENSOR_SETTINGS_CLK_PLL_MIPI, 1200000000)


frame_length = sensor_dll.Read16(VD55G0_REG_CONTEXTS_FRAME_LENGTH)

print(f"Frame length: {frame_length} lines")


sensor_dll.Write16(VD55G0_REG_CONTEXTS_FRAME_LENGTH, Frame_length)
sensor_dll.Write8(VD55G0_REG_CONTEXTS_EXP_MODE, EXP_AUTOMATIC_MODE)


print("wait delay : ",sensor_dll.Read16(VD55G0_REG_CONTEXTS_WAIT_DELAY))
print(f"DPHYTX_CTRL : {sensor_dll.Read8(VD55G0_REG_DPHYTX_CTRL)}")
print(f"MAX_EXPOSURE_LINES : {sensor_dll.Read8(VD55G0_REG_MAX_EXPOSURE_LINES)}")
print(f"LOW_POWER_MODE : {sensor_dll.Read8(VD55G0_REG_LOW_POWER_MODE)}")


# Start the stream

sensor_dll.start_stream()
print(f"FRAMERATE : {sensor_dll.Read16(VD55G0_REG_STATUS_FRAME_RATE)//16}")
start_frames = sensor_dll.Read16(VD55G0_REG_STATUS_FRAME_COUNTER)

raw_images = []

# Grab N first frames

# Split images
t = time.time()

for i in range(10):
    sensor_dll.get_raw_image()


end_frames = sensor_dll.Read16(VD55G0_REG_STATUS_FRAME_COUNTER)
finish = time.time()
no_frames = end_frames - start_frames
print(f"in {finish-t} we got {end_frames - start_frames} frames")
print(f"{no_frames/(finish-t)} FPS")

print("Processing time for 10 frames: ", finish - t)
print(f"FPS : {10/(finish - t)}")

sensor_dll.stop_stream()
sensor_dll.deinit_board() 

And these are the outputs : 

Loading st_brightsense_sdk_vd55g0.dll
IMG platform version 305 loaded :)
Boards connected: (0,P4405C-00A#0543)
No board id was specified, selecting board 0.
Initiating CCI communication at address 32
Sensor detected on interface 0
Cut 1.1 detected loading patch 2.11
Frame length: 2216 lines
wait delay : 0
DPHYTX_CTRL : 24
MAX_EXPOSURE_LINES : 232
LOW_POWER_MODE : 0
FRAMERATE : 531
in 0.10777831077575684 we got 57 frames
528.8633639711982 FPS
Processing time for 10 frames: 0.10777831077575684
FPS : 92.78304631073652

Process finished with exit code -1073740940 (0xC0000374)

Best Regards,

Maciej Lewandowski