Clarification on VD16GZ VIS and NIR Imaging Modes

[Hunghc2]

Dear Technical Support Team,

I would like to clarify how the VD16GZ sensor handles VIS and NIR imaging.

Does the VD16GZ output a single RAW image containing both VIS and NIR information, which must then be separated through software processing, or does it support two independent output modes, VIS-only and NIR-only?

More specifically, I would appreciate your clarification on the following questions:

1. Is it possible to obtain two separate images, one RGB (VIS) and one monochrome (NIR), from the same captured frame?

2. Does the sensor support switching between VIS-only and NIR-only output modes through software commands, without restarting the camera?

3. Is additional image processing required on the host processor, such as NVIDIA Jetson AGX Orin, to generate separate VIS and NIR images?

4. Does STMicroelectronics provide an SDK or reference algorithm for separating VIS and NIR images?

Thank you for your support. I look forward to your clarification.

[michaeltang]

Hello Hunghc2,

Thank you for your interest in ST products and welcome to the ST Community.

The VD16GZ uses a 4 × 4 RGB-NIR pixel pattern, with visible-light pixels and NIR pixels interleaved in the same array. In RAW output mode, a captured frame therefore contains both VIS and NIR samples. Because this differs from a conventional Bayer pattern, a standard Bayer demosaicing algorithm is not suitable.

Please find our answers to your questions below:

1. Yes. The RGB and NIR samples can be extracted from the same RAW frame and processed into separate RGB and monochrome NIR images. Image reconstruction must account for the VD16GZ’s specific 4 × 4 pattern; the resulting images are not two independently captured, full-resolution images.

2. Our recommended approach is to keep the sensor in RAW output mode and select the required image in software: RGB, NIR, or both. This allows you to change which image your application uses without changing the sensor’s acquisition mode or restarting the camera. This is host-side selection, rather than switching between dedicated VIS-only and NIR-only sensor output modes.

3. Yes. For the recommended RAW approach, the host must separate the RGB and NIR samples and perform the appropriate image reconstruction. If only NIR is required, a subsampling approach can be used to extract the NIR pixel samples, taking their positions in the 4 × 4 pattern into account.

4. ST provides Python example code in the STSW-IMG507 software package, which you can try as a starting point for processing the RGB-NIR RAW data:
   https://www.st.com/en/embedded-software/stsw-img507.html

We recommend starting with RAW capture and the Python example to evaluate the separation process before integrating it into your Jetson application.

Best regards,

Michael

[Hunghc2]

Dear STMicroelectronics Technical Support Team,

Thank you for your previous response and for providing the technical documentation.

As I understand, the STEVAL-16GZMAI1 camera module uses the VD16GZ RGB-NIR image sensor. I would appreciate further clarification on several technical aspects to determine whether this camera meets our application requirements.

1. VIS and NIR Image Output

Could you clarify the actual image output format of the STEVAL-16GZMAI1?

• Does the camera output a single RAW image containing both VIS and NIR information, or does it already support separating these components into two independent VIS and NIR images?

• Is RGB-NIR separation performed internally by the camera, or does it require additional processing on the host processor?

• Is it possible to switch between VIS-only and NIR-only display modes through software without restarting or reconfiguring the sensor?

• Does ST provide any SDK, ISP pipeline, or reference algorithm for RGB-NIR separation?

2. Frame Rate, ROI, and Image Acquisition Latency

Our application has strict requirements regarding image acquisition latency and frame rate.

Specifically, we require a frame rate of at least 300 FPS at a resolution of 320 × 320 pixels.

• Does the STEVAL-16GZMAI1 support this configuration?

• What is the maximum achievable FPS at 320 × 320 resolution?

• Does the sensor support configurable ROI and windowing to achieve higher frame rates?

• What are the minimum exposure time, sensor readout time, and frame transmission latency at this resolution?

• Is there any additional latency associated with RGB-NIR separation or internal image processing?

We would appreciate detailed timing specifications to help us evaluate the camera's suitability for low-latency, high-speed imaging applications.

3. Built-in Image Processing (DSP/ISP)

We would like to understand which image preprocessing functions are already implemented in the camera hardware.

• Does the STEVAL-16GZMAI1 include an integrated DSP or ISP?

• Are functions such as demosaicing, RGB-NIR separation, noise reduction, black-level correction, and color correction performed internally?

• If these functions are not integrated, does ST provide software libraries or reference implementations for NVIDIA Jetson platforms?

• Is it possible to access unprocessed RAW sensor data for implementing our own image processing algorithms?

4. Integration with NVIDIA Jetson AGX Orin

We intend to integrate the STEVAL-16GZMAI1 with an NVIDIA Jetson AGX Orin platform.

• Does the camera support direct connection to Jetson AGX Orin via MIPI CSI-2?

• What additional hardware, adapter boards, or cables are required?

• Is USB connectivity supported? If so, what additional hardware is needed?

• Are compatible drivers, SDKs, or reference implementations available for NVIDIA JetPack?

• What are the maximum achievable FPS and expected acquisition latency when using MIPI CSI-2 versus USB?

5. Optical Integration and Image Quality

We plan to use our own optical lens system instead of the lens supplied with the evaluation board.

• What optical specifications and compatibility requirements should be considered when integrating a custom lens with the VD16GZ sensor?

• Are there any recommended requirements regarding chief ray angle (CRA), image circle, back focal distance, and optical alignment?

• Does the RGB-NIR sensor require a specific optical filter to ensure proper separation of VIS and NIR information?

• What calibration procedures or image correction algorithms are recommended to maintain image quality when using a custom lens?

• Does ST provide any guidelines, calibration tools, or reference procedures for optimizing image quality with third-party optics?

Our primary concerns are achieving at least 300 FPS at 320 × 320 resolution, minimizing end-to-end image acquisition latency, and obtaining high-quality VIS and NIR images with our custom optical system.

We would greatly appreciate your technical guidance and any relevant documentation, particularly regarding ROI timing, image processing, optical integration, and Jetson AGX Orin compatibility.

Thank you for your continued support.

 

 

[michaeltang]

Hello Hunghc2,

Thank you for providing your application requirements.

Below are the key clarifications. For detailed sensor specifications, register settings, and operating conditions, please refer to the VD16GZ datasheet and user manual. The STEVAL-16GZMAI1 documentation should also be consulted for board-level interfaces and connections.

1. VIS/NIR output and processing

• The VD16GZ outputs a single RAW frame containing interleaved RGB and NIR samples in its 4x4 pixel pattern. It does not output two independently processed RGB and NIR images.
• RGB-NIR separation and reconstruction need to be performed by an external processor. A standard Bayer demosaicing algorithm is not suitable for this pattern.
• Your application can select which processed image to display, RGB, NIR, or both, without restarting or reconfiguring the sensor, provided that the RAW acquisition configuration remains unchanged. This is a software display selection, rather than a sensor output-mode switch.
• As mentioned in our previous reply, ST provides Python example code in STSW-IMG507
  https://www.st.com/en/embedded-software/stsw-img507.html

2. Frame rate, ROI, and latency

• The VD16GZ supports configurable ROI. Please refer to the VD16GZ datasheet and user manual for reference frame rates, exposure settings, and timing constraints.
• The frame rates listed in the documentation apply to specific operating conditions and should be used as reference values. The achievable frame rate for your intended configuration must be evaluated against the documented settings and constraints. For example, in the datasheet we state that 640x480 can achieve 210fps and 320x240 can achieve 360fps. These reference values do not confirm that your complete setup will achieve 300 FPS at 320x320.
• End-to-end acquisition latency also depends on the capture interface, buffering, and host-side processing, including RGB-NIR separation. It therefore needs to be measured on your target system and cannot be determined from the sensor frame rate alone.

3. Built-in ISP and RAW access

• Please refer to the datasheet and user manual for the sensor’s built-in processing functions and their configuration. It doesn’t have RGB-NIR separation.
• RAW output is the recommended starting point for implementing your own RGB-NIR processing. STSW-IMG507 provides reference Python code; it is not a guarantee of a Jetson-optimized implementation or performance at your target frame rate.

4. NVIDIA Jetson AGX Orin integration

• The STEVAL-16GZMAI1 is a MIPI CSI-2 camera module, not a USB camera. Please refer to the board documentation for connector pinout and interface requirements.

5. Custom optics

• Please refer to the sensor datasheet for the available optical specifications, the STEVAL-16GZMAI has a M12 lens mount, you may need to change the mount to adapt the lens with different focal.
• Image-quality calibration depends on the final optical assembly and ISP to be used, please contact your ISP vender for further assist.

 

Please refer to the VD16GZ datasheet, user manual, and STEVAL-16GZMAI1 board documentation when assessing suitability for your application. Achievable frame rate, end-to-end latency, and image quality depends on the target use case and sensor configurations, they must be validated on your target hardware and software setup.

Best regards,

Michael