VL53L1: What is the data format of histogram_ranging_gain_factor?

[MVand.3]

UM2133 says the default value of histogram_ranging_gain_factor should be 0.971. When I get the data for VL53L1_gain_calibration_data_t->histogram_ranging_gain_factor, I get an integer "1987". The data type listed in the typedef in vl53l1_ll_def.h is uint16_t. Is this some fixed point encoding? I don't see any comments in the code giving clues.

All I have is the user manual (UM2133). I haven't found a full API documentation of all the functions and their outputs and data types. Does this documentation exist somewhere?

[John E KVAM]

The VL53L1 sensor uploads the histogram data to the host, and all the processing of the the data is done there. So what is in the code is what you get. But that code is pretty obtuse.

The value is defined by:

#define VL53L1_TUNINGPARM_HIST_GAIN_FACTOR_DEFAULT \

((uint16_t) 1987)

in the file L1_tuning_params.h

And so you are right.

As we don't know what kind of MCU our customers are using we try to stay away from floating point math as much as possible.

I see the lines:

range_mm *= gain_factor;

range_mm += 0x0400;

range_mm /= 0x0800;

and that looks to me like a multiply of 0x7c3 and a divide by 0x800 - the adding was to do a round.

and indeed the 1987/2048 = 0.970

To change it, take your new factor and multiply it by 2048 to get the integer.

[MVand.3 · Accepted solution]

Thanks John, This scaling factor of 2048 works!

I am considering using a large gain factor like 6.5 (UM2133 says it must be <7) to increase the granularity of the distance measurement. The normal data output is to the nearest mm but if I increase the gain to 6.5 and then divide the results by 6.5 in the host, I'm able to get 1mm/6.5 granularity. Do you see any problem with this approach?

For my application, I only need to detect objects up to a max distance of about 300mm. I'm using multiple ROIs and comparing the results. I'm looking to get the highest accuracy possible at close range (80-250mm). Thanks.

Here's the code snippet I'm using to get and modify histogram_ranging_gain_factor in case other users may find it helpful:

  VL53L1_CalibrationData_t CalibrationData;
  VL53L1_CalibrationData_t *pCalibrationData = &CalibrationData;
  status = VL53L1_GetCalibrationData(Dev, pCalibrationData);
  if(status){
      printf("VL53L1_GetCalibrationData failed: error = %d\r\n", status);
      while(1);
    }
 
  printf("original histogram_ranging_gain_factor: %f\r\n", 
        pCalibrationData->gain_cal.histogram_ranging_gain_factor/2048.0); 
 
  float gain = .970215; //default is 0.970215
  pCalibrationData->gain_cal.histogram_ranging_gain_factor = gain * 2048;
  printf("new histogram_ranging_gain_factor: %f\r\n", 
        pCalibrationData->gain_cal.histogram_ranging_gain_factor/2048.0);