Multiple VL53L0X sensor array intermittently returning bad (-1) distances

[FPayn.1]

I have 7 VL53L0X sensors on a robot platform, all connected to a Teensy 3.5 via a very short I2C daisy-chain. I have all 7 configured for continuous measurement with a timing budget of 20mSec. The Teensy loops through all 7 sensors, retrieving distance data as fast as possible. The Teensy code that does this is shown here: (I couldn't put it in-line due to length constraints, but a representative portion is shown below)

In setup():
 
 
  //from Pololu forum post
  lidar_LR.setMeasurementTimingBudget(20000);
  lidar_LR.startContinuous();
 
  lidar_LR.setAddress(DEFAULT_VL53L0X_ADDR + 6);
  Serial.printf("lidar_LR address is 0x%x\n", lidar_LR.getAddress());
 
In Loop():
  LR_Dist_mm = lidar_LR.readRangeContinuousMillimeters();
 

When the Teensy code runs, I get the following output, which indicates that all 7 sensors are capable of reporting distance information very rapidly, as the loop repeat interval is on the order of 15-20 mSec.

Port closed
Uploading to I/O board
Opening port
Port open
lidar_RF address is 0x2a
lidar_RC address is 0x2b
lidar_RR address is 0x2c
lidar_LF address is 0x2d
lidar_LC address is 0x2e
lidar_LR address is 0x2f
lidar_Rear address is 0x30
Msec	LFront	LCtr	LRear	RFront	RCtr	RRear	LSteer	RSteer	Rear
5847	527	8191	598	312	391	365	-0.71	-0.53	951
5865	518	467	498	319	373	370	0.20	-0.51	903
5884	509	484	8191	318	381	356	-76.82	-0.38	942
5902	526	556	699	321	380	360	-1.73	-0.39	940
5920	526	595	589	325	390	360	-0.63	-0.35	956

The Teensy is a slave device up on a separate I2C bus managed by the main robot controller. The main controller requests distance information every 100mSec via a call to the Teensy's requestEvent() function. This function simply transmits the distance readings over the I2C bus to the main controller, as follows:

    //left side  
    I2C_writeAnything(LF_Dist_mm);
    I2C_writeAnything(LC_Dist_mm);
    I2C_writeAnything(LR_Dist_mm);
    I2C_writeAnything(LeftSteeringVal);
 
    //right side
    I2C_writeAnything(RF_Dist_mm);
    I2C_writeAnything(RC_Dist_mm);
    I2C_writeAnything(RR_Dist_mm);
    I2C_writeAnything(RightSteeringVal);
 
    //10/31/20 now also reporting rear sensor value
    I2C_writeAnything(Rear_Dist_mm);

When the main controller first starts up, it waits for VL53L0X data from the Teensy 3.5 to become valid, showing this:

Checking for Teensy 3.5 VL53L0X Controller at I2C addr 0x20
1375: VL53L0X Teensy Not Avail...
1461: VL53l0x - -1, -1, -1, -1, -1, -1, -1
1478: VL53L0X Teensy Not Avail...
1559: VL53l0x - -1, -1, -1, -1, -1, -1, -1
1579: VL53L0X Teensy Not Avail...

and then this:

Teensy setup() finished at 6059 mSec
Msec	LFront	LCtr	LRear	RFront	RCtr	RRear	Rear
6061: 559	540	541	313	393	354	934
VL53L0X Teensy Ready at 6063

During operation, the main controller requests distance information every 100 mSec via an interrupt timer. VL53L0X data is returned looking like this:

Msec	LF	LC	LR	RF	RC	RR
11324: VL53l0x - 513, 550, 553, 316, 392, 358, 912
11421: VL53l0x - 527, 587, 8191, 321, 386, 358, 900
11520: VL53l0x - 533, 585, 565, 314, 376, 362, 945
11618: VL53l0x - 542, 531, 595, 309, 370, 357, 950
11717: VL53l0x - 500, 588, 602, 319, 389, 369, 952
11814: VL53l0x - 522, 637, 642, 314, 379, 355, 926
11912: VL53l0x - 555, 8191, 552, 320, 388, 347, 906

However, under circumstances (and I'm not really sure yet what those are), I start getting bad data from the VL53L0X array. When this happens, one or more (usually all) of the VL53L0X sensors starts reporting '-1' for distance, like the following:

11324: VL53l0x - -1, -1, -1, -1, -1, -1, -1

Once the Teensy starts reporting bad data it usually stays bad forever (or until I reboot the robot). I've been tearing my hair (what's left, anyway) out over this, trying to figure out what is causing this, but haven't made much progress.

The only thing I have been able to come up with is that the I2C requests to the Teensy (via the 100 mSec request interval timer noted above) are interrupting the Teensy measurement loop at some point in the measurement cycle that intermittently causes that sensor (and all the following sensors) to report bad distance data, but I have no idea what that might be.

So, my question is - under what circumstances would a valid sensor measurement request via a 'lidar_RF.readRangeContinuousMillimeters()' call return -1 instead of the actual measurement data, assuming the sensor has been properly initialized and has been running fine for an essentially infinite amount of time.

Any thoughts/clues would be appreciated

TIA,

Frank

[FPayn.1]

As an experiment, I tried powering one 3-element array from a Teensy digital output, so I could turn the power ON & OFF programmatically. When the power line is set to LOW, the modules are stopped, preventing them from responding to I2C requests. When I later restart the array by setting the line to HIGH, then the modules respond at 0x29, and can be reset/re-initialized. However, this isn't a realistic scenario for power loss, as any power loss scenario involves the power line being momentarily disconnected - i.e. open-circuit. As noted above, an open-circuited power line doesn't stop the modules - they get sufficient power from the I2C lines to continue operating at 0x29, but not enough to properly re-initialize.

It seems to me at the moment that the real killer in all of this is that the initialization code will hang permanently unless module power has been restored before the init code runs.

It's kind of a catch-22 situation now; I can't reliably re-init the modules without waiting until I'm sure power has been restored, but there isn't any way to know when that happens - at least no way available via the VL53L0X API that I can find.

Frank