STWINBX1 and HSDATALOG = Corrupt Data when converting to CSV

[WP]

I am running a STWINBX1 with the latest v1.2.0 firm from the STM32CubeFunctionPack_DATALOG2_V1.2.0.

I have compiled and uploaded the DATALOG2-STWIN.Box firmware successfully.  I am using the ST BLE Sensor App v5.0.0 on my iPhone to configure and run the "High Speed Data Log 2" option.  I configure the sensors (using the accelerometers only) and start and stop logging successfully.  

All good, but now I go to covert the .dat files to CSV files for processing using:

python hsdatalog_data_export.py d:\20230729_10_57_37 -f CSV -d

I am flooded with multiple lines like this in the terminal:

2023-07-29 11:00:52,572 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at -3.2819972584109675e+296 sec (HSDatalog_v2.py:564)
2023-07-29 11:00:52,573 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at -3.2819972584109675e+296 sec (HSDatalog_v2.py:564)

Or for another sensor

2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,672 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)
2023-07-29 10:54:47,673 - HSDatalogApp.st_hsdatalog.HSD.HSDatalog_v2 - WARNING - Sensor iis2dlpc_acc: corrupted data at nan sec (HSDatalog_v2.py:564)

Of course the data in the CSV file is corrupt, but mostly it looks like the Time column is the issue......the accelerometer data seems reasonable.

I am at a loss how the time stamp data can be so messed up.  I didn't have this issue with the previous version (v1.1) of the software but can't prove it.

Q:  Any ideas how to troubleshoot this?  No error messages, no debug data to diagnose.

Thank You

[RhSilicon]

Interesting to know that this data collector exists. I made a data collection project (Datalogger 2039) that also output a dat file and I also had to make a dat file to csv converter (also output excel file directly).

The issue of converting the dat file shouldn't be too complicated to get right if the file doesn't use encryption.

It may be possible to get help on the stackoverflow forums.

[SimonePradolini]

Hello @WP ,

can you share the entire acquisition folder or at least only the json files and the iis2dlpc_acc.dat?

By the log it seems that the acquisition is broken. Logging all the sensors at the maximum rate via SD card can fail if the SD card is not a class 10 one (we suggested some specific part number inside the User Manual). But we are not expecting this if you enabled only the IIS2DLPC .

Have you already done different acquisition with the same setup/configuration, by changing SD card or via USB?

Is the issue persisting also by enabling other sensors? Only IIS2DLPC is failing?

Best regards

Simone

[WP]

Here is the entire folder that will not convert without error as an example (and I put a copy of the corrupt CSV for reference).  I'll try with another SD card as I can't say I've tried that.  I did get one successful datalog and conversion done.  I didn't use the iPhone add, rather just used the USR button to start and stop the recording and it converted successfully.  Only tried that once so far so too early to say it always work or whether I just got lucky.

Appreciate the help!

[SimonePradolini]

Hello @WP ,

Sorry for the late reply. I've just analyzed your data: a timestamp is missing, so breaking the file.

Have you already tested your setup with another SD card and with different sensors configurations? I'm using your device_config.json with one of the suggested SD card and I'm generating working logs. The same configuration is working as expected also with acquisition in Python via USB.

Best regards

Simone

[SimonePradolini]

@WP

Thanks to your feedback we found a bug in the firmware. We've just pushed the patch release v1.2.1 on github here.

Full support for v1.2.1 both on st.com and on BLESensor app (both Android and iOS) will be available in few days. Firmware update will be automatically pushed directly in BLESensor app. So, after the upgrade, your issue should be solved.

Please let us know if now your setup is ok. Feel free to contact us for any further feedback

[WP]

Sorry for the slow reply as I've been away for a few days. 

Thank you very much for the help and I will wait for the BLESensor update in the coming days and report back.

Greatly Appreciated

[WP]

Only observation to share quickly is that it seems to work (record and convert to CSV) pressing the USR button on the STWINBX1 to start and stop, however now the folders are named DL2_00001, DL2_00002, etc instead of the nice date/time stamped versions before (eg: 20200811_09_09_22).  I will keep looking but all I've done so far is download v1.2.1 and build the DATALOG2 application for the STWIN.Box.

[nicolas_tudarmstadt]

Hello,

I think I am having a similar issue. I am not using the app and SD card but I am recording the data via USB. The timestamps also get corrupted, especially in measurements that are longer than a few minutes, with the same errors / warnings that you got when trying to convert to .csv. I observe this behavior even in the HS GUI application sometimes. I am using the 1.2.1 version. Any help is much appreciated.

[WP]

I will give the USB option a try as I haven't used it for long durations....most of my testing has been relatively short.

Another question is how many sensors and at what rate are they recording?  For comparison most of my testing have been with only 2 sensors running at max 800Hz.  Just looking to compare/contrast to find out what's going on.