Error when extracting data .dat files : "Missing packets revealed, data corrupted."

[anthonyADV]

Hello, 

I logged data using the STEVAL-STWINKT1 (flashed with DATALOG2_Release.bin) by connecting the sensor to the iOS app ST BLE Sensor.

Then, I used the Python notebook "nb_stdatalog_converters.ipynb" to convert the .dat files to .csv.

However, for some sensors, I can't retrieve the dataframe, and I get the following errors:

Error: No [iis3dwb_acc] sensor available in your selected acquisition folder. Please check the sensor name and try again.

Error extracting data from C:\Users\...\iis3dwb_acc.dat file. Missing packets revealed, data corrupted.

Do you know why this problem occurs, or is the data actually corrupted?

I've noticed that the error occurs when processing large files—specifically, data from sensors ism330dhcx_acc, imp23absu_mic, and imp34dt05_mic, which exceed 129,000 KB. This might be related to the issue.

If you'd like to reproduce the problem, the code is provided below, and you can download my .dat files via this WeTransfer link : https://we.tl/t-OFuem7TOyT (let me know if you prefer another transfer method).

Thanks,

Anthony

import sys
import os

# Add the STDatalog SDK root directory to the sys.path to access the SDK packages
sys.path.append(os.path.abspath(os.path.join(os.getcwd(), '..\\..')))

from stdatalog_core.HSD.HSDatalog import HSDatalog

#acquisition_folder = "path/to/your/acquisition_folder"

hsd = HSDatalog()
hsd_version = hsd.validate_hsd_folder(acquisition_folder)
hsd_instance = hsd.create_hsd(acquisition_folder=acquisition_folder)
print(f"HSD Version: {hsd_version}\n")

sensor_names = ["imp23absu_mic", "imp34dt05_mic", # "ism330dhcx_acc" ,"ism330dhcx_gyro","iis3dwb_acc", 
                "lps22hh_press" ,"lps22hh_temp", "stts751_temp", "iis2dh_acc" ,"iis2mdc_mag"
                ]


for sensor_name in sensor_names : #
    print(sensor_name)

    try :
        sensor =  hsd.get_sensor(hsd_instance, sensor_name)
        dataframe = hsd.get_dataframe(hsd_instance, sensor)
        print(dataframe)

    except Exception as e:
        print(f"Error: No [{sensor_name}] sensor available in your selected acquisition folder. Please check the sensor name and try again.")
        print(e)

[SimonePradolini]

Hello @anthonyADV 

I'll replicate your setup locally and download the files you shared so to come back to you with more specific feedback.

Best regards,

Simone

[anthonyADV]

Hello @SimonePradolini, 

The wetransfer link has expired so in case you did not have time to download the files, here is a new one : https://we.tl/t-VxcmqAkIvG

Bests regards, 

Anthony

[SimonePradolini]

Hello @anthonyADV 

Your dataset seems valid and I could convert them also if the SDK returns the error you highlighted. Have you also tried the conversion script stdatalog_examples\cli_applications\stdatalog_data_export.py?

I'm having the same error, but the files are properly converted. Be aware that very long acquisitions will create very big data also (that is: IMP34DT05 file in txt is 3.4 GB!). The SDK also offers the possibility to cut and convert only portions of data through the --start_time and --end_time parameters. By launching any distributed example with the -h parameter, it will show you the helper with all the supported feature.

I'm debugging the SDK to find the issue and propose a solution to the team involved in the maintenance of the package.

I'll come back to you soon.

Best regards,

Simone

[SimonePradolini]

Hello @anthonyADV 

Just to confirm you that your dataset is valid and there’s no issue from the firmware side. I already found the issue on the Python SDK side and working for the solution. The error is related to the stdatalog_core wheel only. We're working on a patch that we'll release soon here.

Once you got "Error extracting data from C:\Users\...\iis3dwb_acc.dat file. Missing packets revealed, data corrupted.", are you able to see the data converted or not? I replicated your same error, but the data are correctly converted also when the error is raised. I can see the dataframe, txt or csv and I can confirm you that no data are lost.

Best regards,

Simone