How exactly are datasets larger than 250 MB reduced during benchmarking? What sampling strategy is used?

[liu_pzh]

Hello ST Community,

 

I noticed that when using the benchmarking tool, datasets larger than 250 MB are 

automatically reduced using a "representative sampling strategy" that preserves 

the diversity and distribution of the original data.

 

I would like to understand more details about this process:

 

1. What specific sampling algorithm/strategy is used? (e.g., random sampling, 

   stratified sampling, systematic sampling, etc.)

 

2. How is the target size of 250 MB determined? Is this configurable?

 

3. How does the tool ensure that the sampled dataset preserves:

   - The original data distribution?

   - The diversity of data samples?

   - Edge cases and boundary values?

 

4. After optimization, when we validate the model on the full dataset, what 

   performance differences should we typically expect between the sampled 

   dataset and the full dataset?

 

5. Is there any documentation that describes this sampling mechanism in detail?

 

Any information or pointers to relevant documentation would be greatly appreciated.

 

Thank you!

[Julian E.]

Hi ​@liu_pzh,

 

Here are some details:

 

1. Sampling algorithm

The reduction proceeds in two stages:

• Per-file budget allocation across input files, using an iterative water-filling procedure with a hard minimum of 5 lines per file. Small files are kept in full; large files absorb the reduction. 
• Per-file line selection, chosen automatically depending on the task: 
  • Regression / target-aware tasks: 1D stratified sampling on the target's empirical distribution (quantile spacing). Min/max target values are always kept. 
  • Classification / anomaly detection: seeded uniform random sampling without replacement.

2. The 250 MB cap

Today it is not configurable. In the longer term, it will also be exposed directly in the NanoEdge AI Studio so end users can adjust it themselves. The default was chosen to bound RAM usage during parallel HyperParameter Optimization (HPO), keep per-trial wall-time reasonable, and stay above the point where accuracy plateaus for the supported model families.

3. Preservation properties

• Distribution: exactly preserved for regression (quantile sampling); preserved in expectation for classification (uniform random sampling). 
• Diversity: quantile sampling guarantees one sample per bucket; uniform sampling is unbiased across the feature space. 
• Edge cases: for regression, target min and max are guaranteed to survive. For classification, since one input file always belongs to a single class, class balance is preserved by construction via the per-file budget allocation — no class disappears. Rare/short input files are protected by the per-file minimum. 
• Determinism: everything is seeded, results are reproducible.

4. Expected difference vs. the full dataset

• Regression: gap on R² / MAE / RMSE typically < 1–2 %. 
• Balanced classification: gap < 1 %. 
• Imbalanced classification: the per-file water-filling mechanically rebalances the sample toward rare classes, so performance on the full dataset can even be equal to or better than an HPO run without capping. 
• Selected hyperparameters generally transfer to the full dataset without re-tuning.

In all internal benchmarks we have run so far, the final evaluation on the full (uncapped) dataset has consistently matched the cross-validation results obtained on the subsampled dataset during HPO.

 

 

Have a good day,

Julian

[ethonbrooks0707]

That's an interesting question. If the tool says it uses a "representative sampling strategy," I'd also like to know whether it's based on random, stratified, or diversity-aware sampling. Understanding how rare cases and class balance are preserved would help users better interpret benchmark results and trust that the reduced dataset reflects real-world performance. Hopefully, the ST team can share some implementation details or documentation.

[Julian E.]

Hi ​@ethonbrooks0707,

 

I will look if we can share more details.

 

For this statement though : “ trust that the reduced dataset reflects real-world performance.”

We highly advise using the Validation step that is made to import test dataset and check the performance of the libraries on new unseen data.

 

You can also test your initial train dataset to see the perfs.

 

Have a good day,

Julian