STM32 FAST DATA COMPRESSION AND LOW MEMORY FOOTPRINT

[irmakci]

Posted on January 24, 2013 at 17:03

Hello,

I need to design a system which will compress a data with rate of 1000:1 minimum in the range of  determined time.

I want to compress data (tera bytes to gigabytes) and to save to SD card.

I searched about open source data compression algorithms from internet... 

I think ZLIB suits for my application but i am not sure that it suits for my STM32f107 microcontroller with respect to memory footprint .

Are there any open source data compression libraries for STM32 series ? 

Thank you very much in advance

Best regards

#keep-dreaming #zlib #data-compression

[Tesla DeLorean]

Posted on January 24, 2013 at 17:20

Yeah, you can get 1000:1 out of ZLIB, I don't think so.

If your store vast amounts of repeating data that could be decimated like that, you'd presumably be able code better to identify those conditions when you generated it.

Figure out how LZ compression works, and how large a window you can accommodate, it will likely be bounded by search time (ie Fastness) rather than available memory.

ASCII data compresses at about 2:1 in streams (in single pass, one shot), binary data perhaps 1.5:1. Compressed/Encrypted/Random data tends to get bigger.

[irmakci]

Posted on January 24, 2013 at 22:36

Sorry but this means that i can use zlib or other libraries for my application with this microcontroller?....

 for example i want to compress 1 terabyte data to 1 gigabyte with  1000:1 compression rate? do you suggest a library for this ? for example 

i will measure 2 byte(16 bit data ) resolution data with 50 us sampling rate, and i will save this data as block in my memory then compress to sd card with 1000:1 compression rate.... 

Best regards

[Tesla DeLorean]

Posted on January 24, 2013 at 23:19

Why wouldn't you delta encode the samples? Or RLE? You need to look at the nature of the data, and where the natural redundancy is. If you have a data stream that has 1000:1 decimation prospects you need to seriously analyze your representation method.

ZLIB takes ~54K on an x86, I'd imagine the code foot print in ARM/THUMB would be similar. ZLIB however is designed for general binary data, not analogue samples. I've seen ZLIB and inflate/deflate algorithms used in firmware for years.