High-Speed algorithms added to Pcompress

Pcompress is currently in a first 0.5 Alpha release with a bugfix 0.6 coming up soon.

I have added a couple of very fast compression algorithms into the mix in Pcompress. The first and probably the fastest one is LZFX (http://code.google.com/p/lzfx/). It is based on LZF with some optimizations. The original implementation had fixed size hash on the stack defined at compiled time. I made it dynamic to support 5 compression levels with increasing hash size.

The other one is LZ4 (http://code.google.com/p/lz4/) which is finding mention in many forums. LZ4 also has compile-time defined parameters which I have not yet gone about tweaking. However I have still added 3 compression levels based on which LZ4 routines are being called. LZ4 provides 2 compression variants. A very fast LZ4_compress() and a slower high-compression LZ4_compressHC(). In pcompress at present level one is plain LZ4_compress(). Level 2 is a multi-pass version where LZ4_compress() and LZ4_compressHC() are called in sequence to compress twice. Interestingly enough this yields almost 90% of the compression ratio achieved by LZ4_compressHC() at a fraction of the time taken (see also http://fastcompression.blogspot.in/2012/07/log-file-compression.html). Level 3 is of course LZ4_compressHC() alone.

While reading http://fastcompression.blogspot.in/ I noticed these two posts: http://fastcompression.blogspot.in/2012/05/members-properties.html and http://fastcompression.blogspot.in/2012/05/useful-compressed-streaming-properties.html. Very interesting observations and Pcompress in fact implements majority of those features. One very useful feature, not yet implemented in Pcompress, is the concept of Authorized Members which allows a compressed file chunk to be something other than actual compressed data. This for example can be used to hold things like digital signatures and certificates when implementing security features.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s