This project is read-only.

What is the competition?

For testing I used other algorithms available for .NET. They needed to have a .NET implementation or a .NET wrapper. There are three categories they can compete in:
  • native: The algorithm uses native code, so it's not a real .NET assembly (includes P/Invoke and Mixed Mode)
  • unsafe: The algorithm has been implemented in .NET but uses unsafe code (C++/CLI technically should be in this category but I ignored it)
  • safe: The algorithm has been implemented in .NET and uses safe code only.


Testing

I used the same method as in Performance Testing. This I didn't spend that much time and I ran the test for 3 hours only.
Please note: I used Silesia Corpus so results are averages for compressing different kind of data (binary, database dumps, text data, XML, images, incompressible) in some ratios. If you want to check how algorithms perform on your data - use your data. The results might be different.


Pareto frontier

For 'real' definition please refer to Pareto frontier.
Simply speaking Pareto frontier for compression algorithms is a set of algorithms which cannot be beaten by other algorithms.
Take a look at this picture:
pareto.png
  • Algoritm B beats algorithm D because D offers the same compression ratio (56%) but is much slower (250MB/s vs 100MB/s).
  • Algorithms B also beats algorithm E because E work as fast as B (both 250MB/s) but offers worse compression ratio (57% vs 56%).
  • Obviously B also beats algorithm F because F is not only slower but also offers worse compression ratio.
  • Algorithm B does not beat algorithm A. Although A is slower it offers better compression ratio.
  • Algorithm B does not beat algorithm C. Although C offers worse compression ration it is faster.
  • Algorithms A, B and C are on Pareto frontier - meaning they are most effective in their category.


Native

The competitors are:
  • LZ4 - Mixed mode, this one
  • Snappy - P/Invoke wrapper, can be found here
  • LZO - actually multiple versions of LZO: LZO1X, LZO1X11, LZO1X12, LZO1X15; P/Invoke wrapper, can be found here; Note: this wrapper does not have 64-bit assembly which means it can be used only in 32-bit applications (it's becomes a little bit 'no-no')
  • QuickLZ - P/Invoke wrapper; can be found here; Note: there two implementations on this page - one is safe implementation and the other one is a wrapper

Compression

compare-other-encoding-64.png
Pareto frontier: QuickLZ, LZ4

Decompression

compare-other-decoding-64.png
Pareto frontier: QuickLZ, LZO1X, LZ4
Note: LZO1X15 is in fact on Pareto frontier but as you can see it's speed advantage over LZO1X and compression ratio advantage over LZ4 is negligible.


Safe

The competitors are:
  • LZ4 - Safe, this one
  • QuickLZ - Safe, can be found here
  • LZF - Safe, can be be found here
  • Deflate - standard .NET defalte algorithm; not really 'fast compression algorithm' and I'm not sure if it's implementation is safe, but because it is available from safe code it belong here

Compression

safe-compare-other-encoding-64.png
Pareto frontier: Deflate (because of compression ratio, by no means because it's speed), QuickLZ (absolutely great performance here)
Note: LZ4 has been overshadowed by QuickLZ, maybe not by a mile but still. No Pareto frontier award. I'll have to take a look at safe compression implementation again, maybe something can be done

Decompression

safe-compare-other-decoding-64.png
Pareto frontier: Deflate (again, compression ratio only), QuickLZ (good compression ratio, decent speed), LZ4 (not so much worse compression ratio and great speed)


Unsafe

I couldn't find any unsafe compression implementations. They all are native (P/Invoke) or Safe.
So the only competitor is different implementation of LZ4.
  • LZ4 - unsafe, this one
  • LZ4Sharp - unsafe, can be found here

Compression

unsafe-compare-other-encoding-64.png
Note: My implementation of LZ4 is a little bit better. I used a little bit more mature C version as a template and I spent some time on tuning it. When you a take a closer look you'll notice that differences are quite small, though.

Decompression

unsafe-compare-other-decoding-64.png
Note: My implementation of LZ4 is a little bit better. I used a little bit more mature C version as a template and I spent some time on tuning it. When you a take a closer look you'll notice that differences are quite small, though.


Last edited Feb 6, 2013 at 3:35 PM by Krashan, version 26

Comments

No comments yet.