Featured

▼

Tuesday, March 1, 2011

Multi-threading compression available : LZ4 v0.9

 Previous versions of LZ4 were displaying promising results using multi-threading, but only in benchmark mode. Now this feature is made fully accessible, with real "file i/o" interface.

You will need a very fast disk drive to experiment with it. In effect, only a RAM Drive can expect to keep fast enough steam to feed LZ4, especially when using multi-threading modes.

Block subdivision method was selected, due to its design simplicity and scalability. Compared with previous versions, even single thread mode greatly benefits, since multi-threading requires asynchronous data loading, which means that reading and writing to the disk is done in parallel with compression. 

I can't really test beyond 2 threads, since i only have dual-core systems within reach, but the code should prove pretty scalable with quad-core systems and more (up to 32 threads).

You can download and test this new version on the LZ4 homepage.
at

3 comments:

  1. Roger PackJanuary 29, 2015 at 7:50 PM

    Do you have a multi-core box to test it on these days? (Though I guess you could just rent a digitalocean [or what not] box with many cores temporarily...

    ReplyDelete
    Replies
    1. Roger PackJanuary 29, 2015 at 7:58 PM

      As a note for followers, apparently this "multi threading" never actually made it into the core source [?] https://groups.google.com/forum/#!topic/lz4c/SPVAC62_6WU or clarification welcome here, thanks!

      Delete
    2. CyanFebruary 9, 2015 at 4:21 AM

      No, unfortunately, this code is Windows + Visual specific, and not even remotely portable. It's not open sourced.

      Long term plan is to produce an equivalent open source version using portable library (pthread typically). But available time is my most scarce resource these days.

      Delete