You will need a very fast disk drive to experiment with it. In effect, only a RAM Drive can expect to keep fast enough steam to feed LZ4, especially when using multi-threading modes.
Block subdivision method was selected, due to its design simplicity and scalability. Compared with previous versions, even single thread mode greatly benefits, since multi-threading requires asynchronous data loading, which means that reading and writing to the disk is done in parallel with compression.
You can download and test this new version on the LZ4 homepage.
Do you have a multi-core box to test it on these days? (Though I guess you could just rent a digitalocean [or what not] box with many cores temporarily...
ReplyDeleteAs a note for followers, apparently this "multi threading" never actually made it into the core source [?] https://groups.google.com/forum/#!topic/lz4c/SPVAC62_6WU or clarification welcome here, thanks!
DeleteNo, unfortunately, this code is Windows + Visual specific, and not even remotely portable. It's not open sourced.
DeleteLong term plan is to produce an equivalent open source version using portable library (pthread typically). But available time is my most scarce resource these days.