Zstd, short for Zstandard, is a new lossless compression algorithm, which provides both good compression ratio and speed.
"Standard" translates into everyday situations which neither look for highest possible ratio (which LZMA and ZPAQ cover) nor extreme speeds (which LZ4 covers).
"Standard" translates into everyday situations which neither look for highest possible ratio (which LZMA and ZPAQ cover) nor extreme speeds (which LZ4 covers).
A reference source code is provided as a BSD-license package, hosted on Github which also host some up to date compression benchmark.
You can download latest release here.
If you prefer a Graphical Interface, Tino Reichardt hosts a version of 7-zip with Zstd codec included.
Zstd entropy stage is provided by FSE (Finite State Entropy).
If you prefer a Graphical Interface, Tino Reichardt hosts a version of 7-zip with Zstd codec included.
Zstd entropy stage is provided by FSE (Finite State Entropy).
How does this differ from Zhuff? Zhuff also uses FSE for the entropy stage, but uses LZ4 for deduplication, and seems to be slightly faster at decompression while slightly slower at compression.
ReplyDeleteWill you have any time to write a blog post on what is different/new with zstd compared to the combination of LZ4+FSE ?
It would take a lot of space to explain the differences in detail, but to summarize, Zstd is an evolved version of Zhuff. It should surpass it on every point when it's completed, which is not the case yet.
DeleteThanks Yann. I don't think an extremely detailed explanation is as important as a quick overview. A conceptual overview would make it easier to understand the code.
DeletePerhaps such a thing should live with the code on github rather than be a separate blog post.
Can you please benchmark this against brotli? http://google-opensource.blogspot.com/2015/09/introducing-brotli-new-compression.html
ReplyDeleteHi Zeev :)
Deleteyes sure.
I'll update the benchmark to compare with Brotli.
( Note that my dev station has changed, and so has zstd since this publication, so you'll find newer figures at http://www.zstd.net. )
In the meantime, you may also have a look at Evan Nemerson's benchmark, at : https://quixdb.github.io/squash-benchmark/
where Brotli is compared to a bunch of other compressors.
In general, Zstd is faster, and Brotli compress more. And given the selected design for both, I guess it will stay this way in the foreseeable future.
Can you please throw an post to explain why the zstd is better than zlib in the speed of compressing and compress Ratio? I havn't found any about zstd algorithm;
ReplyDeletethanks in advance;
I suspect the graphs on ZSTD project page :
Deletehttp://www.zstd.net/
would likely help answer this question.
I have a file which zstd -4 compressed worse than zstd -3 (compressed file is larger). Is this expected to occur with some probability or is this a bug?
ReplyDeleteHi Zeev
DeleteThis *could* happen.
Compression levels hide a fairly complex machinery with lots of parameters. Levels simplify that, by selecting into precomputed rows.
In general, higher compression level should be slower and stronger. But there are exceptions, depending on files, and depending on computing station too.
Level 4 is one of the worst compression levels, being at the edge between 2 different strategies, so none really fit in. It will likely be modified in the future.
Regards
images seem dead
ReplyDeleteYou're right, this is fixed, thanks !
Deleteand, dead again, in this blog at least :-D
Deletecan i plz explain how this algorithm works
ReplyDelete