tag:blogger.com,1999:blog-834134852788085492.post5250784155125383722..comments2024-03-02T07:59:30.808+01:00Comments on RealTime Data Compression: When to use Dictionary CompressionCyanhttp://www.blogger.com/profile/02905407922640810117noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-834134852788085492.post-69383229457486137142020-07-28T01:17:33.723+02:002020-07-28T01:17:33.723+02:00> I imagine each compressed message (block) wo...> I imagine each compressed message (block) would have to have a dictionary uuid stamped into it,<br /><br />That's exactly how it works today<br /><br />> the current implementation assumes there is only one dictionary loaded/used for the compression/decompression of a given message.<br /><br />Indeed, for a given "frame", only one dictionary can be used.Cyanhttps://www.blogger.com/profile/02905407922640810117noreply@blogger.comtag:blogger.com,1999:blog-834134852788085492.post-4756841602927044562020-07-28T00:06:27.914+02:002020-07-28T00:06:27.914+02:00Took a look at the command line interface to the d...Took a look at the command line interface to the dictionary feature, and I infer the following: <br /><br />Seems to me that the dictionary contains some "probable" string literals, so it is for the modeling phase of compression. It looks like it is oriented towards file (stateful) compression as it is now envisioned. I imagine that an implementation would have to have some dictionary repository and at least a file naming convention, to make available the common dictionaries. An implementation relying on custom generated dictionaries would have to transmit these along with the compressed message(s) if the recipient is to have any hope of decompressing. Managing the dictionaries for stateless compression would be a bit of a challenge, I imagine each compressed message (block) would have to have a dictionary uuid stamped into it, so the decompress code can select the correct (probably cached) dictionary at decompress time. To make custom dictionaries useful between independent entities would be a bit of a challenge. there would have to be an agreed upon protocol to distribute these along with the compressed data. (I there a standard method envisioned for dictionary identification/distribution? I will go read the rfc see if there is such thing already proposed). <br />Also: The shifting of the storage of some of the unique substring literals from the message to the dictionary has to be a "best effort". In other words, the dictionary procedure is inherently opportunistic. Unique substring literals would still have to be stored in the compressed message, but some of these substring could be replaced by dictionary references, if they are already present in the dictionary that is in use. I imagine the current implementation assumes there is only one dictionary loaded/used for the compression/decompression of a given message. <br /><br />Please correct me if I got this wrong. <br />Anonymoushttps://www.blogger.com/profile/03169520210922766280noreply@blogger.comtag:blogger.com,1999:blog-834134852788085492.post-16560831102336343042020-07-27T23:40:29.479+02:002020-07-27T23:40:29.479+02:001) I'm not sure about what is meant by separat...1) I'm not sure about what is meant by separating modeling and encoding<br />2) Yes, dictionaries contain anticipated strings<br />3-4) The whole mechanism of shipping, sending, synchronizing dictionaries is implementation dependent. There are a lot of possibilities here, with distinct trade-offs. For a more detailed presentation of one possible solution, I recommend reading the following white paper, chapter "Managed Compression" : <br />https://engineering.fb.com/core-data/zstandard/<br />5) Dictionaries make a lot of sense when trying to remove stateful compression, since they replace point-to-point dedicated states with generic states<br />6) Dictionaries are especially useful for small blocks<br />Cyanhttps://www.blogger.com/profile/02905407922640810117noreply@blogger.comtag:blogger.com,1999:blog-834134852788085492.post-63721458115854731422020-07-27T23:30:45.299+02:002020-07-27T23:30:45.299+02:00I would like to understand a few basic things abou...I would like to understand a few basic things about dictionary based zstd compression:<br />1) Are these dictionaries used for modeling or encoding?<br />2) If they help with modeling, do they contain actual/anticipated unique sub-strings? (Did <br /> we move the unique string literals from the compressed message into a database?)<br />3) Where are these dictionaries stored? Esp. custom generated dictionaries? <br />4) How are the dictionaries made available to decompression? Example: <br /> We compressed a message using a custom dictionary on one system. Trying to decompress said <br /> message on a different system. Where is the dictionary coming from? Is it attached to the <br /> message? <br />5) Do these dictionaries make much sense for stateless compression?<br />6) Do dictionaries makes sense for small_block e.g.: 4 or 8k stateless compression?Anonymoushttps://www.blogger.com/profile/03169520210922766280noreply@blogger.comtag:blogger.com,1999:blog-834134852788085492.post-26191192688082216752018-02-21T15:43:14.068+01:002018-02-21T15:43:14.068+01:00> Could that be a sign that choosing different ...> Could that be a sign that choosing different compression settings (worse compression ratio but less cpu usage) could further improve performance in this scenario?<br /><br />All tests have been performed at compression level 1.<br />It's the fastest compression setting available.<br /><br />Using higher compression levels would improve compression ratio, and generally as a consequence, slightly improve decompression speed. So all graphs would improve.<br /><br />In general, if your workload is io-read-limited, and if you have margin on the write side, it's better to increase compression level, to improve compression ratio _and_ read speed.Cyanhttps://www.blogger.com/profile/02905407922640810117noreply@blogger.comtag:blogger.com,1999:blog-834134852788085492.post-26558733695016185282018-02-21T15:07:25.435+01:002018-02-21T15:07:25.435+01:00> Plot: Block Size vs Random Reads / s
Thanks ...> Plot: Block Size vs Random Reads / s<br /><br />Thanks for pointing that out David.<br />After verification, the issue is, I over-estimated the average size of individual records (which span in the [250-550] range). It does not matter when comparing speed at different block sizes, but it does matter when comparing blocks with single-record compression.<br /><br />After fixing this value, I'm getting an almost flat ending : it's barely faster to read single records than to read a (small) block of multiple records and throw away the useless ones. But it's nonetheless *slightly* faster.<br />Which feels more logical.<br /><br />I'll fix the graphs with the new values.Cyanhttps://www.blogger.com/profile/02905407922640810117noreply@blogger.comtag:blogger.com,1999:blog-834134852788085492.post-10550969563150024962018-02-21T08:52:13.745+01:002018-02-21T08:52:13.745+01:00> Plot: Block Size vs Random Reads / s
Do I un...> Plot: Block Size vs Random Reads / s<br /><br />Do I understand correctly that the dip on the right side of the blue (non-dictionary) line is because I/O is becoming the bottleneck? In other words compression ratio is getting so bad that reading and decompressing individually compressed records is more expensive than reading a compressed block of multiple records, (partially) decompressing it, and throwing away most of the data. For me that's a very remarkable result.<br /><br />> Plot: Compression Ratio vs Random Reads / s<br /><br />I feel like this would be easier to understand if you had switched the axis. Doing that in my head, the blue line has a 'peak' where both lower and higher compression ratio provides worse results. The orange line does not have such a peak, the optimal point is the lowest compression (individual records). Could that be a sign that choosing different compression settings (worse compression ratio but less cpu usage) could further improve performance in this scenario?Anonymoushttps://www.blogger.com/profile/04007626658363210501noreply@blogger.comtag:blogger.com,1999:blog-834134852788085492.post-89283798192556327982018-02-18T19:39:47.161+01:002018-02-18T19:39:47.161+01:00Thanks for the link, it's a great read !Thanks for the link, it's a great read !Cyanhttps://www.blogger.com/profile/02905407922640810117noreply@blogger.comtag:blogger.com,1999:blog-834134852788085492.post-34813013753254881772018-02-18T15:40:19.015+01:002018-02-18T15:40:19.015+01:00Thanks for writing about Dictionary compression. W...Thanks for writing about Dictionary compression. We recently used it to store data efficiently in RAM, and wrote about it here - https://clevertap.com/blog/clevertap-engineering-behavioral-messaging-at-scale/Anand Jainhttps://anandjain.wordpress.comnoreply@blogger.com