Saturday, June 28, 2014

Vulnerability assessment

[Edit] There is a new follow up to this story.

I've received an answer from Don Bailey. He blames the situation on a lack of communication. OK. In an attempt to bring the discussion to a more neutral level, this post is dedicated at providing a hopefully clear, concise and factual description of the situation. I hope it will help the reader to properly assess the risk level.

  • There was a vulnerability, it was fully described by Ludwig Strigeus and tracked for correction on the LZ4 issue board
  • The vulnerability is fixed. Update is recommended for 32-bits applications
  • The vulnerability requires uncommon conditions to be exploitable :
    • Input data supplied by untrusted source
    • Large block sizes, preferably ~16MB (beyond LZ4 file format specification)
    • 32-bits applications only
  • These conditions are not met by currently known programs using LZ4
    • For a list of them, get to the LZ4 homepage. It's correct to state that some yet unknown application may match these conditions. That's why the reference version has been fixed
    • Edit : One use case has been found, involving custom scripts to receive untrusted data from external sources using a Python version of LZ4 while not enforcing block size limits. The targeted Python version was using an unprotected version of the decoder. Conclusion remains the same : the Python version has been updated to cover such situation, and is a recommended upgrade
  • The Linux Kernel version, which is different and supplied by LG, has this vulnerability too
  • Linux Kernel programs (which are the only ones to access kernel functions) do not match the conditions that would make it exploitable
  • The Linux Kernel version is nonetheless being fixed too, to avoid any future risk for future versions of the kernel

Hope it helps.

Thursday, June 26, 2014

Debunking the LZ4 "20 years old bug" myth

[Edit] The below post was redacted in the heat, right after the initial publication. Should you wish a summarized, and arguably more neutral tone, view of the situation, please consider reading the follow up.

 A recent post on a security blog has made a wonderful job at sexing up a subtle bug affecting LZ4, claiming that it would result in remote code execution. Given the current spread of use of LZ4, from every modern Linux distro, including critically Android, to modern file systems such as ZFS, shipped with FreeBSD and Illumos, through widely deployed databases such as MySQL, or big data nodes powered by Hadoop, the implication is that it must be a pretty big deal.

The detailed article then makes a minimal job at describing the strong conditions required to trigger that risk, but these explanations are lost, scattered within the content of an overly long and boring technical article, ensuring most readers will stop at the dramatic headline. The present article has the objective to counterweight those "high stakes" claims.

I won't spend too much time underlining the vast difference between reporting a bug for correction to the relevant team, and creating a dramatic communication for maximum coverage without even bothering if a fix is available for the exploit. The first behavior is valuable, helpful and welcomed. The second one is at best irresponsible, if not downright harmful. A security auditor is expected to know this simple common sense, and behave accordingly.

But perhaps more importantly, this bug wasn't unknown, which contradicts the statement that it was found through careful code study. The risk was identified and openly published on the LZ4 public issue board by Ludwig Strigeus, the creator of ĀµTorrent.  Instead of trying to make a headline, he even proposed a solution for it (unfortunately with some side effects). Ludwig made a fine job at describing the risk and opening a discussion.The bug was classified minor (rightly or wrongly) for reasons which will be detailed below. Thus, it was tracked for correction, but with no urgency, trying to design a fix without the initial side effects. After multiple partial improvements, a change of code related to the new streaming mode finally allowed such a fix to be created, closing the issue (by chance, merely a few days before DonB second "disclosure").

Let's now get into the technical details.
In order to use the vulnerability, a number of conditions must be met.
A first minor one is : it is necessary to work within a 32-bits environment. 64-bits is totally unaffected. That basically put most server-side applications outside of the risk zone.

Second, the attacker need to forge a special compressed block to overflow 32-bits address space. This is possible ... if the compressed block is something like 16 MB.

There is just a little problem with that : the legacy LZ4 file format is limited to 8 MB blocks. Any value larger than that just stops the decoding process. 8MB is not enough to trigger a problem, since 32-bits application are restricted to low address ranges. The newer streaming format is even stricter, with a hard limit at 4 MB. As a consequence, it's not possible to exploit that vulnerability using the documented LZ4 file/streaming format.

Well, but what about programs which use their own, undocumented, format ?
Indeed, same condition applies. To be exposed to this risk, very large blocks of 16MB or larger must be read by the decoder.
Does that happen ?
Let's have a look at several high-profile programs using LZ4. ZFS ? Max block size is 128 KB. Lucene ? Typical index size is 16 KB. It could be a bit more, say 64 KB, that's still far short of the objective. zram maybe ? nope, 4 KB memory segments. Linux kernel ? The boot section has to decompress a multi-megabytes kernel into memory, surely it can match the limit ? Yes, it does, but it uses the LZ4 Legacy File format, which limits each block to 8 MB maximum. OK, maybe AAA games such as Guild Wars 2 ? nope, real-time protocol is the realm of short messages, the protocol itself doesn't allow anything beyond a few KB. And so on, and on.

At the end of the day, none of the known implementation of LZ4 is exposed to this risk.
Sure, someone creating some private program at home could get into this pitfall, but this situation is not exactly a "security risk". Only widely deployed programs actually matter, since they are the potential targets by hackers or surveillance organizations.
Basically, most user programs employ LZ4 for small data packet structure, way below the critical limit. Programs which generate and distribute large compressed blocks (notably the lz4c pos-x compression utility, distributed within Linux Distro) use the documented framing format, which limits block size to 4 or 8 MB. Remove also from the list programs which never take "externally provided" data as input, they can't be targeted either.

So sorry, this is not a "new heartbleed" situation the author seems to dream for.

Sure, since a fix is currently available, it's nonetheless a good move to close this risk. But I wouldn't flag that action as "critical", since most applications stand outside of the "custom compression format using large blocks of > 8 MB on 32-bits system, and receiving data from untrusted external sources" scenario.

It's one thing to tell there is a potential vulnerability that should be fixed, to ensure it does not become exploitable in the future. It's a totally different thing to pretend there is a dangerous RCE (Remote Code Execution) exploit currently active on the Internet, which is scary. DonB article cleverly conflates the two, implying the second to create a flashy headline, while disseminating some minor disclaimer elements throughout the long article to pretend, whenever necessary, having said the first. That's clever, but conflating gravity level to grab some free ad is not a respectful behavior from a security firm.

The world is already full of real security threats, from which heartbleed is just one of the most recent examples. Triggering too many alarms to grab a bit of fame is a good way to weaken the power of future alarms. You can guess that when every headline claim a "new heartbleed" situation, no one will pay attention to the real next one which will matter. And that's dangerous.

The long list of "credits" at the end of the article is another reason for caution. It happens I know some the influential names listed there, and they told me they barely heard indirectly about the guy. So why are they listed there ? Fame by association ? Sure, please thank Linus Torvald for "coordinating and advising" on the issue. This is becoming plain ridiculous.

Anyway, should you feel nonetheless at risk now, please don't hesitate to update your LZ4 version. It's a good thing to do anyway, and as stated previously, the vulnerability was already patched.