Skip to content

Bugs found with Address Sanitizer in syslog-ng, monit, cmake

Lately a couple of updates have fixed issues that I discovered while testing software with Address Sanitizer (ASAN). I'll publish these combined. Once again these show that a lot of software hasn't been tested with ASAN and what kind of bugs it can uncover.

In syslog-ng a function returns an error -1 that gets stored in an unsigned variable. This cannot be stored and a subsequent check for the error value fails, this later leads to a heap out of bounds memory read. This was fixed in syslog-ng 3.7.3.
Upstream bug report
Pull request / patch
syslog-ng 3.7.3 release notes

In the monit system monitoring tool a heap overflow was discovered. A call to sscanf would write a string of length 256 into a 256 byte long buffer. However this fails to consider the trailing zero byte of that string, therefore causing a one byte heap overflow. This was fixed in monit 5.16.
Commit / fix

In the compilation tool cmake a use after free error was found. This happened becaused a variable was used in a comparison and was free'd during that comparison operation. This was fixed in cmake 3.5.0.
Upstream bug report
Commit / fix

Several out of bounds reads in ProFTPD

The latest releases of ProFTPD 1.3.5a and 1.3.6rc2 fix several out of bounds read issues. I discovered these issues by running the test suite with Address Sanitizer enabled.

An invalid off by one read can happen in the function pr_fs_dircat(). This affects both 1.3.5a and 1.3.6rc1 and earlier.
Upstream bug report
Git commit / fix

An invalid off by one read can happen in the string handling function pr_ascii_ftp_to_crlf(). This code is not present in the stable 1.3.5 release series and only affects 1.3.6 release candidates before rc2.
Upstream bug report
Git commit / fix

A missing null termination of a string causes an out of bounds memory read in a test. This does not affect the ProFTPD code itself, it's just an issue in the test suite.
Upstream bug report
Git commit / fix

Independent of these memory access issues I discovered an issue in ProFTPD regarding Diffie Hellman parameters. The server will choose risky 1024 bit parameters and ignore a user-supplied parameter file.

Heap use after free in Pidgin-OTR plugin (CVE-2015-8833)

The pidgin-otr plugin version 4.0.2 fixes a heap use after free error. The bug is triggered when a user tries to authenticate a buddy and happens in the function create_smp_dialog.

The bug was discovered with Address Sanitizer. This is yet another example why all C/C++ code should be tested with Address Sanitizer enabled.

This bug was already independently discovered and reported by Stefan Sperling in the otr bug tracker.

Independend of this bug another more severe bug in Libotr itself was also disclosed today, please make sure you update both libotr (4.1.1) and the pidgin-otr plugin (4.0.2).

Upstream bug report (contains Address Sanitizer stack trace)
Commit / fix
CVE-2015-8833

Miscomputations of elliptic curve scalar multiplications in Nettle

The Nettle library is a library for basic cryptographic functions. Its most prominent user is GnuTLS.

Through fuzzing of elliptic curve scalar multiplications (multiplying a point on an elliptic curve with a scalar number) I discovered two carry propagation bugs that would lead the cauculations to produce wrong results. They affect the NIST P-256 and P-384 curves. The P-256 bug is in the C code and affects multiple architectures. The P-384 bug is in the assembly code and only affects 64 bit x86. Both bugs were found with the help of american fuzzy lop.

While analyzing these bugs Nettle developer Niels Möller discovered another carry propagation bug in P-256 that was fixed in the same commit (CVE-2015-8805). Nettle 3.2 fixes all three bugs.

The impact is currently unclear, but miscalculations in cryptographic functions should generally be considered security issues. I'd like to encourage cryptographers to try to analyze whether these bugs can lead to cryptographic breaks.

I have published a code example on how to fuzz elliptic curve multiplications. It can compare the output of OpenSSL with either Nettle or NSS. It currently works only with prime field curves, but it can probably be adapted to other curves.

P-256 bug:
Mailing list post with code sample
Commit / fix for P-256 bug
CVE-2015-8803

P-384 bug:
Mailing list post with code sample
Commit / fix for P-384 bug
CVE-2015-8804

Nettle 3.2 release notes

Mozilla NSS: Wrong calculation results in mp_div() and mp_exptmod()

A bug in the NSS library can cause certain cryptographic calculations to produce wrong results. The bug is in the function mp_div(), this function gets used by the function mp_exptmod(), a combination of an exponentiation and a modulo operation. The mp_exptmod() function is used in several cryptographic algorithms, most notably it's the function to sign and encrypt with RSA and to calculate Diffie Hellman key exchanges. The NSS library is used for TLS connections in various Mozilla products.

This bug was found by comparing the results of NSS and OpenSSL bignum calculations and fuzzing the input with american fuzzy lop. A similar bug was also found and fixed in OpenSSL recently.

I first reported both issues separately, not knowing that they were caused by the same bug. In a first analysis both the Mozilla developers and I thought that the impact of the mp_div() bug would be minor, because it wasn't used in the cryptography code of NSS. But we overlooked that it was indirectly used through the mp_exptmod() function.

It is unclear what the exact impact and severity of this bug is. I am not aware of a practical way to exploit it, but as it affects important crypto code it might be possible to find exploitable scenarios.

I have published code examples with test inputs triggering the bug for both functions on Github. This bug is fixed in NSS 3.21 and in Firefox 44.

Mozilla Foundation Security Advisory 2016-07
Commit with the fix
Bug report for mp_div() (still private)
Bug report for mp_exptmod() (still private)
Example for mp_div()
Example for mp_exptmod()
CVE-2016-1938

Heap buffer overflow in fgetwln function of libbsd

libbsd is a library to provide common functions from BSD systems on Linux.

libbsd 0.8.1 and earlier contains a buffer overflow in the function fgetwln(). An if checks if it is necessary to reallocate memory in the target buffer. However this check is off by one, therefore an out of bounds write happens.

Upstream has released version 0.8.2 to fix this.

I have checked where this function gets used. I didn't find any code using it, so I assume the impact is limited.

This bug was found with the help of Address Sanitizer.

Bug report
Git commit / fix
CVE-2016-2090

Out of bounds heap read in shred / coreutils

The GNU Coreutils project has just released the new version 8.25 which fixes an out of bounds heap read bug in the shred tool that I reported. It is a nice example of the subtle bugs one can find by testing code with address sanitizer.

shred is a tool to overwrite files with random data before deleting them. It generates a random memory pattern and in this pattern generation there was a heap overread. Due to the random pattern generation this bug is not deterministic and one has to run shred with certain parameters (for example -n 20) multiple times to trigger it.

Upstream bug report
Git commit / fix
Coreutils 8.25 release notes

Talk / Session at 32C3

I'm currently at the 32C3. Tomorrow (Day 3, 28th December) I will give a small talk about the Fuzzing Project. This will be hosted by the Free Software Foundation Assembly at the congress.

Where? Room A.1
When? 2015-12-28, 19:00

The talk will give a short introduction to Fuzzing and the motivation of the Fuzzing Project. I'll also cover Address Sanitizer and my current efforts in creating a Gentoo Linux system with Address Sanitizer. And finally I'll talk a bit about fuzzing bignum libraries to find crypto vulnerabilities.

There's also a wiki page for the talk, but the 32c3 wiki is currently down.

Update: I've made the slides available as a PDF and on Slideshare.

I will repeat the talk on January 5th, 7 p.m., in the Hackerspace AFRA in Berlin.

Out of bounds read in OpenVPN

OpenVPN versions before 2.3.9 contain an out of bounds read error. The bug happens in the function resolve_remote() in the file socket.c.

I reported this bug to the OpenVPN security team on December 6th. I was informed that this was already reported to them previously and fixed in the repository. The new release 2.3.9 fixes this. The current git head code of OpenVPN has this code part completely reworked, it is thus not affected.

The reason for this bug is that for both IPv4 and IPv6 connections OpenVPN will read a struct sockaddr_in6, but in the IPv4 case the data structure is smaller than in the IPv6 case. The bug was found by trying to run OpenVPN with Address Sanitizer.

I don't know whether this is in any way exploitable, but as OpenVPN is a security sensitive software I found it worthy to make it public.

Fuzzing Math - miscalculations in OpenSSL's BN_mod_exp (CVE-2015-3193)

Today OpenSSL released a security advisory and updates for a carry propagation bug that I discovered in the BN_mod_exp() function. The bug is in the 1.0.2 branch of OpenSSL and is fixed in 1.0.2e. It only affects the x86_64 assembly optimizations. Other architectures and older versions are not affected.

The bug was introduced in commit this commit and fixed in this one. It got CVE-2015-0860 assigned. A simple proof of concept test can be found here.

Fuzzing Bignum libraries

This is not the first time a miscalculation bug was found in the bignum library of OpenSSL. In January OpenSSL already had to fix a bug in the squaring function BN_sqr(). Back then I already asked myself if it would be worthwhile to use fuzzing to find such bugs. The BN_sqr() bug was special that it only occurred on very rare occasions. Only one out of 2^128 inputs would produce a wrong result. That effectively means random testing will never find such a bug. However american fuzzy lop has shown to be surprisingly successful in finding hard to find bugs. In a talk given at the Black Hat conference Ralph-Philipp Weinmann showed that with a very simple test tool he was able to re-find the BN_sqr() bug in OpenSSL with american fuzzy lop.

Finding bugs we already know may give interesting insights, but what we really want to do is to find new bugs. I tried various strategies to fuzz bignum libraries. There are two basic options to do so:

1. Do a calculation with one bignum library and check it for consistencies. This depends on the calculation you do. An example would be a division function. If you divide a by b, store the result in r and the remainder in s then r*b+s must be a again. In case of the BN_sqr() bug a possibility is simply to compare the result of the squaring with a multiplication of a number by itself. They should produce the same result.
2. Do differential testing with two different implementations. You simply take two different bignum libraries, do the same operation and compare the results.

One small challenge is how you structure the input data. When you have a single input value it is easy: Just take the whole file and interpret it as a number. But for most functions you will have different input values. What I did was that I simply took the first two bytes and used them to decide how to split the rest of the file in pieces. To compare the results I used a simple assert call. In case an assert failure happens american fuzzy lop will detect that.

The BN_mod_exp() bug was found by comparing libgcrypt with OpenSSL. Unfortunately I have been sloppy with archiving my code and I lost the exact code that I used to fuzz the bug. But I think I recreated an almost functionally equivalent example. (I should mention that libfuzzer might be the better tool for this job, but I still haven't gotten around trying it out.)

Fuzzing is usually associated with typical memory corruption bugs. What these examples show is that you can use fuzzing to target entirely different classes of bugs. Essentially fuzz testing can target any kind of bug class that depends on an input and that has a testable failure state. For mathematics the failure state is pretty obvious: If the result of a calculation is wrong then there is a bug.

Fuzzing versus branch-free code

After reporting the bug I was asked by the OpenSSL developers if I could do a similar test on their HMAC implementation. I did that and the result is interesting. At first I was confused: A while after the fuzzing started american fuzzy lop was only reporting two code paths. Usually it finds dozends of code paths within seconds.

This happens because cryptographic code is often implemented in a branch-free way. That means that there are no if-blocks that will execute different parts of the code depending on the input. The reason this is done is to protect against all sorts of sidechannel attacks. This conflicts with the way modern fuzzers like american fuzzy lop or libfuzzer work. They use the detection of new code paths as a way to be smart about their inputs.

I don't want to suggest here that branch-free code is bad. I think the advantages of branch-free code are undisputed, but it's interesting to see that it can make fuzz testing harder.

In case you wonder why american fuzzy lop still found two code paths: The reason is likely the input length. The HMAC code is branch-free for each block, but if the block number changes you will get a different code path.

What's the impact?

Finally you may ask what the impact of the BN_mod_exp() bug is. This is in part still unknown and I can only offer a preliminary analysis.

The BN_mod_exp() function is used to exponentiate a number in a modulus (a^b mod m) and is used in many algorithms. It is the core of both RSA and Diffie Hellman. In the case of RSA I think it's unlikely that there is a vulnerability. A potential attacker has basically no control over the input values. The base is either random (RSA exchange) or a hash (DHE/ECDHE exchange). The exponent and the modulus are part of the key. I haven't looked into DSA, because nobody uses it.

Diffie Hellman looks more interesting. I first thought it's not interesting, because usually in a Diffie Hellman key exchange the secret key is only used for one connection. Therefore the only thing an attacker could do is attacking a connection that he himself is part of. That is unlikely to give him anything interesting. But Juraj Somorovsky pointed out to me that OpenSSL caches and reuses the ephemeral key for several Diffie Hellman exchanges until the application restarts. So it might be possible to construct an oracle that will extract this cached ephemeral key. I leave it to people who know more about cryptography and x64 assembly to decide whether that is the case.

The conclusions of the OpenSSl team in the advisory are similar to mine.

OpenSSL has an option to disable this key caching. This can be done by passing the SSL_OP_SINGLE_DH_USE (for classic Diffie Hellman) and SSL_OP_SINGLE_ECDH_USE (for Elliptic Curve Diffie Hellman) values to SSL_CTX_set_options(). In my opinion this should be the default, reusing the ephemeral key seems quite dangerous. Many popular applications, including the Apache web server, already set this option.

I invite everyone to analyze this further and try to come up with a practical attack.

Thanks to Tom Ritter, Ralph-Philipp Weinmann and Juraj Somorovsky for valuable discussions on the topic.

Stack overflows and out of bounds read in dpkg (Debian)

Two stack overflows and one stack out of bounds access were fixed in dpkg, the package management tool from Debian.

A call to the function read_line didn't consider a trailing zero byte in the target buffer and thus could cause a one byte stack overflow with a zero byte. This issue was already fixed in the testing code when I reported it, but the fix wasn't backported to stable yet.
Git commit / fix
Minimal PoC file
The Debian developers consider this as non-exploitable, therefore no CVE got assigned.

A second almost identical stack overflow due to a call to the function read_line was in the same file.
Minimal PoC file
This issues got the id CVE-2015-0860.

A stack out of bounds read can happen in the function dpkg_ar_normalize_name. There is a read access to an array where the index can have the value -1. A check if the index is a positive value fixes this.
Minimal PoC file

All issues were found with the help of american fuzzy lop and address sanitizer.

Debian has published the advisory DSA 3407-1. Fixes packages for both stable (Jessie) and oldstable (Wheezy) have been published.

Ubuntu has published the advisory USN-2820-1. Fixed packages for Ubuntu 15.10, 15.04 and the LTS versions 14.04 and 12.04 have been published.

The updates fix all three issues. All users of Ubuntu, Debian and other dpkg/apt-based distributions should update.

Heap Overflow in PCRE

The Perl Compatible Regular Expressions (PCRE) library has just released a new version which fixes a number of security issues.

Fuzzing the pcretest tool uncovered an input leading to a heap overflow in the function pcre_exec. This bug was found with the help of american fuzzy lop and address sanitizer.
Upstream bug #1637

This is fixed in PCRE 8.38. There are two variants of PCRE, the classic one and PCRE2. PCRE2 is not affected.

Appart from that a couple of other vulnerabilities found by other people have been fixed in this release:
Stack overflow in compile_regex (bug #1503)
Heap overflow in compile_regex (bug #1672)
Stack overflow in compile_regex (bug #1515)
Heap overflow in compile_regex (bug #1636, CVE-2015-3210)
Stack overflow in match (bug #1638, CVE-2015-3217)
Heap overflow in compile_regex (bug #1667)
(this list may be incomplete)

If you use PCRE with potentially untrusted regular expressions you should update immediately. There is no immediate risk if you use regular expressions from a trusted source with an untrusted input.

Libxml2: Several out of bounds reads

I discovered several out of bounds read issues in Libxml2. The upstream developers have just released version 2.9.3, which fixes all relevant issues.

A malformed XML file can cause a heap out of bounds read access in the function xmlParseXMLDecl.
Upstream bug #751603 (sample input attached)
Git commit / fix

A second, very similar issue in the same function xmlParseXMLDecl.
Upstream bug #751631 (sample input attached)
Git commit / fix

A malformed XML file can cause a global out of bounds read access in the function xmlNextChar. This only affected the git code and was never an issue in any release version.
Upstream bug #751643 (sample input attached)

All three issues above were found with american fuzzy lop and address sanitizer.

Some inputs can cause a stack out of bounds read. This was found by running the test suite with Address Sanitizer (make check). The issue was re-found by fuzzing independently by Hugh Davenport:
Upstream bug #752191
Upstream bug #756372 (duplicate)
Git commit / fix
CVE-2015-8242

Unfortunately there is another issue affecting the test suite (also documented in upstream bug #752191) that isn't fixed yet, but the bug is in the code of the test itself, therefore it's not affecting the use of Libxml2.

A large number of other issues have been fixed, many of them found with american fuzzy lop and libfuzzer. The release notes of 2.9.3 mention 10 CVEs. If you use Libxml2 please update as soon as possible.

Network fuzzing with american fuzzy lop

American fuzzy lop is a remarkable tool, but it always had a big limitation: It only worked for file inputs.

There had been different attempts to adapt networking to afl. There's a tool called preeny that works by preloading a library. I created a similar approach myself, however I never published it, the approach was very error-prone and only worked on very few applications.

Now there is a new attempts for fuzzing network input with afl and based on my first experiences it seems to work much better. Doug Birdwell created a modified version of afl that allows to fuzz networking inputs. It's relatively simple to use, just check out the documentation. For example I fuzzed wget with this command line:
afl-fuzz -i in -o out -t 30+ -D 7 -m none -L -Ntcp://localhost:8082 ./wget -O - -t 1 http://localhost:8082/test.htm

This doesn't just work in theory, Doug Birdwell reported on the afl-users mailing list that one of the bugs fixed with the latest release of ntp (CVE-2015-7855) was found with this new afl variant.

Having a networking variant of afl is a huge step to make it even more useful.