The Fuzzing Project

I've lately played a lot with Address Sanitizer, a feature of the gcc and clang compilers that enables additional checks for memory access errors. Surprisingly quite often you don't even have to fuzz an application to find errors, just running it already exposes bugs.

In response to this I wrote a small tutorial describing how Address Sanitizer and other compiler flags can improve software testing and development. I think this is a low hanging fruit measure to improve software quality.

Know your CFLAGS - simple Tips to find Bugs with Compiler Features

OpenSSH 6.9 has been released yesterday and fixes two out of bounds read issues I had reported.

The first one is not very interesting. A null byte in a configuration file can cause an off-by-one read access. As a config file is not attacker controlled this doesn't have any security impact.

The second issue is an out of bounds read access in the function match_pattern_list() through the ssh handshake. I found this while experimenting with a way to do network fuzzing with american fuzzy lop via preloading a library that will intercept networking functions. I hope to be able to release the tool I use for it at some point, right now it's not really in a usable state. (I later learned that there is a similar tool called Preeny.)

The match_pattern_list() function had two strings and the length of the second string as parameters. The length of the first string was determined by strlen(). A call in the file combat.c called that function with the length of the first string as the parameter, causing an out of bounds read.

This issue can be reproduced by:
a) Compiling openssh 6.8p1 with address sanitizer (./configure CFLAGS="-fsanitize=address" LDFLAGS="-fsanitize=address"; make)
b) Running this netcat-command:
echo "SSH-2.0-OpenSSH_6.5"|nc -l -p 22
c) Connecting to it: ./ssh 127.0.0.1

OpenSSH fixed this by changing the function match_pattern_list() to accept only the two strings and checking both lengths with strlen().

Now there's a backstory to this. When I reported this to Damien Miller he considered it not security relevant and published the fix right away to the public git. It wasn't noticed by anyone until I briefly mentioned it in a post to the american fuzzy lop users mailing list. People started linking to this post on twitter so I thought I should explain it more detailed and wrote a message to oss-security. That only caused more (probably undeserved) attention, up to the point where some people would call it a high risk vulnerability.

Some people criticised me for calling this issue a heap overflow. This highlights a problem: The term buffer overflow and the related terms stack / heap overflow aren't defined in a strict way. Some consider only out of bounds writes an overflow (e. g. the CWE-120 definition), others also call out of bounds reads an overflow (e. g. Address Sanitizer). See also David Wheeler's discussion on Heartbleed. To avoid any confusion I decided for myself to use the term out of bounds read in the future for similar issues.

There are two takeaways for me from this story:
a) If you write about potential vulnerabilities in important security software it's hard to avoid that others will exaggerate the issue.
b) Vulnerability terminology can be ambiguous and is not always strictly defined.

Two memory access issues were found in the Courier mail server. These issues were discovered by compiling the software with Address Sanitizer (-fsanitize=address) and running the test suite.

In the file mailboxlist.c, part of the IMAP folder parser, there is a memcmp call checking whether the fist six bytes of a string match "SHARED". However the string can be less than six bytes, which will cause an out of bounds read access. This issue is unlikely to cause much trouble.

In the mailbot tool (mailbot.c) there is a memory allocation for a zero-terminated list of pointers. The allocation only reserves one byte for the zero termination, however it must be the size of the pointer (8 bytes on 64 bit systems). Therefore it causes a write heap overflow of seven zero bytes. The code parses command line data, therefore it is unlikely that any attacker controlled input is affected.

Both issues have been reported to Courier's developer Sam Varshavchik on 27th June 2015 and were fixed with the release of courier 0.75 on 29th June 2015.

Commit / Patch (bundles some unrelated changes, the relevant parts are in mailbot.c and mailboxlist.c)
Courier 0.75 release announcement

I'm happy to announce today that the Linux Foundation's Core Infrastructure Initiative has decided to financially support the work I'm doing for the Fuzzing Project. The Core Infrastructure Initiative (CII) was established last year after the Heartbleed bug in OpenSSL. A number of companies came together to support people improving the security of free and open source software.

For the Fuzzing Project this of course means that I will report many more bugs in the future. While in the beginning all the fuzzing work was done on my personal laptop I recently moved that to dedicated servers that fuzz applications all the time. Appart from that I'm investigating methods to further increase the ways to find bugs in large quantities. I have some preliminary experiments to use american fuzzy lop on network input. I also want to encourage developers of free software projects to get in touch with me if they think their project could profit from fuzz testing.

I also want to spend a few words on the other two projects that got supported by the CII.

One problem of free software is that you can in theory verify what the code is doing and check it for backdoors, but usually you don't compile the code yourself. You still get it in binary form from somewhere (unless you use a source-based system like Gentoo, but that's a small minority). In order to bridge this trust gap between source code and binaries the idea of reproducible builds emerged. If different people can compile the same code and will always get the same output then it can be independently verified that the code matches the binary. This is easier said than done, because you have to consider all kinds of non-deterministic behaviour, e. g. added timestamps, different compiler versions and more.

The Debian Linux distribution is currently working on reproducible builds. Debian-developers Holger Levsen and Jérémy Bobbio (Lunar) will be supported by the CII and I am very optimistic that they will make it happen and the next Debian version will be reproducible. I am glad about this because I think if we really want trustworthy software in the future we need reproducible builds. I hope that in the future we can make it a norm that trustworthy, secure software contains a reproducible build process (which by definition means open code).

Pascal Cuoq is working on improving automated, formal methods to find bugs in software. He's worked on Frama-C before and he is currently working on a new code analyzer tool which is not public yet, but it already found a number of potential issues in OpenSSL.

Lately I started an effort to systematically fuzz all possible file input vectors of OpenSSL. This led to the discovery of one potential security issue and two minor non-security fixes.

Malformed inputs can cause an out of bounds heap read access in the function X509_cmp_time. This issue was reported to the OpenSSL developers on 11th March. It was independently discovered three days earlier by Google developer Robert Swiecki.

During the fuzzing I also discovered several issues in the parser of ASN1 definition files. These can be used to create ASN1 data structures with OpenSSL. It is unlikely that there is any situation where ASN1 definitions are attacker controlled, therefore these are not considered security issues.

The latest security updates of OpenSSL (1.0.2b, 1.0.1n, 1.0.0s, 0.9.8zg) fix all three issues. These releases also fix a number of other security issues. Shortly after publishing these updates OpenSSL issued another update (1.0.2c, 1.0.1o), because the versions contained an ABI change which should not happen in minor releases.

I am aware that a couple of other people were also fuzzing OpenSSL lately. Noteworthy is one issue that was found by Joseph Birr-Pixton in the parser of elliptic curve parameters. It is an endless loop and can be used to hang processes with a high CPU load. Endless loop issues tend to get ignored because they are often false positives.

It is definitely getting harder finding any new issues through fuzzing in OpenSSL. This is good news.

Out of bounds read in X509_cmp_time
CVE-2015-1789
Git commit / fix
OpenSSL Security Advisory
Sample malformed cert (test with openssl verify [input])

Samples for issues in ASN1 definition parser (test with openssl asn1parse -genconf [input]):
Out of bounds read heap
Stack overflow through endless recursion
Uninitialized memory access

Update: Around one year after these bugs were reported it turned out that one of the issues is actually more severe than we thought.

The nghttp2 library ships two tools to parse http headers packed with the hpack algorithm. An invalid input file can crash the inflatehd tool. This is a bug in the tool, there is no issue in the library.

This issue was fixed in version 0.7.15 of nghttp2.

One day of fuzzing both the inflatehd and deflatehd turned up no other issues.

Sample input file
Git commit / patch
Upstream bug report
nghttp 0.7.15 release notes

uudecode is a tool to decode uuencoded data. It is shipped with the package sharutils.

An invalid input file can cause an out of bounds stack read access in the function expand_tilde(). This issue has been reported to the developers on 2015-03-04. It has been fixed in sharutils 4.15.2 (2015-05-30).

To see this bug one needs to use a tool like valgrind or address sanitizer that detects out of bounds memory reads. The bug was found with american fuzzy lop.

Sample file

Address sanitizer output:

==8209==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff8a4a8690 at pc 0x40738d bp 0x7fff8a4a44a0 sp 0x7fff8a4a4490
READ of size 1 at 0x7fff8a4a8690 thread T0
#0 0x40738c in expand_tilde /mnt/ram/sharutils-4.14/src/uudecode.c:252
#1 0x40738c in decode /mnt/ram/sharutils-4.14/src/uudecode.c:437
#2 0x403660 in main /mnt/ram/sharutils-4.14/src/uudecode.c:530
#3 0x7f13d97fff9f in __libc_start_main (/lib64/libc.so.6+0x1ff9f)
#4 0x403c81 (/mnt/ram/sharutils-4.14/src/uudecode+0x403c81)

Address 0x7fff8a4a8690 is located in stack of thread T0 at offset 16800 in frame
#0 0x403da7 in decode /mnt/ram/sharutils-4.14/src/uudecode.c:362

This frame has 7 object(s):
[32, 36) 'mode'
[96, 104) 'outlen'
[160, 168) 'ctx'
[224, 368) 'attr'
[416, 16800) 'buf' <== Memory access at offset 16800 overflows this variable
[16832, 33216) 'buf_in'
[33248, 49632) 'buf'

A while ago I did a little experiment trying to fuzz the OpenSSL handshake with the intent to test whether Heartbleed could've been found with fuzzing. At some point while developing the sample code I discovered that american fuzzy lop would find a lot of crashes. However, on careful investigation I found that these were not due to bugs in OpenSSL, they were due to a bug in my test code: I didn't properly check for errors.

My code used TLSv1_method() to initialize the handshake. The packets the fuzzer created had a changed version number, so OpenSSL interpreted them as SSLv3 packets. OpenSSL wouldn't do a handshake with them and the function SSL_do_handshake() would return an error. However, as my code didn't check that error it later called SSL_do_handshake() again - and that will cause a crash. I've uploaded some example code to show this behaviour.

This is not really a bug in OpenSSL, but one may consider this as unexpected behaviour. OpenSSL has now changed that and the new code will not crash in that situation.

I wondered if other software would do similar mistakes and found out that trying to use the handshake packets in question on a dovecot mail server would cause crashes in the imap-login or pop3-login processes. However, I didn't analyze it further at that point.

Some time later I chattet with Aaron Zauner who is involved in a research project scanning for TLS server settings. He got feedback that their scans caused crashing dovecot processes, so I remembered my findings and reinvestigated the issue.

It turned out that on errors dovecot would try to finish the handshake. This caused the same error that I observed. I posted a detailed explanation on the dovecot mailing list, which is linked below. The dovecot developers have fixed this in release 2.2.17 (which was quickly followed by a 2.2.18 release due to unrelated bugs).

The crash is caused by a null pointer access. It is not exploitable. It can only be used to remotely crash server processes. As these login processes are automatically restarted by dovecot the impact is probably low.

Explanation on Dovecot mailing list
Upstream commit / patch
OpenSSL bug report
Dovecot 2.2.17 release notes
CVE-2015-3420

The Wireshark parser code for Android Logcat network packages contained a read heap overflow in the function detect_version().

This issue was reported to the Wireshark developers on May 5th. It was fixed in the 1.12.5 release of Wireshark, published on May 12th. The beta release 1.99.5 and the Git head code are not affected.

Apart from this issue Wireshark 1.12.5 fixes seven other security issues.

Wireshark 1.12.5 Release Notes
Wireshark bug #11188
Commit / fix
Sample pcap file triggering the overflow (test with tshark -r [input], can be seen with valgrind or address sanitizer)
CVE-2015-3815

While fuzzing SQLite I discovered two read heap overflow errors. One is in the database file parser, one in the sql command parser. Both issues are present in SQLite 3.8.9 and are fixed in SQLite 3.8.10.1. These bugs can be seen with either valgrind or address sanitizer.

Passing the command ".\" will cause a one byte heap overflow in the function resolve_backslashes().
Sample input file (test with sqlite3 < [inputfile])
Upstream commit / patch

Parsing a malformed database file will cause a heap overflow of several bytes in the function sqlite3VdbeExec(). This only matters if your attack scenario involves parsing untrusted database files.
Sample input file (test with sqlite3 [inputfile] .dump)
Upstream commit / patch

Please also note:
Finding bugs in SQLite, the easy way - Michal Zalewski fuzzed SQLite with a dictionary - most of these were already fixed in 3.8.9, the version I was testing.
SQL Fuzz Using The American Fuzzy Lop Fuzzer - SQLite developers themselve now use regular fuzz testing to find further bugs.
SQLite 3.8.10.1 release notes mention fixes for "many obscure problems discovered while SQL fuzzing", so there are likely more fixes than the two I mentioned above.

While fuzzing GnuTLS I discovered a malformed certificate input sample that would cause a heap overflow read of 99 bytes in the DER decoding functions of Libtasn1. The heap overflow happens in the function _asn1_extract_der_octet().

This issue was reported to the Libtasn1 developer on 16th April. A fix was committed on 20th April and is part of the Libtasn1 4.5 release. This issue was found with american fuzzy lop and address sanitizer.

Git commit / fix
Libtasn1 4.5 release notes
Sample malformed certificate exposing heap overflow (test with certtool -i --inder --infile=[sample] and address sanitizer or valgrind)
CVE-2015-3622

Today version 7.42.0 of curl was released. It fixes two vulnerabilities and one not security relevant bug that I found via fuzzing and reported.

When fuzzing an application the question is always which input vector you fuzz. The tool american fuzzy lop is purely file-based, therefore one needs some kind of file input to fuzz an application. Curl has a couple of potentially interesting file inputs.

One feature of curl is to save cookies to a file and later re-use that cookie file for followup connections. By fuzzing these inputs I found out that using a double-quote character (") as the path causes a memory access to a -1 index of an array. This could also be triggered by a malicious webpage.

The other input vector I fuzzed was the config file. Usually config files are not attacker-controlled, so one may question whether this makes sense at all. But here's what happened: The config file allows to pass an URL to curl and by that I found an out ouf bounds memory access in the URL parser. And an URL can be attacker controlled in many situations (for example through HTTP forwards). So the takeaway message here is that testing input vectors that are never attacker controlled (like config files) may still uncover potential security issues, because they might trigger code paths that matter in other situations.

All issues were reported to the curl security team on 16th April 2015 and are fixed in curl 7.42.0. Thanks to Daniel Stenberg of the curl team who quickly fixed all the issues I reported. Please also note that curl 7.42.0 fixes two other unrelated security issues.

cookie parser out of boundary memory access (cURL Security Advisory)
Patch
Git commit
CVE-2015-3145
Sample file (test with curl -b)

host name out of boundary memory access (cURL Security Advisory)
Patch
Git commit
CVE-2015-3144
Sample file (test with curl -K on the sample or just curl :0

Git commit for non-security config file parser issue
Sample file (test with curl -K)

GnuPG is a popular E-Mail encryption solution. GnuPG also ships a tool gpgsm that can be used to sign and encrypt mails according to the S/MIME standard. To parse S/MIME data structures the library libksba is used. GpgOL is a GnuPG-based plugin to use OpenPGP-encryption in Microsoft Outlook.

Fuzzing various input vectors of GnuPG led to the discovery of several potential security issues.

An integer overflow in the BER decoder of libksba can lead to an invalid write / heap overflow. This is fixed in libksba 1.3.3.

Git commit / fix
Sample file (test with gpgsm --verify)

An invalid UTF-8 encoding can cause an assert in libksba. This may be abused to crash / DoS an application. This is fixed in libksba 1.3.3.

Git commit / fix
Sample file (test with gpgsm --verify)

A stack overflow in an internal stack of the BER decoder of libksba can be abused to crash / DoS an application. This is fixed in libksba 1.3.3.

Git commit / fix
Sample file (test with gpgsm --verify)

An invalid mail header can cause an off by one read access / heap overflow in the tool gpgparsemail, which is part of GnuPG. The same code is present in GpgOL, a plugin to use OpenPGP encryption in Microsoft Outlook. The gpgparsemail issue is fixed in GnuPG 2.1.3. There is no new release of GpgOL yet.

(Due to an incomplete fix there are two commits each.)

Git commit 1 for gpgparsemail / GnuPG
Git commit 2 for gpgparsemail / GnuPG
Git commit 1 for GpgOL
Git commit 2 for GpgOL
Sample file for original issue
Sample file for incomplete fix

A malformed packet could cause GnuPG to detect a negative packet length. The size variable is unsigned, resulting in an malloc call to a very large value and a crash / DoS. This is fixed in GnuPG 2.1.3.

Git commit / fix
Sample file for malloc DoS (test with gpg --import or gpg --list-packets)

A malformed private key can cause a null pointer access when trying to import the key. This came too late for GnuPG 2.1.3 and is not yet fixed in a release.

Git commit / fix
Sample file (test with gpg --verify)

libtasn1 is a library to parse ASN.1 data structures. Its most prominent user is GnuTLS.

Fuzzing libtasn1 led to the discovery of a stack write overflow in the function _asn1_ltostr (file parser_aux.c). It overflows a temporary buffer variable on certain inputs. This issue has been reported to the developers on 2015-03-26. A fix was released on 2015-03-29.

The issue can be exposed with Valgrind or Address Sanitizer. The Address Sanitizer output with detailed info is given below.

Git commit / fix
Release notes libtasn1 4.4
Sample input for stack overflow (to be used with examples/pkix.asn from libtasn1 source, e.g. src/asn1Decoding examples/pkix.asn TFPA-2015-002-libtasn1-4.3-stack-overflow.crt PKIX1Implicit88.Certificate)
CVE-2015-2806

An earlier fuzzing effort led to the discovery of a null pointer derefenence error in the ASN.1 definition parser. This is unlikely to have any security impact. Null pointer errors are usually not exploitable and there are probably no scenarios where ASN.1 definitions are attacker controlled. This issue has been reported to the libtasn1 developers on 2015-01-25 and was fixed on 2015-02-05. The fix was delivered with the 4.3 release of libtasn1.

Report on mailing list
Git commit / fix
Sample input for null ptr (can be tested with asn1Decoding TFPA-2015-002-libtasn1-4.2-null-ptr.asn x x)

I want to thank libtasn1 developer Nikos Mavrogiannopoulos for the quick fixes. Both issues were found with american fuzzy lop.

==4372==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff85a08084 at pc 0x43c180 bp 0x7fff85a07d10 sp 0x7fff85a07d00
WRITE of size 1 at 0x7fff85a08084 thread T0
#0 0x43c17f in _asn1_ltostr /data/libtasn1/libtasn1-4.3/lib/parser_aux.c:574
#1 0x41ee31 in _asn1_get_objectid_der /data/libtasn1/libtasn1-4.3/lib/decoding.c:397
#2 0x41ee31 in asn1_der_decoding2 /data/libtasn1/libtasn1-4.3/lib/decoding.c:1225
#3 0x423b0e in asn1_der_decoding /data/libtasn1/libtasn1-4.3/lib/decoding.c:1602
#4 0x403692 in simple_decode /data/libtasn1/libtasn1-4.3/src/asn1Decoding.c:251
#5 0x403692 in decode /data/libtasn1/libtasn1-4.3/src/asn1Decoding.c:280
#6 0x403692 in main /data/libtasn1/libtasn1-4.3/src/asn1Decoding.c:205
#7 0x7f94cb39af9f in __libc_start_main (/lib64/libc.so.6+0x1ff9f)
#8 0x4046a1 (/data/libtasn1/libtasn1-4.3/src/asn1Decoding+0x4046a1)

Address 0x7fff85a08084 is located in stack of thread T0 at offset 564 in frame
#0 0x419bdf in asn1_der_decoding2 /data/libtasn1/libtasn1-4.3/lib/decoding.c:980

This frame has 10 object(s):
[32, 36) 'len2'
[96, 100) 'tag_len'
[160, 164) 'len2'
[224, 232) 'p'
[288, 296) 'p2'
[352, 360) 'ptail'
[416, 424) 'p'
[480, 489) 'temp'
[544, 564) 'temp' <== Memory access at offset 564 overflows this variable
[608, 736) 'temp'

A complex tool like GnuPG has many ways to parse input data. I previously had fuzzed GnuPG which had led to the detection of a Buffer Overflow vulnerability in GnuPG and libksba (CVE-2014-9087) and various other minor issues. Recently I tried to fuzz less obvious inputs of GnuPG: Keyrings and configuration files.

GnuPG allows to specify a non-standard keyring on the command line. Fuzzing GPG with
gpg --export --no-default-keyring --keyring [input keyring]
led to the detection of various issues. (Please note that the keyring parameter needs the full path and does not like filenames with unusual characters like the ones generated by american fuzzy lop.)

NULL pointer deref in parse_trust (parse-packet.c)
http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=39978487863066e59bb657f5fe4e8baab510da7e
https://crashes.fuzzing-project.org/TFPA-2015-01-gnupg-keyring-null-ptr-1

NULL pointer deref in do_key (build-packet.c)
http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=0835d2f44ef62eab51fce6a927908f544e01cf8f
https://crashes.fuzzing-project.org/TFPA-2015-01-gnupg-keyring-null-ptr-2

Use after free (build-packet.c)
http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=f0f71a721ccd7ab9e40b8b6b028b59632c0cc648
https://crashes.fuzzing-project.org/TFPA-2015-01-gnupg-keyring-use-after-free
CVE-2015-1606

memcpy with overlapping ranges (keybox_search.c)
http://git.gnupg.org/cgi-bin/gitweb.cgi?p=gnupg.git;a=commit;h=2183683bd633818dd031b090b5530951de76f392
https://crashes.fuzzing-project.org/TFPA-2015-01-gnupg-keyring-memcpy-overlap
CVE-2015-1607

All issues found with american fuzzy lop. Fuzzing of the configuration file parser showed no issues.

While keyrings are usually not user-submitted data, some of these can be reached through other code paths. None of the issues looks severe, however judging the exact security would require further analysis.

Timeline:
2015-02-06 Reported three issues to GnuPG developer Werner Koch
2015-02-09 ALl reported issues fixed in git
2015-02-09 Reported one more issue to Werner Koch
2015-02-11 Last issue fixed in git
2015-02-11 Release of GnuPG 2.1.2 containing all fixes