The Fuzzing Project

During some tests of TLS libraries I found a stack buffer overflow vulnerability in the WolfSSL library.
Finding this one was surprisingly simple: I had a wolfssl server that was compiled with address sanitizer and ran the SSL Labs test against it.

The bug happens in the parsing of the signature hash algorithm list that is sent in a ClientHello and is basically a textbook stack buffer overflow. WolfSSL simply tries to store that in an array with 32 elements. If one sends more than 32 hash algorithms it overflows.

With the SSL Labs scan the bug only causes WolfSSL to terminate if it's compiled with address sanitizer, but if one sends a very large list of hash algorithms it also crashes in a normal compile. In situations where WolfSSL is used without ASLR this bug is probably trivially exploitable.

I have created a simple bash proof of concept (using netcat and xxd) that crashes a WolfSSL server.

The bug was fixed in this commit and in version 3.13.0 of WolfSSL.

If you're using the HTTP protocol in everday Internet use you are usually only using two of its methods: GET and POST. However HTTP has a number of other methods, so I wondered what you can do with them and if there are any vulnerabilities.

One HTTP method is called OPTIONS. It simply allows asking a server which other HTTP methods it supports. The server answers with the "Allow" header and gives us a comma separated list of supported methods.

A scan of the Alexa Top 1 Million revealed something strange: Plenty of servers sent out an "Allow" header with what looked like corrupted data. Some examples:
Allow: ,GET,,,POST,OPTIONS,HEAD,,
Allow: POST,OPTIONS,,HEAD,:09:44 GMT
Allow: GET,HEAD,OPTIONS,,HEAD,,HEAD,,HEAD,, HEAD,,HEAD,,HEAD,,HEAD,POST,,HEAD,, HEAD,!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"
Allow: GET,HEAD,OPTIONS,=write HTTP/1.0,HEAD,,HEAD,POST,,HEAD,TRACE

That clearly looked interesting - and dangerous. It suspiciously looked like a "bleed"-style bug, which has become a name for bugs where arbitrary pieces of memory are leaked to a potential attacker. However these were random servers on the Internet, so at first I didn't know what software was causing this.

Sometimes HTTP servers send a "Server" header telling the software. However one needs to be aware that the "Server" header can lie. It's quite common to have one HTTP server proxying another. I got all kinds of different "Server" headers back, but I very much suspected that these were all from the same bug.

I tried to contact the affected server operators, but only one of them answered, and he was extremely reluctant to tell me anything about his setup, so that wasn't very helpful either.

However I got one clue: Some of the corrupted headers contained strings that were clearly configuration options from Apache. It seemed quite unlikely that those would show up in the memory of other server software. But I was unable to reproduce anything alike on my own Apache servers. I also tried reading the code that put together the Allow header to see if I can find any clues, but with no success. So without knowing any details I contacted the Apache security team.

Fortunately Apache developer Jacob Champion digged into it and figured out what was going on: Apache supports a configuration directive Limit that allows restricting access to certain HTTP methods to a specific user. And if one sets the Limit directive in an .htaccess file for an HTTP method that's not globally registered in the server then the corruption happens. After that I was able to reproduce it myself. Setting a Limit directive for any invalid HTTP method in an .htaccess file caused a use after free error in the construction of the Allow header which was also detectable with Address Sanitizer. (However ASAN doesn't work reliably due to the memory allocation abstraction done by APR.)

FAQ

What's Optionsbleed?

Optionsbleed is a use after free error in Apache HTTP that causes a corrupted Allow header to be constructed in response to HTTP OPTIONS requests. This can leak pieces of arbitrary memory from the server process that may contain secrets. The memory pieces change after multiple requests, so for a vulnerable host an arbitrary number of memory chunks can be leaked.

The bug appears if a webmaster tries to use the "Limit" directive with an invalid HTTP method.

Example .htaccess:

<Limit abcxyz>
</Limit>

How prevalent is it?

Scanning the Alexa Top 1 Million revealed 466 hosts with corrupted Allow headers. In theory it's possible that other server software has similar bugs. On the other hand this bug is nondeterministic, so not all vulnerable hosts may have been caught.

So it only happens if you set a quite unusual configuration option?

There's an additional risk in shared hosting environments. The corruption is not limited to a single virtual host. One customer of a shared hosting provider could deliberately create an .htaccess file causing this corruption hoping to be able to extract secret data from other hosts on the same system.

I can't reproduce it!

Due to its nature the bug doesn't appear deterministically. It only seems to appear on busy servers. Sometimes it only appears after multiple requests.

Does it have a CVE?

CVE-2017-9798.

I'm seeing Allow headers containing HEAD multiple times!

This is actually a different Apache bug (#61207) that I found during this investigation. It causes HEAD to appear three times instead of once. However it's harmless and not a security bug.

Launchpad also has a harmless bug that produces a malformed Allow header, using a space-separated list instead of a comma-separated one.

How can I test it?

A simple way is to use Curl in a loop and send OPTIONS requests:

for i in {1..100}; do curl -sI -X OPTIONS https://www.google.com/|grep -i "allow:"; done

Depending on the server configuration it may not answer to OPTIONS requests on some URLs. Try different paths, HTTP versus HTTPS hosts, non-www versus www etc. may lead to different results.

Please note that this bug does not show up with the "*" OPTIONS target, you need a specific path.

Here's a python proof of concept script.

What shall I do?

If you run an Apache web server you should update. Most distributions should have updated packages by now or very soon. A patch can be found here. A patch for Apache 2.2 is available here (thanks to Thomas Deutschmann for backporting it).

Unfortunately the communication with the Apache security team wasn't ideal. They were unable to provide a timeline for a coordinated release with a fix, so I decided to define a disclosure date on my own without an upstream fix.

If you run an Apache web server in a shared hosting environment that allows users to create .htaccess files you should drop everything you are doing right now, update immediately and make sure you restart the server afterwards.

Is this as bad as Heartbleed?

No. Although similar in nature, this bug leaks only small chunks of memory and more importantly only affects a small number of hosts by default.

It's still a pretty bad bug, particularly for shared hosting environments.

Updates:

Analysis by Apache developer William A. Rowe Jr.

Distribution updates:
Gentoo: Commit (2.2.34 / 2.4.27-r1 fixed), Bug
NetBSD/pkgsrc: Commit
Guix: Commit
Arch Linux: Commit (2.4.27-2 fixed)
Slackware: Advisory
NixOS: Commit
Debian: Security Tracker, Advisory (2.4.10-10+deb8u11, 2.4.25-3+deb9u3)
Ubuntu: Advisory (2.4.25-3ubuntu2.3, 2.4.18-2ubuntu3.5, 2.4.7-1ubuntu4.18)

Media:
Apache-Webserver blutet (Golem.de)
Apache Webserver: "Optionsbleed"-Bug legt Speicherinhalte offen (heise online)
Risks Limited With Latest Apache Bug, Optionsbleed (Threatpost)
Apache “Optionsbleed” vulnerability – what you need to know (Naked Security)
Apache bug leaks contents of server memory for all to see—Patch now (Ars Technica)

This post first appeared on oss-security.

I'm reporting three issues here in exiv2, a parser libary for image metadata. These are only examples, exiv2 is full of memory safety bugs that can trivially be found by running afl with asan for a few hours.

I have not reported those issues upstream. When I previously tried to report bugs in exiv2 found via fuzzing the upstream author made it clear to me that he has little interest in fixing those issues and doesn't consider his software suitable to parse defect files (which basically means it's unsuitable for untrusted input). The discussion can be read here. (the page is sometimes not available, searching for it in the google cache usually works though)

exiv2 is to my knowledge used by the major Linux Desktops GNOME and KDE. I'll also inform their security teams. I leave it up to Linux distros how to handle this, but it certainly is problematic that a crucial parser used by major desktop applications is not interested in fixing potential security issues.


Heap overflow (write) in tiff parser

A malformed tiff file can cause a one byte heap overflow in exiv2.

Stack trace:

==22873==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000df1 at pc 0x000000842091 bp 0x7fff51b3ee70 sp 0x7fff51b3ee68
WRITE of size 1 at 0x602000000df1 thread T0
#0 0x842090 in Exiv2::ul2Data(unsigned char*, unsigned int, Exiv2::ByteOrder) /f/exiv2-trunk/src/types.cpp:362:20
#1 0x68beac in long Exiv2::toData<unsigned int>(unsigned char*, unsigned int, Exiv2::ByteOrder) /f/exiv2-trunk/src/../include/exiv2/value.hpp:1459:16
#2 0x68beac in Exiv2::ValueType<unsigned int>::copy(unsigned char*, Exiv2::ByteOrder) const /f/exiv2-trunk/src/../include/exiv2/value.hpp:1612
#3 0x6742b2 in Exiv2::Exifdatum::copy(unsigned char*, Exiv2::ByteOrder) const /f/exiv2-trunk/src/exif.cpp:362:48
#4 0x7f794d in Exiv2::TiffImage::readMetadata() /f/exiv2-trunk/src/tiffimage.cpp:204:18
#5 0x59786a in Action::Print::printSummary() /f/exiv2-trunk/src/actions.cpp:289:16
#6 0x596ef8 in Action::Print::run(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /f/exiv2-trunk/src/actions.cpp:244:44
#7 0x55fb3f in main /f/exiv2-trunk/src/exiv2.cpp:170:25
#8 0x7f91c1e571d0 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.24-r2/work/glibc-2.24/csu/../csu/libc-start.c:289
#9 0x468979 in _start (/r/exiv2/exiv2+0x468979)

0x602000000df1 is located 0 bytes to the right of 1-byte region [0x602000000df0,0x602000000df1)
allocated by thread T0 here:
#0 0x55af00 in operator new[](unsigned long) (/r/exiv2/exiv2+0x55af00)
#1 0x83fadf in Exiv2::DataBuf::alloc(long) /f/exiv2-trunk/src/types.cpp:158:22


Heap out of bounds read in jp2 / JPEG2000 parser

A malformed jpeg2000 file causes a (large) out of bounds read.

Stack trace:

==32038==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f5e099e6838 at pc 0x00000050f22c bp 0x7ffdf7f3dcd0 sp 0x7ffdf7f3d480
READ of size 808464432 at 0x7f5e099e6838 thread T0
#0 0x50f22b in __asan_memcpy (/r/exiv2/exiv2+0x50f22b)
#1 0x6e82bc in Exiv2::Jp2Image::readMetadata() /f/exiv2-trunk/src/jp2image.cpp:277:29
#2 0x59786a in Action::Print::printSummary() /f/exiv2-trunk/src/actions.cpp:289:16
#3 0x596ef8 in Action::Print::run(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /f/exiv2-trunk/src/actions.cpp:244:44
#4 0x55fb3f in main /f/exiv2-trunk/src/exiv2.cpp:170:25
#5 0x7f5e130a71d0 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.24-r2/work/glibc-2.24/csu/../csu/libc-start.c:289
#6 0x468979 in _start (/r/exiv2/exiv2+0x468979)

0x7f5e099e6838 is located 0 bytes to the right of 808452152-byte region [0x7f5dd96e6800,0x7f5e099e6838)
allocated by thread T0 here:
#0 0x55af00 in operator new[](unsigned long) (/r/exiv2/exiv2+0x55af00)
#1 0x6e8176 in Exiv2::DataBuf::DataBuf(long) /f/exiv2-trunk/src/../include/exiv2/types.hpp:204:46
#2 0x6e8176 in Exiv2::Jp2Image::readMetadata() /f/exiv2-trunk/src/jp2image.cpp:273
#3 0x59786a in Action::Print::printSummary() /f/exiv2-trunk/src/actions.cpp:289:16
#4 0x596ef8 in Action::Print::run(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /f/exiv2-trunk/src/actions.cpp:244:44
#5 0x7f5e130a71d0 in
__libc_start_main /var/tmp/portage/sys-libs/glibc-2.24-r2/work/glibc-2.24/csu/../csu/libc-start.c:289


Stack out of bounds read in webp parser

A malformed webp file causes a six bytes stack out of bounds read.

Stack trace:

==598==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7ffcc12aa054 at pc 0x0000004fe311 bp 0x7ffcc12a9f90 sp 0x7ffcc12a9740
READ of size 6 at 0x7ffcc12aa054 thread T0
#0 0x4fe310 in __interceptor_memcmp.part.76 (/r/exiv2/exiv2+0x4fe310)
#1 0x8889d0 in Exiv2::WebPImage::getHeaderOffset(unsigned char*, long, unsigned char*, long) /f/exiv2-trunk/src/webpimage.cpp:798:17
#2 0x8889d0 in Exiv2::WebPImage::decodeChunks(unsigned long) /f/exiv2-trunk/src/webpimage.cpp:601
#3 0x884ff2 in Exiv2::WebPImage::readMetadata() /f/exiv2-trunk/src/webpimage.cpp:496:20
#4 0x59786a in Action::Print::printSummary() /f/exiv2-trunk/src/actions.cpp:289:16
#5 0x596ef8 in Action::Print::run(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /f/exiv2-trunk/src/actions.cpp:244:44
#6 0x55fb3f in main /f/exiv2-trunk/src/exiv2.cpp:170:25
#7 0x7f7f9cc9f1d0 in __libc_start_main /var/tmp/portage/sys-libs/glibc-2.24-r2/work/glibc-2.24/csu/../csu/libc-start.c:289
#8 0x468979 in _start (/r/exiv2/exiv2+0x468979)

Address 0x7ffcc12aa054 is located in stack of thread T0 at offset 180 in frame
#0 0x885a0f in Exiv2::WebPImage::decodeChunks(unsigned long) /f/exiv2-trunk/src/webpimage.cpp:501

This frame has 13 object(s):
[32, 36) 'size_buff' (line 503)
[48, 64) 'payload' (line 516)
[80, 84) 'size_buf' (line 520)
[96, 100) 'size_buf48' (line 536)
[112, 114) 'size_buf_w' (line 551)
[128, 131) 'size_buf_h' (line 552)
[144, 148) 'size_buf112' (line 568)
[160, 162) 'size_buff152' (line 587)
[176, 180) 'exifLongHeader295' (line 588) <== Memory access at offset 180 overflows this variable
[192, 196) 'exifTiffBEHeader297' (line 591)
[208, 272) 'xmpData' (line 650)
[304, 688) 'temp.lvalue'
[752, 1136) 'temp.lvalue232'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
(longjmp and C++ exceptions are supported)


The way the Apache httpd web server handles password hashes can be abused by a malicious user on a server to cause resource exhaustion and denial of service of the web server. I reported this a while ago to the Apache security team - which led to a lengthy discussion where I was a bit appalled about some of the statements I got from the Apache developers. They can be summed up in a way that major use cases of Apache - especially in web hosting environments - are not recommended by their developers.

Apache supports HTTP basic authentication, a simple login mechanism with username and password that is part of the HTTP protocol. This can be configured via the .htaccess file on a web server. A very simple htaccess file looks like this:

AuthType Basic
AuthName "privat"
AuthUserFile /home/user/pass
require valid-user

The file "/home/user/pass" is a file containing usernames and password hashes. It can be created with the htpasswd tool. It supports several different password hashing schemes. An entry looks like this:

guest:$2y$05$sCcklxS8F1mvB/B2R090IOjqC0/i2FdhlAOJ0ujy.yfXswXIjQwGe

bcrypt hash with insane running time

By fuzzing htpasswd I recognized that some inputs would cause a very long running time of the application. This was caused by bcrypt hashes with a very long computing time.

The hash above uses the bcrypt hash function, indicated by the $2y. The number after that - the $05 - indicates the computing time of the hash function. It can have values between 04 and 31. The purpose of password hashing function is to make brute force attacks slow in case of a breach. Therefore it is desirable for password hashing functions to be slow (which is a very different requirement from many other use cases, where hash functions should be fast). However they also can't be too slow, because they still have to be calculated once for every login attempt.

A user who wants to cause disturbance on a server can now choose to set the computing time of the hash to an insanely large value. The hash value doesn't have to make any sense for this. A user can simply create a password file and change the 05 to 31:

guest:$2y$31$sCcklxS8F1mvB/B2R090IOjqC0/i2FdhlAOJ0ujy.yfXswXIjQwGe

For every login attempt with the right username the server will calculate the hash. The running time of bcrypt doubles with every increase of the computing time value. On my system calculating a hash with the maximum value 31 takes around 30 hours. Therefore with this a user can cause a server process to consume lots of resources for more than a day.

Two things are notable about the Apache behavior here:

• The hash calculation is neither limited by a connection timeout or by a termination of the connection. Once a connection tries to log in the hashing starts and won't stop even if the user closes his browser.
• The calculation happens within the server process. In common configurations this means it is not owned by the user, instead it's running under a system-wide user for the httpd server. Apache has functionalities to make sure that user scripts can only be executed under their own user account (SuExec), but these don't matter for the password hashing. This means any resource limit the server administrator has applied to the user account is irrelevant here.

So in summary a user that has the ability to host content on a server is able to severely slow down the whole server for more than a day with a single http request. This can only be stopped by restarting the server. For an administrator it will be nontrivial to figure out what's going on, because he'll only see a server process running amok, he won't have any easy way to trace that back to a user account.

Obviously the malicious user can also open multiple connections, but in my tests at least this didn't cause more resource exhaustion, so it seems all password hashing jobs were processed by the same Apache process. It will probably run longer, but I haven't investigated that in detail. This behavior may differ depending on the server configuration.

As an easy fix I proposed to the Apache developers to limit the computing time of the bcrypt hash. While a slower password hash can be a security improvement, there is no reason to have a hash function that runs longer than a few seconds at best. Here's a patch against apr-util - the package that contains Apache's bcrypt function - that will reject computing time values larger than 17. On my system that runs for 8 seconds, which some may argue is already too much. But it will prevent very simple DoS scenarios.

Is Apache inherently unable to protect against malicious users?

The Apache developers mostly rejected the notion that this is a security problem or any problem at all. I got statements that argue that Apache is inherently unable to defend against a user DoS'ing the server if a user is allowed to use .htaccess and that server operators shouldn't give untrusted users access to configuration files.

This is notable, because the ability to allow users a certain kind of configurability via .htaccess is actually a very distinctive feature of Apache and is widely used in shared web hosting environments. It has lost a lot of market share to Nginx in the past years, yet one of the major reasons some people don't want to switch to Nginx is that it has no comparable feature. In essence I get the feeling that the Apache developers consider the one feature that distincts them from many competitors as being inherently dangerous and discouraged.

One concern that was raised by one of the Apache developers was that even if the bcrypt password hash is capped in its execution time a user can do the same thing via password hashes supported by the C standard library. Apart from its own implementations Apache supports all password hashes provided by the crypt() function. However the concern here is much smaller. The maximum one can achieve is a running time of several minutes with the SHA512-based password hash and 999,999,999 iterations. I submitted a patch to Glibc that limits the execution time to a sane value (no reaction in the bug report yet).

To my surprise the musl libc already capped the running time of the SHA256 and SHA512 password hashing functions - and the code comment by Rich Felker explicitly mentions that this was done to avoid DoS.

Another issue that was raised and that I haven't investigated further is that a user could cause a similar denial of service by creating a crazy regular expression in the .htaccess file (the mod_rewrite functionality supports regular expression, see also Regular expression Denial of Service - ReDoS).

Separating from other issues

I want to point out that there are other security issues that shouldn't be confused with this one. There's an issue with hash table implementations named HashDoS (see this 29C3 talk) that can cause very slow running times of hash tables. It sounds similar, but it's a very different issue.

Another issue is the run time of password hashes for long inputs. Especially the SHA512-based password hash supported by glibc is exposed to this, because its running time grows quadratically with the size of the input. OpenSSH recently has restricted the password length to 1024 characters. There are probably other applications affected by this. Yet again this is a different issue.

Conclusion and comment

While the Apache developers were unwilling to accept that this is a potential security issue, they eventually patched it (the patch is incomplete, but we'll sort that out). So it will be fixed eventually.

The problem highlights several issues. It shows that the user permission concept of Apache is quite questionable. A user has a lot of control over the operations of a server - this also leads to other security problems in multi-user settings that are very hard to avoid. Ideally all user-controlled actions should run under the user account. This can be achieved with the nonstandard mpm-itk, but it can't properly handle use cases where different hosts are accessed over the same connection. It also seems non-ideal that server processes can execute code that can continue to run for hours after a connection has been terminated.

Apart from the specific issue in Apache the idea of abusing password hashes for Denial of Service might work in other settings. It can be applied everywhere where a user is in control of the hash of his password. Commonly this is not the case, but it would be worthwhile to investigate whether there are similar situations.

I've published some proof of concept examples here.

Logo source, Copyright: The Apache Software Foundation, License: Apache License, Version 2.0

When using fuzzing tools like afl a common challenge is how you can pass input to the interesting parts of the application you want to fuzz. In easy situations we have a tool that will accept our input as a file or via stdin. However sometimes this is not easily possible.

Let's have a look at Irssi, an irc chat client. The only input you can pass on the command line is a config file. Fuzzing Irssi 0.8.10 easily led to a segfault caused by a null pointer access. However while bugs in config file parsers probably still should be fixed, usually they are not very interesting. (There can be exceptions.)

So what else might be interesting? Irssi does some parsing on all output, e.g. due to color codes. However we can't just print text that is passed via the command line as an input file. We have to abuse Irssi's perl scripting capability for that.

We can place a simple perl script that will read a file (fuzzp.txt) and print it into Irssi's autorun directory (default location is ~/.Irssi/scripts/autorun/). We can then place some examples of Irssi color codes into the directory "in/". I have installed an afl/asan-instrumented Irssi into my system to /usr/local/, because for running perl scripts it needs more than just the executable. So we can run afl like this:

afl-fuzz -i in -o out -m none -f fuzzp.txt Irssi

afl will put the fuzzed output into fuzzp.txt and our autoload script will read it from there. Doing this lets us learn that the sequence "%[" causes Irssi to read an invalid memory byte. For reasons unclear to me this only happens if a script outputs this sequence, not if a user types it in. (This issue got CVE-2017-5196 assigned.)

We can go further and do a similar script that executes input as a command. Commands are things like "/QUIT" that control the application and the channel behavior. I named the input file fuzzc.txt, so we can place some simple Irssi commands into in/ and run:

afl-fuzz -i in -o out -m none -f fuzzc.txt Irssi

Thus we will now fuzz Irssi's command processing.

As we have seen, scripting functionality can be used to fuzz an application. So if you want to fuzz something and don't know how to pass input: See if there's a scripting functionality.

Irssi has issued a security advisory for several security vulnerabilities, including the out of bounds read mentioned above. All vulnerabilities and the config file parser segfault are fixed in 0.8.21 and 1.0.0.

By testing GNOME-related packages with Address Sanitizer I recently discovered several trivial to find bugs.

Two out of bounds bugs in the glib library were uncovered by running the test suite with Address Sanitizer enabled. One heap buffer overflow in the parameter parsing of gnome-session was uncovered by trying to start GNOME. Given that these bugs weren't discovered earlier means that most likely nobody ever used Address Sanitizer to test GNOME components.

I strongly recommend to GNOME and to other software communities to use Address Sanitizer testing in order to improve the quality of their software.

Out of bounds read in g_unichar_iswide_bsearch() / glib
Upstream bug report (again reported here)
Commit / fix
Fixed in 2.48.2.

Out of bounds read in token_stream_prepare() / glib
Upstream bug report
Commit / fix
Fixed in 2.48.0.

Heap buffer overflow in gnome-session
Upstream bug report
Commit / fix
Fixed in 3.20.2.

Last year in November I decided that it might be a good idea to fuzz the parsers of package management tools in Linux distributions. I quickly found a couple of issues in DPKG and RPM. For DPKG the process went very smooth. I reported them to Debian's security team, eight days later fixes and security advisories were published by both Debian and Ubuntu, the main distributions using DPKG. For RPM the process was a bit more difficult.

If you want to report a bug to RPM you first may wonder where to report it. The RPM webpage [1] is a trac installation which has its own bug tracker. However if you try to register an account there you'll get forwarded to an HTTPS site with an expired certificate that doesn't match the domain name. In case you are brave and tell your browser to ignore all warnings you'll be greeted by a broken-looking trac without any CSS. Should you proceed and create an account you will learn that this doesn't help you, because in order to be allowed to report a bug you first have to ask on the mailing list or in the IRC channel for permission [2]. That's probably the point where many well-meaning bug reporters give up.

Okay, but RPM originally stood for “Red Hat package manager” (I've been told that today it stands for RPM Package Manager), so maybe Red Hat feels responsible. So I reported three bugs with sample files triggering them to the Red Hat security team on November 20th. The answer was – to put it mildly – a bit dissatisfying. I'll just fully quote it: “Thanks for the report. We also received about 30+ crash reports in RPM from
a different reporter recently so processing all of them (yours and the
others) will take quite a bit of time. We simply don't have the resources
to spend hours upon hours analyzing all crash reports.”

Okay, so I wasn't the only one fuzzing RPM and the maybe bugs will be fixed some day. I waited. In the meantime I got contacted by another person who also had tried to report fuzzing bugs in RPM and who has made similar experiences (maybe the same person who reported the 30+ crashers, I don't know).

In February I decided to ask what the state of things is. I also gave them a 30 day period until I'd publish the bugs (I know that it's now long past that, I shouldn't have let this issue wait so long). I ended up having a call with a Red Hat security team member and exchanged a couple of mails. I learned that RPM has a Github repository [3], which contains fixes for some (but not all) of the issues I reported, however that's nowhere referenced on its webpage. I then fuzzed the current RPM git code again and found two more issues I also reported to the Red Hat security team.

Status today is that the latest release of RPM on its webpage – 4.12.0.1 - is from July 2015, so all of the bugs still affect this release. However it seems there is an unofficial 4.13 release that's nowhere to be found on the RPM webpage, but Red Hat is using it together with some fixes [4]. And the Github repository says the latest release is 4.12.0, so according to three different sources three different versions are the current one (4.12.0, 4.12.0.1, 4.13).

One of the bugs – a stack overflow (write) - is still present in the latest code on Github.

Commend and Conclusion

This blog post probably reads like a big rant about how unprofessional Red Hat is in handling potential security issues. But this is contrary to my usual experience. I often see Red Hat developers being very active in the free software security community and often contributing in a positive way. Quite simply I expect better from Red Hat. This is not some dubious Enterprise vendor where I wouldn't be the least bit surprised of such a reaction.

The development process of RPM seems to be totally chaotic, it's neither clear where one reports bugs nor where one gets the latest code and security bugs don't get fixed within a reasonable time.

There's been some recent events that make me feel especially worried about this: An unknown person has created an entry in the Libarchive issue tracker [5] that points to an anonymous document [6] with a very detailed description of various security weaknesses in the FreeBSD update process (most of them are still unfixed). The most worrying thing about this is however that the anonymous post mentions the existence similar documents affecting multiple Linux distributions. These documents haven't shown up publicly yet and given the unclear nature of this incident it's hard to know whether they ever will become public or exist at all. But this should still be reason enough to have a closer look at potential security vulnerabilities in all pieces of Linux package management systems.

I haven't analyzed the RPM installation process in detail, so I can't say how likely it is that the RPM tool ever sees a malformed input file. It seems downloads happen over HTTP, but the first thing that happens is a signature check. As the signature is part of the RPM file it already needs to be parsed for this. The exact impact of these bugs would require further analysis. But independent of how likely this is I think the parser in such a crucial piece of software should be robust. It should be safe to use the rpm tool to show info about a file on the command line.


[1] http://rpm.org/
[2] http://rpm.org/wiki/ReportingBugs
[3] https://github.com/rpm-software-management/rpm
[4] http://pkgs.fedoraproject.org/cgit/rpms/rpm.git/diff/rpm-4.13.0-rpmtd-out-of-bounds.patch?h=f22&id=165614f3dd42caa188f78b55e7723dad2900b2f4
[5] https://github.com/libarchive/libarchive/issues/743
[6] https://gist.github.com/anonymous/e48209b03f1dd9625a992717e7b89c4f

All bugs were found with the help of american fuzzy lop. Here are the bugs:

Stack Overflow in glob() / rpmglob.c.
Sample file (test with rpm -i [input]):
https://crashes.fuzzing-project.org/rpm-stackoverflow-glob.rpm
Unfixed in the current Git code.

Heap out of bounds read in headerVerifyInfo() / header.c.
Sample file (test with “rpm -i [input]”):
https://crashes.fuzzing-project.org/rpm-heap-oob-read-headerVerifyInfo.rpm
Git commit:
https://github.com/rpm-software-management/rpm/commit/8e847d52c811e9a57239e18672d40f781e0ec48e

Null pointer access / segfault in stringFormat() / formats.c
Sample file (test with “rpm -i [input]”):
https://crashes.fuzzing-project.org/rpm-nullptr-rpmtdFormat.rpm
Git commit:
https://github.com/rpm-software-management/rpm/commit/cddf43a56f19711866371f02f378dc4095b0fadd

Out of bounds read in rpmtdGetNumber() / rpmtd.c
Sample file (test with “rpm -qi -p -- [input]”)
https://crashes.fuzzing-project.org/rpm-heap-oob-read-rpmtdGetNumber.rpm
Git commit:
https://github.com/rpm-software-management/rpm/commit/b722cf86200505b3e3fcbb2095c4ff61f1f5a2ab

Finally one annoying thing to admit: In my original report I included another segfault in headerVerifyInfo() with unclear reasons. However I am now unable to reproduce this one. It may be due to compiler options, different command line parameters or dependencies on my system that have changed. For completeness I'm still providing the sample file:
https://crashes.fuzzing-project.org/rpm-segfault-headerVerifyInfo.rpm
(Ideally the RPM developers should include all those sample files in a test suite they regularly run against an address sanitizer build of RPM.)

Please also note that I expect this list to be incomplete and there are likely more issues that could be uncovered with further fuzzing. I'll test that once all the existing issues are fixed.

I reported several bugs discovered with afl in p7zip, the portable command line version of the 7-Zip archiving tool, one null pointer access causing a segfault and one out of bounds heap read.

At the time I reported the first issues these were already fixed in the main 7-Zip code, but the portable version hadn't been updated yet. Now there is version 16.02, unfortunately there are still issues left.

Null pointer access / segfault
An access to a null pointer in NArchive::N7z::CStreamSwitch::Set can cause p7zip to crash.

Out of bounds read (bug is not publicly visible yet)
An out of bounds heap read access can happen in the function NArchive::N7z::CDecoder::Decode.

Null pointer acctess / segfault (unfixed in 16.02)
A memory allocation isn't checked for a failing allocation. If the file's header indicates a large data size the allocation will fail and subsequently p7zip will try to write to a null pointer. This will likely only lead to a crash and not to further security issues, however this problem seems prevalent in the 7-zip codebase - there are several memory allocations without a check for failures.

p7zip is often used in security appliances to scan inputs, because it supports a large number of archiving file formats. Vulnerabilities in it have played a role in previous attacks on Fireeye appliances. Therefore it's worrying that it's so easy to find crashers in it via fuzzing.

There were also other issues in 7-Zip discovered lately.

I recently wrote about a large number of bugs and potential security issues in libarchive. The release 3.2.0 missed one fix for an out of bounds read in the rar parser. Also I discovered one additional signed integer overflow issue with ubsan. Both issues are now fixed in libarchive 3.2.1.
All issues were discovered with the help of american fuzzy lop.

Out of bounds heap read in RAR parser
Sample rar file
CVE-2015-8934

Signed integer overflow in ISO parser
Sample ISO file
CVE-2016-5844

Also a couple of other security issues in libarchive were found by Cisco.

With the release of version 3.2.1 I consider libarchive to be reasonably robust against fuzzing. I've tested all supported file formats and fuzzed each one with afl/asan for at least one day. Of course that doesn't mean that no security issues are left - but the easy to find ones should be wiped out.

Recently the ImageTragick vulnerability shed some light on the security status of ImageMagick.

This made me wonder how resilient to fuzzing ImageMagick is these days. It's pretty much a posterchild example for a good fuzzing target: Lots of supported complex binary file formats.

I already did some fuzzing on ImageMagick, but as far as I remember that was before I used american fuzzy lop and was done with zzuf. I was also aware that others did some more thorough fuzzing on ImageMagick.

What I did now was relatively simple: I took a trivial, few pixels PNG and used ImageMagick's "convert" tool to convert it into all file formats that have both read and write support in ImageMagick. I used that to run a fuzzing job with afl and asan. By design ImageMagick will sometimes do huge memory allocations, these can be prevented by setting limits for the width, height and memory usage in the policy.xml file.

I discovered one heap buffer overflow in the PICT parser and one heap out of bounds read in the PSD parser. Given how big the attack surface is this is not terrible, but it shows that despite previous efforts there's still potential to fuzz ImageMagick.

Sample file for heap buffer overflow in WritePixelCachePixels() (PICT format)
Git commit / fix

Sample file for heap out of bounds read in PushShortPixel() (PSD format)
Git commit / fix

Both issues have been fixed in the versions 6.9.4-0 and 7.0.1-2. In the meantime new versions (6.9.4-1, 7.0.1-3) came out that, as far as I understand the ChangeLog, remove another potential vector for the ImageTragick vulnerabilities, so you should preferrably update to those.

I lately fuzzed various filesystem check tools. This uncovered a number of issues in dosfstools / fsck.fat that have now been fixed in the new version 4.0. All issues were found with american fuzzy lop and address sanitizer.

Global out of bounds read file_stat() / check_dir()
Git commit / fix

Unclear invalid memory access in get_fat()
Git commit / fix
CVE-2015-8872

Heap overflow in read_fat()
Heap out of bounds read in get_fat()
Git commit / fix for both issues
CVE-2016-4804

These bugs can pose a security risk if a system automatically checks attached storage media with fsck or in situations where filesystems on untrusted devices get checked. The new version dosfstools 4.0 fixes all four bugs.

PHP recently released the security updates 7.0.6, 5.6.21 and 5.5.35 that fix - among a couple of other security issues - a couple of out of bounds issues in the EXIF parser I reported.

Sample file (CVE-2016-4542)
Sample file (CVE-2016-4543)
Sample file (also CVE-2016-4543)
Sample file (CVE-2016-4544)
Bug report
Git commit

These bugs are not exceptionally interesting, but there is something to know when fuzzing PHP and the same applies also to a number of other applications: It uses a custom memory allocator that can sometimes mask issues from memory safety tools like Address Sanitizer. It is therefore good to know about them and disable them during fuzz testing.

With PHP this can be circumvented by setting the environment variable USE_ZEND_ALLOC=0 while fuzzing. This disables the Zend allocator from PHP and uses normal libc memory allocation calls.

I started documenting such issues and workarounds.

Lately a couple of updates have fixed issues that I discovered while testing software with Address Sanitizer (ASAN). I'll publish these combined. Once again these show that a lot of software hasn't been tested with ASAN and what kind of bugs it can uncover.

In syslog-ng a function returns an error -1 that gets stored in an unsigned variable. This cannot be stored and a subsequent check for the error value fails, this later leads to a heap out of bounds memory read. This was fixed in syslog-ng 3.7.3.
Upstream bug report
Pull request / patch
syslog-ng 3.7.3 release notes

In the monit system monitoring tool a heap overflow was discovered. A call to sscanf would write a string of length 256 into a 256 byte long buffer. However this fails to consider the trailing zero byte of that string, therefore causing a one byte heap overflow. This was fixed in monit 5.16.
Commit / fix

In the compilation tool cmake a use after free error was found. This happened becaused a variable was used in a comparison and was free'd during that comparison operation. This was fixed in cmake 3.5.0.
Upstream bug report
Commit / fix

The pidgin-otr plugin version 4.0.2 fixes a heap use after free error. The bug is triggered when a user tries to authenticate a buddy and happens in the function create_smp_dialog.

The bug was discovered with Address Sanitizer. This is yet another example why all C/C++ code should be tested with Address Sanitizer enabled.

This bug was already independently discovered and reported by Stefan Sperling in the otr bug tracker.

Independend of this bug another more severe bug in Libotr itself was also disclosed today, please make sure you update both libotr (4.1.1) and the pidgin-otr plugin (4.0.2).

Upstream bug report (contains Address Sanitizer stack trace)
Commit / fix
CVE-2015-8833

The Nettle library is a library for basic cryptographic functions. Its most prominent user is GnuTLS.

Through fuzzing of elliptic curve scalar multiplications (multiplying a point on an elliptic curve with a scalar number) I discovered two carry propagation bugs that would lead the cauculations to produce wrong results. They affect the NIST P-256 and P-384 curves. The P-256 bug is in the C code and affects multiple architectures. The P-384 bug is in the assembly code and only affects 64 bit x86. Both bugs were found with the help of american fuzzy lop.

While analyzing these bugs Nettle developer Niels Möller discovered another carry propagation bug in P-256 that was fixed in the same commit (CVE-2015-8805). Nettle 3.2 fixes all three bugs.

The impact is currently unclear, but miscalculations in cryptographic functions should generally be considered security issues. I'd like to encourage cryptographers to try to analyze whether these bugs can lead to cryptographic breaks.

I have published a code example on how to fuzz elliptic curve multiplications. It can compare the output of OpenSSL with either Nettle or NSS. It currently works only with prime field curves, but it can probably be adapted to other curves.

P-256 bug:
Mailing list post with code sample
Commit / fix for P-256 bug
CVE-2015-8803

P-384 bug:
Mailing list post with code sample
Commit / fix for P-384 bug
CVE-2015-8804

Nettle 3.2 release notes