The Fuzzing Project

Recently I shared some information about potential memory safety bugs in the Apache web server together with Craig Young. One issue that came up in that context is the so-called pool allocator Apache is using.

What is this pool allocator? Apache’s APR library has a feature where you can allocate a pool, which is a larger area of memory, and then do memory allocations within that pool. It’s essentially a separate memory allocation functionality by the library. Similar concepts exist in other software.

Why would anyone do that? It might bring performance benefits to have memory allocation that’s optimized for a specific application. It also can make programming more convenient when you can allocate many small buffers in a pool and then not bothering about freeing each one of then and instead just free the whole pool with all allocations within.

There’s a disadvantage with the pool allocator, and that is that it may hide bugs. Let’s look at a simple code example:



We can compile this with:
gcc $(pkg-config --cflags --libs apr-1) input.c

What we’re doing here is that we create a pool p and we create two buffers (b1, b2) within that pool, each six byte. Now we fill those buffers with strings. However for b1 we fill it with a string that is larger than its size. This is thus a classic buffer overflow. The printf at the end which outputs both strings will show garbled output, because the two buffers interfere.

Now the question is how do we find such bugs? Of course we can carefully analyze the code, and in the simple example above this is easy to do. But in complex software such bugs are hard to find manually, therefore there are tools to detect unsafe memory access (e.g. buffer overflows, use after free) during execution. The state of the art tool is Address Sanitizer (ASAN). If you write C code and don’t use it for testing yet, you should start doing so now.

Address Sanitizer is part of both the gcc and clang compiler and it can be enabled by passing -fsanitize=address on the command line. We’ll also add -g, which adds debugging information and will give us better error messages. So let’s try:
gcc -g -fsanitize=address $(pkg-config --cflags --libs apr-1) input.c

If you try this you will find out that nothing has changed. We still see the garbled string and Address Sanitizer has not detected the buffer overflow.

Let’s try rewriting the above code in plain C without the pool allocator:



If we compile and run this with ASAN it will give us a nice error message that tells us what’s going on:



So why didn’t the error show up when we used the pool allocator? The reason is that ASAN is built on top of the normal C memory allocation functions like malloc/free. It does not know anything about APR’s pools. From ASAN’s point of view the pool is just one large block of memory, and what’s happening inside is not relevant.

Thus we have a buffer overflow, but the state of the art tool to detect buffer overflows is unable to detect it. This is obviously not good, it means the pool allocator takes one of the most effective ways to discover an important class of security bugs away from us.

If you’re looking for solutions for that problem you may find old documentation about "Debugging Memory Allocation in APR". However it relies on flags that have been removed from the APR library, so it’s not helpful. However there’s a not very well documented option of the APR library that allows us to gain memory safety checks back. Passing --enable-pool-debug=yes to the configure script will effectively disable the pool allocator and create a separate memory allocation for each call to the pool allocator.

If we compile our first example again, this time with the pool debugger and ASAN, we’ll see the error:



Apache is not alone in having a custom memory allocation that can hide bugs. Mozilla’s NSPR and NSS libraries have something called an Arena Pool, Glib has memory slices and PHP has the Zend allocator. All of them have the potential of hiding memory safety bugs from ASAN, yet luckily all have an option to be turned off for testing. I maintain a collection of information about such custom allocators and how to disable them.

But back to Apache. When we started reporting use after free bugs we saw with the debugging option for the pool allocator we learned from the Apache developers that there are incompatibilities with the http2 module and the pool debugger. This has led to replies after our disclosure that these are non-issues, because nobody should run the pool debugger in production.

It should be noted that we were also able to reproduce some bugs without the pool debugger in the latest Apache version (we have shared this information with Apache and will share it publicly later), and that indeed it seems some people did run the pool debugger in production (OpenBSD).

But I have another problem with this. If we consider that parts of the current Apache code are incompatible with the APR pool debugger then we end up with an unfortunate situation: If a tool like ASAN reports memory safety bugs with the pool debugger we don’t know if they are real issues or just incompatibilities. If we turn off the pool debugger we won’t see most of the memory safety bugs.

That’s a situation where testing Apache for memory safety bugs becomes practically very difficult. In my opinion that by itself is a worrying and severe security issue.

Image source: Dreamstime, CC0