When your Memory Allocator hides Security Bugs
Recently I shared some information about potential memory safety bugs in the Apache web server together with Craig Young. One issue that came up in that context is the so-called pool allocator Apache is using.What is this pool allocator? Apache’s APR library has a feature where you can allocate a pool, which is a larger area of memory, and then do memory allocations within that pool. It’s essentially a separate memory allocation functionality by the library. Similar concepts exist in other software.
Why would anyone do that? It might bring performance benefits to have memory allocation that’s optimized for a specific application. It also can make programming more convenient when you can allocate many small buffers in a pool and then not bothering about freeing each one of then and instead just free the whole pool with all allocations within.
There’s a disadvantage with the pool allocator, and that is that it may hide bugs. Let’s look at a simple code example:
#include <apr_pools.h>
#include <stdio.h>
#include <string.h>
int main() {
apr_pool_t *p;
char *b1, *b2;
apr_initialize();
apr_pool_create(&p, NULL);
b1 = apr_palloc(p, 6);
b2 = apr_palloc(p, 6);
strcpy(b1, "This is too long");
strcpy(b2, "Short");
printf("%s %s\n", b1, b2);
}
#include <stdio.h>
#include <string.h>
int main() {
apr_pool_t *p;
char *b1, *b2;
apr_initialize();
apr_pool_create(&p, NULL);
b1 = apr_palloc(p, 6);
b2 = apr_palloc(p, 6);
strcpy(b1, "This is too long");
strcpy(b2, "Short");
printf("%s %s\n", b1, b2);
}
We can compile this with:
gcc $(pkg-config --cflags --libs apr-1) input.cWhat we’re doing here is that we create a pool p and we create two buffers (b1, b2) within that pool, each six byte. Now we fill those buffers with strings. However for b1 we fill it with a string that is larger than its size. This is thus a classic buffer overflow. The
printf at the end which outputs both strings will show garbled output, because the two buffers interfere.Now the question is how do we find such bugs? Of course we can carefully analyze the code, and in the simple example above this is easy to do. But in complex software such bugs are hard to find manually, therefore there are tools to detect unsafe memory access (e.g. buffer overflows, use after free) during execution. The state of the art tool is Address Sanitizer (ASAN). If you write C code and don’t use it for testing yet, you should start doing so now.
Address Sanitizer is part of both the gcc and clang compiler and it can be enabled by passing
-fsanitize=address on the command line. We’ll also add -g, which adds debugging information and will give us better error messages. So let’s try:gcc -g -fsanitize=address $(pkg-config --cflags --libs apr-1) input.cIf you try this you will find out that nothing has changed. We still see the garbled string and Address Sanitizer has not detected the buffer overflow.
Let’s try rewriting the above code in plain C without the pool allocator:
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
int main() {
char *b1, *b2;
b1 = malloc(6);
b2 = malloc(6);
strcpy(b1, "This is too long");
strcpy(b2, "Short");
printf("%s %s\n", b1, b2);
}
#include <string.h>
#include <stdlib.h>
int main() {
char *b1, *b2;
b1 = malloc(6);
b2 = malloc(6);
strcpy(b1, "This is too long");
strcpy(b2, "Short");
printf("%s %s\n", b1, b2);
}
If we compile and run this with ASAN it will give us a nice error message that tells us what’s going on:
==9319==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000016 at pc 0x7f81fdd08c9d bp 0x7ffe82881930 sp 0x7ffe828810d8
WRITE of size 17 at 0x602000000016 thread T0
#0 0x7f81fdd08c9c in __interceptor_memcpy /var/tmp/portage/sys-devel/gcc-8.2.0-r6/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:737
#1 0x5636994851e0 in main /tmp/input.c:10
#2 0x7f81fdb204ea in __libc_start_main (/lib64/libc.so.6+0x244ea)
#3 0x5636994850e9 in _start (/tmp/a.out+0x10e9)
0x602000000016 is located 0 bytes to the right of 6-byte region [0x602000000010,0x602000000016)
allocated by thread T0 here:
#0 0x7f81fddb6b10 in __interceptor_malloc /var/tmp/portage/sys-devel/gcc-8.2.0-r6/work/gcc-8.2.0/libsanitizer/asan/asan_malloc_linux.cc:86
#1 0x5636994851b6 in main /tmp/input.c:7
#2 0x7f81fdb204ea in __libc_start_main (/lib64/libc.so.6+0x244ea)
WRITE of size 17 at 0x602000000016 thread T0
#0 0x7f81fdd08c9c in __interceptor_memcpy /var/tmp/portage/sys-devel/gcc-8.2.0-r6/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:737
#1 0x5636994851e0 in main /tmp/input.c:10
#2 0x7f81fdb204ea in __libc_start_main (/lib64/libc.so.6+0x244ea)
#3 0x5636994850e9 in _start (/tmp/a.out+0x10e9)
0x602000000016 is located 0 bytes to the right of 6-byte region [0x602000000010,0x602000000016)
allocated by thread T0 here:
#0 0x7f81fddb6b10 in __interceptor_malloc /var/tmp/portage/sys-devel/gcc-8.2.0-r6/work/gcc-8.2.0/libsanitizer/asan/asan_malloc_linux.cc:86
#1 0x5636994851b6 in main /tmp/input.c:7
#2 0x7f81fdb204ea in __libc_start_main (/lib64/libc.so.6+0x244ea)
So why didn’t the error show up when we used the pool allocator? The reason is that ASAN is built on top of the normal C memory allocation functions like
malloc/free. It does not know anything about APR’s pools. From ASAN’s point of view the pool is just one large block of memory, and what’s happening inside is not relevant.Thus we have a buffer overflow, but the state of the art tool to detect buffer overflows is unable to detect it. This is obviously not good, it means the pool allocator takes one of the most effective ways to discover an important class of security bugs away from us.
If you’re looking for solutions for that problem you may find old documentation about "Debugging Memory Allocation in APR". However it relies on flags that have been removed from the APR library, so it’s not helpful. However there’s a not very well documented option of the APR library that allows us to gain memory safety checks back. Passing
--enable-pool-debug=yes to the configure script will effectively disable the pool allocator and create a separate memory allocation for each call to the pool allocator.If we compile our first example again, this time with the pool debugger and ASAN, we’ll see the error:
==20228==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000000016 at pc 0x7fe2e625dc9d bp 0x7ffe8419a180 sp 0x7ffe84199928
WRITE of size 17 at 0x602000000016 thread T0
#0 0x7fe2e625dc9c in __interceptor_memcpy /var/tmp/portage/sys-devel/gcc-8.2.0-r6/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:737
#1 0x55fe439d132c in main /tmp/input.c:15
#2 0x7fe2e5fc34ea in __libc_start_main (/lib64/libc.so.6+0x244ea)
#3 0x55fe439d1129 in _start (/tmp/a.out+0x1129)
0x602000000016 is located 0 bytes to the right of 6-byte region [0x602000000010,0x602000000016)
allocated by thread T0 here:
#0 0x7fe2e630bb10 in __interceptor_malloc /var/tmp/portage/sys-devel/gcc-8.2.0-r6/work/gcc-8.2.0/libsanitizer/asan/asan_malloc_linux.cc:86
#1 0x7fe2e6203157 (/usr/lib64/libapr-1.so.0+0x1f157)
#2 0x7fe2e5fc34ea in __libc_start_main (/lib64/libc.so.6+0x244ea)
WRITE of size 17 at 0x602000000016 thread T0
#0 0x7fe2e625dc9c in __interceptor_memcpy /var/tmp/portage/sys-devel/gcc-8.2.0-r6/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:737
#1 0x55fe439d132c in main /tmp/input.c:15
#2 0x7fe2e5fc34ea in __libc_start_main (/lib64/libc.so.6+0x244ea)
#3 0x55fe439d1129 in _start (/tmp/a.out+0x1129)
0x602000000016 is located 0 bytes to the right of 6-byte region [0x602000000010,0x602000000016)
allocated by thread T0 here:
#0 0x7fe2e630bb10 in __interceptor_malloc /var/tmp/portage/sys-devel/gcc-8.2.0-r6/work/gcc-8.2.0/libsanitizer/asan/asan_malloc_linux.cc:86
#1 0x7fe2e6203157 (/usr/lib64/libapr-1.so.0+0x1f157)
#2 0x7fe2e5fc34ea in __libc_start_main (/lib64/libc.so.6+0x244ea)
Apache is not alone in having a custom memory allocation that can hide bugs. Mozilla’s NSPR and NSS libraries have something called an Arena Pool, Glib has memory slices and PHP has the Zend allocator. All of them have the potential of hiding memory safety bugs from ASAN, yet luckily all have an option to be turned off for testing. I maintain a collection of information about such custom allocators and how to disable them.
But back to Apache. When we started reporting use after free bugs we saw with the debugging option for the pool allocator we learned from the Apache developers that there are incompatibilities with the http2 module and the pool debugger. This has led to replies after our disclosure that these are non-issues, because nobody should run the pool debugger in production.
It should be noted that we were also able to reproduce some bugs without the pool debugger in the latest Apache version (we have shared this information with Apache and will share it publicly later), and that indeed it seems some people did run the pool debugger in production (OpenBSD).
But I have another problem with this. If we consider that parts of the current Apache code are incompatible with the APR pool debugger then we end up with an unfortunate situation: If a tool like ASAN reports memory safety bugs with the pool debugger we don’t know if they are real issues or just incompatibilities. If we turn off the pool debugger we won’t see most of the memory safety bugs.
That’s a situation where testing Apache for memory safety bugs becomes practically very difficult. In my opinion that by itself is a worrying and severe security issue.
Image source: Dreamstime, CC0
I recently did some testing of the qpdf library with afl and libfuzzer. I discovered an input sample that would generate a high CPU load spike and eventually after several minutes cause an out of memory error. It looked like the parser was caught in some kind of endless loop.
The way the Apache httpd web server handles password hashes can be abused by a malicious user on a server to cause resource exhaustion and denial of service of the web server. I reported this a while ago to the Apache security team - which led to a lengthy discussion where I was a bit appalled about some of the statements I got from the Apache developers. They can be summed up in a way that major use cases of Apache - especially in web hosting environments - are not recommended by their developers.
.svg){kind=link}