The Fuzzing Project

The way the Apache httpd web server handles password hashes can be abused by a malicious user on a server to cause resource exhaustion and denial of service of the web server. I reported this a while ago to the Apache security team - which led to a lengthy discussion where I was a bit appalled about some of the statements I got from the Apache developers. They can be summed up in a way that major use cases of Apache - especially in web hosting environments - are not recommended by their developers.

Apache supports HTTP basic authentication, a simple login mechanism with username and password that is part of the HTTP protocol. This can be configured via the .htaccess file on a web server. A very simple htaccess file looks like this:

AuthType Basic
AuthName "privat"
AuthUserFile /home/user/pass
require valid-user

The file "/home/user/pass" is a file containing usernames and password hashes. It can be created with the htpasswd tool. It supports several different password hashing schemes. An entry looks like this:

guest:$2y$05$sCcklxS8F1mvB/B2R090IOjqC0/i2FdhlAOJ0ujy.yfXswXIjQwGe

bcrypt hash with insane running time

By fuzzing htpasswd I recognized that some inputs would cause a very long running time of the application. This was caused by bcrypt hashes with a very long computing time.

The hash above uses the bcrypt hash function, indicated by the $2y. The number after that - the $05 - indicates the computing time of the hash function. It can have values between 04 and 31. The purpose of password hashing function is to make brute force attacks slow in case of a breach. Therefore it is desirable for password hashing functions to be slow (which is a very different requirement from many other use cases, where hash functions should be fast). However they also can't be too slow, because they still have to be calculated once for every login attempt.

A user who wants to cause disturbance on a server can now choose to set the computing time of the hash to an insanely large value. The hash value doesn't have to make any sense for this. A user can simply create a password file and change the 05 to 31:

guest:$2y$31$sCcklxS8F1mvB/B2R090IOjqC0/i2FdhlAOJ0ujy.yfXswXIjQwGe

For every login attempt with the right username the server will calculate the hash. The running time of bcrypt doubles with every increase of the computing time value. On my system calculating a hash with the maximum value 31 takes around 30 hours. Therefore with this a user can cause a server process to consume lots of resources for more than a day.

Two things are notable about the Apache behavior here:

• The hash calculation is neither limited by a connection timeout or by a termination of the connection. Once a connection tries to log in the hashing starts and won't stop even if the user closes his browser.
• The calculation happens within the server process. In common configurations this means it is not owned by the user, instead it's running under a system-wide user for the httpd server. Apache has functionalities to make sure that user scripts can only be executed under their own user account (SuExec), but these don't matter for the password hashing. This means any resource limit the server administrator has applied to the user account is irrelevant here.

So in summary a user that has the ability to host content on a server is able to severely slow down the whole server for more than a day with a single http request. This can only be stopped by restarting the server. For an administrator it will be nontrivial to figure out what's going on, because he'll only see a server process running amok, he won't have any easy way to trace that back to a user account.

Obviously the malicious user can also open multiple connections, but in my tests at least this didn't cause more resource exhaustion, so it seems all password hashing jobs were processed by the same Apache process. It will probably run longer, but I haven't investigated that in detail. This behavior may differ depending on the server configuration.

As an easy fix I proposed to the Apache developers to limit the computing time of the bcrypt hash. While a slower password hash can be a security improvement, there is no reason to have a hash function that runs longer than a few seconds at best. Here's a patch against apr-util - the package that contains Apache's bcrypt function - that will reject computing time values larger than 17. On my system that runs for 8 seconds, which some may argue is already too much. But it will prevent very simple DoS scenarios.

Is Apache inherently unable to protect against malicious users?

The Apache developers mostly rejected the notion that this is a security problem or any problem at all. I got statements that argue that Apache is inherently unable to defend against a user DoS'ing the server if a user is allowed to use .htaccess and that server operators shouldn't give untrusted users access to configuration files.

This is notable, because the ability to allow users a certain kind of configurability via .htaccess is actually a very distinctive feature of Apache and is widely used in shared web hosting environments. It has lost a lot of market share to Nginx in the past years, yet one of the major reasons some people don't want to switch to Nginx is that it has no comparable feature. In essence I get the feeling that the Apache developers consider the one feature that distincts them from many competitors as being inherently dangerous and discouraged.

One concern that was raised by one of the Apache developers was that even if the bcrypt password hash is capped in its execution time a user can do the same thing via password hashes supported by the C standard library. Apart from its own implementations Apache supports all password hashes provided by the crypt() function. However the concern here is much smaller. The maximum one can achieve is a running time of several minutes with the SHA512-based password hash and 999,999,999 iterations. I submitted a patch to Glibc that limits the execution time to a sane value (no reaction in the bug report yet).

To my surprise the musl libc already capped the running time of the SHA256 and SHA512 password hashing functions - and the code comment by Rich Felker explicitly mentions that this was done to avoid DoS.

Another issue that was raised and that I haven't investigated further is that a user could cause a similar denial of service by creating a crazy regular expression in the .htaccess file (the mod_rewrite functionality supports regular expression, see also Regular expression Denial of Service - ReDoS).

Separating from other issues

I want to point out that there are other security issues that shouldn't be confused with this one. There's an issue with hash table implementations named HashDoS (see this 29C3 talk) that can cause very slow running times of hash tables. It sounds similar, but it's a very different issue.

Another issue is the run time of password hashes for long inputs. Especially the SHA512-based password hash supported by glibc is exposed to this, because its running time grows quadratically with the size of the input. OpenSSH recently has restricted the password length to 1024 characters. There are probably other applications affected by this. Yet again this is a different issue.

Conclusion and comment

While the Apache developers were unwilling to accept that this is a potential security issue, they eventually patched it (the patch is incomplete, but we'll sort that out). So it will be fixed eventually.

The problem highlights several issues. It shows that the user permission concept of Apache is quite questionable. A user has a lot of control over the operations of a server - this also leads to other security problems in multi-user settings that are very hard to avoid. Ideally all user-controlled actions should run under the user account. This can be achieved with the nonstandard mpm-itk, but it can't properly handle use cases where different hosts are accessed over the same connection. It also seems non-ideal that server processes can execute code that can continue to run for hours after a connection has been terminated.

Apart from the specific issue in Apache the idea of abusing password hashes for Denial of Service might work in other settings. It can be applied everywhere where a user is in control of the hash of his password. Commonly this is not the case, but it would be worthwhile to investigate whether there are similar situations.

I've published some proof of concept examples here.

Logo source, Copyright: The Apache Software Foundation, License: Apache License, Version 2.0