Multiple vulnerabilities in RPM – and a rant
Last year in November I decided that it might be a good idea to fuzz the parsers of package management tools in Linux distributions. I quickly found a couple of issues in DPKG and RPM. For DPKG the process went very smooth. I reported them to Debian's security team, eight days later fixes and security advisories were published by both Debian and Ubuntu, the main distributions using DPKG. For RPM the process was a bit more difficult.
If you want to report a bug to RPM you first may wonder where to report it. The RPM webpage [1] is a trac installation which has its own bug tracker. However if you try to register an account there you'll get forwarded to an HTTPS site with an expired certificate that doesn't match the domain name. In case you are brave and tell your browser to ignore all warnings you'll be greeted by a broken-looking trac without any CSS. Should you proceed and create an account you will learn that this doesn't help you, because in order to be allowed to report a bug you first have to ask on the mailing list or in the IRC channel for permission [2]. That's probably the point where many well-meaning bug reporters give up.
Okay, but RPM originally stood for “Red Hat package manager” (I've been told that today it stands for RPM Package Manager), so maybe Red Hat feels responsible. So I reported three bugs with sample files triggering them to the Red Hat security team on November 20th. The answer was – to put it mildly – a bit dissatisfying. I'll just fully quote it: “Thanks for the report. We also received about 30+ crash reports in RPM from
a different reporter recently so processing all of them (yours and the
others) will take quite a bit of time. We simply don't have the resources
to spend hours upon hours analyzing all crash reports.”
Okay, so I wasn't the only one fuzzing RPM and the maybe bugs will be fixed some day. I waited. In the meantime I got contacted by another person who also had tried to report fuzzing bugs in RPM and who has made similar experiences (maybe the same person who reported the 30+ crashers, I don't know).
In February I decided to ask what the state of things is. I also gave them a 30 day period until I'd publish the bugs (I know that it's now long past that, I shouldn't have let this issue wait so long). I ended up having a call with a Red Hat security team member and exchanged a couple of mails. I learned that RPM has a Github repository [3], which contains fixes for some (but not all) of the issues I reported, however that's nowhere referenced on its webpage. I then fuzzed the current RPM git code again and found two more issues I also reported to the Red Hat security team.
Status today is that the latest release of RPM on its webpage – 4.12.0.1 - is from July 2015, so all of the bugs still affect this release. However it seems there is an unofficial 4.13 release that's nowhere to be found on the RPM webpage, but Red Hat is using it together with some fixes [4]. And the Github repository says the latest release is 4.12.0, so according to three different sources three different versions are the current one (4.12.0, 4.12.0.1, 4.13).
One of the bugs – a stack overflow (write) - is still present in the latest code on Github.
Commend and Conclusion
This blog post probably reads like a big rant about how unprofessional Red Hat is in handling potential security issues. But this is contrary to my usual experience. I often see Red Hat developers being very active in the free software security community and often contributing in a positive way. Quite simply I expect better from Red Hat. This is not some dubious Enterprise vendor where I wouldn't be the least bit surprised of such a reaction.
The development process of RPM seems to be totally chaotic, it's neither clear where one reports bugs nor where one gets the latest code and security bugs don't get fixed within a reasonable time.
There's been some recent events that make me feel especially worried about this: An unknown person has created an entry in the Libarchive issue tracker [5] that points to an anonymous document [6] with a very detailed description of various security weaknesses in the FreeBSD update process (most of them are still unfixed). The most worrying thing about this is however that the anonymous post mentions the existence similar documents affecting multiple Linux distributions. These documents haven't shown up publicly yet and given the unclear nature of this incident it's hard to know whether they ever will become public or exist at all. But this should still be reason enough to have a closer look at potential security vulnerabilities in all pieces of Linux package management systems.
I haven't analyzed the RPM installation process in detail, so I can't say how likely it is that the RPM tool ever sees a malformed input file. It seems downloads happen over HTTP, but the first thing that happens is a signature check. As the signature is part of the RPM file it already needs to be parsed for this. The exact impact of these bugs would require further analysis. But independent of how likely this is I think the parser in such a crucial piece of software should be robust. It should be safe to use the rpm tool to show info about a file on the command line.
[1] http://rpm.org/
[2] http://rpm.org/wiki/ReportingBugs
[3] https://github.com/rpm-software-management/rpm
[4] http://pkgs.fedoraproject.org/cgit/rpms/rpm.git/diff/rpm-4.13.0-rpmtd-out-of-bounds.patch?h=f22&id=165614f3dd42caa188f78b55e7723dad2900b2f4
[5] https://github.com/libarchive/libarchive/issues/743
[6] https://gist.github.com/anonymous/e48209b03f1dd9625a992717e7b89c4f
All bugs were found with the help of american fuzzy lop. Here are the bugs:
Stack Overflow in glob() / rpmglob.c.
Sample file (test with rpm -i [input]):
https://crashes.fuzzing-project.org/rpm-stackoverflow-glob.rpm
Unfixed in the current Git code.
Heap out of bounds read in headerVerifyInfo() / header.c.
Sample file (test with “rpm -i [input]”):
https://crashes.fuzzing-project.org/rpm-heap-oob-read-headerVerifyInfo.rpm
Git commit:
https://github.com/rpm-software-management/rpm/commit/8e847d52c811e9a57239e18672d40f781e0ec48e
Null pointer access / segfault in stringFormat() / formats.c
Sample file (test with “rpm -i [input]”):
https://crashes.fuzzing-project.org/rpm-nullptr-rpmtdFormat.rpm
Git commit:
https://github.com/rpm-software-management/rpm/commit/cddf43a56f19711866371f02f378dc4095b0fadd
Out of bounds read in rpmtdGetNumber() / rpmtd.c
Sample file (test with “rpm -qi -p -- [input]”)
https://crashes.fuzzing-project.org/rpm-heap-oob-read-rpmtdGetNumber.rpm
Git commit:
https://github.com/rpm-software-management/rpm/commit/b722cf86200505b3e3fcbb2095c4ff61f1f5a2ab
Finally one annoying thing to admit: In my original report I included another segfault in headerVerifyInfo() with unclear reasons. However I am now unable to reproduce this one. It may be due to compiler options, different command line parameters or dependencies on my system that have changed. For completeness I'm still providing the sample file:
https://crashes.fuzzing-project.org/rpm-segfault-headerVerifyInfo.rpm
(Ideally the RPM developers should include all those sample files in a test suite they regularly run against an address sanitizer build of RPM.)
Please also note that I expect this list to be incomplete and there are likely more issues that could be uncovered with further fuzzing. I'll test that once all the existing issues are fixed.
If you want to report a bug to RPM you first may wonder where to report it. The RPM webpage [1] is a trac installation which has its own bug tracker. However if you try to register an account there you'll get forwarded to an HTTPS site with an expired certificate that doesn't match the domain name. In case you are brave and tell your browser to ignore all warnings you'll be greeted by a broken-looking trac without any CSS. Should you proceed and create an account you will learn that this doesn't help you, because in order to be allowed to report a bug you first have to ask on the mailing list or in the IRC channel for permission [2]. That's probably the point where many well-meaning bug reporters give up.
Okay, but RPM originally stood for “Red Hat package manager” (I've been told that today it stands for RPM Package Manager), so maybe Red Hat feels responsible. So I reported three bugs with sample files triggering them to the Red Hat security team on November 20th. The answer was – to put it mildly – a bit dissatisfying. I'll just fully quote it: “Thanks for the report. We also received about 30+ crash reports in RPM from
a different reporter recently so processing all of them (yours and the
others) will take quite a bit of time. We simply don't have the resources
to spend hours upon hours analyzing all crash reports.”
Okay, so I wasn't the only one fuzzing RPM and the maybe bugs will be fixed some day. I waited. In the meantime I got contacted by another person who also had tried to report fuzzing bugs in RPM and who has made similar experiences (maybe the same person who reported the 30+ crashers, I don't know).
In February I decided to ask what the state of things is. I also gave them a 30 day period until I'd publish the bugs (I know that it's now long past that, I shouldn't have let this issue wait so long). I ended up having a call with a Red Hat security team member and exchanged a couple of mails. I learned that RPM has a Github repository [3], which contains fixes for some (but not all) of the issues I reported, however that's nowhere referenced on its webpage. I then fuzzed the current RPM git code again and found two more issues I also reported to the Red Hat security team.
Status today is that the latest release of RPM on its webpage – 4.12.0.1 - is from July 2015, so all of the bugs still affect this release. However it seems there is an unofficial 4.13 release that's nowhere to be found on the RPM webpage, but Red Hat is using it together with some fixes [4]. And the Github repository says the latest release is 4.12.0, so according to three different sources three different versions are the current one (4.12.0, 4.12.0.1, 4.13).
One of the bugs – a stack overflow (write) - is still present in the latest code on Github.
Commend and Conclusion
This blog post probably reads like a big rant about how unprofessional Red Hat is in handling potential security issues. But this is contrary to my usual experience. I often see Red Hat developers being very active in the free software security community and often contributing in a positive way. Quite simply I expect better from Red Hat. This is not some dubious Enterprise vendor where I wouldn't be the least bit surprised of such a reaction.
The development process of RPM seems to be totally chaotic, it's neither clear where one reports bugs nor where one gets the latest code and security bugs don't get fixed within a reasonable time.
There's been some recent events that make me feel especially worried about this: An unknown person has created an entry in the Libarchive issue tracker [5] that points to an anonymous document [6] with a very detailed description of various security weaknesses in the FreeBSD update process (most of them are still unfixed). The most worrying thing about this is however that the anonymous post mentions the existence similar documents affecting multiple Linux distributions. These documents haven't shown up publicly yet and given the unclear nature of this incident it's hard to know whether they ever will become public or exist at all. But this should still be reason enough to have a closer look at potential security vulnerabilities in all pieces of Linux package management systems.
I haven't analyzed the RPM installation process in detail, so I can't say how likely it is that the RPM tool ever sees a malformed input file. It seems downloads happen over HTTP, but the first thing that happens is a signature check. As the signature is part of the RPM file it already needs to be parsed for this. The exact impact of these bugs would require further analysis. But independent of how likely this is I think the parser in such a crucial piece of software should be robust. It should be safe to use the rpm tool to show info about a file on the command line.
[1] http://rpm.org/
[2] http://rpm.org/wiki/ReportingBugs
[3] https://github.com/rpm-software-management/rpm
[4] http://pkgs.fedoraproject.org/cgit/rpms/rpm.git/diff/rpm-4.13.0-rpmtd-out-of-bounds.patch?h=f22&id=165614f3dd42caa188f78b55e7723dad2900b2f4
[5] https://github.com/libarchive/libarchive/issues/743
[6] https://gist.github.com/anonymous/e48209b03f1dd9625a992717e7b89c4f
All bugs were found with the help of american fuzzy lop. Here are the bugs:
Stack Overflow in glob() / rpmglob.c.
Sample file (test with rpm -i [input]):
https://crashes.fuzzing-project.org/rpm-stackoverflow-glob.rpm
Unfixed in the current Git code.
Heap out of bounds read in headerVerifyInfo() / header.c.
Sample file (test with “rpm -i [input]”):
https://crashes.fuzzing-project.org/rpm-heap-oob-read-headerVerifyInfo.rpm
Git commit:
https://github.com/rpm-software-management/rpm/commit/8e847d52c811e9a57239e18672d40f781e0ec48e
Null pointer access / segfault in stringFormat() / formats.c
Sample file (test with “rpm -i [input]”):
https://crashes.fuzzing-project.org/rpm-nullptr-rpmtdFormat.rpm
Git commit:
https://github.com/rpm-software-management/rpm/commit/cddf43a56f19711866371f02f378dc4095b0fadd
Out of bounds read in rpmtdGetNumber() / rpmtd.c
Sample file (test with “rpm -qi -p -- [input]”)
https://crashes.fuzzing-project.org/rpm-heap-oob-read-rpmtdGetNumber.rpm
Git commit:
https://github.com/rpm-software-management/rpm/commit/b722cf86200505b3e3fcbb2095c4ff61f1f5a2ab
Finally one annoying thing to admit: In my original report I included another segfault in headerVerifyInfo() with unclear reasons. However I am now unable to reproduce this one. It may be due to compiler options, different command line parameters or dependencies on my system that have changed. For completeness I'm still providing the sample file:
https://crashes.fuzzing-project.org/rpm-segfault-headerVerifyInfo.rpm
(Ideally the RPM developers should include all those sample files in a test suite they regularly run against an address sanitizer build of RPM.)
Please also note that I expect this list to be incomplete and there are likely more issues that could be uncovered with further fuzzing. I'll test that once all the existing issues are fixed.