Someone dumped his findings about Use After Free's in Apache httpd on the internet yesterday. I will not go into the social, political and other soft-issues aspects of this here.
In this blog post, I want to provide technical information related to the observed issues in regard to Apache httpd's HTTP/2 implementation . After all, this is what this github repository is about!
There are 2:
__pthread_tpp_change_priority: Assertion `new_prio == -1 || (new_prio &grt;= fifo_min_prio && new_prio <= fifo_max_prio)' failed.This happened without APR pool debugging, but the issue had already been known and fixed in Spring 2018. If you are running a current Apache 2.4.x release, no need to worry.
The reported data dump lists a lot of stack traces for various versions of Apache httpd.
90% of those show stack traces with
apr_pool_walk_tree as the offender. The other, 10 files,
are as follows:
> for f in */*; do fgrep apr_pool_walk_ $f >/dev/null || echo $f; done apache-2.4.29/apache-2.4.29-heap-use-after-free-h2_stream_destroy-asan-error.15581 apache-2.4.29/apache-2.4.29-heap-use-after-free-h2_stream_destroy-asan-error.15582 apache-2.4.29/apache-2.4.29-heap-use-after-free-h2_stream_destroy-asan-error.15908 apache-2.4.29/apache-2.4.29-heap-use-after-free-h2_stream_destroy-asan-error.18719 apache-2.4.33/apache-2.4.33-SEGV-impl_pollset_remove-asan-error.22791 apache-2.4.33/apache-2.4.33-SEGV-impl_pollset_remove-asan-error.28821 apache-2.4.33/apache-2.4.33-heap-use-after-free-add_unless_null-asan-error.6415 apache-2.4.34/apache-2.4.34-SEGV-impl_pollset_remove-asan-error.11548 apache-2.4.34/apache-2.4.34-SEGV-impl_pollset_remove-asan-error.17207 apache-2.4.34/apache-2.4.34-SEGV-impl_pollset_remove-asan-error.2350 apache-2.4.34/apache-2.4.34-heap-use-after-free-abort_socket_nonblocking-asan-error.28259 apache-2.4.37-without-mod_h2/apache-2.4.37-heap-use-after-free-set_neg_headers-asan-error.3921Of those 10, a single one is from the current version, e.g. 2.4.37. That one is not related to
h2, so I will not discuss it here.
Summary: all reported
HTTP/2 related issues that can
be reproduced in a current version are connected to
apr_pool_walk_tree() is an internal function of the Apache Runtime (APR).
The APR encapsulates OS related differences into a common API and also provides several
utility functions and data structures. One of those are the memory pools.
APR memory pools are good for performance since they
reduce the burden on the standard/OS memory allocator. They are also convenient
to use since code does not need to
free() every piece of data. When
the pool is destroyed/cleared this is done for all data of the pool.
These advantages come at a price: the pools are not thread safe (per se) and reuse of the allocated memory prevents modern tools to find coding errors. One of those is 'address sanitation' which tracks allocated/free'ed memory and reports reads after free.
Now, to make such analysis easier again, APR offers a compile time flag named 'pool debugging' that changes the implementation of the APR memory pools. The pools will return memory to the main allocator more frequently and, in addition, it will run several analysis functions on its internal state. These check for inconsistencies, for example.
One of these is
apr_pool_walk_tree(). It traverses the different
memory pools (specifically all children of the current pool) and applies a function.
Mainly, these are:
pool_num_bytes: sums all allocated memory nodes in a pool and all child pools.
pool_find: finds the memory node that contains the given address in this pool or any child pool. This is triggered when you manipulate HTTP headers via the APR functions (read: for every request and response).
Contrary to HTTP/1 h2 uses a pool hierarchy that spans several threads. That means that
apr_pool_walk_tree() reads pool data that is concurrently being modified in
another thread. No surprise that sometimes data is being read by one thread that has been free()'d
Since this debug code was part of the APR long before HTTP/2 was invented, mod_http2 had three options:
I went with the last option and here we are. Until last week, I did not know of any release/distribution that ships APR with pool debugging. Now I know that there are OpenBSD releases that have it. They have not heard of any problems, maybe because Apache and h2 is not a frequent setup for them. They prefer their own web server. They are prepared to push changes should the need arise.
In the meantime, the OpenBSD, APR and httpd devs are working on creating code changes that will work for all of us.
The reported incompatibility between mod_h2 and APR pool debugging are known and a non-issue for almost everyone. People on OpenBSD that use Apache and HTTP/2 will want to check with their distribution. In doubt, disable the h2 protocol unless you get confirmation.
If there is any other distribution that runs h2 and APR pool debugging in a production environment, please report to me and I will update this page. You can also report any related problems or observations at the official bug report we opened for this or just follow any progress there.
Stefan Eissing, greenbytes GmbH
Copyright (C) 2019 greenbytes GmbH
Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without warranty of any kind. See LICENSE for details.