h2 and APR Pool Debugging

Someone dumped his findings about Use After Free's in Apache httpd on the internet yesterday. I will not go into the social, political and other soft-issues aspects of this here.

In this blog post, I want to provide technical information related to the observed issues in regard to Apache httpd's HTTP/2 implementation . After all, this is what this github repository is about!

The h2 Issues Reported

There are 2:

1. The log files showed failed assertions in pthread:

   __pthread_tpp_change_priority: Assertion `new_prio == -1 || (new_prio &grt;= fifo_min_prio && new_prio <= fifo_max_prio)' failed.

   This happened without APR pool debugging, but the issue had already been known and fixed in Spring 2018. If you are running a current Apache 2.4.x release, no need to worry.
2. Together with APR pool debugging, the address sanitation found several use after read. Stack traces can be found in the repository linked above. I'll discuss these in detail.

Types of Use After Free

The reported data dump lists a lot of stack traces for various versions of Apache httpd. 90% of those show stack traces with apr_pool_walk_tree as the offender. The other, 10 files, are as follows:

> for f in */*; do fgrep apr_pool_walk_ $f >/dev/null || echo $f; done
apache-2.4.29/apache-2.4.29-heap-use-after-free-h2_stream_destroy-asan-error.15581
apache-2.4.29/apache-2.4.29-heap-use-after-free-h2_stream_destroy-asan-error.15582
apache-2.4.29/apache-2.4.29-heap-use-after-free-h2_stream_destroy-asan-error.15908
apache-2.4.29/apache-2.4.29-heap-use-after-free-h2_stream_destroy-asan-error.18719
apache-2.4.33/apache-2.4.33-SEGV-impl_pollset_remove-asan-error.22791
apache-2.4.33/apache-2.4.33-SEGV-impl_pollset_remove-asan-error.28821
apache-2.4.33/apache-2.4.33-heap-use-after-free-add_unless_null-asan-error.6415
apache-2.4.34/apache-2.4.34-SEGV-impl_pollset_remove-asan-error.11548
apache-2.4.34/apache-2.4.34-SEGV-impl_pollset_remove-asan-error.17207
apache-2.4.34/apache-2.4.34-SEGV-impl_pollset_remove-asan-error.2350
apache-2.4.34/apache-2.4.34-heap-use-after-free-abort_socket_nonblocking-asan-error.28259
apache-2.4.37-without-mod_h2/apache-2.4.37-heap-use-after-free-set_neg_headers-asan-error.3921

Summary: all reported HTTP/2 related issues that can be reproduced in a current version are connected to apr_pool_walk_tree.

What is apr_pool_walk_tree()?

apr_pool_walk_tree() is an internal function of the Apache Runtime (APR). The APR encapsulates OS related differences into a common API and also provides several utility functions and data structures. One of those are the memory pools.

APR memory pools are good for performance since they reduce the burden on the standard/OS memory allocator. They are also convenient to use since code does not need to free() every piece of data. When the pool is destroyed/cleared this is done for all data of the pool.

These advantages come at a price: the pools are not thread safe (per se) and reuse of the allocated memory prevents modern tools to find coding errors. One of those is 'address sanitation' which tracks allocated/free'ed memory and reports reads after free.

Now, to make such analysis easier again, APR offers a compile time flag named 'pool debugging' that changes the implementation of the APR memory pools. The pools will return memory to the main allocator more frequently and, in addition, it will run several analysis functions on its internal state. These check for inconsistencies, for example.

One of these is apr_pool_walk_tree(). It traverses the different memory pools (specifically all children of the current pool) and applies a function. Mainly, these are:

• pool_num_bytes: sums all allocated memory nodes in a pool and all child pools.
• pool_find: finds the memory node that contains the given address in this pool or any child pool. This is triggered when you manipulate HTTP headers via the APR functions (read: for every request and response).

What's the beef with h2?

Contrary to HTTP/1 h2 uses a pool hierarchy that spans several threads. That means that apr_pool_walk_tree() reads pool data that is concurrently being modified in another thread. No surprise that sometimes data is being read by one thread that has been free()'d by another.

Since this debug code was part of the APR long before HTTP/2 was invented, mod_http2 had three options:

1. Use locking for all pools of a connection. This would slow down parallel processing of requests and needs care to avoid dead locks.
2. Break the parent/child relation between pools. This makes real memory problems harder to analyze and resource de-allocation less automated and error prone.
3. Accept the conflicts and change the debug code in future APR versions to allow for multi-threaded use. Advise against h2 and pool debugging in production setups.

I went with the last option and here we are. Until last week, I did not know of any release/distribution that ships APR with pool debugging. Now I know that there are OpenBSD releases that have it. They have not heard of any problems, maybe because Apache and h2 is not a frequent setup for them. They prefer their own web server. They are prepared to push changes should the need arise.

In the meantime, the OpenBSD, APR and httpd devs are working on creating code changes that will work for all of us.

Conclusions

The reported incompatibility between mod_h2 and APR pool debugging are known and a non-issue for almost everyone. People on OpenBSD that use Apache and HTTP/2 will want to check with their distribution. In doubt, disable the h2 protocol unless you get confirmation.

If there is any other distribution that runs h2 and APR pool debugging in a production environment, please report to me and I will update this page. You can also report any related problems or observations at the official bug report we opened for this or just follow any progress there.

Thanks!

Münster, 22.01.2019,

Stefan Eissing, greenbytes GmbH

Copyright (C) 2019 greenbytes GmbH

Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without warranty of any kind. See LICENSE for details.