So, Apache 2.4.24 is about to get tagged and tested Real Soon Now and I managed to find some time to work on proxy setups and mod_h2. Which was needed badly. The result - so far - is the shiny v1.7.5 which you can drop into an Apache 2.4.23.
If you still run the plain vanilla
mod_h2 that comes with Apache 2.4.23, you
will get several benefits from upgrading (be it from here or via the next Apache release). I
describe those in more detail below:
Connection timeouts are only active while doing IO. When the backend is happily chewing on a request,
mod_http2 will wait very patiently for it to produce something. Only when request processing reads or
writes and is blocked will the timeout happen. This also means that
ProxyTimeout is treated
Timeout, as it should be.
Example: you have timely clients and set a
Timeout of 10 seconds. So when reading DATA
for requests, or sending DATA for responses is blocked for more than 10 seconds, the request fails.
But in the backend, you have a slow, old application server and requests can be quite complex, so
you are willing to let it chew at a response for 5 minutes, before you suspect that something is
wrong. You set the
ProxyTimeout to 300 seconds.
Before it got fixed, such a setup was not possible, since the 10 seconds would always trigger, even while waiting on the backend to produce something. Not so now.
There was a CGI related problem. To be more exact, it was
mod_cgid related. You see,
talks to a daemon and needs some kind of identifier for each CGI process it triggers (so that it can kill a process
it is not happy with, for example). Since several
httpd processes might talk to the same daemon, such
an identifier needs to be unique among the whole server.
Now, connections have a server-wide unique ID and since there is only one request per connection, let's use
the connection id to identify our CGI request. And this works nicely for HTTP/1.1. However in HTTP/2, there are
several requests per connection at the same time. And up till now, when two or more of them where
mod_cgid assigned the same id. And maybe cancelled the wrong one. Yikes!
Since this only messed up CGIs on the same connection, it's not a security risk. You cannot mess with CGIs from someone else.
Happily, connection ids are of type
long. That means on 64-bit systems, and with the way Apache
generates these ids, it is possible to squeeze the HTTP/2 stream id into it and have no collisions. On 32-bit
systems, though, this needs more work. And the final result is not in yet...
Needless to say, mod_http2 smoke tests now include those that run 50 uploads on 2 connections each against a CGI script and check the completeness of all results.
Many clients prefer the '
Expect: 100-continue' request header when performing uploads. This worked
for simple setups, but did not do the neccessary things when acting as a HTTP proxy. It generated the
100 Continue' response way too early and suppressed the header when talking to the backend server.
This has been fixed now.
Expect headers are passed to the backend and only when the backend sends
mod_http2 send this on to the client. So, now also HTTP/2 clients can be
100 Continue means "I am willing to take in your data, please send it".
Graceful server restarts lead to early request terminations. Now the module knows better and will process ongoing requests to the end before shutting down the connection. And it will notify the client that it is no longer willing to accept new ones.
When adding more and more upload tests, those also executed in the
and - to my misery - exhibited spurious failures now and then. What was going on?
A closer look (and tons of trace outputs) showed that the failed requests where all sending data to
mod_proxy_http2 did send it on to the backend and when the backend failed, no
one was in possession of the data any longer and the request had to fail.
But why did the backend fail? Turns out, it was no real failure, but the backend was just closing its
connection after a keepalive timeout expired. The Apache proxy infrastructure keeps connections around
for future reuse, so
mod_proxy_http2 often gets a connection where a HTTP/2 session has
already been established and used. It can fire new requests right away and everyone is happy.
But when the backend closed the connection, it sends a last
GOAWAY frame, which is sitting
in the connection buffers and make the connection look fine and healthy. It is handed to
which immediately sends requests over it, then reads only to find that the other side has already
left the party.
Originally, I though to fix this by letting
mod_proxy_http2 read from a live connection
first. If it finds a GOAWAY, it can chuck the connection and create a new one. But what if the GOAWAY
has not yet arrived, but is alreay on its way? The change would not help here.
What is implemented now is safe even in those cases:
mod_proxy_http2 will continue to
send of requests right away, but will suspend sending and request DATA until it is certain that the
connection is still good. This means that request processing can start without unnecessary delays,
but sending DATA will have the penalty of 1 RTT - if the connection was "old", e.g. did not receive
frames for more than a second.
This seems quite ok, because RTT to backends is usually small and on busy servers, where backend
connections stay busy, no delays are necessary.
Hope to have a new Apache release by the end of the month. If you cannot wait, grab the lastest HTTP/2 support here. And let me know how it worked for you!
Stefan Eissing, greenbytes GmbH
Copyright (C) 2016 greenbytes GmbH
Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without warranty of any kind. See LICENSE for details.