Getting the most out of HAProxy

Getting the most out of HAProxy

HAProxy is a pretty magical piece of software that we use extensively at Twilio. Today we are going to walk through our haproxy configuration and explain some of the settings therein. But first, a short story about our history with HAProxy.

In the beginning, the Founders created the HAProxy Config. The Config let one host in the cluster talk to other hosts, and carried all manner of traffic. And the Config was acceptable, and it let the Founders raise a venture capital Round.

Then the Founders said, "I will make an SMS service, that can send SMS. And the API can talk to the SMS service to send messages, and it will be good." But they needed a Config for the API to talk to the SMS service. And they copied the Config from the Voice service, and the API could talk to SMS, and it was good. But the Config was not adapted for SMS requests.

And the Config was fruitful, and multiplied across the services at Twilio. But nobody really understood the Config, or its options. So when they needed a new Config, they copied the old one. And in this way, problems spread across the Cluster.

This is the state we got into with our HAProxy settings in the olden days of Twilio. All of our HAProxy configs were copies of each other, with some of the timeout values changed. In our quest for five nines of reliability, we tightened up our HAProxy configs and made sure they were appropriate for the service and traffic being carried. Let's walk through a sample configuration.

# Twilio HTTP HAProxy Configuration
# Version: 0.1

These lines let us know at a glance that the config has been touched/updated by someone in the recent history of the company.

mode    http
option  httplog
log 127.0.0.1 local0 info

These two options turn on most of HAProxy's power, and if you are carrying HTTP traffic with HAProxy you should definitely have this setting on. mode http means HAProxy will inspect the incoming HTTP request, and reject it if the request is invalid.

This mode also allows you to log HTTP requests, generating log lines like this:

Oct  9 15:32:25 127.0.0.1 haproxy[32156]: 127.0.0.1:34677
    [09/Oct/2013:15:32:25.744] twilio-frontend twilio-backends/10.10.10.10
    0/0/0/38/71 200 944419 - - ---- 94/0/0/ 0/0 0/0
    "GET http://example.com/ACaaaa?Signature=signature HTTP/1.1"

These logs give us an incredibly rich amount of detail about the request - from the URL requested, to the total time spent connecting, total time for the download, total number of concurrent connections to the downstream service, and more. For a full explanation of what these log messages mean, see the HAProxy logging documentation.

As we convert more and more Twilio tools to a service-oriented architecture, these logs have been invaluable at debugging failures. As an example, a customer wrote in that her recordings downloads were failing occasionally. As the recordings stack hits multiple different services, logs like these which record the timings between network components helped us nail down the failure to a single host.

The third line

log 127.0.0.1 local0 info

enables haproxy logging over syslog. A separate daemon listens on the local0 facility and logs incoming haproxy messages to a log file.

Note, enabling the info log level means HAProxy will log one line per incoming request. For servers that stay up for long periods of time, this means the log files can overwhelm the disk. It is important to have a data purging policy in place - every night, logrotate compresses the previous day's logs and wipes out logs older than 2 weeks.

option http-server-close

Let's say you have one HAProxy frontend communicating with ten different backends, and moreover let's say those ten backends don't support HTTP keep-alive. This setting lets HAProxy keep a persistent connection between the client and the frontend, while rotating the HTTP requests between all of the backend servers. Of course, to get the full value out of this setting it is necessary for the clients to be keep-alive aware, as well.

timeout client 30s
timeout server 30s

These two settings combine to set the timeout for requests made with HAProxy. Essentially, how long will HAProxy wait for the server to return information to the client? Most applications have some kind of a maximum latency, so you want to add a timeout. For us, the API will wait for a response for a maximum of 30 seconds, so every internal service has a timeout set to a value lower than 30 seconds, based on its SLA. Note that if the server is streaming one byte slowly, say once every 29 seconds, you will not trigger the read timeout, so you may want to have a separate thread watching the request to ensure it completes in a sane amount of time. You should also set the client and server timeouts to the same value - the desired read timeout on the socket.

timeout connect 3100

This is a different timeout than the client and server timeouts! This is the amount of time HAProxy should spend trying to connect to a host. In the olden days of Twilio, this was set to the same value as the server timeout - 30 seconds - so if a host died, HAProxy would try to connect to the same dang host for 30 seconds. The connect usually happens in a matter of milliseconds when the hosts are on the same machine, or the same LAN (or in nearby AZ's). We allow 3 seconds for the default TCP retransmission window to proceed, and a small amount of buffer.

Unlike server timeouts, connection timeouts imply the request is safe for the client to retry.

retries 2
option redispatch

When I said a 30 second connect timeout meant HAProxy would try a bad connection for 30 seconds, I lied. It turns out that by default HAProxy will retry the connect attempt 3 times. So our 30 second connect timeout is actually a 120 second connect timeout, blowing through our SLA and meaning we're returning an empty response to the customer.

It's a common assumption that HAProxy will automatically retry requests to a second host if the first one goes down. But this is only true if HAProxy has marked the host as DOWN after a healthcheck. If a host goes down and your healthchecks take 20 seconds to pull it, you are looking at a possible 20 seconds of failed requests to a host. The redispatch option makes the final connect request to a different downstream host, so individual requests have some degree of protection from a host going bad.

These two settings in combination cut down the number of retries to 2, and mean that we only attempt connections to a single host for a maximum of eight seconds before giving up and trying a different host.

option httpchk GET /healthcheck

By default HAProxy will simply open a TCP connection to a host to check whether it is up. That ping will detect if a host is down, but not a host that is unhealthy (with a bad disk, bad networking connection). The httpchk option will make a HTTP request to an endpoint on the backend. The backend can query itself and answer whether it is healthy or not. Note, healthchecks should be fairly conservative, and generally scoped to the health of an individual host. A failed healthcheck will lead HAProxy to send zero traffic to a host, and if all of your hosts "go bad" at the same time, you will be left with no backends.

I hope these are helpful - I've posted the HAProxy config with all of our updates here. As with anything, reading the manual can lead to significant improvements in performance, reliability, usability and stability of your server. Hope this walkthrough saved you some time.