Yesterday evening, a client of the company I work for was under what I would consider a small-scale DDoS attack. There were definite signs of malicious behavior including: multiple concurrent connections, high request rates and a number of HTTP error responses from clients requesting pages that aren’t there. Rather than running and hiding, I thought that I would make a reasonable attempt at slowing the attack.
I have been using HAProxy for the better part of a year now as it is our current software load balancing solution between our back-end ”worker” machines in Fusion Tracker. However, I’ve noticed that in the development branch there are now ACL tools that you can use to flag malicious behavior and effectively block the traffic from ever hitting your server.
Now, if you don’t know much about HAProxy, the one thing you need to know is that it is very good at what it does. 100,000 requests per second good. This makes it an ideal candidate for being placed in front of our client’s back-end server.
Willy, the creator of HAProxy, wrote:
HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for web sites crawling under very high loads while needing persistence or Layer7 processing. Supporting tens of thousands of connections is clearly realistic with todays hardware. Its mode of operation makes its integration into existing architectures very easy and riskless, while still offering the possibility not to expose fragile web servers to the net.
Download
The current non-developmental release of HAProxy is sitting at 1.4.23 (at this time of writing), and unfortunately our ACL tools don’t pop up until the 1.5 branch. This means that it is not included in any RHEL repository and we’ll have to build the 1.5-dev18 branch ourselves.
Start by retrieving the source code and extracting it
wget http://haproxy.1wt.eu/download/1.5/src/devel/haproxy-1.5-dev18.tar.gz tar xzvf haproxy-1.5-dev18.tar.gz
Bug Work-Around
There is an issue when you compile and run this, that after a few requests HAProxy will segfault (or at least this happened in my case). I did quite a bit of Googling and was able to find that the author pushed a file change to temporarily fix it and we’re going to do that.
Change to the HAProxy directory and edit the acl.c file
cd haproxy-1.5-dev18 vi src/acl.c
Inside this file, we need to navigate to line 1137. The file should look something like this:
/* Build an arg list containing the type as an empty string
* and the usual STOP.
*/
expr->args = calloc(2, sizeof(*expr->args));
expr->args[0].type = type;
expr->args[0].unresolved = 1;
expr->args[0].data.str.str = strdup("");
expr->args[0].data.str.len = 1;
expr->args[0].data.str.len = 0;
expr->args[1].type = ARGT_STOP;
We need to edit the file and change it to look like this (I’ve highlighted the line we are inserting):
/* Build an arg list containing the type as an empty string
* and the usual STOP.
*/
expr->args = calloc(2, sizeof(*expr->args));
expr->args[0].type = type;
expr->args[0].unresolved = 1;
expr->args[0].data.str.str = strdup("");
expr->args[0].data.str.len = 1;
expr->args[0].data.str.len = 0;
arg_list_add(al, &expr->args[0], 0);
expr->args[1].type = ARGT_STOP;
Compilation & Installation
Now that we’ve edited the source code, we must compile HAProxy so that we can use it to filter incoming HTTP requests. The compilation line is platform and architecture dependent. My machine has a fairly recent kernel and runs in 64 bit, so I’ve added them to make’s parameters.
make TARGET=linux2628 USE_EPOLL=1 USE_OPENSSL=1 ARCH=x86_64 && make install
If you do not know your architecture or just want to use a generic version, you can run the following and it will build a non-specialized version of HAProxy:
make TARGET=generic && make install
I suggest that you do a little research before going the generic route, there are a lot of little features that HAProxy takes advantage of to meet the 100,000 requests per second benchmark that you may or may not receive if you go the generic route.
Configuring HAProxy
Now that HAProxy is installed on our system, we are free to configure it and place it in front of our services. I went ahead and created a generic configuration file that you can use to get started. I’ll highlight a few choices that I made.
Global and Defaults
global
daemon
nbproc 1
maxconn 100000
defaults
option http-server-close
mode http
timeout http-request 5s
timeout connect 5s
timeout server 10s
timeout client 10s
In the global section, I’ve told HAProxy that I want it to run as a daemon (in the background), that I want it to only use 1 process to accept and process incoming requests, and that the max connections that I want it to handle is 100,000. Now, you will get a message from HAProxy if you do not have this many available file descriptors allocated in your limits.conf file.
The defaults tell HAProxy how we want to handle incoming traffic. In this case, we’re telling HAProxy that by default, the traffic we will be dealing with is HTTP traffic, that we want to wait a maximum of 5 seconds for HAProxy to connect to the backend server, that we will wait a maximum of 10 seconds for the server to reply back after connected, and that we will wait 10 seconds before kicking the client for inactivity. These values are somewhat high, but they assure that even when the server is extremely loaded that clients still have an opportunity to get the page being requested.
By setting timeout http-request, we address slowloris-like attacks where the client will connect to the server and very slowly send the HTTP request. In this situation, they will be dropped for not completing the HTTP request and will not impact the availability of your site.
The option http-server-close is an useful one, because it allows HAProxy to analyze the HTTP header for the keep-alive option and keep the session alive with the client but close it on the back-end to free up the server’s available connections. This means that it is low latency for the client as they keep the connection open but it makes the server available for other clients connecting.
I want some stats!
HAProxy has a read-only interface that you can use to monitor incoming connections and how they are hitting the back-end. I’ve went ahead and added this to the configuration file:
listen stats 0.0.0.0:8080
mode http
stats enable
stats hide-version
stats realm Haproxy Statistics
stats uri /
stats auth admin:chooseanewpassword
This tells HAProxy that you want to be able to connect to your server on port 8080 and view stats but that you want to have it authenticated with the username admin and the password chooseanewpassword. I suggest that you choose a new password!
Handle Incoming HTTP Connections
By defining frontends in HAProxy, we allow HAProxy to talk to the outside world. These frontends determine where the incoming traffic goes to and if it reaches the backend via blocking, etc.
frontend httpFrontEnd 0.0.0.0:80
maxconn 100000
option forwardfor header x-forwarded-for
stick-table type ip size 100k expire 30s store conn_rate(3s)
stick-table type ip size 1m expire 10s store gpc0,http_req_rate(10s)
stick-table type ip size 100k expire 30s store conn_cur
tcp-request connection track-sc1 src
tcp-request connection reject if { src_get_gpc0(httpFrontEnd) gt 2 }
tcp-request connection reject if { src_conn_cur ge 10 }
tcp-request connection reject if { src_conn_rate ge 10 }
default_backend httpBackEnd
The configuration file snippet that I have placed above features a lot of new things in HAProxy. Right off the bat, we are telling HA that we want it to bind to all of the interfaces on port 80 (this is something you may need to change depending upon how many interfaces you have and what other services you are running). Now, inside of this frontend, we set the maximum connections to 100,000 — this determines how many concurrent connections this frontend can accept. Anything more than this and the clients will receive an error indicating that there is no server available to process their request. This snippet defaults the backend to the one defined below as httpBackEnd.
In the above configuration, there are three things that we are monitoring: concurrent connections, the client’s connection rate, and the HTTP request rate. We track these for 30 seconds, 30 seconds, and 10 seconds respectively. This means that if a client has too many concurrent connections, too many connections per time period, or too many HTTP requests, that it will drop their connection and continue to until the expiry duration has passed.
Process Incoming Requests
Now that we’ve configured HAProxy to accept incoming connections and on what port to do so, we need to tell HAProxy what backend servers that it needs to use to process these connections.
backend httpBackEnd
acl abuse src_http_req_rate(httpFrontEnd) ge 10
acl flag_abuser src_inc_gpc0(httpFrontEnd)
tcp-request content reject if abuse flag_abuser
server httpd 127.0.0.1:81 check maxconn 250
The backend here is very simple and defines one server available to process these HTTP requests. It starts by using our acl rules to determine if the person connected is flagged as an abuser and if they are that we should reject their connection.
If you have multiple back-end servers, you can define multiple server entries with their name (in HAProxy), their address and port, whether you want to check their health, and how many connections you want to send to it. (There are more options available, read the configuration documentation!)
By setting the maximum amount of connections allowed to hit our HTTP server, we effectively prevent an “overloaded” situation where the server simply cannot keep up with all of the incoming requests and everything slows to a halt.
Putting it Into Place
Now that we’ve compiled and installed HAProxy and created a configuration for it, we need to start it. For the following command, I am going to assume that you created your configuration file in /etc/haproxy/haproxy.cfg. You can change the command-line parameters to better suit where you placed yours. This tells HAProxy to load the configuration file and to run as a daemon.
haproxy -f /etc/haproxy/haproxy.cfg -D
A Few Thoughts
Admittedly, no configuration will be perfect, especially one that hasn’t been thoroughly tested, so don’t hold me 100% liable for this configuration. It is a good starting point that you can test and refine as needed.
There are a ton of really awesome features that HAProxy supports and I don’t think I could ever write a post that covered all of them. I suggest that you read the configuration document or browse the public wiki. You will find a wealth of information which you can use to further refine your configuration.
Happy configuring!