Defending Against DoS/DDoS Attacks in Apache Server With mod_evasive Module

Posted on by Rick

Your infrastructure must be protected against DoS/DDoS attacks. Major cloud providers these days protect their customers from such attacks to a decent threshold but even that threshold could be huge for your web application. Your limited bandwidth, processor and memory resources may not be able to take that much of load. A massive load of illegit requests can run tons of scripts that connects to other services like databases, caching systems, queues, etc., taking you offline.

This is why you must employ tools (or write your own) that can do a generic rate limiting and block requests altogether at an IP address level at the very least. For starters, there’s an Apache module that can help us to some extent – mod_evasive.

The third party mod_evasive module is fairly easy to use and can help us block requests from malicious user agents that try to overload our servers. It can provide evasive actions against HTTP DoS/DDoS attacks or brute force attacks. It can even be configured to talk to firewalls, routers, etc. on detection of an attack from one or more IP addresses so that blacklisting can happen at a higher layer.

Installation

There are packages available for your favourite Linux distros.

Ubuntu / Debian

# apt install libapache2-mod-evasive

Installing the package also takes care of enabling the module, so no need of a2enmod evasive.

CentOS / Fedora

dnf install mod_evasive
# or
yum install mod_evasive

# Followed by (in your configuration file)
LoadModule evasive20_module <path-to-so-file>

The module will be automatically loaded, i.e., LoadModule will be added to the configiguration. If for some reason that doesn’t happen then read this article to learn how to find and enable a module.

FreeBSD

# pkg search mod_evasive

ap24-mod_evasive-1.10.1_1 Apache module to try to protect the HTTP Server from DoS/DDoS attacks

# pkg install ap24-mod_evasive-1.10.1_1

Hopefully, the module will be automatically loaded via LoadModule in the configs.

From Source

Cases where the package manager’s repository doesn’t have mod_evasive (like Alpine for docker), you can simply install it from the source. Simply get the source file from here. We just need the mod_evasive20.c file. For Apache 2.4 we’ll to make a minor modification to the source code.

# sed s/remote_ip/client_ip/g -i mod_evasive24.c

With that change in effect, simply run the apxs program to build a .so module out of the code.

# apxs -cia mod_evasive20.c
...
chmod 644 /usr/lib/apache2/modules/mod_evasive.so

In the end you’ll find the path to the .so. To enable the module, put this in the main config file:

LoadModule evasive20_module /usr/lib/apache2/modules/mod_evasive.so

Usage

Once the module is loaded and enabled, you’ll need to put a bunch of configurations to control the module’s functioning.

<IfModule mod_evasive20.c>
    DOSHashTableSize    3097
    DOSPageCount        2
    DOSSiteCount        50
    DOSPageInterval     1
    DOSSiteInterval     1
    DOSBlockingPeriod   10

    #DOSEmailNotify      [email protected]
    #DOSSystemCommand    "su - someuser -c '/sbin/... %s ...'"
    DOSLogDir           "/var/log/mod_evasive"
</IfModule>

In case of standard package manager based installations, the config will be automatically added to your httpd installation, but commented. So whether you add it manually or know where it was added post a package manager installation, just uncomment everything except DOSEmailNotify and DOSSystemCommand for now.

I’ll explain the configuration options in a while (below).

Server Restart

After the module is installed and config changes have been made, we must restart httpd for the changes to take effect.

apachectl restart

Feel free to use systemctl or service instead for restarts if you’d like to.

Testing

This should be enough for the module to get into action. You can test it by writing a short script that hits your server pretty fast. For instance, try the following PHP script:

# cat dos.php
<?php

foreach (range(1, 50) as $i) {
	echo file_get_contents("http://yoursite.com");
}

Assuming http://yoursite.com just prints Welcome to yoursite.com, this is what’ll happen:

# php dos.php
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
Welcome to yoursite.com
PHP Warning:  file_get_contents(http://yoursite.com): failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden
 in /etc/apache2/dos.php on line 4
PHP Warning:  file_get_contents(http://yoursite.com): failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden
 in /etc/apache2/dos.php on line 4
PHP Warning:  file_get_contents(http://yoursite.com): failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden
 in /etc/apache2/dos.php on line 4
PHP Warning:  file_get_contents(http://yoursite.com): failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden
 in /etc/apache2/dos.php on line 4
PHP Warning:  file_get_contents(http://yoursite.com): failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden
 in /etc/apache2/dos.php on line 4
...
... and the request keeps failing with a 403 Forbidden
...

Beyond a certain threshold, mod_evasive started throwing 403 Forbidden and blocked the IP address. Fantastic! Let’s now understand in detail how it actually works.

How Does It Work ?

mod_evasive maintains a hash table where the keys are of two types:

  1. Requestor IP Address and the requested URI hashed together.
  2. Just the Requestor IP Address hashed.

Against these keys, the page count values and a timestamp are held. If the page count increased beyond whats specified in the configuration via DOSPageCount or DOSSiteCount then the requestor IP will be blocked for DOSBlockingPeriod seconds. Storing the requestor IP + requested URI as key allows blocking when the threshold at a page level is breached (DOSPageCount), where as storing it at a requestor IP level helps with blocking site-wide (DOSSiteCount). The blocked IPs are maintained in a temporary blacklist.

So when a web request comes in, this is what happens:

  1. Lookup the client IP in the temporary blacklist. If it doesn’t exists carry on, otherwise reset the timestamp which prolongs the blocking period and throw 403 Forbidden.
  2. Hash the IP + URI into a key and check if the page-level threshold has been breached in the hash table. If yes then blacklist the IP and throw 403 Forbidden or carry on.
  3. Hash just the IP address into a key and check if the site-wide threshold has been breached according to the hash table. If yes, then you know what happens. If no, then let httpd serve the legit request.

It is important to know that the hash table is maintained by the child processes, each of them have their own copies. Nothing is shared or global. Hence the configuration is effective at a per child process level rather than the entire server.. So if there are say 20 Apache worker processes and the allowed traffic per IP is 2 request per second per page (DOSPageCount and DOSPageInterval) then your server will allow:

20 workers x 2 requests/second/page
= 40 requests/second/page (best case)

So your server will allow 40 requests per second per page before blocking an IP. This is the best case number because the master Apache process that listens to incoming connections, may forward more than 2 requests per second for a page to the same child process leading to 403 Forbidden after the second request in a given second. So it’s possible a few requests out of those 40 will actually get blocked.

Hopefully now that we understand what happens under the hood, let’s thoroughly go through the different configuration options.

Configuration Options

There are a bunch of them, all easy to understand.

DOSHashTableSize

DOSHashTableSize    <hash-table-size>

Defines the size of the hash table. Whatever value you pass will be reset to the closest prime number on the higher side (greater in value than what you specified). Prime numbers help with better distribution of values across the hash tables.

Anyway, increasing this number improves performance naturally due to decreased iterations in nodes holding multiple values. At the same time memory consumption will increase. So if you have a fairly sized box and a busy web server then go for a larger value.

DOSPageCount

DOSPageCount        <request-count-per-page>

Threshold at a per page, per child process level for number of allowed requests in an interval of DOSPageInterval. If breached, the client will be blocked by the child.

DOSPageInterval

DOSPageInterval     <interval-in-seconds>

Interval in seconds in which DOSPageCount number of requests are allowed.

DOSSiteCount

DOSSiteCount     <request-count-site-wide>

Site-wide (any endpoint/object/resource), per child process threshold for max number of requests allowed in a timespan of DOSSiteInterval. Exceeding this will blacklist the IP temporarily, by the child.

DOSSiteInterval

DOSSiteInterval     <interval-in-seconds>

Interval in seconds in which DOSSiteCount number of requests are allowed.

DOSBlockingPeriod

DOSBlockingPeriod   <block-period-in-seconds>

Number of seconds an IP will be temporarily blocked for, by a particular child. If the IP keeps hitting the server even after getting blocked then the timer keeps resetting, increasing the blocking period.

DOSEmailNotify

DOSEmailNotify      <email-address>

Enable email notifications to the specified email address upon an IP blacklisting. To prevent email bombing for a particular IP moving back and forth to the blacklist, the module acquires a lock by creating a file per IP address in /tmp or DOSLogDir. The email sent is pretty simple:

From: ...
To: <email-address>
Subject: HTTP BLACKLIST 157.245.192.96

mod_evasive HTTP Blacklisted 157.245.192.96

DOSSystemCommand

DOSSystemCommand    <system-command>

System command to be executed when an IP address is blacklisted. Super useful if you want to take actions at a stricter or higher level like blocking IPs at a firewall or router level. The IP will be available via the %s string formatter. For instance:

DOSSystemCommand    "su - someuser -c '/sbin/... %s ...'"

So basically whenever an IP is blacklisted, you could pipe it to another tool for a bunch of extra useful operations. The module doesn’t repeatedly execute the system command if an IP gets blacklisted multiple times. It takes care of that by acquiring a lock (same mechanism as DOSEmailNotify).

In fact integrating mod_evasive with a firewall or router for blocking traffic is a pretty good idea to further lower down the bandwidth and processor usage which can spike up like crazy under a mass distributed attack.

DOSLogDir

DOSLogDir           "/var/log/mod_evasive"

Directory where lock files will be created to prevent continuous emails from being sent or system commands from getting executed. Make sure the folder is writable by the Apache system user (www-data).

Whitelisting

If you want to whitelist a bunch of trusted clients, then that is surely possible. Although make use of this feature cautiously, for instance you may whitelist a customer whose developer might make a tiny mistake ending up abusing your service. You don’t want that.

Put this in your configuration:

DOSWhitelist	127.0.0.1
DOSWhitelist	127.0.0.*

As you can see in the second line, it is possible to use wildcards. In fact you can use them upto the last 3 octets.

Conclusion

For a relatively small setup, like when you’re serving a decent amount of traffic off a few servers, mod_evasive is a fine option to safeguard yourself against (distributed) denial-of-service attacks. But for large scale setups, without a solid infrastructure that actually prevents your system against such attacks at different layers and rate limits user agents at a global level instead of per-httpd-child-process levels, a heavy distributed attack can still likely bring you down.

Personally, until I have a sophisticated infra in place, I’d totally go for this module as a primitive solution as its super quick to setup and definitely effective.

Leave a Reply

Your email address will not be published. Required fields are marked *