We are setting up shell tools and a method enabling us to efficiently edit Apache configurations and test them in just a few seconds without using a browser or having to constantly search through files by hand.<\/p>\n
Successfully configuring the Apache web server requires a lot of know-how and experience. When ModSecurity is added and intervenes in processing this make configuration even more complicated. That\u2019s why it\u2019s necessary to set up the right tools and a systematic workflow. This is the objective of this tutorial.<\/p>\n
The tutorial is a bit pedantic, since it takes optimizing individual key presses in all seriousness. Since dozens, if not hundreds of actions have to be initiated in sequence in the configuration of a web server, there is plenty of room for optimizing the workflow, as ridiculous as it may seem. The advantage of this is that we can remove unnecessary ballast, clearing the view to the actual configuration problem. To this end this tutorial will be presenting some tricks and ideas that promise to be of benefit.<\/p>\n
Curl is the right tool for making HTTP requests. HTTP must of course first work in the browser or the application. But when debugging or configuring new features the browser normally proves to be very cumbersome. Cooking handling, the size of the window, automatically following redirects, etc. all contribute towards making work in the browser very time consuming. That\u2019s why when you have identified a problem in the browser it\u2019s better to reproduce it using We\u2019ve used The first example works with a cookie file, writes the cookies received and reads new requests from the cookies. In the second example, which works only with a relatively new version of The Apache configuration used in this series of tutorials is intended for the lab. By this I mean a test environment in which you can quickly try out the configuration or perform debugging regardless of productive HTTP traffic. One essential feature of the Apache configuration being used was the method for accommodating the entire configuration (with the exception of the OWASP ModSecurity Core Rule Set) in one file. The advantage of this is that we are able to quickly find our way around in the file and get to specific passages via standard search commands. However, this principle is not only beneficial in a lab-like setting. Working this way is also useful in a production environment for eliminating most conflicts and redundancies. This however runs counter to the widespread trend towards modularization of the web server configuration into a variety of files and induces errors. Indeed, I have many times been confronted by setups that configured the VirtualHosts multiple times, set up conflicting access restrictions and whose administrators seemed surprised to learn that the directives configured were being overridden in other places. This is the reason why whenever possible I try to work on one configuration in a single file.<\/p>\n In any case, it\u2019s important to have a clear structure and a stringent approach. It must be clear where each directive is included in the configuration. Marked out sections have to be sufficiently commented (without hiding the directive in a jungle of documentation as in the default Apache configuration). The example configuration in the preceding tutorials attempted to do just that.<\/p>\n In a lab-like setting, it is probably best to set up a template. For my own work I have come up with a configuration named The normal procedure for working with a single process Apache is to alternate starting the binary with the configuration file configured above and stopping it via The problem is not the lack of space, but the lack of free semaphores. They have to be recovered before a restart. The following construction has proven useful:<\/p>\n We use This is how to get a handle on this problem, the error message is still a nuisance, but repeatedly having to go back through the history to keep from having to re-type the name of the configuration file is no longer necessary. It\u2019s better to have both done by a script: The script first attempts to identify the most recently edited local Apache configuration file. This is very likely to be the configuration we want to test. If the assumption made is incorrect, it may be necessary to exit the script and Now, before we can start a new iteration of the loop, we pause for a bit and give the user the option of exiting the script either via the The script thus provides threefold functionality: * It identifies the Apache configuration file we want to test. * It stops and starts a web server process at the press of a key * In ensures that the supply of semaphores is not exhausted<\/p>\n In summary: One press of the Enter key and Apache will be restarted using the most recently edited configuration:<\/p>\n We still lack intelligent access to the log files. curl<\/code>, find the bug, fix it in the server configuration and finally, verify it in the browser.<\/p>\ncurl<\/code> a number of different ways in the preceding tutorials. This puts us in a strong position. But adding one or two features to your toolbox is still worthwhile.<\/p>\n<\/a>$><\/span> curl<\/span> --cookie-jar \/tmp\/cookies.txt --cookie \/tmp\/cookies.txt --data "username=test"<\/span>\\<\/span><\/span>\n<\/a> --data<\/span> "password=xxxxxxxx"<\/span> http:\/\/localhost\/login.action<\/span>\n<\/a>...<\/span><\/span>\n<\/a>$><\/span> curl<\/span> http:\/\/localhost\/login.html --next --cookie-jar \/tmp\/cookies.txt --cookie \/tmp\/cookies.txt\\<\/span><\/span>\n<\/a> --data<\/span> "username=test"<\/span> --data "password=xxxxxxxx"<\/span> http:\/\/localhost\/login.action<\/span>\n<\/a>...<\/span><\/span><\/code><\/pre><\/div>\ncurl<\/code>, multiple requests are put together on a single line. What\u2019s interesting is the --next<\/code> option dividing the command line arguments. The parameters following --next<\/code> apply only to the right. This means that the rules above initially made a GET request, and then followed with a POST request on the same TCP connection and afterwards stored the session cookie in the cookie jar.<\/p>\nStep 2: Single Apache configuration file<\/h3>\n
httpd.conf_template<\/code>. It\u2019s a ready-to-go configuration including reverse proxy configuration, the core rules, the performance log supported by ModSecurity, etc. and is based on the configurations from the preceding tutorials. When I start working on a new problem, I duplicate this template file and then slightly adjust this configuration to bend it towards the right scenario.<\/p>\n<\/a>$><\/span> cp<\/span> httpd.conf_template httpd.conf_problem-of-the-day<\/span>\n<\/a>$><\/span> vim<\/span> httpd.conf_problem-of-the-day<\/span><\/code><\/pre><\/div>\nVim<\/code> is perfectly suited for editing Apache configurations, but the editor does not play such a key role so long as it supports syntax highlighting, which in my opinion is an invaluable feature.<\/p>\nStep 3: Apachex<\/h3>\n
curl<\/code> and a single Apache file provide us a good foundation for quickly customizing configurations and testing them just as fast. This usually involves a somewhat annoying step between these often repeated steps: restarting the web server. In the first two tutorials we started the sever each time using the -X<\/code> command line flag. I often work with -X<\/code>, because it does not put the server into daemon mode and instead allows it to run as a single process in the foreground. Any server crash shows up immediately in the shell. If we start the web server as usual and run it as a daemon, we then have to watch the error log and make sure that we don\u2019t miss a crash, which can involve all kinds of strange effects. Although it\u2019s possible to monitor for crashes this way, in my experience it is a surprisingly serious drawback. So I work with -X<\/code>.<\/p>\nCTRL-C<\/code>. This is accompanied by two more disadvantages: First, we have to enter the name of the configuration file, or access it from the shell history starting from the second iteration. What\u2019s even less pleasant is the loss of semaphores caused by using CTRL-C<\/code> to quit the web server. Semaphores are a Linux operating system communication structure used by the web server. The number of semaphores is finite and when we quit the web server, it often fails to release a reserved semaphore. Eventually, the supply of semaphores is exhausted and the server can longer be started. It will instead issue the following error message, which can easily be attributed to the semaphore problem:<\/p>\n<\/a>[emerg<\/span>] (28)No<\/span> space left on device: Couldn't create accept lock<\/span><\/span><\/code><\/pre><\/div>\n<\/a>$><\/span> sudo<\/span> ipcs -s |<\/span> grep<\/span> www-data |<\/span> awk<\/span> '{ print $2 }'<\/span> |<\/span> xargs<\/span> -n 1 sudo ipcrm sem<\/span><\/code><\/pre><\/div>\nipcs -s<\/code> to read a list of semaphores, select the right lines via the web server\u2019s user name and then use awk<\/code> to select the second column of the output. We extract the ID of the semaphore, which when then use xargs and ipcrm sem<\/code> to delete.<\/p>\napachex<\/code>. This script is available online (apachex<\/a>).<\/p>\n<\/a>$><\/span> <\/span>\n<\/a>.\/bin\/apachex<\/span> -h<\/span>\n<\/a><\/span>\n<\/a>.\/bin\/apachex<\/span><\/span>\n<\/a><\/span>\n<\/a>Script<\/span> to launch apache repeatedly via httpd -X.<\/span>\n<\/a><\/span>\n<\/a>The<\/span> script guesses the desired apache config and handles<\/span>\n<\/a> -<\/span> sudo<\/span>\n<\/a> -<\/span> pidfile<\/span>\n<\/a> -<\/span> semaphore cleanup<\/span><\/code><\/pre><\/div>\ntouch<\/code> the file you want. The script is aware of the different locations for configuration files and selects one file each time. It derives the httpd binary from this file. It then uses this information to start a loop. Then in the loop any web server process running is stopped, the semaphores are retrieved and the server restarted, with the help of sudo<\/code> if necessary. The script uses the configuration file identified at the beginning. We send the single process Apache to the background, but keep it running in the current shell. It is thus not yet running as a separate daemon in the background, but will continue outputting it\u2019s information to our shell. We will then continue to be informed if the process crashes.<\/p>\nq<\/code> key or initiating another iteration of the loop via the Enter key.<\/p>\n<\/a>$><\/span> apachex<\/span><\/span>\n<\/a><\/span>\n<\/a>Launching<\/span> apache on config file \/apache\/conf\/httpd.conf_problem-of-the-day ... ok<\/span>\n<\/a><\/span>\n<\/a>Press<\/span> [enter] to restart apache, enter [q] to stop apache and exit: <\/span>\n<\/a><\/span>\n<\/a>Stopping<\/span> active apache process ... ok<\/span>\n<\/a>Launching<\/span> apache on config file \/apache\/conf\/httpd.conf_problem-of-the-day ... ok<\/span>\n<\/a><\/span>\n<\/a>Press<\/span> [enter] to restart apache, enter [q] to stop apache and exit: q<\/span>\n<\/a><\/span>\n<\/a>Bailing<\/span> out ... ok<\/span><\/code><\/pre><\/div>\nStep 4: lastrequestsummary<\/h3>\n
curl -v<\/code> does provide us feedback about the results of a request, but with ModSecurity rules it\u2019s important to be able to follow processing on the server side. And with its chatty and unclear log file entries, ModSecurity poses a challenge, especially since a single request can quickly generate more logging that can fit in a window and most of this information is of no interest. What we are missing is a summary of a request including information from both the access log<\/code> and the error log<\/code>. The lastrequestsummary<\/code> adds just such an analysis (lastrequestsummary<\/a>):<\/p>\n<\/a>$><\/span> cat<\/span> lastrequestsummary<\/span>\n<\/a><\/span>\n<\/a>#!\/bin\/bash<\/span><\/span>\n<\/a>#<\/span><\/span>\n<\/a># Script that extracts information from the latest request in the access<\/span><\/span>\n<\/a># log and enriches this with info from the error-log.<\/span><\/span>\n<\/a>#<\/span><\/span>\n<\/a># Script is meant to be called regularly via watch.<\/span><\/span>\n<\/a>#<\/span><\/span>\n<\/a><\/span>\n<\/a>ACCESSLOG=<\/span>"<\/span>$1<\/span>"<\/span><\/span>\n<\/a>ERRORLOG=<\/span>"<\/span>$2<\/span>"<\/span><\/span>\n<\/a><\/span>\n<\/a>ACCESS_IGNORE_REGEX=<\/span>"(heartbeat)"<\/span><\/span>\n<\/a><\/span>\n<\/a>if<\/span> [<\/span> -z<\/span> "<\/span>$ACCESSLOG<\/span>"<\/span> ]<\/span>; then<\/span> <\/span>\n<\/a> echo<\/span> "Accesslog not passed via commandline. Please pass path to accesslog as first parameter. \\<\/span><\/span>\n<\/a>This is fatal. Aborting."<\/span><\/span>\n<\/a> exit<\/span> 1<\/span>\n