The Elliquiy LAMP Stack: Introduction and Overview

[Vekseid]

The Elliquiy LAMP Stack

1: Introduction and Overview
2: General Configuration
3: General Security
4: IPTables configuration
5: Postfix configuration
6: ntp configuration
7: Apache compilation and configuration
8: MySQL compilation and configuration
9: PHP compilation and configuration
10: Conclusion and future plans

Introduction

This 'series' covers the LAMP (Linux - Apache - MySQL - PHP) setup and configuration of the Debian Lenny installation I have running Elliquiy. As that might imply, it is tuned rather specifically to running Elliquiy proper - if you use this for your own configuration, you will of course want to adjust your parameters accordingly. As a general disclaimer, I am not responsible for anything you do. : )

So, in order to increase reliability and uptime, and reduce lag spikes, I ordered a dedicated server from Ubiquity for Elliquiy.

When I first started looking for VPS and dedicated hosts back in July, I was pretty rusty with *nix working as an actual, high-capacity server. Don't get me wrong - Acceleratedweb used FreeBSD, I've gotten Linux From Scratch installs working on multiple machines, and at one point or another I've toyed with AIX, Solaris, and CentOS. Obviously, I have a rather disturbing tendancy to keep returning to Debian. May be a character flaw.

None of that really teaches one what involves a high-load server, however. Elliquiy's server processes roughly a million database-driven php page landings per day - each including multiple writes - and some fifteen million database queries per day, on average. A piece of rather poorly thought-out AJAX turned into a most wonderful DDOS simulation.

I did not let it kill the server, but here are the statistics for when I finally took down the bad ajax, during 7:31 PM CST, March 3rd, 2009:

elliquiy.com
Shoutbox POSTs: 4,137 (The Ajax script used POST to retrieve data... I'll talk about what I learned from that later)
Other POSTs: 3
PHP GETs: 126
Other GETs: 1,069
Total: 5,335

bluemoonroleplaying.com
POSTs: 5
PHP GETs: 167
Other GETs: 751
Total: 918

MySQL
Connections: 134
Queries: 62,317

For one minute. For one minute, this little Core2 Duo machine served over a hundred requests and a thousand database queries per second, eighty percent of the requests dynamic. And despite what should have been a DDOS, the other major community on this server hardly even noticed. Hell, Elliquiy hardly even noticed up until I pulled the plug on it. Except that the shoutbox was there and things really were getting to the point of collapse - at that point Elliquiy was functioning at 80% of its capacity and climbing.

I learned quite a bit that day, about that minute, about what this server's peak performance really was. I routinely see people talking about needing to purchase additional servers, often for loads involving far less than Elliquiy's average. Aside from wishing that I had the resources to spare for that,  I am struck with curiosity. I have seen one individual brag about a cluster of nineteen dual Xeon FreeBSD machines having a peak load of 4,400 requests per minute. I let Elliquiy get to 4,438 dynamic requests in a minute and the thin lady did not even sing yet.

Now, that shoutbox no longer exists - I've replaced it with an AJAX Chat install which is far superior, though as with everything I have my bones to pick with it. Still, I am forced to wonder - am I doing something exceptionally right, are others doing something exceptionally wrong, am I just misunderstanding, or is there a mix involved?

To help get to the bottom of that - and maybe save some people some money - I am posting the full construction and configuration I built for the LAMP stack here. I've documented nearly everything, performed tests, and seen the results. Since this is my sixth Lenny setup, this also works as a handy reference for myself, as well. I hate wondering what I did the last time. : )

I have tweaked performance a bit since the above incident, of course. The result appears to be approximately optimal. There is little remaining for me to take away without compromising security or removing a desired feature, though a lot of values may benefit from slight tweaks.

Currently, everything except for php uses less than 5% of one core on average. Sphinx, MySQL, Postfix, Apache, the kernel, everything. MySQL uses about 60% of that and Apache most of the rest. PHP, on the other hand, happily swallows several times that. I will go over some of the implications of that in the conclusion, but the wall my optimization efforts have hit is a pretty clear one - I did a lot of testing on various php compilations.

Security wise, I have a lot of confidence in the stack itself, if not so much for the applications I run on it (currently SMF, phpBB, MediaWiki, MyBB, and until recently Drupal). Suhosin goes a long way towards mitigating this, however, and it goes even further towards pointing out potential problems. Local php.ini files permit a different Suhosin (and other) configuration for every site on the machine, which are in turn slaved to the master configuration. Php is called through mod_fcgid + suexec, and a custom iptables script I wrote has proven to be a surprisingly effective firewall, so far stopping one DOS attempt and notifying me of an intrusion attempt by the same individual.

The reasons for my lack of confidence in web application security is probably a subject for a different sort of article if there is interest.

Overview

We are hosting elliquiy.com off of what amounts to a low-end (in 2009) computer system - a 2.4GHz Core2Duo machine with 4 gigabytes of DDR2 667 ram. It does have a pair of 250GB SATA II disks in a hardware RAID 1 configuration - which is nice. The chosen chassis only has room for two drives, however, so no tripling, and no hotswap. The failings of being poor : /

We were given five ips with the server, and we currently host four active or planned-active projects. Elliquiy and Blue Moon Roleplaying are the major two, though we are hoping that Personal Roleplaying will also get to a significant size. The fifth I'm testing other things that need an ip - nginx, etc.

We are hosted by Ubiquity Hosting out of their data center in Chicago. I use them for several reasons.

1) Chicago is both rather conveniently located in the middle of the United States, and happens to be the nearest major Internet hub to where I live. It's eight hops from my router to elliquiy.com - my own ISP's website is twelve.
2) Chicago, being in Illinois, is a bit more liberal than some other states in the union. Florida does a lot of hosting, and Chris Wilson of nowthatsfuckedup.com may have wished for a less conservative venue. The same is true of Pennsylvania and Karen Fletcher of Red Rose Stories. While I am willing to face obscenity charges for what we host here, I would rather not throw myself in that path.
3) Going along with the above, Ubiquity permits legal adult traffic. Most dedicated hosts permit this, of course, but not all VPS providers do so. In Ubiquity's case, this was pretty rare.
4) Ubiquity does not have a retarded IRC policy. It's even rather permissive - if you set up an IRC server they place you on a segregated network. Several hosts I had contacted explicitly banned even the embedded Mibbit client we use, as well as the pjIRC embedded java client. I can understand not wanting to permit servers hooked to a network, individual servers or even bots, but if a company's technical staff cannot comprehend such a basic concept as embedded client software, do you really want to trust your data with them? I find some vindication in that one such company, previously receiving rave reviews, is not doing so hot any longer.

Ubiquity isn't the only company that meets these needs out of Chicago, of course, but they have my money and have given me a server with 100% uptime save for when I fuck it up, and started me off at the most reasonable price point I could find.

About Compiling in Debian

I discuss compiling from source a bit, which is not always described, and I have been asked. The practice I use for this is:

Code Select Expand


apt-get install fakeroot build-essential debhelper
apt-get build-dep apache2
apt-get build-dep php5
apt-get build-dep mysql-server


I switch to my administrative user (see General Security), cd /usr/src (remember they've been added to the src group) and run apt=get source package where the package is one of the above - php5, apache2 or mysql-server. apt-get automatically downloads and untars the resulting source tarball. I enter the directory and edit debian/changes to add my own version number (remember it needs a number at the end...) and debian/rules appropriately.

dpkg-buildpackage -b package builds 'package', which dumps one or more .deb files into the parent directory to be installed normally.

The specific edits to a given package's debian/rules file are given in the appropriate sections.

Notes

These documents were written with a dedicated, 64-bit server in mind, running on the AMD64 distribution of Debian Lenny. I do document some things about CentOS, VPSes, or 32 bit platforms at some points, but do keep in mind what this was written for. I don't go over every last, final detail in these documents. Stuff will need to be extrapolated accordingly - if you apply some of the ideas here to your server, you will have to adjust them to your needs appropriately.

I make no warranties or guarantees about the information provided here - use it at your own risk. I fully acknowledge that this setup is less than perfect, something I'll talk more about later, but I welcome criticism anyway.

Thank you : )

[Vekseid]

Alright, this took some time >_>

It is a bit outside the scope of this place as a role playing forum, so it may be a bit on the technical side. I apologize. But, hopefully some people may find it useful : )