Elliquiy Security, part I

[Vekseid]

Elliquiy's time at Accelerated Web was not a bad one. The new server is faster, more reliable, and more customizable. However, there is something you do not see in that $50/month price tag and feature list for that host. Of every shared host I have seen, they were the most security-conscious by far. It is hard to fully express just how much I learned from them.

Not just because it would take awhile, but discussing the finer details of how Elliquiy is set up always bothers me - knowledge can be used against its giver. I also have a moral objection to throwing out technical details that I am not willing to make at least a courteous attempt at explaining. I hope that, if you are curious, you do not feel that I throw out random terms that hold no meaning without explanation. I am going to do my best to explain a lot of terms here, but if I fail and you are still curious, please feel free to ask.

This does not keep me from discussing concepts, things that a technically adept attacker would know, or someone who takes the time to read the public statements I have made so far, however. That still leaves quite a lot to talk about, so I am going to begin somewhat randomly. : )



The cute little xkcd strip there describes a type of injection attack - specifically a SQL injection. I found myself referring to this term quite a lot, so I figured a preemptive explanation is in order. The first apostrophe ('), when unescaped (escaping in most programming languages is usually done by putting a backslash - \ - before a special character), terminates the string, and the rest of it begins executing SQL commands, with the result described in the strip.

You can learn a lot about what runs Elliquiy by taking a good look at it. Not just on a technical level, but also socially. When I mention I am having issues with fcgi, that tells a technically experienced intruder some things, and in cyberspace, knowledge is both your sword and your shield.

There is no such thing as a perfectly secure system, however, not all measures are equally secure. It can even depend on the situation itself.

Case in point: Proactive versus reactive security.

Reactive security is used to respond to an attack - database backups, code integrity checks, and so on. Up to half a decade ago, some people would respond to attacks with attacks in kind, however, few serious intruders are going to be using their own machines, and that sort of behavior is often redirected against innocent targets. Reactive policies are generally in place to mitigate damage, not prevent it. Elliquiy does a remote database backup once a day - if everything within a 50 mile radius of Chicago vanished tomorrow, we would only lose a single day's worth of posts, at the most. We could be up and running again within 48 hours.

Proactive security is about preventing damage in the first place. It is represented by good passwords, input sanitization - preventing the injections described above - and so on. It is also the surge protector on your machine, locks... physical security is important to note as well, and more than just hackers can damage a system.

In terms of actually preventing someone from gaining access to a database, however, which would you rather have? A rather retarded example would be running a check every minute of every day checking for unauthorized database calls - reactively - rather than simply making sure that they cannot happen in the first place - proactively.

Elliquiy's database is one gigabyte in size. If it takes a minute for a reactive system to realize what is going on, there is a problem. It takes twenty seconds for Elliquiy's server to transfer the entirety of its database to a local machine on our host's network. Even across networks, access is fast - moving the entire database to our current host from Acceleratedweb took ninety seconds.

It is called the information superhighway for a reason. If your goal is to prevent data access, it is not enough to see an issue and react to it. You need to act before the issue even occurs. On the other hand, if your goal is to recover from acts of God, a reactive solution is appropriate.

They are complimentary, but it is important to be mindful of what methods are best suited to which purpose.

The following discusses a 'proactive' security issue - choice of software, specifically, mod_php versus cgi and fcgi. I will make an attempt at explaining these below.

Elliquiy as you see it largely runs on a set of what are called 'content management systems' or CMSes. This blog is a part of the Drupal CMS, and of the three that Elliquiy runs on, is probably the most well-structured, code wise. The wiki runs on Mediawiki - the same software that lies behind Wikipedia. The forums run Simple Machines Forum, which, when I chose it, was the best-coded out of the free forums that I could find. Each of these parse, display, and organize content in their own special ways, and have a slightly different use. Drupal controls the home page, the forums run the community, and the wiki is a nice feature for a handful of people here.

All of those are well-known applications that are written in php. Php originally stood for 'personal home page', and was not originally constructed as a language, so much as it was a collection of useful scripts that eventually morphed into something resembling a programming language. There is a rather subtle hint, there - which sounds more secure to you, a master-planned fortress, or a pile of pretty sticks thrown together?

As you might imagine, this is a tempting prospect for anyone looking to get into a site. A given script might be secure, a given site might be secure...

...there's a fun feature of Simple Machines Forum called 'Search Engine Friendly urls'. They are not particularly pretty:

This changes the format of URLs a little so search engines will like them better. They will look like index.php/topic,1.html.

This feature will not work on your server.

Bolded part, for what I see, but not every smf administrator may see, and instead be excited by a feature that they think will make their forum better.

It really seems so innocuous, does it not? Just making the urls slightly prettier, for some supposed search engine benefit. Not nearly so pretty as what we have under drupal, here. Enabling that, however, requires that Apache be used as the webserver, and that php code must be executed through an Apache module called mod_php. The former is probably not an issue - half the Internet runs on Apache.

To explain why the latter is a problem, however, I must go into greater detail.

Modern *nix systems run the webserver as a separate user than 'root', the equivalent of NT AUTHORITY\SYSTEM on Windows NT based machines, except that it is actually intended for human use. Now, you can read that page and get a good idea of just why compromising the webserver would be bad if it were running as root - the attacker would have full control of the system.

So instead we run apache as a much weaker user - named www-data, apache, httpd, whatever. Weak as this user is, it still needs to have read access to every file you wish for it to serve. And if you want to allow uploads, it needs to have write access at least somewhere. Chances are, if you are running a website like this, the sensitive information is actually stored in a database whose access credentials need to be stored in a file readable by a script that the webserver - or a process it calls - can read.

Note the mod_php thing, above. If that sort of syntax is permitted, that means Apache is computing the php script, not some lower-level, potentially more secure function which I will describe shortly.

That means anyone with access to the webserver potentially has access to the entire site. Access to the webserver includes anyone who is capable of compromising any script on that server. You might wonder, then, how would someone, if they are after a specific site, find such a compromised script?

It turns out that our friends at Microsoft will happily tell you who else is running on a given server. Naturally, the only thing that appears to be on Elliquiy's server is more Elliquiy, but, pick a random Dreamhost server...

Which is actually rather typical of oversellers like that. Dreamhost, for its reputation, is actually one of the better ones.

Most hosting companies that run mod_php like to tell their clients that they are safe because PHP's safe mode has been enabled, which, in theory, would limit the damage to a single account. So have a look at a random Server header and guess what could possibly go wrong with that theory:

Server: Apache/2.2.9 (FreeBSD) PHP/5.2.6 with Suhosin-Patch DAV/2 mod_ssl/2.2.9 OpenSSL/0.9.8i mod_perl/2.0.4 Perl/v5.8.8

So you can configure php to be 'secure', perhaps. Perl, however, is not going to hear php's configuration limits. It will happily execute a script uploaded by a compromised php script. You do not necessarily even need to have mod_perl installed, execing that perl script from an injected php upload will work just as well.

And now someone has free, unfettered access to your data. You will not even know, unless they make an actual change, because unless you control the server, you cannot even log the transactions you make inside the database, and you can bet that an oversold server is not going to be logging all database transactions.

Elliquiy gets injection attempts that attempt to take advantage of the above scenario on a daily basis - when you see a guest doing Nothing, or nothing you can see... that is often some script kiddie trying to get Elliquiy to run perl. I am fairly certain this sort of thing is common on any popular site.

Explanation: The first time I noticed this, it was done by way of attempting to craft a url that, were Elliquiy's code sufficiently broken, would execute code that called a perl script on a remote machine. Not very likely to work, but it only needs to work on one single site on a given server using a mod_php setup. When said server is running hundreds or thousands of scripts, you might see the problem.

So, how else can php be run?

Suppose instead that, rather than being given read access to the script itself, the webserver is instead only permitted to attempt to initiate an execution, and read the resulting output. We might then restrict who can actually execute the script, and prevent the webserver from even being capable of making writes to the website - which may be used for code injection.

The technical terms for those two situations are CGI - for Common Gateway Interface - and suexec, or 'Switch User, Execute'. I have linked to the Wikipedia entries there if you want more details.

Thus, compromising a single user account can then no longer compromise the entire server. Likewise, compromising a webserver is then incapable of actually compromising the actual accounts. That may not be seriously necessary for Elliquiy proper, but I do want to host stexxx, ewrimo and other sites on this machine, and I would hate to see one of them get compromised and expose Elliquiy's data.

And Elliquiy's data is your data.

Fcgi, or fastcgi, is a variant of the CGI described above that allows a site like Elliquiy to actually run on a single server without choking to death. It is picky, and has a lot of personality issues. People were getting 500 errors for awhile until I worked out the last of that, and it still causes problems for some.

A little temporary inconvenience is worth a little permanent safety. I apologize to those affected, but I know better now. I know a lot better now, about a great many things. It can be frightening just how seemingly innocent something is. A simple / instead of a ? after .php probably meant nothing to you before - on the surface, it seems just as safe. That / on a Simple Machines forum tells me something, though. About the website, its operator, the server it runs on and the people who run it - far more than merely using a less secure scripting model.

Elliquiy's security setup is not perfect, I will not claim it is. I do endeavor to perfect it, however, and perfecting security involves taking good lessons to heart. If a few people learn something from this, so much the better.

There is a great deal more that can be said, obviously. If this article interested some of you, please let me know. If so, I will be adding additional articles to this series in the future.

Old forum topic for feedback and comments.