Elliquiy's Server Layout and Backup Policy

[Vekseid]

This thread may be a bit technical, but it may help some people in understanding what is going on in the event of a service outage. Also, it makes my backup policy clear, for the paranoid.

Elliquiy and all other sites I host use three servers:

Isis: Intel Nehalem E5520 (quad core Xeon), 12GB of RAM and Four 300gb WD Velociraptor drives in two RAID 1 arrays. Located in Chicago. Runs Debian Squeeze.
- Runs every dynamic website I own or host, at this point. Elliquiy, Blue Moon Roleplaying, Lunar Radiance, Personal Roleplaying, World Enough and Time, Fantasy Compendium, Whip and Apple, Vekseid.com, Cecilia's blog, and a number of others.
- It's the mail server, database server, the scripting server, and the primary webserver. It does just about everything you expect a server to do. If it goes down, everything of note, except for voice chat, is effectively down.
- Server can hold a maximum of 96gb of RAM. Almost all slowdowns are caused by being low on RAM, at this point, so a new server is probably some distance off.

Hera: Intel Lynnfield E5520 (quad core Xeon), 2GB of RAM, two 250gb SATA drives. Linked to Isis on a dedicated gigabit connection. Located in Chicago, second server down from Isis, in the same cabinet. Runs Debian Wheezy.
- It's primary function is as a MySQL replication slave for Isis. This allows me to do backups, search indexing, and other functions without impacting Isis's performance.
- Runs a few static sites, most notably http://roleplayingerotica.com/ which I add to as more people have me host them.
- Runs Sphinx Search, which powers the search engine for Elliquiy and Blue Moon. This saves a lot of disk load for E and BM, especially during reindexing. Of course this also means that if the server is down, search won't work.
- Runs the voice chat server, Murmur, for Mumble.
- Runs the DNS resolver. This being down will make some sites I run a bit slower for certain things, but not in ways most people care about.
- Also serves as a handy test platform.
- Server can hold a maximum of 6gb of RAM. There's a bit to be gained from upgrading, the primary advantage being stability for the voice chat during read intensive tasks.

Amaterasu: A 60GB VPS in Los Angeles. Runs Debian Squeeze.
- I mostly use it as a remote backup, in the off chance that something actually e.g. takes out the entire cabinet in Chicago.



If all forums and sites are down (not even the ip address works), but the voice chat is still up: Either the server or software is being rebooted, or something serious happened. I tend to be pretty up front about when these occur.

If Mumble and Search is down: Chances are I'm rebooting the backup server or something. If it's toasted, it isn't like we've lost a whole lot.

If Mumble and all websites are down: Chances are the host is having an outage (again). Remote backups of the database are pulled daily, in the event of a disaster we might loose the last day of posts. The files are backed up less often (avatars, image uploads to the wiki, etc), but I'm gunning for weekly for them.



Backup Policy
- The database for all sites is effectively backed up to the slave (Hera), in real time. The slave makes two snapshots of the database each day.
- A backup of all website files (Image uploads, avatars, etc) and then some is made once each day.

- Amaterasu pulls the 1 AM database snapshot every morning.
- Amaterasu pulls the website snapshot three times a week.