The Elliquiy LAMP Stack: Apache compilation and configuration

Started by Vekseid, March 28, 2009, 05:03:27 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Vekseid

The Elliquiy LAMP Stack

1: Introduction and Overview
2: General Configuration
3: General Security
4: IPTables configuration
5: Postfix configuration
6: ntp configuration
7: Apache compilation and configuration
8: MySQL compilation and configuration
9: PHP compilation and configuration
10: Conclusion and future plans




Apache

Six sections covering 'L' and we finally get to 'A'.

Our goal, here, is to permit fastcgi (via mod_fcgid) + suexec to execute php. Even though I am in full control of the system here, I operate under the assumption that it is possible for an account on the machine to be compromised. This gives some peace of mind in the event of that occurance.

In addition to apache, you will want to grab mod-evasive and mod-fcgid.

Tuning apache, for my server, is about reducing its memory footprint. During the 'DDOS', Elliquiy began to collapse not because it was running out of horsepower, but because it was being driven deeply into swap, primarily by the combination of php, apache and mysql. I use mpm-worker for that reason - it's stable enough, and while slower under peak, pure-webserving load, its smaller footprint lets me devote a more resources to php and MySQL instead.

Obviously, nginx fans will have something to say about that - but that's a topic for the conclusion : )




Apache Compilation

As explained above, compiling Apache is not done so much for the speed boost (though there is some) but rather to reduce its memory footprint. I use mpm-worker from this but it should work with any mpm.

My configuration section in debian/rules looks as follows:

CFLAGS = -O2 -march=core2

AP2_COMMON_CONFARGS = --enable-layout=Debian --enable-so \
                      --with-program-name=apache2  \
                      --with-ldap=no --with-ldap-include=/usr/include \
                      --with-ldap-lib=/usr/lib \
                      --with-suexec-caller=www-data \
                      --with-suexec-bin=/usr/lib/apache2/suexec \
                      --with-suexec-docroot=/var/www \
                      --with-suexec-userdir=public_html \
                      --with-suexec-logfile=/var/log/apache2/suexec.log \
                      --with-suexec-uidmin=1015 \
                      --with-suexec-gidmin=1015 \
                      --enable-suexec=static \
                      --enable-log-config=static --enable-logio=static \
                      --with-apr=/usr/bin/apr-1-config \
                      --with-apr-util=/usr/bin/apu-1-config \
                      --with-pcre=yes

AP2_MODS_CONFARGS =   --disable-authn-alias --disable-authnz-ldap  \
                      --disable-disk-cache --disable-cache \
                      --disable-mem-cache --disable-file-cache \
                      --disable-cern-meta --disable-dumpio --disable-ext-filter \
                      --disable-charset-lite --disable-cgi \
                      --disable-dav-lock --disable-log-forensic \
                      --disable-ldap --disable-proxy \
                      --disable-proxy-connect --disable-proxy-ftp \
                      --disable-proxy-http --disable-proxy-ajp \
                      --disable-proxy-balancer --enable-ssl=static \
                      --disable-authn-dbm --disable-authn-anon \
                      --disable-authn-dbd --disable-authn-file \
                      --disable-authn-default --enable-authz-host=static \
                      --disable-authz-groupfile --disable-authz-user \
                      --disable-authz-dbm --disable-authz-owner \
                      --disable-authnz-ldap --disable-authz-default \
                      --disable-auth-basic --disable-auth-digest \
                      --disable-dbd --enable-deflate=static \
                      --enable-include=static --disable-filter \
                      --enable-env=static --disable-mime-magic \
                      --enable-expires=static --disable-headers \
                      --disable-ident --disable-usertrack \
                      --disable-unique-id --enable-setenvif=static \
                      --disable-version --disable-status \
                      --disable-autoindex --disable-asis \
                      --disable-info --disable-cgid \
                      --disable-dav --disable-dav-fs \
                      --disable-vhost-alias --disable-negotiation \
                      --enable-dir=static --disable-imagemap \
                      --enable-actions=static --disable-speling \
                      --disable-userdir --enable-alias=static \
                      --enable-rewrite=static --enable-mime=static \
                      --disable-substitute

AP2_CFLAGS = $(CFLAGS) -g -pipe -I/usr/include/xmltok -I/usr/include/openssl -Wall
AP2_LDFLAGS = -Wl,--as-needed

#support noopt building
ifneq (,$(findstring noopt,$(DEB_BUILD_OPTIONS)))
  AP2_CFLAGS += -O0
else
  AP2_CFLAGS += -O2 -march=core2
endif


This is pretty straightforward. Obviously, if you use this, make sure to change core2 to whatever architecture you are running on, and note the vast swath of disabled features - enable anything you might think you will miss. Apache is not quite the hog that php is, so enable accordingly. You will also want to set the minimum uid and gid values for suexec to your lowest 'web user' id - don't let it be possible for someone to suexec to your admin account.

Also, make sure that other mpms are taking the same configuration parameters as mpm-worker. This was a rather annoying gotcha.


mpm-%: patch-stamp
        dh_testdir
        mkdir -p $(BUILD)/$*
        cd $(BUILD)/$* ;\
        CFLAGS="$(AP2_CFLAGS)" LDFLAGS="$(AP2_LDFLAGS)" $(CONFFLAGS) $(REALCURDIR)/configure --srcdir=$(REALCURDIR) $(AP2_COMMON_CONFARGS) $(AP2_MODS_CONFARGS) --with-mpm=$*  ;\
        $(MAKE) ;\
        ./apache2 -l |grep -v $* > mods.list
        touch $@


After you compile, you are going to want to move the configuration for the modules you just compiled in into your httpd.conf file (see below).




Configuring Apache

The first order of business is to make sure your compiled-in modules work correctly. For me this means copying the configurations into httpd.conf for the needed modules - mod_authz_host, mod_deflate, mod_dir, mod_expires, mod_mime, mod_setenvif, and mod_ssl. Most of these are fairly straightforward, though you probably want to tune deflate, dir, and expires specifically.


# apache.conf

Timeout 45
KeepAlive On
MaxKeepAliveRequests 0
KeepAliveTimeout 6

<IfModule mpm_worker_module>
    StartServers          8
    ServerLimit           8
    MaxClients          512
    MinSpareThreads      64
    MaxSpareThreads     512
    ThreadsPerChild      64
    ThreadLimit          64
    MaxRequestsPerChild   0
</IfModule>


I set timeout to 45 because php and mod_fcgid will die after 40 seconds anyway, basd on my current configuration. KeepAlive is very useful, though it is important to keep KeepAliveTimeout fairly low, unless you want to increase the amount of threads you are working with dramatically.

ThreadLimit = ThreadsPerChild and StartServers = ServerLimit save on memory usage.


# /etc/apache2/ports.conf
NameVirtualHost 1.0.0.4:80
NameVirtualHost 1.0.0.5:80
Listen 80

<IfModule mod_ssl.c>
    # SSL name based virtual hosts are not yet supported, therefore no
    # NameVirtualHost statement here
    Listen 443
</IfModule>


I listen on two of my five ips. Two more ips are hardcoded and the fifth is reserved for testing other webserver apps >_>


# /etc/apache2/conf.d/security
ServerTokens Prod
ServerSignature Off
TraceEnable Off
EnableSendfile Off
LimitRequestBody 16777216
<Directory />
  Order Deny,Allow
  Deny from all
  Options None
  AllowOverride None
</Directory>


As minimal as possible. No snooping, no uploads larger than what we'll let php handle, no advertising.




mod-evasive

I usually give mod-evasive some pretty generous default settings. It's just nice to have available in the event of minor DDOS situation.


# mod-evasive.conf
<IfModule mod_evasive20.c>
  DOSHashTableSize 8191
  DOSPageCount 4
  DOSSiteCount 64
  DOSPageInterval 1
  DOSSiteInterval 1
  DOSBlockingPeriod 6
  DOSEmailNotify admin
</IfModule>


Where admin is the name of your administrator account.




mod_fcgid

Along with suexec, mod_fcgid permits us to have comparable speed to mod_php while keeping a more secure layout.


# fcgid.conf
<IfModule mod_fcgid.c>
  AddHandler    fcgid-script .fcgi
#  BusyTimeout 300
#  IdleTimeout 300
  IdleScanInterval 60
  BusyScanInterval 60
  ProcessLifeTime 3600
  MaxProcessCount 128
  DefaultMinClassProcessCount 0
  DefaultMaxClassProcessCount 64
#  IPCCommTimeout 40
#  IPCConnectTimeout 3
  ErrorScanInterval 3
  ZombieScanInterval 3
  PHP_Fix_Pathinfo_Enable 1
  MaxRequestsPerProcess 0
</IfModule>


DefaultInitEnv, IPCCommTimeout and IPCConnectTimeout need to be specified within the virtualhost itself if you are using virtualhosts - and I think BusyTimeout and IdleTimeout are similar.

MaxProcessCount determines the maximum number of master processes that will be created in total. Children created through your php calling script do not count towards this limit.

DefaultMinClassProcessCount is self-explanatory, as is MaxRequestsPerProcess. There is little problem with setting them both to zero, at least with modern php versions. Using the busy and idle timeouts to refresh processes is far superior to force killing them after x requests, performance wise.

PHP_Fix_Pathinfo_Enable 1 corrects Path information for php.

DefaultMaxClassProcessCount is the per-user limit for master processes. So if you skip the suexec step, this will be the total - but with suexec the real total is going to depend on how many users (not virtualhost declarations) you have. You use this to cap the ability of one 'user' to dominate your server.

Setting it to 1 has the unique feature of letting you use APC or xcache's shared memory, but this has some flaws, at least with mod_fcgid:
1) You need to have php handle threading... which is distressingly slow compared to mod_fcgid's management.
2) Perhaps it is mod_fcgid rather than php, but this configuration has a difficult time properly handling POST requests, as if they require the full attention of a master php process.

Otherwise, choose 4*number of processor cores as a minimum. If you have a lot of uploaders or downloads that go through php, you will want to increase this value accordingly. You can still make use of APC or XCache's opcode cache, though this is not very memory efficient. eAccelerator handles shared memory properly on a per-user basis, however it has not been updated to handle php 5.2.




VirtualHost Configuration


<VirtualHost 1.0.0.4:80>
  ServerAdmin email@elliquiy.com
  ServerName elliquiy.com
  ServerAlias *.elliquiy.com

  SuexecUserGroup elliuser elliuser

  DocumentRoot /var/www/ellidir/docs/
  <Directory /var/www/ellidir/docs/>
    Options FollowSymLinks IncludesNoExec
    <FilesMatch "\.php$">
      SetHandler fcgid-script
      FCGIWrapper /var/www/ellidir/fcgi/php.fcgi .php
      Options +ExecCGI
    </FilesMatch>
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>

  <Location /elluiki/images>
    Options None
    SetHandler None
  </Location>
 
  <Location /forums/avs>
    Options None
    SetHandler None
  </Location>
 
  RewriteEngine on
  RewriteCond %{HTTP_HOST} !^(elliquiy\.com|1\.0\.0\.1)$ [NC]
  RewriteRule ^(.*)$ http://elliquiy.com$1 [R=301,L]

  RewriteCond %{HTTP_REFERER} !^(https?://([a-z0-9\.\-]*\.)?(elliquiy)\.com/|https?://1\.0\.0\.1|$) [NC]
  RewriteCond %{REQUEST_URI} !^/pics/
  RewriteCond %{REQUEST_URI} !\.php
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} -f
  RewriteRule .*\.(.*)$ /pics/nohotlink.gif [R,NC,L]

  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
  RewriteRule ^/wiki/?(.*)$ /elluiki/index.php?title=$1 [L,QSA]

  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_URI} !^/wiki
  RewriteRule ^(.*)$ /index.php?q=$1 [L,QSA]

  ErrorLog "|sudo -u elliuser /usr/bin/cronolog /home/elliuser/logs/%Y/%b/error-%d.log"
  CustomLog "|sudo -u elliuser /usr/bin/cronolog /home/elliuser/logs/%Y/%b/access-%d.log" combined
</VirtualHost>

<IfModule mod_ssl.c>
<VirtualHost 1.0.0.1:443>
  ServerAdmin email@elliquiy.com
  ServerName elliquiy.com
  ServerAlias *.elliquiy.com

  SuexecUserGroup elliuser elliuser

  DocumentRoot /var/www/ellidir/docs/
  <Directory /var/www/ellidir/docs/>
    Options FollowSymLinks IncludesNoExec
    <FilesMatch "\.php$">
      SetHandler fcgid-script
      FCGIWrapper /var/www/ellidir/fcgi/php.fcgi .php
      Options +ExecCGI
    </FilesMatch>
    AllowOverride None
    Order allow,deny
    allow from all
  </Directory>

  <Location /elluiki/images>
    Options None
    SetHandler None
  </Location>

  RewriteEngine on
  RewriteCond %{HTTP_HOST} !^(elliquiy\.com|1\.0\.0\.1)$ [NC]
  RewriteRule ^(.*)$ [url]https://elliquiy.com[/url]$1 [R=301,L]

  RewriteCond %{HTTP_REFERER} !^(https?://([a-z0-9\.\-]*\.)?(elliquiy)\.com/|https?://1\.0\.0\.1|$) [NC]
  RewriteCond %{REQUEST_URI} !^/pics/
  RewriteCond %{REQUEST_URI} !\.php
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} -f
  RewriteRule .*\.(.*)$ /pics/nohotlink.gif [R,NC,L]

  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
  RewriteRule ^/wiki/?(.*)$ /elluiki/index.php?title=$1 [L,QSA]

  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
  RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
  RewriteCond %{REQUEST_URI} !^/wiki
  RewriteRule ^(.*)$ /index.php?q=$1 [L,QSA]

  ErrorLog "|sudo -u elliuser /usr/bin/cronolog /home/elliuser/logs/%Y/%b/error-ssl-%d.log"
  CustomLog "|sudo -u elliuser /usr/bin/cronolog /home/elliuser/logs/%Y/%b/access-ssl-%d.log" combined

  SSLEngine               on
  SSLCertificateFile      /etc/apache2/certs/elli.crt
  SSLCertificateKeyFile   /etc/apache2/certs/elli.key
  SSLCertificateChainFile /etc/apache2/certs/sub.class1.server.ca.crt
  SSLCACertificateFile    /etc/apache2/certs/ca.crt

  <FilesMatch "\.(cgi|shtml|phtml|php|pl|py)$">
    SSLOptions +StdEnvVars
  </FilesMatch>

  BrowserMatch ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
</VirtualHost>
</IfModule>


So... where to begin.

1) Set email, servername, alias.
2) Set docroot
3) We only execute files in the docs directory with a .php extension.
4) I like includes, but Exec can eat flaming death and die.
5) I don't use or permit .htaccess files.
6) Don't allow upload directories to have executable files. Or any shennanigans, at all.
7) All requests are forced to http(s)://elliquiy.com
8) If it's a file, and doesn't have a .php extension, and not in my intended sharable directory, hotlinkers see a nice image spamming my site in their face.  Note that, because we don't use .htaccess, %{DOCUMENT_ROOT} needs to go in front as well.
9) Pretty wiki url redirect for our wiki
10) Same for the custom CMS I wrote to replace Drupal.
11) Logs get piped through sudo to cronolog. Logs go into the home directory. Probably going to move ErrorLog to syslog instead, in order to be cleaner.
12) SSL - fairly straightforward.

php.fcgi needs to have the same user/group as the person you are suexecing to. It's simply a shell script that calls php - I go into it in more detail in the PHP chapter.

If I go further with Apache I would probably look into using multiple handlers, particularly if I could split GET, HEAD and POST requests between multiple php processes - possibly even calling different configurations for each.