A.J. Clark

Today I woke up to several alerts from Linode informing me that one of my VPS nodes was exceeding the Disk I/O threshold that I had set. Curiously this VPS is used as a HTTP web proxy and whilst it gets about 300-400 visitors per day (mainly china) this morning I was seeing over 800 visitors in Google Analytics.

Attempting to ssh to the server failed with timeouts although the PHP web application was still responding to requests over HTTP fine. I suspect sshd was failing to reverse-lookup my IP address in any reasonable amount of time, or perhaps IP Tables – (Note to self: Look in to why that happened). Thankfully Linode provide out of band / console access via SSH and AJax so all was not lost.

Looking at the Network rrdgraph it shows that the server was approaching 7Mbit/s of HTTP traffic and almost 50GB had been consumed today alone. Whilst the server seemed to handle the load without problem (minus ssh access) consuming 50GB+ per day would quickly max out my monthly data transfer allowance with Linode – this wasn’t acceptable. I modified the firewall to accept HTTP/HTTPS traffic from my IP only in order to investigate and the load suddenly stopped and SSH was alive again.

Initially I had suspected that some sort of automated bot was using ehproxy.info to do automated scans and attacks but a closer inspection of the traffic showed an even number of distributed IPs (all from China – as Google Analytics confirms) all clicking various porn sites. I guess everyone in China was feeling horny this afternoon!

Further analysis of the access.log shows that the server (Linode XenU VPS with 720MB of ram) was handling 62 hits sec (2428863/39600) and lighttpd was dealing with the load no problem. Pretty good considering this is a pure PHP application utilising php-cgi.

For the record, the top five IP addresses were:

Hits : IP address

A rather simple but often asked question was put forward to me today: How can I see the maximum amount of file descriptors my running process can open? (without killing the process!)

Typically one would say ‘check ulimit -n’ but lets say that a thread-driven or event-driven application like varnish or lighttpd is configured with an arbitrary amount of open file descriptors and you want to verify that they have taken effect before the application crashes.

A simple way to check this (atleast on Linux 2.6.26-1 or later) is to run:

svr1:~# awk '/Max open files/{ print $4}' /proc/$(pgrep -n lighttpd)/limits

1024

As you can see the above command returned the value of max open files for the running process. This means you can be sure that your lighttpd or varnish application will not suddenly die after being starved of file descriptors!

I have included the entire output of the limits table for the lighttpd process for completeness:

svr1:~# cat /proc/$(pgrep -n lighttpd)/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            ms
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             5824                 5824                 processes
Max open files            1024                 1024                 files
Max locked memory         32768                32768                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       5824                 5824                 signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Are you running Xen / “xm create” and you get this error?

Try this (RHEL/CentOS): service haldaemon start

- and have a nice day!

Recently, I investigated a security-compromised Apache web server. This particular web server hosted 500+ web sites and had fallen victim to script kiddies / attackers. An investigation revealed that the attackers exploitedan old Joomla CMS installation that a customer failed to update.  The attackers used the Joomla vulnerability to upload and execute a binary to /var/tmp.

An analysis of this nasty executable revealed that it was apparently capable of performing SYN floods and UDP floods as well as other nasty attacks. When launched, the executable connected to a private IRC server (active-sound.de:6667) where the bot received it’s orders. The Joomla installation was subsequently brought up to date, and the binary was removed. The owner of the web server realises that these problems are typical of an unmanaged hosting business, but never the less he would like to take steps to mitigate and/or prevent this happening again.

A simple and effective solution to this problem is to remount /tmp && /var/tmp file systems with noexec,nosuid options. If it is not possible to repartition the disk, you can mount these two directories as tmpfs. This protects against any backdoors or irc DDoS bots from actually executing thus preventing the server from falling in to the hands of attackers. This protection can be mean the difference between your server being rooted and/or being turned in to a DDoS / spam drone.

I have modified my VBulletin config file and enabled the use of APC as a VBulletin datastore. Smokeping is now reporting latency of 90-95ms. Not an immediately noticeable improvement but the average load on the server is 0.00 0.00 0.00 even with 100,000 hits per day. The performance improvements should be more measurable as the load increases.

To configure APC as a VBulletin datastore I simply uncommented the following line from includes/config.php:

$config['Datastore']['class'] = 'vB_Datastore_APC';

I run a small-medium VBulletin based web forum that receives a modest 110,000 hits / 2,000 – 3,000 unique visitors per day. I run this forum from an even more modest Xen-based VPS from Linode. Originally the forum started out on a big dedicated machine with 2GB of ram and a beefy processor, running VBulletin / Apache 2 mod_php / FreeBSD. I wasn’t happy with this solution, Apache 2 / mod_php could easily consume 2GB of ram due to the prefork MPM and the notion of running a complete PHP interpreter in each Apache process. I was convinced that it could create a much more efficient platform to host the VBulletin forum. 

I decided to move the forum to a more modern, efficient, platform consisting of a Xen-based VPS from Linode,  Debian GNU/Linux, and lighttpd. Not only is lighttpd measurably faster than Apache, using a VPS allows me to attain higher availability by default since most VPS servers (atleast at Linode) are usually of better specification (quad-core, RAID-1, dual PSU) than your low-end single disk dedicated server. Linode also allows me to rapidly deploy VPS instances in different data centres, and create HA/failover solutions. A VPS is also substantially cheaper than a dedicated server. The average dedicated server costs $150-$250 per month – the average VPS costs $20 per month. Win!

Whilst lighttpd and it’s FastCGI architecture based PHP happily serve out over 110,000 hits per day, running ApacheBench against the forum revealed that the server would max-out at serving 12 concurrent requests of the forum per second. Although due to the nature of HTTP and Web surfing users do not notice any problems. To gather a better understanding of how long the page took to load I installed smokeping and echoping. Smokeping reported that the forum took 150ms to load. This accounted for the 12 requests per second that ApacheBench reported. Not good… but at least I had a clearer picture of how long things were taking to load.

In an attempt to further optimize the forum I installed the XCache PHP Accelerator. Smokeping showed a measurable improvement of 50ms, taking the forum load time from 150ms to 100ms. Although XCache was working perfectly out of the box I decided to compare it with PHP-APC. After I installed PHP-APC smokeping reported the same drop from 150ms to 100ms, and the APC admin URL reported a 98% cache hit rate within 20 minutes of running, and with a default setting of 30MB of cache. Overall, a satisfactory performance improvement.

Below you can see the two ‘dips’ in the graph where adding a PHP accelerator improved VBulletin performance. The first dip is XCache, the second is APC:

Today I provisioned a new RedHat VMware ESX 2.5 guest at the request of the web hosting department. They requested a standard build which consists of a 9GB disk for the Operating System. They also requested an additional 80GB disk for data. The standard build consists of (only) OS on local disk, with application data on the SAN. This is build is platform agnostic and thus applies to both physical servers and VMware guests.

The normal procedure is to attach all of the storage while the VMware guest is powered off, but today I used this as an opportunity to get RedHat to add the 80GB SCSI data disk on the fly.

I powered up the VM and installed the OS according to the build specification. I then attached the 80GB SCSI data disk while the VMware guest was running. VMware reported this operation a success. The next task was to get Red Hat to “re-scan” the SCSI bus and detect the new disk.

The command: echo “- – -” > /sys/class/scsi_host/host0/scan worked without error. I then ran “fdisk  -l” and immediately  I was presented with a new /dev/sdb disk ready to be configured.

NOTE: The Red Hat Knowledge Base (RHKB) states that the recommend way to add new SCSI disk under Red Hat Linux is to switch off the server before attaching the new disk. The above procedure was tested on RedHat EL ES 4.6 (Nahant)