A.J. Clark

Today I provisioned a new RedHat VMware ESX 2.5 guest at the request of the web hosting department. They requested a standard build which consists of a 9GB disk for the Operating System. They also requested an additional 80GB disk for data. The standard build consists of (only) OS on local disk, with application data on the SAN. This is build is platform agnostic and thus applies to both physical servers and VMware guests.

The normal procedure is to attach all of the storage while the VMware guest is powered off, but today I used this as an opportunity to get RedHat to add the 80GB SCSI data disk on the fly.

I powered up the VM and installed the OS according to the build specification. I then attached the 80GB SCSI data disk while the VMware guest was running. VMware reported this operation a success. The next task was to get Red Hat to “re-scan” the SCSI bus and detect the new disk.

The command: echo “- – -” > /sys/class/scsi_host/host0/scan worked without error. I then ran “fdisk  -l” and immediately  I was presented with a new /dev/sdb disk ready to be configured.

NOTE: The Red Hat Knowledge Base (RHKB) states that the recommend way to add new SCSI disk under Red Hat Linux is to switch off the server before attaching the new disk. The above procedure was tested on RedHat EL ES 4.6 (Nahant)

Update:  I have updated my Highly Available Mail Cluster Architecture.

Checkout: http://ajclark.wordpress.com/2009/03/05/highly-available-mail-cluster-v2/

In my spare time I look after a mail cluster for a small-ish hosting / consulting company. The mail infrastructure used to consist of one physical dedicated server which was rented by the consulting company. It was my job to manage this server. The server was basically a FreeBSD box running Plesk 7.3 which worked very well for a year or so.

Unfortunately with the rising costs of energy most dedicated / colocated server ISPs are raising their prices. I know Layered Tech are one such company that did this and angered many of their customers. The consulting company were faced with a decision, either pay the increased ISP prices or migrate to a virtual Xen based platform which was a fraction of the cost and could be made Highly Available.

Design goals

Since this is a new project I felt it was important to have design goals.

1. Keep it simple.
2. Keep it consistent.
3. Use stock software packages – All system builds should adhere to Debian release configurations.
4. Platform must scale.
5. KEEP IT SIMPLE!

The advantages of a Xen platform

The plan was to purchase for Xen domU “linodes” systems from the ISP linode.com. Linode are a great ISP who provide Xen (and previously UML) domUs across four different data centres in the US. Their ingenious control panel lets you quickly provision a Xen domU in a data centre of your choosing, and/or clone/migrate existing domUs. The advantages of using a Xen platform:

• Cost saving – Able to purchase four domU linodes for less than the cost of one dedicated server.
• High Availability – Linode assist in HA by placing your domUs on different physical servers. They also allow IP takeover.
• Simple management – Using Debian as opposed to FreeBSD allows us to quickly patch and update the OS.
• Efficient resource usage  - A single “dom0″ physical server hosting eight domUs is more efficient than a single dedicated server hosting Web / Mail. This also gives us that ‘Green Energy’ feel good factor.

The Mail Cluster Architecture

The Mail Cluster design was based on Linux-HA & Heartbeat. There would be a total of four Xen domUs in two different data centres. In each data centre there would be a primary (active) and secondary (inactive) domU. All domUs would have identical configurations. There would only be one Xen node serving Mail services at any one time. This means that three domUs are effectively inactive until they are required.

Failover Architecture

Should the primary server DC1-SVR1 fail DC1-SVR2 will instantly take over using Linux Heartbeat. This provides a near sub-second failover. Should the entire DC1 go offline DNS will be manually switched over to DC2-SVR1. Primary DNS is hosted on DC1-SVR1 and slave DNS on DC2-SVR1. All DNS TTLs are 300 seconds.

• DNS is kept in sync using standard master / slave configuration.
• MySQL is kept in sync with master / slave replication (1 master, 3 slaves)
• Web htdocs are kept in sync with rsync at regular intervals.

This architecture gives us protection against both a physical server failure and a complete data centre failure.

The failover architecture was designed on the principle that it is not acceptable to have a multi-hour server outage (e.g. physical server failure) but it is acceptable to have a 10 minute outage while DNS fails over (e.g. complete DC failure)

Mail Cluster Software Components

One of the key goals of the fail over design is to make as many applications as possible use MySQL to store data. This simplifies data synchronization.

• Debian stable
• Apache
• Postfix – Using MySQL for all maps
• Dovecot – Using MySQL Postfix database
• MySQL – Replication enabled. Master: DC1-SVR1, Slaves: DC1-SVR2, DC2-SVR1, DC2-SVR2.
• BIND
• Roundcube mail
• Postfix Admin
• Bindgraph
• Mailgraph

Future improvements to the Mail Cluster

The main thing that bugs me is that there are three domUs that are largely inactive, except for MySQL and htdoc replication. This complies with the number one design goal of keep it simple, however, I can’t help but feel this is wasteful.

I would like to replace the rsync data syncs with iSCSI. Each set of servers could fail over to an iSCSI target within their respective data centres. This would be more efficient than periodic rsyncs from cron.

This weekend I decided to migrate Antihoe.org from a typical self-hosted Apache / MySQL / PHP (AMP) stack running on a dedicated server to wordpress.com. I have also moved my image gallery away from AMP and over to Google Picasa, although I may change over to Flickr.

The original plan was to migrate everything in to the Google Cloud (mail, docs, RSS reader, blog, image gallery) to remove the burden of system administration. Having my digital life in the Google Cloud would not only free up a vast amount of my time but it would also give me the advantages of Single Sign On (SSO) for all apps and a consistent User Interface (UI).

Unfortunately it started going down hill with Blogger and Picasaweb. I originally thought that having my blog in Blogger and images in Picasaweb albums would be perfect because they were both Google apps and would interoperate nicely. However, having created a Blogger account I felt that the interoperability that I desired was not there, and Blogger lacked the quality that WordPress gives you.

I also experienced problems using Picasaweb. While I think the ability to export Events and Albums directly from iPhoto in to Picassaweb is brilliant I encountered a few problems with server error messages from Google. Half way through large(!) photo album uploads I would get “Error 500: Storage not available” messages and have to start again – thank goodness for high speed Broadband.

Eventually I got my albums uploaded to Picasaweb but it left a sour taste in my mouth. Depending on the interoperability between WordPress and Flickr, and Flickr/iPhoto I may be moving to Flickr Pro. This means a more quality Blog and image gallery for Antihoe.org but ultimately my Cloud is fractured between Google, WordPress.com, and Flickr.