The perfect load balanced & high availability web cluster with 2 servers running Xen on Ubuntu 8.04 hardy heron

Posted Fri, 09/05/2008 - 00:36 by ma

In this howto we will build a load balanced and high availability web cluster on 2 real servers with Xen, hearbeat and ldirectord. The cluster will do http, mail, DNS, MySQL database and will be completely monitored. This is currently used on a production server with a couple of websites.

Introduction

The goal of this tutorial is to achieve load balancing & high availability with less real servers as possible and of course, with open-source software. More servers means more hardware & hosting cost.

Most of the information you will find here has been copy / pasted from a dozen howtos, many of them from howtoforge.com, but some important details have been modified to make this possible and to put everything together.

Here is a quick list of services & applications that will be installed :

  • Apache
  • MySQL + phpmyadmin
  • Postfix (SMTP) with web based users configuration and Spamassassin
  • Courier (IMAP & POP) and squirrelmail
  • Bind (DNS server)
  • Munin and monit for web based monitoring
  • Homemade scripts for monitoring

What you need :

2 servers with dual lan, at least 7 ip. IPs will be used like this :

  • dom01.example.com : 192.168.1.100
  • dom02.example.com : 192.168.1.101
  • lb1.example.com : 192.168.1.102
  • lb2.example.com : 192.168.1.103
  • web1.example.com : 192.168.1.104
  • web2.example.com : 192.168.1.105
  • example.com : 192.168.1.106
  • yousite.com (optionnal) : 192.168.1.107

DomO will be separated from load balancers and web servers. I didnt try it but I believe it would be possible to put load balancers on Dom0.

I suggest at least 2GB ram and RAID 1 or 10 hard drives for a production server.

Limitations

  1. This worked for me. Doesnt mean it will work for you but rest assured that the howto is 100% tested to work on a production and test server !
  2. This setup is scalable over 2 servers but you will need to find another way for MySQL replication if you do so.
  3. No control panel such as ISPConfig, CPanel, etc...
  4. Some websites can break MySQL Master to Master replication. It happend to me with Drupal but I fixed it either by disabling cache or by setting a minimum cache lifetime. Please read this before you go further :


A: MySQL replication currently does not support any locking protocol between master and slave to guarantee the atomicity of a distributed (cross-server) update. In other words, it is possible for client A to make an update to co-master 1, and in the meantime, before it propagates to co-master 2, client B could make an update to co-master 2 that makes the update of client A work differently than it did on co-master 1. Thus, when the update of client A makes it to co-master 2, it produces tables that are different from what you have on co-master 1, even after all the updates from co-master 2 have also propagated. This means that you should not chain two servers together in a two-way replication relationship unless you are sure that your updates can safely happen in any order, or unless you take care of mis-ordered updates somehow in the client code.


1. Installing Ubuntu

Do a basic install of Ubuntu 8.04 LTS server edition.

If you want to install with software RAID 1 please read this howto I wrote :

Install Ubuntu 8.04 with software raid 1


2. Installing Xen

You can run Xen from image files or from dedicated partition. Both have pros and cons.

From image files disk I/O is slower but its easier to do backups and to manage. Its the other way around when working on a partition.

What I suggest doing is starting with image file and to end with partition when your setup is finished. This way you can do backups of your image files and rollback if necessary when testing.

To install on image files please refer to this great tutorial from the howto master Falko :

Installing Xen On An Ubuntu 8.04 (Hardy Heron) Server From The Ubuntu Repositories

To install directly on a partition (my modified version of Falko's howto) :

High Performance XEN On An Ubuntu Hardy Heron (8.04) Server System AMD64 or i386

You need to do 2 Xen domain on each server (dom01 and dom02 are Dom0 or VM controller) :

***On server 1 dom01.example.com ip : 192.168.1.100***

  • lb1.example.com (256MB RAM - 5GB HD is enough)
  • ip : 192.168.1.102
  • web1.example.com (the more RAM the better, keep 512MB for Dom0)
  • ip : 192.168.1.104

***On server 2 dom02.example.com ip : 192.168.1.101***

  • lb2.example.com (256MB RAM - 5GB HD is enough)
  • ip : 192.168.1.103
  • web2.example.com (the more RAM the better, keep 512MB for Dom0)
  • ip : 192.168.1.105


3. Creating Xen Bridges for local data transfers (OPTIONAL)

By default only one network card is enabled on virtual machine with Xen. For local transfer such as rsync, MySQL replication and backups I use a gigabit crossover cable between the 2 servers. Its not necessary but it will save bandwidth costs and replication will be faster.

Please refer to this howto to create xen bridge :

Creating new xen bridges on Ubuntu 8.04

In this howto ip used on the second network card (crossover) will be the following :

  • dom01.example.com : 192.168.0.100
  • dom02.example.com : 192.168.0.101
  • lb1.example.com : 192.168.0.102
  • lb2.example.com : 192.168.0.103
  • web1.example.com : 192.168.0.104
  • web2.example.com : 192.168.0.105


4. Node preparation (dom01, dom02, lb1, lb2, web1, web2)


4.1 Installing openssh server and VIM

Run : 

sudo su
apt-get install vim ssh openssh-server


4.2 Updating the repositories

mv /etc/apt/sources.list /etc/apt/sources.list.bak
vi /etc/apt/sources.list

make source.list look like this (here I use mirrors.portafixe.com in Canada, use the closest one from your location ): 

#
# deb cdrom:[Ubuntu-Server 8.04 _Hardy Heron_ - Release i386 (20080423.2)]/ hardy main restricted
#deb cdrom:[Ubuntu-Server 8.04 _Hardy Heron_ - Release i386 (20080423.2)]/ hardy main restricted
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://de.archive.ubuntu.com/ubuntu/ hardy main restricted
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy main restricted
## Major bug fix updates produced after the final release of the
## distribution.
deb http://de.archive.ubuntu.com/ubuntu/ hardy-updates main restricted
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy-updates main restricted
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## universe WILL NOT receive any review or updates from the Ubuntu security
## team.
deb http://de.archive.ubuntu.com/ubuntu/ hardy universe
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy universe
deb http://de.archive.ubuntu.com/ubuntu/ hardy-updates universe
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy-updates universe
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://de.archive.ubuntu.com/ubuntu/ hardy multiverse
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy multiverse
deb http://de.archive.ubuntu.com/ubuntu/ hardy-updates multiverse
deb-src http://de.archive.ubuntu.com/ubuntu/ hardy-updates multiverse
## Uncomment the following two lines to add software from the 'backports'
## repository.
## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
# deb http://de.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
# deb-src http://de.archive.ubuntu.com/ubuntu/ hardy-backports main restricted universe multiverse
## Uncomment the following two lines to add software from Canonical's
## 'partner' repository. This software is not part of Ubuntu, but is
## offered by Canonical and the respective vendors as a service to Ubuntu
## users.
# deb http://archive.canonical.com/ubuntu hardy partner
# deb-src http://archive.canonical.com/ubuntu hardy partner
deb http://security.ubuntu.com/ubuntu hardy-security main restricted
deb-src http://security.ubuntu.com/ubuntu hardy-security main restricted
deb http://security.ubuntu.com/ubuntu hardy-security universe
deb-src http://security.ubuntu.com/ubuntu hardy-security universe
deb http://security.ubuntu.com/ubuntu hardy-security multiverse
deb-src http://security.ubuntu.com/ubuntu hardy-security multiverse

now do :

apt-get update
apt-get upgrade


4.3 Modifications (dom01, dom02, lb1, lb2, web1, web2)

/bin/sh is a symlink to /bin/dash, however we need /bin/bash, not /bin/dash. Therefore we do this:

ln -sf /bin/bash /bin/sh

We will disable AppArmor (on dom01 and dom02) by doing the following : 

/etc/init.d/apparmor stop
update-rc.d -f apparmor remove


5. Network configuration (dom01, dom02, lb1, lb2, web1, web2)


5.1 Setting up IPs

To edit network configuration under ubuntu do : 

vi /etc/network/interfaces

We will now do each network configuration one by one. I assume you use 2 network card, eth0 is the one connected to the internet and eth1 the one with the crossover cable. I wont write the config file individually, only for dom01.example.com, please modify accordingly to this list :

dom01.example.com
eth0 : 192.168.1.100
eth1 : 192.168.0.100

dom02.example.com
eth0 : 192.168.1.101
eth1 : 192.168.0.101

lb1.example.com
eth0 : 192.168.1.102
eth1 : 192.168.0.102

lb2.example.com
eth0 : 192.168.1.103
eth1 : 192.168.0.103

web1.example.com
eth0 : 192.168.1.104
eth1 : 192.168.0.104

web2.example.com
eth0 : 192.168.1.105
eth1 : 192.168.0.105

***Example network configuration of dom01.example.com***

Make the file /etc/network/interfaces look like this : 

# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface connected to the internet
auto eth0
iface eth0 inet static
        address 192.168.1.100
        netmask 255.255.255.0
        network 192.168.1.0
        broadcast 192.168.1.255
        gateway 192.168.1.1
# The secondary network interface connected by a crossover cable on the other server
auto eth1
iface eth1 inet static
        address 192.168.0.100
        netmask 255.255.255.0
        network 192.168.0.0
        broadcast 192.168.0.255

Now save the file and do : 

/etc/init.d/networking restart


5.2 Hostname

vi /etc/hosts

and make it look like this :

***On dom01.example.com***

127.0.0.1       localhost.localdomain   localhost
127.0.1.1       dom01.example.com dom01
 
192.168.1.101   dom02.example.com     dom02
192.168.1.102   lb1.example.com     lb1
192.168.1.103   lb2.example.com     lb2
192.168.1.104   web1.example.com     web1
192.168.1.105   web2.example.com     web2
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

echo dom01.example.com > /etc/hostname
/etc/init.d/hostname.sh start

***On dom02.example.com***

127.0.0.1       localhost.localdomain   localhost
127.0.1.1       dom02.example.com dom02
 
192.168.1.100   dom01.example.com     dom01
192.168.1.102   lb1.example.com     lb1
192.168.1.103   lb2.example.com     lb2
192.168.1.104   web1.example.com     web1
192.168.1.105   web2.example.com     web2
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

echo dom02.example.com > /etc/hostname
/etc/init.d/hostname.sh start

***On lb1.example.com***

127.0.0.1       localhost.localdomain   localhost
127.0.1.1       lb1.example.com lb1
 
192.168.1.100   dom01.example.com     dom01
192.168.1.101   dom02.example.com     dom02
192.168.1.103   lb2.example.com     lb2
192.168.1.104   web1.example.com     web1
192.168.1.105   web2.example.com     web2
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

echo lb1.example.com > /etc/hostname
/etc/init.d/hostname.sh start

***On lb2.example.com***

127.0.0.1       localhost.localdomain   localhost
127.0.1.1       lb2.example.com lb2
 
192.168.1.100   dom01.example.com     dom01
192.168.1.101   dom02.example.com     dom02
192.168.1.102   lb1.example.com     lb1
192.168.1.104   web1.example.com     web1
192.168.1.105   web2.example.com     web2
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

echo lb2.example.com > /etc/hostname
/etc/init.d/hostname.sh start

***On web1.example.com***

127.0.0.1       localhost.localdomain   localhost
127.0.1.1       web1.example.com web1
 
192.168.1.100   dom01.example.com     dom01
192.168.1.101   dom02.example.com     dom02
192.168.1.102   lb1.example.com     lb1
192.168.1.103   lb2.example.com     lb2
192.168.1.105   web2.example.com     web2
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

echo web1.example.com > /etc/hostname
/etc/init.d/hostname.sh start

***On web2.example.com***

127.0.0.1       localhost.localdomain   localhost
127.0.1.1       web2.example.com web2
 
192.168.1.100   dom01.example.com     dom01
192.168.1.101   dom02.example.com     dom02
192.168.1.102   lb1.example.com     lb1
192.168.1.103   lb2.example.com     lb2
192.168.1.104   web1.example.com     web1
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

echo web2.example.com > /etc/hostname
/etc/init.d/hostname.sh start


6. Software installation (dom01, dom02, lb1, lb2, web1, web2)

Run :

apt-get install binutils cpp fetchmail flex gcc libarchive-zip-perl libc6-dev libcompress-zlib-perl libdb4.3-dev libpcre3 libpopt-dev lynx m4 make ncftp nmap openssl perl perl-modules unzip zip zlib1g-dev autoconf automake1.9 libtool bison autotools-dev g++ build-essential


NEXT PAGE

Or jump to the following :

1. Installing Ubuntu
2. Installing Xen
3. Creating Xen Bridges for local data transfers
4. Preparation
5. Network configuration
6. Software installation
7. Apache/PHP5/Ruby
8. DNS Server
9. Proftpd
10. MySQL replication
11. Mail server
12. Setting up the load balancers
13. Server Monitoring With munin And monit
14. Mirroring web & mail files with rsync
15. Custom scripts for monitoring