Multiple Concurrent Linux Distributions

You can run multiple Linux distributions at the same time, on the same computer, without a virtual machine. Milo's production environment is a mix of Ubuntu Hardy and Lucid, while eBay's production Linux is Red Hat. Eventually, this will all converge on one environment, but in the mean time while we port, we need a way to rapidly iterate changes on a handful of Linux distributions. A virtual machine seems like the obvious answer, but that's overkill for this situation.

A Linux distribution is a kernel, some libraries, binaries, and a package manager. The kernel is the lowest level of abstraction over the hardware; everything else is fairly interchangeable. Apache running on Red Hat will make the same system calls as Apache running on Ubuntu. In theory, as long as shared library paths are managed correctly and the package managers don't trample on each other, you can have multiple distributions "running" under one kernel, no virtual machine needed.

Unix provides the chroot mechanism to keep all of the distribution files in order. There are some tools that build on chroot to support these virtual environments so that you don't have to do any bookkeeping or stupid shit with /dev or /proc. I am using Ubuntu as my "host" operating system, and a Debian tool called schroot to manage it all.

Ubuntu In Ubuntu

I use Ubuntu Maverick as my desktop operating system, while production uses either Hardy or Lucid because they are long-term support releases. Here's how you install one Ubuntu distribution within another:

sudo apt-get install debootstrap schroot
sudo mkdir -p /var/chroot/lucid
sudo debootstrap --variant=buildd --arch [amd64|i386] lucid /var/chroot/lucid http://archive.ubuntu.com/ubuntu/

Now open up /etc/schroot/schroot.conf in a text editor and add a configuration stanza that looks like this:

[lucid]
directory=/var/chroot/lucid
description=Ubuntu Lucid
root-users=root,ted
users=ted
type=directory

In practice this can be any Ubuntu release you want, just change the argument to debootstrap and your configurations accordingly. I have one stanza for Lucid and one for Hardy, but you can have as many as you've got disk space for.

Now you can use your virtual environment with schroot -c lucid:

ted@duke:~$ lsb_release -dc
Description:	Ubuntu 10.10
Codename:	maverick
ted@duke:~$ schroot -c lucid
(lucid)ted@duke:~$ lsb_release -dc
Description:	Ubuntu 10.04 LTS
Codename:	lucid
(lucid)ted@duke:~$

At this point you should probably apt-get install ubuntu-minimal as debootstrap doesn't install a whole lot.

Neat.

Red Hat in Ubuntu

Now that we've got ourselves bothered with schroot, let's make it jump. Installing Red Hat in a chroot environment is going to be tricker because we don't have debootstrap, which is a Debian tool for bootstrapping an installation. Fortunately, some enterprising programmer has unfucked the Red Hat bootstrap system with a utility called rinse.

sudo apt-get install rinse
sudo mkdir -p /var/chroot/centos
sudo rinse --distribution centos-5 --arch [amd64|i386] --directory /var/chroot/centos

If CentOS isn't your game, there are others to choose from. Run rinse --list-distributions to see what is available.

Add a schroot.conf stanza for Red Hat like so:

[centos]
description=CentOS 5 amd64
directory=/var/chroot/centos
root-users=root,ted
users=ted
type=directory

And now I can jump into CentOS 5 with schroot -c centos, and I'll prove it, too:

ted@duke:~$ schroot -c centos
(centos)ted@duke:~$ lsb_release -dc
Description:	CentOS release 5.5 (Final)
Codename:	Final

Of course, CentOS uses RPM and some derelict dependency manager called yum. You will have to yum install lsb to get the Linux Standard Base.

This is Not Virtualization

If you're asking yourself how you can get a private network interface for a chroot environment, then you've got to take a little more time to understand the problem we are solving. When I first started digging into this, my goal was to be able to run flymake in a hermetic environment. Just the tip, just for a little bit, just to see how it feels. And look where it lead.

This doesn't solve a whole lot of production problems, in fact, it will probably create more than it solves. If you're interested in chrooting your way out of a jam in production, check out FreeBSD jails. This trick is useful for development, when you need to target more than one Linux distribution, or you want a clean environment to test in without booting up a VM.

Ain't Unix grand?

Ted Dziuba