Painful lessons – Porting Centos 5.8 from Virtualbox to Amazon EC2

Notice – this post will save you a massive amount of headbanging if you ever need to do anything similar to this.  (Thanks to HN for posting… almost makes it worthwhile)

Amazon has a really nice cloud infrastructure.  It’s really easy to spin up machines and attach volumes and do stuff.

I have one caveat.  Sometimes I can’t get trusted versions of an operating system and version that I need.  By trusted, I mean from the distributor, as opposed to some fine third parties.   The facinating thing about Amazon is you can spin up all sorts of machine images (AMIs) from all over the place.  They can literally contain anything.  Really.  Anything.

So recently I wanted version 5.8 of Centos, a flavor of linux, and I couldn’t get a pristine copy from AWS from the Centos people, so I had to make one myself.

There’s a great free program out there – Virtualbox – http://virtualbox.org, (originally from Sun, which is now owned by Oracle).  Install this on your machine and you can run other machines inside your computer… this is called Virtualization.  So for example, if you’re running a Mac, you could have a version of Windows running on your computer, in it’s own little window, at the same time.  Or if you’re running Windows, you could spin up your very own little linux box to play with, and so forth.

So my thinking was, spin up my trusted Centos 5.8 using virtualbox on my mac, then export the disk image to Amazon, and coax it into running there.

I’m by no means the first one to do this.

Carson McDonald has the definitive post here http://www.ioncannon.net/system-administration/1246/converting-from-virtualbo…

Michael Martinez took a shot at is as well: http://amazonaws.michael–martinez.com/

I had a remarkably difficult time with this, due to some really subtle issues which were buried inside of moving gigabytes of crap around repeatedly over the course of a pile of hours.  I’ll list the problems here first, for any poor soul searching for the same solutions, the actual instructions are below:

1. My virtualbox image is the vanilla Centos 5.8 installation. Amazon needs the xen kernel installed:

yum install kernel-xen

2. The hard disk consists of a boot partition, and a logical volume located on the second partition.  This means that the menu.lst file will be different than any of the examples out there – the key is root=/dev/VolGroup00/LogVol00.  Symptom: kernel panic at AWS.  Settings which worked for me below:

default=0
fallback=1
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-308.4.1.el5xen)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-308.4.1.el5xen ro root=/dev/VolGroup00/LogVol00
        initrd /initrd-2.6.18-308.4.1.el5xen.img

3. Even if you have the right kernel installed with the correct menu.lst file, it doesn’t matter if it’s not associated correctly.  Get this wrong you get a kernel panic. We need the magic mkinitrd command to make it work:

mkinitrd -f  –preload=xenblk –preload=xennet /boot/initrd-2.6.18-308.4.1.el5xen.img 2.6.18-308.4.1.el5xen

4. Because I’ve got /boot in it’s own partition, Amazon will not end up looking in the correct location for my kernel.  You need to copy the /boot stuff into /boot/boot (really):

cp -Rp /boot /boot

5. Now just because you’re set up correctly doesn’t mean that Amazon will recognize this.  You need to boot up your instance with a special Amazon kernel.

Amazon can boot a Xen kernel using pv-grub which we installed at step 1, then you need to find which of Amazon’s initial kernels will allow you to boot your kernel.  Here’s the list: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/UserProvidedkernels… – there are 4 kernels per availability zone, 2 for 32 bit versions, 2 for 64 bit versions:

Separate /boot: kernels with “hd00” in their name like pv-grub-hd00_1.03-i386
One big disk: kernels with “hd0” in their name like pv-grub-hd0_1.03-i386

Once you’ve located the correct kernel, in the correct availability zone, permanently associate it with your snapshot using the ec2-register command – otherwise if you launch your kernel from the AWS dashboard with the defaults, you’ll get, wait for it, Kernel panic:

ec2-register -n “centos5.8″ –kernel “aki-b2aa75db” -d “Centos 5.8″ –root-device-name /dev/sda2 -b /dev/sda=SNAPSHOTID:10:true`

6. If you’re paranoid like I am, then you’ll be taking all sorts of snapshots of your Virtualbox instance, just in case things go badly.  Good practice.  Of course, when we need to get a copy of the running machine’s disk we use the following command:

vBoxManage internalcommands converttoraw centos5.8.vdi centos5.8.img

This command takes a copy of the virtual hard disk, and converts it to an img file, which we can then compress and transfer up to AWS and write onto a volume which we snapshot and turn into an AMI.  I’ll outline this process later, don’t panic.

I did everything correctly.  What did I get?  Kernel panic. It’s especially heartwarming since this happens after having waited the excruciating amount of time to upload 500mb to Amazon.  So this debugging is a slow and painful process, which is why I’m documenting it.

Here’s what went wrong.  If you’ve snapshotted a Virtualbox machine, the changes are not reflected at the “top level” in my case, centos5.8.vdi.  That file is still the state of your machine before any snapshots.

You must delete all snapshots from the top, down.  Each time you delete a snapshot, your changes are merged with the later snapshots.  When you no longer have any snapshots, your changes have been rolled into the top level machine instance, which you can now copy and send to Amazon.  You can verify this by checking the date on your .vdi file.

This pearl of wisdom can be found in the Virtualnbox blog here: https://forums.virtualbox.org/viewtopic.php?f=2&t=20810

* * *

Here’s a step by step guide

Steps done on your local machine

1. default install of Centos 5.8 on virtualbox 4.1.12
2. Download netinstall iso http://mirrors.bluehost.com/centos/5.8/isos/i386/CentOS-5.8-i386-netinstall.iso
3. Select HTTP and these parameters for the rest mirror.centos.org / centos/5/os/i386
4. yum -y update
5. yum -y install kernel-xen
6. Edit /boot/grub/menu.lst (see note 2 above)
7. cp -Rp /boot /boot
8. Powerdown your Centos virtual machine
9. cd to the location of your virtual images (under Virtualbox VMs/YOURMACHINE)
10. vBoxManage internalcommands converttoraw YOURMACHINE.vdi YOURMACHINE.img
11. gzip YOURMACHINE.img
12. scp -i YOURKEY.pem YOURMACHINE.img.gz YOURSERVER.amazonaws.com

Steps done on Amazon AWS

For this part you’ll need an instance on AWS running some form of linux, and a volume at least as big as your Centos image – mine was the default 8mb so…

1. Create a 10 gb volume on AWS
2. Associate it with your running instance – note it’s name, like /dev/sdg
3. cat YOURMACHINE.img.gz | gzip -d | dd of=/dev/sdg bs=10M
4. Snapshot your volume
5. Create an AMI from your snapshot
6. Run the AMI

script: volume to running instance (steps 4-6 above)

This script is run from your local machine – make any changes to your volume, then run the script and it’ll try to fire up an instance… which may or may not work.  At least you’ll know what the steps to follow are.

You’ll need to have the EC2 tools for AWS installed: http://aws.amazon.com/developertools/351

And for the script, you’ll need to have your volume-ID from above, plus a keypair.  The stuff you need to set is shown inside the shell script.  No warranty, public domain, enjoy.

http://kimosabe.net/aws-vol2ami.sh