Java 2 Ada

Migrating a virtual machine from one server to another

By stephane.carrez

OVH is providing new offers that are cheaper and provide more CPU power so it was time for me to migrate and pick another server and reduce the cost by 30%. I'm using 7 virtual machines that run either NetBSD, OpenBSD, FreeBSD, Ubuntu or Debian. Most are Intel based, but some of them are Sparc or Arm virtual machines. I've colllected below the main steps that must be done for the migration.

LVM volume creation on the new server

The first step is to create the LVM volume on the new server. The volume should have the same size as the original. The following command creates a 20G volume labeled netbsd.

$ sudo lvcreate -L 20G -Z n -n netbsd vg01

WARNING: "netbsd" not zeroed

Logical volume "netbsd created

Copying the VM image

After stopping the VM, we can copy the system image from one server to another server by using a combination of dd and ssh. The command must be executed as root otherwise some temporary file and additional copy steps could be necessary.

$ sudo dd if=/dev/vg01/netbsd bs=8192 |

ssh root@master.vacs.fr dd bs=8192 of=/dev/vg01/netbsd


root@master.vacs.fr's password:

2621440+0 records in

2621440+0 records out

21474836480 bytes (21 GB) copied, 1858.33 s, 11.6 MB/s

2621440+0 records in

2621440+0 records out

21474836480 bytes (21 GB) copied, 1848.62 s, 11.6 MB/s

By compressing the image on the fly, the remote copy is faster (4 times faster). The following command does this:
$ sudo dd if=/dev/vg01/netbsd bs=8192 |

gzip -c | ssh root@master.vacs.fr \

'gzip -c -d | dd bs=8192 of=/dev/vg01/netbsd'


root@master.vacs.fr's password:

2621440+0 records in

2621440+0 records out

21474836480 bytes (21 GB) copied, 427.313 s, 50.3 MB/s

2621440+0 records in

2621440+0 records out

21474836480 bytes (21 GB) copied, 436.128 s, 49.2 MB/s

Once the copy is done, it's good to verify the integrity of the copy. For this, we can run the sha1sum on the source image and on the destination image and compare the SHA1 checksum: they must match.

$ sudo sha1sum /dev/vg01/netbsd

04e23ccc1d22cb1de439b43535855b2d1331da6a /dev/vg01/netbsd
(run this command on both servers and compare the result).

Importing the virtual machine definition

The last step is to copy the virtual machine definition from one server to the other. The definition is an XML file located in the /etc/libvirt/qemu directory. Once copied, run the virsh command on the target server and import the definition:

$ sudo virsh

virsh# define netbsd.xml

virsh# start netbsd
That's it, the virtual machine was migrated at a reasonable small cost: the whole process took less than one hour!

To add a comment, you must be connected. Login to add a comment

Installation of FreeBSD for a jenkins build node

By stephane.carrez

A few days ago, I did a fresh installation of my Jenkins build environment for my Ada projects (this was necessary after a disk crash on my OVH server). I took this opportunity to setup a FreeBSD build node. This article is probably incomplete but tends to collect a number of tips for the installation.

Virtual machine setup

The FreeBSD build node is running within a QEMU virtual machine. The choice of the host turns out to be important since not all versions of QEMU are able to run a FreeBSD/NetBSD or OpenBSD system. There is a bug in QEMU PCI emulation that prevents the NetBSD network driver to recognize the emulated network cards (See qemu-kvm 1.0 breaks openbsd, netbsd, freebsd). Ubuntu 12.04 and 12.10 provide a version of Qemu that has the problem. This is solved in Ubuntu 13.04, so this is the host linux distribution that I've installed.

For the virtual machine disk, I've setup some LVM partition on the host as follows:

sudo lvcreate -Z n -L 20G -n freebsd vg01

this creates a disk volume of 20G and label it freebsd.

The next step is to download the FreeBSD Installation CD (I've installed the FreeBSD-10.0-RC2). To manage the virtual machines, one can use the virsh command but the virt-manager graphical front-end provides an easier setup.

sudo virt-manager

The virtual machine is configured with:

    CPU: x86_64
    
    • Memory: 1048576
    • Disk type: raw, source: /dev/vg01/freebsd
    • Network card model: e1000
    • Boot on the CD image

    After the virtual machine starts, the FreeBSD installation proceeds (it was so simple that I took no screenshot at all).

    Post installation

    After the FreeBSD system is installed, it is almost ready to be used. Some additional packages are added by using the pkg install command (which is very close to the Debian apt-get command).

    pkg install jed
    pkg install sudo bash tcpdump
    

    By default the /proc is not setup and some application like the OpenJDK need to access it. Edit the file /etc/fstab and add the following lines:

    fdesc   /dev/fd         fdescfs         rw      0       0
    proc    /proc           procfs          rw      0       0
    

    and mount the new partitions with:

    mount -a
    

    GNAT installation

    The FreeBSD repository provides some packages for Ada development. They are easily installed as follows:

    pkg install gmake
    pkg install gnat-aux-20130412_1 gprbuild-20120510
    pkg install xmlada-4.4.0.0_1 zip-ada-45
    pkg install aws-3.1.0.0
    pkg install gdb-7.6.1_1
    

    After the installation, change the path and setup the ADA_PROJECT_PATH variables to be able to use gnatmake:

    export PATH=/usr/local/gcc-aux/bin:$PATH
    export ADA_PROJECT_PATH=/usr/local/lib/gnat
    

    Jenkins slave node installation

    Jenkins uses a Java application that runs on each build node. It is necessary to install some Java JRE. To use subversion on the build node, we must make sure to install some 1.6 version since the 1.8 and 1.7 version have incompatibilities with the Jenkins master. The following packages are necessary:

    pkg install openjdk6-jre-b28_7
    pkg install subversion-1.6.23_2
    

    Jenkins needs a user to connect to the build node. The user is created by the adduser command. The Jenkins user does not need any privilege.

    Jenkins master will use SSH to connect to the slave node. During the first connection, it installs the slave.jar file which manages the launch of remote builds on the slave. For the SSH connection, the password authentication is possible but I've setup a public key authentication that I've setup on the FreeBSD node by using ssh-copy-id.

    At this stage, the FreeBSD build node is ready to be added on the Jenkins master node (through the Jenkins UI Manage Jenkins/Manage Nodes).

    MySQL Installation

    The MySQL installation is necessary for some of my projects. This is easily done as follows:

    pkg install mysql55-server-5.5.35 mysql55-client-5.5.35
    

    Then add the following line to /etc/rc.conf

    mysql_enable="YES"
    

    and start the server manyally:

    /usr/local/etc/rc.d/mysql-server onestart
    

    The database tables are setup during the first start.

    Other packages

    Some packages that are necessary for some projets:

    pkg install autoconf-2.69 curl-7.33.0_1
    pkg install ImageMagick-nox11-6.8.0.7_3
    

    Jenkins jobs

    The jenkins master is now building 7 projects automatically for FreeBSD 10: FreeBSD Ada Jobs

    To add a comment, you must be connected. Login to add a comment

    Migration of KVM virtual machine image to a raw disk partition

    By stephane.carrez 4 comments

    This article explains how to move a KVM virtual disk image file from a plain file to a raw hard disk partition. It then explains how to grow the virtual disk to use the full partition size.

    Why using a disk partition for the virtual machine image

    Using a plain file for a virtual machine disk image is the easiest configuration when you setup some virtual machine environment. It allows to start quickly for the setup and you can copy easily the virtual machine for a backup.

    However, using a raw disk partition for the virtual machine provides better performance in general. The overhead of the guest file system is avoided as we have a direct access to the partition.

    Copy the virtual machine image on the partition

    To copy the virtual machine image on our partition, the easiest way is to use the dd command. This step assumes that the virtual machine is stopped. In the example, the partition is /dev/sdb10, this partition is bigger than the image file (if this is not the case, the image will be truncated).

    $ sudo dd if=windows-xp.img of=/dev/sdb10 bs=1048576

    5120+1 records in

    5120+1 records out

    5368709121 bytes (5.4 GB) copied, 331.51 s, 16.2 MB/s

    Resize the virtual disk to the full partition size

    The virtual disk partition should be changed to use the full disk space provided by our /dev/sdb10 partition. For this, we can use the fdisk command:

    $ sudo fdisk /dev/sdb10

    Command (m for help): p

    Disk /dev/sdb10: 22.0 GB, 22019042304 bytes

    ...

    Device Boot Start End Blocks Id System

    /dev/sdb10p1 * 1 651 5229126 7 HPFS/NTFS

    You can easily change the partition to use the full disk by deleting the partition and creating it again so that you get something such as:

    Device Boot Start End Blocks Id System

    /dev/sdb10p1 1 2676 21494938+ 7 HPFS/NTFS
    Now, we have to resize the file system on the virtual disk partition /dev/sdb10p1. For this, we will use kpartx to get access to the disk partitions provided by our /dev/sdb10 partition:

    $ sudo kpartx -v -a /dev/sdb10

    add map sdb10p1 (251:1): 0 42989877 linear /dev/sdb10 63
    After the partitions are mapped, we can look at the filesystem before resizing it with the ntfsresize command. We use this command to know the good size for resizing the file system.

    $ sudo ntfsresize --info /dev/mapper/sdb10p1

    ntfsresize v2.0.0 (libntfs 10:0:0)

    Device name : /dev/mapper/sdb10p1

    NTFS volume version: 3.1

    Cluster size : 4096 bytes

    Current volume size: 5354623488 bytes (5355 MB)

    Current device size: 22010817024 bytes (22011 MB)

    Checking filesystem consistency ...

    100.00 percent completed

    Accounting clusters ...

    Space in use : 4786 MB (89.4%)

    Collecting resizing constraints ...

    You might resize at 4785565696 bytes or 4786 MB (freeing 569 MB).

    Please make a test run using both the -n and -s options before real resizing!
    And we can do the resize by using the Current device size as the new file system size.

    $ sudo ntfsresize -s 22010817024 /dev/mapper/sdb10p1

    ntfsresize v2.0.0 (libntfs 10:0:0)

    Device name : /dev/mapper/sdb10p1

    NTFS volume version: 3.1

    Cluster size : 4096 bytes

    Current volume size: 5354623488 bytes (5355 MB)

    Current device size: 22010817024 bytes (22011 MB)

    New volume size : 22010810880 bytes (22011 MB)

    Checking filesystem consistency ...

    100.00 percent completed

    Accounting clusters ...

    Space in use : 4786 MB (89.4%)

    Collecting resizing constraints ...

    WARNING: Every sanity check passed and only the dangerous operations left.

    Make sure that important data has been backed up! Power outage or computer

    crash may result major data loss!

    Are you sure you want to proceed (y/[n])? y

    Schedule chkdsk for NTFS consistency check at Windows boot time ...

    Resetting $LogFile ... (this might take a while)

    Updating $BadClust file ...

    Updating $Bitmap file ...

    Updating Boot record ...

    Syncing device ...

    Successfully resized NTFS on device '/dev/mapper/sdb10p1'.

    At this stage, our virtual machine disk image was moved from a plain file to a raw disk partition that it uses entirely.

    Change the virtual machine definition

    The virtual machine definition must now be changed to use our partition. You can do this by copying the XML definition to another file, thus creating a new virtual machine. This is the best thing to do so that you can still use the old configuration. If you do such copy, you have to change the uuid as well as the network mac address.

    The disk type parameter must be changed to block and the dev parameter must now point to the device partition.

    <domain type='kvm'>

    ...

    <disk type='block' device='disk'>

    <source dev='/dev/sdb10'/>

    <target dev='hda' bus='ide'/>

    </disk>

    ...

    </domain>
    After this, start the virtual machine!

    The next step is to setup virtio to boost performance by using paravirtualization.

    4 comments
    To add a comment, you must be connected. Login to add a comment