Java 2 Ada - Tag KVM2014-01-25T13:19:00+00:00Stephane Carrezurn:md5:d12e23c53b2436d6becce3d51ddbdf38AWAMigrating a virtual machine from one server to anotherurn:md5:09f381dd27d1977d7b5f06a68d7aaffc2014-01-25T13:19:00+00:002014-01-25T13:19:00+00:00Stephane CarrezKVMmigration
<div class="post-text"><p><a href="http://www.ovh.com">OVH</a> is providing new offers that are cheaper and provide more CPU power so it was time for me to migrate and pick another server and reduce the cost by 30%. I'm using 7 virtual machines that run either <a href="http://www.netbsd.org/">NetBSD</a>, <a href="http://www.openbsd.org/">OpenBSD</a>, <a href="http://www.freebsd.org/">FreeBSD</a>, <a href="http://www.ubuntu.com/">Ubuntu</a> or <a href="http://www.debian.org/">Debian</a>. Most are Intel based, but some of them are Sparc or Arm virtual machines. I've colllected below the main steps that must be done for the migration.</p><h3>LVM volume creation on the new server</h3><p>The first step is to create the LVM volume on the new server. The volume should have the same size as the original. The following command creates a 20G volume labeled <code>netbsd</code>.</p><pre><code class="lang-sh">$ sudo lvcreate -L 20G -Z n -n netbsd vg01
WARNING: "netbsd" not zeroed
Logical volume "netbsd created
</code></pre><h3>Copying the VM image</h3><p>After stopping the VM, we can copy the system image from one server to another server by using a combination of <code>dd</code> and <code>ssh</code>. The command must be executed as <code>root</code> otherwise some temporary file and additional copy steps could be necessary.</p><pre><code class="lang-sh">$ sudo dd if=/dev/vg01/netbsd bs=8192 |
ssh root@master.vacs.fr dd bs=8192 of=/dev/vg01/netbsd
root@master.vacs.fr's password:
2621440+0 records in
2621440+0 records out
21474836480 bytes (21 GB) copied, 1858.33 s, 11.6 MB/s
2621440+0 records in
2621440+0 records out
21474836480 bytes (21 GB) copied, 1848.62 s, 11.6 MB/s
</code></pre><p>By compressing the image on the fly, the remote copy is faster (4 times faster). The following command does this:</p><pre><code class="lang-sh">$ sudo dd if=/dev/vg01/netbsd bs=8192 |
gzip -c | ssh root@master.vacs.fr \
'gzip -c -d | dd bs=8192 of=/dev/vg01/netbsd'
root@master.vacs.fr's password:
2621440+0 records in
2621440+0 records out
21474836480 bytes (21 GB) copied, 427.313 s, 50.3 MB/s
2621440+0 records in
2621440+0 records out
21474836480 bytes (21 GB) copied, 436.128 s, 49.2 MB/s
</code></pre><p>Once the copy is done, it's good to verify the integrity of the copy. For this, we can run the <code>sha1sum</code> on the source image and on the destination image and compare the SHA1 checksum: they must match.</p><pre><code class="lang-sh">$ sudo sha1sum /dev/vg01/netbsd
04e23ccc1d22cb1de439b43535855b2d1331da6a /dev/vg01/netbsd
</code></pre><p>(run this command on both servers and compare the result).</p><h3>Importing the virtual machine definition</h3><p>The last step is to copy the virtual machine definition from one server to the other. The definition is an XML file located in the <code>/etc/libvirt/qemu</code> directory. Once copied, run the <code>virsh</code> command on the target server and import the definition:</p><pre><code class="lang-sh">$ sudo virsh
virsh# define netbsd.xml
virsh# start netbsd
</code></pre><p>That's it, the virtual machine was migrated at a reasonable small cost: the whole process took less than one hour!</p></div> Installation of FreeBSD for a jenkins build nodeurn:md5:39d824bf9f537085e92f4ca8eafcf2a82013-12-31T14:43:19+00:002013-12-31T14:43:19+00:00Stephane CarrezAdaFreeBSDKVMJenkins
<div class="post-text"><p>A few days ago, I did a fresh installation of my <a href="http://jenkins-ci.org/">Jenkins</a> build environment for my Ada projects (this was necessary after a disk crash on my <a href="http://www.ovh.com/fr/index.xml">OVH</a> server). I took this opportunity to setup a <a href="http://www.freebsd.org/">FreeBSD</a> build node. This article is probably incomplete but tends to collect a number of tips for the installation.</p><h2>Virtual machine setup</h2><p>The FreeBSD build node is running within a <a href="http://wiki.qemu.org/KVM">QEMU</a> virtual machine. The choice of the host turns out to be important since not all versions of QEMU are able to run a FreeBSD/NetBSD or OpenBSD system. There is a bug in QEMU PCI emulation that prevents the NetBSD network driver to recognize the emulated network cards (See <a href="http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=653473">qemu-kvm 1.0 breaks openbsd, netbsd, freebsd</a>). Ubuntu 12.04 and 12.10 provide a version of Qemu that has the problem. This is solved in Ubuntu 13.04, so this is the host linux distribution that I've installed.</p><p>For the virtual machine disk, I've setup some LVM partition on the host as follows:</p><pre><code>sudo lvcreate -Z n -L 20G -n freebsd vg01
</code></pre><p>this creates a disk volume of 20G and label it <code>freebsd</code>.</p><p>The next step is to download the <a href="http://www.freebsd.org/where.html">FreeBSD Installation CD</a> (I've installed the FreeBSD-10.0-RC2). To manage the virtual machines, one can use the <code>virsh</code> command but the <code>virt-manager</code> graphical front-end provides an easier setup.</p><pre><code>sudo virt-manager
</code></pre><p>The virtual machine is configured with:</p><ul><li> CPU: x86_64</li><li>Memory: 1048576</li><li>Disk type: raw, source: /dev/vg01/freebsd</li><li>Network card model: e1000</li><li>Boot on the CD image</li></ul><p>After the virtual machine starts, the FreeBSD installation proceeds (it was so simple that I took no screenshot at all).</p><h2>Post installation</h2><p>After the FreeBSD system is installed, it is almost ready to be used. Some additional packages are added by using the <code>pkg install</code> command (which is very close to the Debian <code>apt-get</code> command).</p><pre><code>pkg install jed
pkg install sudo bash tcpdump
</code></pre><p>By default the <code>/proc</code> is not setup and some application like the OpenJDK need to access it. Edit the file <code>/etc/fstab</code> and add the following lines:</p><pre><code>fdesc /dev/fd fdescfs rw 0 0
proc /proc procfs rw 0 0
</code></pre><p>and mount the new partitions with:</p><pre><code>mount -a
</code></pre><h2>GNAT installation</h2><p>The FreeBSD repository provides some packages for Ada development. They are easily installed as follows:</p><pre><code>pkg install gmake
pkg install gnat-aux-20130412_1 gprbuild-20120510
pkg install xmlada-4.4.0.0_1 zip-ada-45
pkg install aws-3.1.0.0
pkg install gdb-7.6.1_1
</code></pre><p>After the installation, change the path and setup the <code>ADA_PROJECT_PATH</code> variables to be able to use <code>gnatmake</code>:</p><pre><code>export PATH=/usr/local/gcc-aux/bin:$PATH
export ADA_PROJECT_PATH=/usr/local/lib/gnat
</code></pre><h2>Jenkins slave node installation</h2><p>Jenkins uses a Java application that runs on each build node. It is necessary to install some Java JRE. To use subversion on the build node, we must make sure to install some 1.6 version since the 1.8 and 1.7 version have incompatibilities with the Jenkins master. The following packages are necessary:</p><pre><code>pkg install openjdk6-jre-b28_7
pkg install subversion-1.6.23_2
</code></pre><p>Jenkins needs a user to connect to the build node. The user is created by the <a href="http://www.freebsd.org/cgi/man.cgi?query=adduser&amp;sektion=8">adduser</a> command. The Jenkins user does not need any privilege.</p><p>Jenkins master will use <code>SSH</code> to connect to the slave node. During the first connection, it installs the <code>slave.jar</code> file which manages the launch of remote builds on the slave. For the <code>SSH</code> connection, the password authentication is possible but I've setup a public key authentication that I've setup on the FreeBSD node by using <code>ssh-copy-id</code>.</p><p>At this stage, the FreeBSD build node is ready to be added on the Jenkins master node (through the Jenkins UI <i>Manage Jenkins</i>/<i>Manage Nodes</i>).</p><h2>MySQL Installation</h2><p>The MySQL installation is necessary for some of my projects. This is easily done as follows:</p><pre><code>pkg install mysql55-server-5.5.35 mysql55-client-5.5.35
</code></pre><p>Then add the following line to <code>/etc/rc.conf</code></p><pre><code>mysql_enable="YES"
</code></pre><p>and start the server manyally:</p><pre><code>/usr/local/etc/rc.d/mysql-server onestart
</code></pre><p>The database tables are setup during the first start.</p><h2>Other packages</h2><p>Some packages that are necessary for some projets:</p><pre><code>pkg install autoconf-2.69 curl-7.33.0_1
pkg install ImageMagick-nox11-6.8.0.7_3
</code></pre><h2>Jenkins jobs</h2><p>The jenkins master is now building 7 projects automatically for FreeBSD 10: <a href="http://jenkins.vacs.fr/view/FreeBSD/">FreeBSD Ada Jobs</a></p></div> Migration of KVM virtual machine image to a raw disk partitionurn:md5:70324804d2578e7cce45253821145e392010-10-10T21:03:00+00:002010-10-10T21:03:00+00:00Stephane CarrezKVMLinuxUbuntu
<div class="post-text"><p>This article explains how to move a KVM virtual disk image file from a plain file to a raw hard disk partition. It then explains how to grow the virtual disk to use the full partition size.</p><h2>Why using a disk partition for the virtual machine image</h2><p>Using a plain file for a virtual machine disk image is the easiest configuration when you setup some virtual machine environment. It allows to start quickly for the setup and you can copy easily the virtual machine for a backup.</p><p>However, using a raw disk partition for the virtual machine provides better performance in general. The overhead of the guest file system is avoided as we have a direct access to the partition.</p><h2>Copy the virtual machine image on the partition</h2><p>To copy the virtual machine image on our partition, the easiest way is to use the <code>dd</code> command. This step assumes that the virtual machine is stopped. In the example, the partition is <code>/dev/sdb10</code>, this partition is bigger than the image file (if this is not the case, the image will be truncated).</p><pre><code>$ sudo dd if=windows-xp.img of=/dev/sdb10 bs=1048576
5120+1 records in
5120+1 records out
5368709121 bytes (5.4 GB) copied, 331.51 s, 16.2 MB/s
</code></pre><h2>Resize the virtual disk to the full partition size</h2><p>The virtual disk partition should be changed to use the full disk space provided by our <code>/dev/sdb10</code> partition. For this, we can use the <code>fdisk</code> command:</p><pre><code>$ sudo fdisk /dev/sdb10
Command (m for help): p
Disk /dev/sdb10: 22.0 GB, 22019042304 bytes
...
Device Boot Start End Blocks Id System
/dev/sdb10p1 * 1 651 5229126 7 HPFS/NTFS
</code></pre><p>You can easily change the partition to use the full disk by deleting the partition and creating it again so that you get something such as:</p><div style='float: left; border: 1px solid #aaa; width: 99%; margin-bottom: 10px;'>
Device Boot Start End Blocks Id System<br></br>
/dev/sdb10p1 1 2676 21494938+ 7 HPFS/NTFS
</div>
<p>Now, we have to resize the file system on the virtual disk partition <code>/dev/sdb10p1</code>. For this, we will use <code>kpartx</code> to get access to the disk partitions provided by our <code>/dev/sdb10</code> partition:</p><pre><code>$ sudo kpartx -v -a /dev/sdb10
add map sdb10p1 (251:1): 0 42989877 linear /dev/sdb10 63
</code></pre><p>After the partitions are mapped, we can look at the filesystem before resizing it with the <code>ntfsresize</code> command. We use this command to know the good size for resizing the file system.</p><pre><code>$ sudo ntfsresize --info /dev/mapper/sdb10p1
ntfsresize v2.0.0 (libntfs 10:0:0)
Device name : /dev/mapper/sdb10p1
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 5354623488 bytes (5355 MB)
Current device size: <b>22010817024</b> bytes (22011 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 4786 MB (89.4%)
Collecting resizing constraints ...
You might resize at 4785565696 bytes or 4786 MB (freeing 569 MB).
Please make a test run using both the -n and -s options before real resizing!
</code></pre><p>And we can do the resize by using the <code>Current device size</code> as the new file system size.</p><pre><code>$ sudo ntfsresize -s 22010817024 /dev/mapper/sdb10p1
ntfsresize v2.0.0 (libntfs 10:0:0)
Device name : /dev/mapper/sdb10p1
NTFS volume version: 3.1
Cluster size : 4096 bytes
Current volume size: 5354623488 bytes (5355 MB)
Current device size: 22010817024 bytes (22011 MB)
New volume size : 22010810880 bytes (22011 MB)
Checking filesystem consistency ...
100.00 percent completed
Accounting clusters ...
Space in use : 4786 MB (89.4%)
Collecting resizing constraints ...
WARNING: Every sanity check passed and only the dangerous operations left.
Make sure that important data has been backed up! Power outage or computer
crash may result major data loss!
Are you sure you want to proceed (y/[n])? y
Schedule chkdsk for NTFS consistency check at Windows boot time ...
Resetting $LogFile ... (this might take a while)
Updating $BadClust file ...
Updating $Bitmap file ...
Updating Boot record ...
Syncing device ...
Successfully resized NTFS on device '/dev/mapper/sdb10p1'.
</code></pre><p>At this stage, our virtual machine disk image was moved from a plain file to a raw disk partition that it uses entirely.</p><h2>Change the virtual machine definition</h2><p>The virtual machine definition must now be changed to use our partition. You can do this by copying the XML definition to another file, thus creating a new virtual machine. This is the best thing to do so that you can still use the old configuration. If you do such copy, you have to change the <code>uuid</code> as well as the network <code>mac</code> address.</p><p>The disk <code>type</code> parameter must be changed to <code>block</code> and the <code>dev</code> parameter must now point to the device partition.</p><pre><code class="lang-XML"><domain type='kvm'>
...
<disk type='block' device='disk'>
<source dev='/dev/sdb10'/>
<target dev='hda' bus='ide'/>
</disk>
...
</domain>
</code></pre><p>After this, start the virtual machine!</p><p>The next step is to setup <a href="http://wiki.libvirt.org/page/Virtio">virtio</a> to boost performance by using paravirtualization.</p></div>