Java 2 Ada

Migration of KVM virtual machine image to a raw disk partition

By stephane.carrez 4 comments

This article explains how to move a KVM virtual disk image file from a plain file to a raw hard disk partition. It then explains how to grow the virtual disk to use the full partition size.

Why using a disk partition for the virtual machine image

Using a plain file for a virtual machine disk image is the easiest configuration when you setup some virtual machine environment. It allows to start quickly for the setup and you can copy easily the virtual machine for a backup.

However, using a raw disk partition for the virtual machine provides better performance in general. The overhead of the guest file system is avoided as we have a direct access to the partition.

Copy the virtual machine image on the partition

To copy the virtual machine image on our partition, the easiest way is to use the dd command. This step assumes that the virtual machine is stopped. In the example, the partition is /dev/sdb10, this partition is bigger than the image file (if this is not the case, the image will be truncated).

$ sudo dd if=windows-xp.img of=/dev/sdb10 bs=1048576

5120+1 records in

5120+1 records out

5368709121 bytes (5.4 GB) copied, 331.51 s, 16.2 MB/s

Resize the virtual disk to the full partition size

The virtual disk partition should be changed to use the full disk space provided by our /dev/sdb10 partition. For this, we can use the fdisk command:

$ sudo fdisk /dev/sdb10

Command (m for help): p

Disk /dev/sdb10: 22.0 GB, 22019042304 bytes

...

Device Boot Start End Blocks Id System

/dev/sdb10p1 * 1 651 5229126 7 HPFS/NTFS

You can easily change the partition to use the full disk by deleting the partition and creating it again so that you get something such as:

Device Boot Start End Blocks Id System

/dev/sdb10p1 1 2676 21494938+ 7 HPFS/NTFS
Now, we have to resize the file system on the virtual disk partition /dev/sdb10p1. For this, we will use kpartx to get access to the disk partitions provided by our /dev/sdb10 partition:

$ sudo kpartx -v -a /dev/sdb10

add map sdb10p1 (251:1): 0 42989877 linear /dev/sdb10 63
After the partitions are mapped, we can look at the filesystem before resizing it with the ntfsresize command. We use this command to know the good size for resizing the file system.

$ sudo ntfsresize --info /dev/mapper/sdb10p1

ntfsresize v2.0.0 (libntfs 10:0:0)

Device name : /dev/mapper/sdb10p1

NTFS volume version: 3.1

Cluster size : 4096 bytes

Current volume size: 5354623488 bytes (5355 MB)

Current device size: 22010817024 bytes (22011 MB)

Checking filesystem consistency ...

100.00 percent completed

Accounting clusters ...

Space in use : 4786 MB (89.4%)

Collecting resizing constraints ...

You might resize at 4785565696 bytes or 4786 MB (freeing 569 MB).

Please make a test run using both the -n and -s options before real resizing!
And we can do the resize by using the Current device size as the new file system size.

$ sudo ntfsresize -s 22010817024 /dev/mapper/sdb10p1

ntfsresize v2.0.0 (libntfs 10:0:0)

Device name : /dev/mapper/sdb10p1

NTFS volume version: 3.1

Cluster size : 4096 bytes

Current volume size: 5354623488 bytes (5355 MB)

Current device size: 22010817024 bytes (22011 MB)

New volume size : 22010810880 bytes (22011 MB)

Checking filesystem consistency ...

100.00 percent completed

Accounting clusters ...

Space in use : 4786 MB (89.4%)

Collecting resizing constraints ...

WARNING: Every sanity check passed and only the dangerous operations left.

Make sure that important data has been backed up! Power outage or computer

crash may result major data loss!

Are you sure you want to proceed (y/[n])? y

Schedule chkdsk for NTFS consistency check at Windows boot time ...

Resetting $LogFile ... (this might take a while)

Updating $BadClust file ...

Updating $Bitmap file ...

Updating Boot record ...

Syncing device ...

Successfully resized NTFS on device '/dev/mapper/sdb10p1'.

At this stage, our virtual machine disk image was moved from a plain file to a raw disk partition that it uses entirely.

Change the virtual machine definition

The virtual machine definition must now be changed to use our partition. You can do this by copying the XML definition to another file, thus creating a new virtual machine. This is the best thing to do so that you can still use the old configuration. If you do such copy, you have to change the uuid as well as the network mac address.

The disk type parameter must be changed to block and the dev parameter must now point to the device partition.

<domain type='kvm'>

...

<disk type='block' device='disk'>

<source dev='/dev/sdb10'/>

<target dev='hda' bus='ide'/>

</disk>

...

</domain>
After this, start the virtual machine!

The next step is to setup virtio to boost performance by using paravirtualization.

4 comments
To add a comment, you must be connected. Login to add a comment

Solving Linux system lock up when intensive disk I/O are performed

By stephane.carrez

When a system lock up occurs, we often blame applications but when you look carefully you may see that despite your multi-core CPU, your applications are sleeping! No cpu activity! So what happens then? Check the I/Os, it could be the root cause!

With Ubuntu 10.04, my desktop computer was freezing when the ReadyNAS Bacula backup was running. Indeed, the Bacula daemon was performing intensive disk operations (on a fast SATA hard disk). The situation was such that it was impossible to use the system, the interface was freezing for a several seconds then working for a few seconds and freezing again.

Linux I/O Scheduler

The I/O scheduler is responsible for organizing the order in which disk operations are performed. Some algorithms allow to minimize the disk head moves, other algorithms tend to anticipate read operations,

When I/O operations are not scheduled correctly, an interactive application such as a desktop or a browser can be blocked until its I/O operations are scheduled and executed (the situation can be even worse for those applications that use the O_SYNC writing mode).

By default, the Linux kernel is configured to use the Completely Fair Queuing scheduler. This I/O scheduler does not provide any time guarantee but it gives in general good performances. Linux provides other I/O schedulers such as the Noop scheduler, the Anticipatory scheduler and the Deadline scheduler.

The deadline scheduler puts an execution time limit to requests to make sure the I/O operation is executed before an expiration time. Typically, a read operation will wait at most 500 ms. This is the I/O scheduler we need to avoid the system lock up.

Checking the I/O Scheduler

To check which I/O scheduler you are using, you can use the following command:

$ cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]

where sda is the device name of your hard disk (or try hda).

The result indicates the list of supported I/O scheduler as well as the current scheduler used (here the Completely Fair Queuing).

Changing the I/O Scheduler

To change the scheduler, you can echo the desired scheduler name to activate it (you must be root):

# echo deadline >  /sys/block/sda/queue/scheduler

To make sure the I/O scheduler is configured after each system startup, you can add the following lines to your /etc/rc.local startup script:

test -f /sys/block/sda/queue/scheduler &&
  echo deadline > /sys/block/sda/queue/scheduler

test -f /sys/block/sdb/queue/scheduler &&
   echo deadline > /sys/block/sdb/queue/scheduler

test -f /sys/block/hda/queue/scheduler &&
   echo deadline > /sys/block/hda/queue/scheduler

You may have to change the sda and sdb into hda and hdb if you have an IDE hard disk.

Conclusion

After changing the I/O scheduler to use the Deadline scheduler, the desktop was not freezing any more when backups are running.

To add a comment, you must be connected. Login to add a comment

Tuning mysql configuration for the ReadyNAS duo

By stephane.carrez

After installing mysql server on a Ready NAS duo, it is necessary to tune the configuration to make the server run well on this small hardware. This article describes a possible configuration for tuning the Mysql server.

Mysql Temporary directory

Mysql uses files in the temporary directory to store temporary tables. Depending on your database and your queries, temporary tables could be quite large. To avoid problems in the /tmp partition becomming full, the best thing is to use a directory in the /c partition

tmpdir          = /c/backup/tmp

Make sure the directory exist before starting mysql:

# mkdir -p /c/backup/tmp

Mysql storage engine

After playing with a reasonably big database and the MyISAM storage engine, it turns out that the mysql server was sometimes crashing and barking at some corrupted myisam tables. I switched to the InnoDB storage engine, which is better for transactions anyway. Since the readynas does not have a lot of memory I've used the following configuration:

default_storage_engine = InnoDB
thread_cache_size = 0

innodb_buffer_pool_size = 6M
innodb_thread_concurrency = 1

Other mysql settings

To reduce the resources used by the mysql server to the minimum, I changed the max number of connections to a small number.

key_buffer_size = 16k
sort_buffer_size = 100k
max_connection = 10

I'm using these settings for almost 6 months now; my bacula database now contains a table with 5 milions of rows. Of course you can't expect big performance but the mysql server is stable.

To add a comment, you must be connected. Login to add a comment

Installing Mysql server on a ReadyNAS duo

By stephane.carrez 11 comments

Installing the APT extension

The ReadyNAS duo runs a Debian Sarge distribution. The apt-get commands are not available and we must install the APT addon:

  1. Download the APT extension. I've used the following link:

    http://www.readynas.com/download/addons/4.00/APT_1.0.bin
  2. Go in the ReadyNAS FrontView with your browser and go to System -> Update -> Local Update
  3. Upload the APT binary file. The ReadyNAS verifies that content and if it is correct it displays a description of the addon.
  4. Acknowledge the installation of the addon

After installation, the ReadyNAS must be restarted. Shortly after, you will receive an email:

Subject: Addon Package Progress (nas-XX-XX-XX)
Successfully installed APT.

Debian Sarge Package update

It may be good to check the debian packages. Connect to the ReadyNAS using ssh and run the following commands:

apt-get update

Installation of Mysql server

The mysql-server-5.0 is supposed to be installed according to dpkg -i command. However, the files are not there and they have probably be removed. This is also the case for some utilities which are used by some installation scripts.

Preparation

The /usr/bin/chfn utility is missing and we need it for Mysql installation. We must re-install the passwd package. Download it and install it as follows;

# dpkg -i passwd_4.0.3-31sarge9_sparc.deb

The /usr/bin/logger is also missing. We must re-install the bsdutils_2.12p-4sarge2_sparc.deb package. Download it and install it as follows:

# dpkg -i bsdutils_2.12p-4sarge2_sparc.deb

Edit the /etc/mysql/my.cnf file and change the line:

user            = admin

into

user           = mysql

Additional packages

As reported by RoB (see comments), the following packages are also necessary for mysql-server:

# apt-get install libreadline5 libdbi-perl

Get Mysql packages

The mysql packages are part of backports.org (backport from Etch). You should download the following:

    Mysql installation

    Install the Mysql server debian packages:

    # dpkg -i libmysqlclient15off_5.0.32-7etch5~bpo31+1_sparc.deb \
    mysql-common_5.0.32-7etch5~bpo31+1_all.deb  \
    mysql-server-5.0_5.0.32-7etch5~bpo31+1_sparc.deb \
    mysql-client-5.0_5.0.32-7etch5~bpo31+1_sparc.deb \
    mysql-client_5.0.32-7etch5~bpo31+1_all.deb
    
    Installation Problem and Solution

    If in the installation process you see a message '/etc/init.d/mysql' not found, force the extraction of mysql-server files without the execution of the installation script:

    # dpkg --extract mysql-server-5.0_5.0.32-7etch5~bpo31+1_sparc.deb /
    

    After that, redo the dpkg -i command.

    If you see some errors in the logs:

    [ERROR] /usr/sbin/mysqld: Can't find file: './mysql/host.frm' (errno: 13)
    

    Fix ownership of some existing directories:

    # chown mysql /var/lib/mysql
    # chown mysql /var/lib/mysql/mysql
    

    and restart mysql

    Fix /etc/mysql/debian.cnf

    After installation, the debian.cnf file used by mysqlcheck uses a user that does not exist. You can either create the debian-sys-maint user in mysql or change it to 'root' by editing the file /etc/mysql/debian.cnf and change user and password to use 'root'.

    Fix startup scripts

    Remove the following two startup scripts, they are not necessary for us:

    1. rm /etc/rc2.d/S18mysql-ndb
    2. rm /etc/rc2.d/S17mysql-ndb-mgm

    Testing the database

    Verify that the Mysql database is running:

    nas-D2-24-F2:/var/log# mysql
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 6
    Server version: 5.0.32-Debian_7etch5~bpo31+1 Debian etch distribution
    
    Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
    
    mysql>    
    
    11 comments
    To add a comment, you must be connected. Login to add a comment

    Connecting to a ReadyNAS duo using SSH

    By stephane.carrez 6 comments

    Before you start, you must be aware that there is a risk that you break your ReadyNAS. You should not do this unless you really understand what it is doing.

    Installing the EnableRootSSH extension

    The first step is to install the addon which allows you to connect to your ReadyNAS using ssh:

    1. Download the EnableRootSSH extension. I've used the following link: http://www.readynas.com/download/addons/4.00/EnableRootSSH_1.0.bin
    2. Go in the ReadyNAS FrontView with your browser and go to System -> Update -> Local Update
    3. Upload the EnableRootSSH binary file. The ReadyNAS verifies that content and if it is correct it displays a description of the addon.
    4. Acknowledge the installation of the addon

    After installation, the ReadyNAS must be restarted. Shortly after, you will receive an email:

    Subject: Addon Package Progress (nas-XX-XX-XX)
    Successfully enabled root SSH access.  The root password is now the same as your admin password.
    

    Connecting to the ReadyNAS using ssh

    With the EnableRootSSH extension in place, you can easily connect using ssh. The RSA key fingerprint of your ReadyNAS is prompted and you must accept it in your known_hosts.

    $ ssh -l root pollux
    The authenticity of host 'pollux (192.168.1.6)' can't be established.
    RSA key fingerprint is 01:c8:00:b4:56:5a:f9:fe:2d:73:9a:b0:55:a1:31:2f.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'pollux,192.168.1.6' (RSA) to the list of known hosts.
    root@pollux's password:
    Linux nas-D2-24-F2 2.6.17.8ReadyNAS #1 Fri Sep 19 15:04:06 PDT 2008 padre unknown
    nas-D2-24-F2:~# 
    

    Exploring the ReadyNAS

    Since the ReadyNAS runs a Debian Sarge with a GNU/Linux 2.6.17 kernel, you can easily explore the system.

    CPU and Memory

    The CPU is a Sparc-V8 (LEON) that Infrant has optimized for their needs. It integrates hardware RAID, the gigabit Ethernet and 4 SATA channels, a 64-bit DDR SRAM controller, a DMA, a 3 DES engine and a PCI/USB interface.

    nas-D2-24-F2:~# cat /proc/cpuinfo
    cpu             : Infrant Technologics, Inc. - neon version: 0
    fpu             : Softfpu
    ncpus probed    : 1
    ncpus active    : 1
    BogoMips        : 186.36
    MMU             : version: 0
    LP              : HW.FW version: 0.1
    FPGA            : fpga000000-0 Configuration: 0
    AHB arbitraion  : 7
    CPU id          : 0
    Switch          : 0
    ASIC            : IT3107
    

    And the memory:

    nas-D2-24-F2:~# cat /proc/meminfo
    MemTotal:       226384 kB
    MemFree:        146560 kB
    Buffers:         15440 kB
    Cached:          42352 kB
    SwapCached:          0 kB
    Active:          61776 kB
    Inactive:        22944 kB
    HighTotal:           0 kB
    HighFree:            0 kB
    LowTotal:       226384 kB
    LowFree:        146560 kB
    SwapTotal:      255968 kB
    SwapFree:       255968 kB
    Dirty:               0 kB
    Writeback:           0 kB
    Mapped:          39712 kB
    Slab:             5488 kB
    CommitLimit:    391792 kB
    Committed_AS:    72048 kB
    PageTables:          0 kB
    VmallocTotal:   131008 kB
    VmallocUsed:      1120 kB
    VmallocChunk:   129408 kB
    

    Disks

    The system is installed on the hard disk. It appears to use arround 1.9G from my 1Tb disk.

    nas-D2-24-F2:/usr# df -h
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/hdc1             1.9G  224M  1.7G  12% /
    tmpfs                  16k     0   16k   0% /USB
    /dev/c/c              925G  543M  924G   1% /c
    

    Other interesting commands:

    The following commands are interesting to explore the system and learn more about it:

    dpkg -l         List of installed packages
    netstat -ln   Network open ports (TCP/UDP/Unix)
    top              Top running processes
    ps aux         List all running processes
    

    Next step

    The next step for me is to see if I can install the Bacula Storage Daemon and see if my bacula server is able to connect to it directly.

    6 comments
    To add a comment, you must be connected. Login to add a comment

    Server configuration management: track changes with subversion and be notified

    By stephane.carrez 1 comments

    The overall idea is to put the server configuration files stored in /etc directory under a version control system: subversion. The VCS is configured to send an email to the system administrators. The email contains the differences with a previous version. A cron script is executed every day to automatically commit the changes, thus triggering the email.

    The best practice is of course that each system administrator commits their changes after they validated the new running configuration. If they do so, they are able to specify a comment which is helpful to understand what was done.

    Install subversion

    First, you should install subversion with its tools.

     sudo apt-get install -y subversion subversion-tools
    

    Mail notification

    For the mail notification, you may use postfix, exim or sendmail. But to avoid to setup a complete mail system, you may just use a simple mail client. For this, you can use the combination of esmtp and procmail.

     sudo apt-get install -y procmail esmtp
    

    Create the subversion repository

    The subversion repository will contain all the version and history of your /etc. It must be protected carefully because it contains sensitive information.

     sudo mkdir /home/svn
     sudo svnadmin create /home/svn/repos
     sudo chmod 700 /home/svn
     sudo chmod 700 /home/svn/repos
    

    Now, setup the subversion repository to send an email for each commit. For this, copy or rename the post-commit.tmpl file and edit it to specify to whom you want the email to be sent:

     sudo cp /home/svn/repos/hooks/post-commit.tmpl  \
               /home/svn/repos/hooks/post-commit
    

    and change the last line to something like (with your email address)

     /usr/share/subversion/hook-scripts/commit-email.pl \
      --from yoda+mercure@alliance.com \
      "$REPOS" "$REV" yoda@alliance.com
    

    Initial import

    To initialize the repository, we can use the svn import command:

     sudo svn import -m 'Initial import of /etc' \
                   /etc file:///home/svn/repos/etc
    

    Subversion repository setup in /etc

    Now the hard stuff is to turn /etc into a subversion environment without breaking the server. For this, we extract the subversion /etc repository somewhere and copy only the subversion files in /etc.

     sudo mkdir /home/svn/last
     sudo sh -c "cd /home/svn/last && svn co file:///home/svn/repos/etc"
     sudo sh -c "cd /home/svn/last/etc && tar cf - `find . -name .svn` | (cd /etc && tar xvf -)"
    

    At this step, everything is ready. You can go in /etc directory and use all the subversion commands. Example:

     sudo svn log /etc/hosts
    

    to see the changes in the hosts file.

    Auto-commit and detection of changes

    The goal now is to detect every day the changes that were made and send a mail with the changes to the supervisor. For this, you create a cron script that you put in /etc/cron.daily. The script will be executed every day at 6:25am. It will commit the changes that were made and send an email for the new files.

     #!/bin/sh
     SVN_ETC=/etc
     HOST=`hostname`
     # Commit those changes
     cd $SVN_ETC && svn commit -m "Saving changes in /etc on $HOST"
     # Email address to which changes are sent
     EMAIL_TO="TO_EMAIL"
     STATUS=`cd $SVN_ETC && svn status`
     if test "T$STATUS" != "T"; then
       (echo "Subject: New files in /etc on $HOST";
        echo "To: $EMAIL_TO";
        echo "The following files are new and should be checked in:";
        echo "$STATUS") | sendmail -f'FROM_EMAIL' $EMAIL_TO
     fi
    

    In this script you will replace TO_EMAIL and FROM_EMAIL by real email addresses.

    Complete setup script

    To help setup and configure all this easily, I'm now using a script that configures everything. You can download it: mk-etc-repository. The usage of the script is really simple, you just need to specify the email address for the notification:

     sudo sh mk-etc-repository sysadmin@mycompany.com
    
    1 comments
    To add a comment, you must be connected. Login to add a comment

    Audit errors reported by linux kernel - why you must care

    By stephane.carrez

    Today I had to migrate the mysql storage to another partition because the /var partition was not large enough and the database was growing. After moving the files, updating the mysql configuration files to point to the new partition, mysql refused to start: it pretend it had no permission to access the directory. The directory was owned by mysql and it had the all the rights to write on files. What could happen?

    After looking at the kernel logs, I saw this kind of message:

    [173919.699270] audit(1229883052.863:39): type=1503 operation="inode_create" requested_mask="w::" denied_mask="w::" name="/data/var/mysql" pid=21625 profile="/usr/sbin/mysqld" namespace="default"

    This kernel log is produced by the AppArmor kernel extension which restricts the access to resources to programs. Indeed, it tells that /usr/sbin/mysqld is not able to access the file /data/var/mysql. To fix the problem, you have to change the AppArmor configuration by editing the file /etc/apparmor.d/usr.sbin.mysqld.

     # vim:syntax=apparmor
     # Last Modified: Tue Jun 19 17:37:30 2007
     #include <tunables/global>
    
     /usr/sbin/mysqld {
      #include <abstractions/base>
      #include <abstractions/nameservice>
      #include <abstractions/user-tmp>
      #include <abstractions/mysql>
    
      capability dac_override,
      capability setgid,
      capability setuid,
    
      /etc/hosts.allow r,
      /etc/hosts.deny r,
    
      /etc/group              m,
      /etc/passwd             m,
    
      /etc/mysql/*.pem r,
      /etc/mysql/conf.d/ r,
      /etc/mysql/conf.d/* r,
      /etc/mysql/my.cnf r,
      /usr/sbin/mysqld mr,
      /usr/share/mysql/** r,
     __ /var/lib/mysql/ r,__     #  ''Must be updated''
      __/var/lib/mysql/** rwk,__  # ''Must be updated''
      /var/log/mysql/ r,
      /var/log/mysql/* rw,
      /var/run/mysqld/mysqld.pid w,
      /var/run/mysqld/mysqld.sock w,
    }
    

    The two lines must be fixed to point to the new directory, in the example:

     __ /data/var/mysql/ r,__
      __/data/var/mysql/* rw,__
    

    After changing the files, you must restart Apparmor:

    $ sudo /etc/init.d/apparmor restart
    

    After the fix, the mysql server was able to start again and the audit error was not reported any more.

    To add a comment, you must be connected. Login to add a comment

    Restoring a complete system after a hard disk failure: bacula to the rescue!!!

    By stephane.carrez

    Step 1: Boot on your Ubuntu 8.04 CD

    Since the disk that crashed contained the system, my computer was not even able to boot. A first step for me was to boot on the Ubuntu CDrom without installing Ubuntu again. After booting I was able to check my other disks, look at the kernel logs to realize that the disk was really completely dead without any hope to recover anything. By looking at my second hard disk, I was able to evaluate what was lost and needed to be recovered. If you have no other disk, you have to setup a new disk to proceed. Booting on the CD also helped me discover some room on my second disk where I would install a new system.

    Step 2: Install the system

    If the system has gone, you may have to re-install it from scratch. This is what I had to do. Having found an old debian partition on my second hard disk, I decided to install Ubuntu 8.0.4 Desktop on it. After 15 minutes, my computer was working again, running Ubuntu 8.0.4 as before. Still, my data were lost.

    Step 3: Restore with bacula

    Bacula is a great network backup solution that I put in place 2 years ago. Every night my bacula server is creating an incremental, differential or full backup of my computer (zebulon). It is the first time thought that I had to recover a full content. For the recovery, you have to use the Bacula Console and use the restore command.

    ciceron $ bconsole

    Every action made in bacula creates a job that is recorded in the database. The first thing is to identify those jobs that did the full, differential and incremental backups.

    • list jobs
     | JobId | Name      | StartTime           | Type | Level | JobFiles  | JobBytes       | JobStatus |
     |   877 | Zebulon   | 2007-12-02 02:22:27 | B    | F     | 1,245,258 | 31,026,036,274 | T         |
     | 1,067 | Zebulon   | 2008-02-03 00:52:18 | B    | F     |         0 |              0 | f         |
     | 1,319 | Zebulon   | 2008-04-26 22:28:29 | B    | D     |   207,801 |  6,048,511,830 | T         |
     | 1,328 | Zebulon   | 2008-04-29 22:17:04 | B    | I     |         0 |              0 | E         |
     | 1,331 | Zebulon   | 2008-04-30 22:17:04 | B    | I     |     1,025 |    761,323,545 | T         |
     | 1,511 | Zebulon   | 2008-06-29 22:47:57 | B    | I     |    77,997 |  9,050,108,256 | T         |
     | 1,514 | Zebulon   | 2008-06-30 22:16:40 | B    | I     |       968 |    613,957,318 | T         |
     | 1,517 | Zebulon   | 2008-07-01 22:16:38 | B    | I     |    16,710 |    866,232,575 | T         |
     | 1,520 | Zebulon   | 2008-07-02 22:17:00 | B    | I     |    11,530 |    887,021,057 | T         |
    

    In result above is just an extract of the list command. Job 877 is a full backup (level F) and I had no other recent full backups than this one. It must be restored first. Since bacula has pruned the files, it has lost all the information about its contain (my backup could have been improved). Anyway, it is possible to restore completely this full backup. Jobs 1067 and 1328 cannot be used because they were in errors (I had many of them because the computer is off when the daily backup is started or for some other reasons). This is not a problem, bacula just ignores those jobs for the restore. To restore the full backup use the restore command:

      * __restore__
     
      First you select one or more JobIds that contain files
      to be restored. You will be presented several methods
      of specifying the JobIds. Then you will be allowed to
      select which files from those JobIds are to be restored.
    

    After this, the bacula restore command prompts for a restore method. You can restore a files selectively, find files or restore a complete job or complete client. For me, I had to restore the full backup (job 877) so I selected the Enter list of comma separated JobIds to select method with my full backup job id:

     To select the JobIds, you have the following choices:
       1: List last 20 Jobs run
       2: List Jobs where a given File is saved
       3: Enter list of comma separated JobIds to select
       4: Enter SQL list command
       5: Select the most recent backup for a client
       6: Select backup for a client before a specified time
       7: Enter a list of files to restore
       8: Enter a list of files to restore before a specified time
       9: Find the JobIds of the most recent backup for a client
      10: Find the JobIds for a backup for a client before a specified time
      11: Enter a list of directories to restore for found JobIds
      12: Cancel
     Select item:  (1-12): __3__
       Enter JobId(s), comma separated, to restore: __877__
       You have selected the following JobId: 877
       
       Building directory tree for JobId 877 ...
       There were no files inserted into the tree, so file selection
       is not possible.Most likely your retention policy pruned the files
       
       Do you want to restore all the files? (yes|no):     __yes__
    

    After this step, bacula searches which volumes (backup files, DVD, tapes) contain the backup:

       Bootstrap records written to /var/lib/bacula/janus-dir.restore.12.bsr
       
       The job will require the following
         Volume(s)            Storage(s)                SD Device(s)
       ===================================================
       
         Full-0013            File                      FileStorage
         Full-0014            File                      FileStorage
         Full-0015            File                      FileStorage
         Full-0016            File                      FileStorage
         Full-0017            File                      FileStorage
         Full-0035            File                      FileStorage
         Full-0036            File                      FileStorage
         Full-0037            File                      FileStorage
       
       1,245,258 files selected to be restored.
    

    Now, I had to choose the client for the restore. For some reasons, I had to choose my crashed computer (zebulon):

       Defined Clients:
           1: janus-fd
           2: zebulon-fd
       Select the Client (1-2):    __2__
    

    Bacula describes the restore job and you have a chance to change some parameters. In general, the restore process is made by the bacula daemon on the computer that you want to restore (ie, the client). This is natural, your computer X crashed and you want to recover on it. In my case, I wanted to recover on bacula server (called janus).

        Run Restore job
        JobName:         RestoreFiles
        Bootstrap:       /var/lib/bacula/janus-dir.restore.13.bsr
        Where:           /tmp/bacula-restores
        Replace:         always
        FileSet:         Janus Files
        Backup Client:   zebulon-fd
        Restore Client:  zebulon-fd
        Storage:         File
        When:            2008-07-05 14:16:28
        Catalog:         MyCatalog
        Priority:        10
        OK to run? (yes/mod/no): __mod__
        Parameters to modify:
         1: Level
         2: Storage
         3: Job
         4: FileSet
         5: Restore Client
         6: When
         7: Priority
         8: Bootstrap
         9: Where
        10: File Relocation
        11: Replace
        12: JobId
        Select parameter to modify (1-12): __5__
        The defined Client resources are:
         1: janus-fd
         2: zebulon-fd
       Select Client (File daemon) resource (1-2):__ 1__
        Run Restore job
        JobName:         RestoreFiles 
        Bootstrap:       /var/lib/bacula/janus-dir.restore.13.bsr
        Where:           /tmp/bacula-restores
        Replace:         always
        FileSet:         Janus Files
        Backup Client:   zebulon-fd
        Restore Client:  janus-fd
        Storage:         File
        When:            2008-07-05 14:16:28
        Catalog:         MyCatalog
        Priority:        10
        OK to run? (yes/mod/no):  __yes__
    

    The restore process runs in background and a message and an email are sent after the restore job has finished. In my case, the files were restored on my bacula server in a /tmp/bacula-restores directory. When the restore process finished, that directory contained all my files.... back in December 2007. The differential backup was restored in the same say because the files were pruned too. Other jobs were restored as follows, using the same restore command:

        * __restore__
       
        First you select one or more JobIds that contain files
        to be restored. You will be presented several methods
        of specifying the JobIds. Then you will be allowed to
        select which files from those JobIds are to be restored.
       
        To select the JobIds, you have the following choices:
         1: List last 20 Jobs run
         2: List Jobs where a given File is saved
         3: Enter list of comma separated JobIds to select
         4: Enter SQL list command
         5: Select the most recent backup for a client
         6: Select backup for a client before a specified time
         7: Enter a list of files to restore
         8: Enter a list of files to restore before a specified time
         9: Find the JobIds of the most recent backup for a client
        10: Find the JobIds for a backup for a client before a specified time
        11: Enter a list of directories to restore for found JobIds
        12: Cancel
        Select item:  (1-12):__ 3__
        Enter JobId(s), comma separated, to restore: __1331,1511,1514,1517,1520__
        You have selected the following JobIds: 1331,1511,1514,1517,1520
       
        Building directory tree for JobId 1331 ...
        Building directory tree for JobId 1511 ...  +++++++++++++++++++++++++++++++++
        Building directory tree for JobId 1517 ...  +++++++++++++++++++++++++ 
        Building directory tree for JobId 1520 ...  +++++++++++++++++++++++++++++
        5 Jobs, 75,552 files inserted into the tree.
       
        You are now entering file selection mode where you add (mark) and
        remove (unmark) files to be restored. No files are initially added, unless
        you used the "all" keyword on the command line.
        Enter "done" to leave this mode.
       
        cwd is: /
        $ __mark *__
        79,536 files marked.
        $ __done__
        Bootstrap records written to /var/lib/bacula/janus-dir.restore.14.bsr
       
        The job will require the following
       Volume(s)            Storage(s)                SD Device(s)
        ======================================================
       
       Incr-0002            File                      FileStorage
       Incr-0005            File                      FileStorage
       Incr-0001            File                      FileStorage
       Incr-0006            File                      FileStorage
       
       79,536 files selected to be restored.
    

    After the restore jobs finished, all my files were restored back to July 2nd 2008.

    Lesson learned and conclusion

    1. Backup is vital in computer world. You don't want to loose your photos, emails and documents. When you loose one of them, you just cry. When you loose everything, you....die.
    2. My bacula configuration is not perfect. In particular it should do a full backup every 3 or 6 months. In the past I only used some file recovery but I've never tested a full recovery. This was an error (without bad consequences hopefully). Every change in bacula configuration must be followed by a full recovery test.
    3. The system partitions (/ and /usr) were not backup. Even if we can restore them with an installation, this may not be a good idea. You loose the configuration files and the knowledge of all the packages you have installed. Loosing this is not a big deal but it is a matter of time.
    4. It is necessary to test on a regular basis that we can recover from the backup. The problem is absolutely not the software itself. The problem is the backup configuration and backup needs that change over the time.

    I am very thankful to the Bacula development team for their software. It is really a professional backup solution. I knew that for sure but now I can say I tested it in real situation. The hard disk failure only costs me time: time to install, time to recover the backup and time to write this story....

    Deploying a J2EE application behind an Apache server in a production environment

    By stephane.carrez

    In a production environment, you should not put your JBoss application as a Web front-end. Instead, you should use an Apache server and configure it to redirect specific Web application requests to your J2EE server. There are many many advantages in doing this:

    • The Apache server can serve static files (CSS, images, javascript files) faster than JBoss/Tomcat.
    • When you need it, you can activate SSL on Apache without having to change your application.
    • The Apache SSL implementation is faster compared to the Tomcat implementation (and a lot easier to configure!).
    • You can have a better control of HTTP headers. No need to develop any servlet filter for that.
    • You can get compression out of the box. No need to develop another servlet filter either (no need to configure Tomcat connector either!).

    I assume here that the Apache server is already installed with the following modules and these modules are enabled.

    jk headers expires ssl deflate rewrite

    If they are not enabled, you can enable them using the command:

    sudo a2enmod jk

    Step 1: Explode your Web or J2EE application

    For Apache to serve the static files, it is necessary to have those files available in a directory that the Apache server can access. For this, explode your J2EE application (EAR file) and all the Web applications which have static files to be served by Apache. You will do this in a directory somewhere with one of the command:

    mkdir ''myapplication''
    cd myapplication && jar xf ../''myapplication''.ear
    mv ''mywebapp''.war ''mywebapp''-new.war && mkdir ''mywebapp.war''
    cd ''mywebapp''.war && jar xf ../''mywebapp''-new.war
    

    If your EAR file contains a WAR file, you have to explode it as well (the static files to be served are there!). It is also a good practice to explode it in a directory having the same name as the WAR. Once everything is exploded, you can also configure JBoss to directly use the exploded directory (this will speed up JBoss startup significantly).

    Step 2: Create a site configuration file

    For good practices, you should write a configuration file that corresponds to the site that you are going to manage. This allows to enable or not a server configuration which will be useful during the maintenance. For this, create a file in /etc/apache2/sites-available and put an initial content (replace myserver.mydomain.com with your server name and server-installation-dir with the path of your installation directory):

     <VirtualHost _default_:80>
           ServerAdmin webmaster@localhost
           ServerAlias ''myserver.mydomain''.com
           ServerName ''myserver.mydomain''.com
           DocumentRoot /''server-installation-dir''
           <Directory />
                   Options FollowSymLinks
                   AllowOverride None
           </Directory>
           ErrorLog /var/log/apache2/''myserver''-error.log
           LogLevel warn
           CustomLog /var/log/apache2/''myserver''-access.log combined
     </VirtualHost>
    

    The server-installation-dir should point to the WAR exploded directory.

    It is also a good practice to use a specific log file for each server (virtual host) that you configure in Apache. Restrict the number of options to the minimum so that you do not activate an option that could compromise the security and also to keep the configuration understandable and manageable.

    You may find additional information about virtual hosts on Apache Virtual Host documentation.

    Step 3: Configure Apache mod_jk

    The Apache server can redirect requests to JBoss/Tomcat by using the mod_jk module (jk). Edit the file /etc/apache2/mods-available/jk.load and define the following properties:

     
    JkWorkersFile /etc/apache2/worker.properties
    JkShmFile     /var/log/apache2/mod_jk.shm
    JkLogFile     /var/log/apache2/mod_jk.log
    JkLogLevel    info
    JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
    

    Write a /etc/apache2/worker.properties file with the following example (be sure to replace some of the paths):

    workers.tomcat_home=''directory where Tomcat is installed''
    workers.java_home=''directory of the JDK installation home'' 
    ps=/
    worker.list=workerName
    worker.workerName.port=8009
    worker.workerName.host=localhost
    worker.workerName.type=ajp13
    worker.wokerName.lbfactor=1
    

    The workerName is the name you are going to use within your site configuration file to tell Apache to which JBoss/Tomcat the requests are going to be forwarded.

    You may find additional information on The Apache Tomcat Connector - Webserver HowTo

    Step 4: Configure mod_jk in your site configuration file

    Now that mod_jk is configured, you have to setup your site configuration file to redirect some of your URLs to your JBoss/Tomcat server through the AJP connector. This is done by the JkMount and JkUnMount directives. For this, add the following lines:

    <VirtualHost _default_:80>
     ....
     JkMount /''mywebapp''/* ''workerName''
     JkOptions +ForwardURICompat
    </VirtualHost>
    

    where mywebap is the Web application context of your Web application when it is running. All request to /mywebapp will be redirected to JBoss/Tomcat. If Apache has to serve static files located in the same context, you have to use:

       JkUnMount /mywebapp/*.css workerName
       JkUnMount /mywebapp/*.js workerName
       JkUnMount /mywebapp/*.html workerName
       JkUnMount /mywebapp/*.png workerName
    

    The Javascript, CSS, images and HTML files will not be served by JBoss/Tomcat because the JK connector is not activated for these links.

    You could also only mount the dynamic files that your Web application is serving (like *.jsp, *.do, *.jsf or *.seam). This may not be the best solution if your Web application has specific servlets that are mapped without any extension (like an XMLRPC servlet, a Seam resource servlet and others). This is why, it is best to mount everything to your Web application and then manually specify what is static and served by Apache directly. This will prevent you from big surprises!

    Step 5: Configure caching and compression

    Compression can be activated easily by adding the following line at the top of your site configuration file (see the deflate module):

           AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/x-javascript
    

    Browser caching is activated and controlled by the expires module in Apache. You may add the following options within the <Directory> section controlling the static files.

           # enable expirations
           ExpiresActive On
    
           # Activate the browser caching (CSS, images and scripts should not
           # change)
           ExpiresByType text/css A1296000
           ExpiresByType image/png A1296000
           ExpiresByType image/gif A1296000
           ExpiresByType image/jpg A1296000
           ExpiresByType text/javascript A1296000
    

    You may find additional information on Apache Module mod_expires and Apache Module mod_deflate.

    Step 6: Harden you Apache configuration

    JBoss can add headers in the HTTP response. The X-Powered-By header exposes what implementation is behind your site. This header is created by a servlet filter that is activated by default in JBoss web configuration files (server/default/deploy/jbossweb-tomcat55.sar/conf/web.xml). You can either disable this filter by commenting the following lines:

      <!-- <filter-mapping>
         <filter-name>CommonHeadersFilter</filter-name>
         <url-pattern>/*</url-pattern>
      </filter-mapping> -->
    

    If you cannot change this, don't worry! The Apache server can remove those headers for you. Just add the following directives in your site configuration file:

       # For security reasons, do not expose who serves the page
       <LocationMatch '^/''mywebap''/.*'>
           Header unset 'X-Powered-By'
       </LocationMatch>
    

    Removing this header is also good for performance as it reduces the size of responses. You may have a look at the Apache documentation Apache Module mod_headers.

    By default, the Apache server sends a complete signature in the Server header response. You should verify the /etc/apache2/apache2.conf file and make sure you have the following options:

    ServerSignature Off
    ServerTokens Prod
    

    To verify that your server generates the good response headers, you may use the wget -S command or Firebug to look at those headers.

    Ubuntu Server and Ubuntu Desktop

    By stephane.carrez

    I started to use Linux in 1994 with a Debian distribution. It worked quite well on a 133Mhz Pentium with only 128Mb of memory. It was stable and gave good performance (given the memory and speed of cpu at that time).

    Then, I have used a Linux Red Hat distribution, starting at version 5 and then 6. It was in 1998 on a 300Mhz Pentium II with 256Mb of memory. KDE and Gnome were not there at that time (if my memory is right) and the X11 environment was not as powerful as today but worked reasonably well.

    In 2002, I decided to switch to a Mandrake distribution because it was close to RedHat and it offered a better support for French language, more applications. The switch was easy: the packages are managed in the same way, some administration tools are specific to Mandrake but with a simpler and easier interface. Mandrake did a good job at simplifying the interface to end-users. I started with a Mandrake 8.1, upgraded to a 9.1 and then 10.2. However, each time I did an upgrade, I made a complete installation (because I added a disk or switched some hardware). One of my system was not upgraded and it stayed in Mandrake 8 because the system was remote (it did not have a display) and the remote upgrade was difficult (at least, I didn't know how to do it). Each time I upgraded, the KDE environment had different behaviors. Not big differences, but still annoying for a end-user (different behaviors on the mouse, the keyboard and how you type accented letters, keyboard shortcuts that changed, menus that changed, the session restore that worked differently or did not worked, ...).

    In 2005, I have been convinced by my colleges at Solsoft to use a Debian distribution. The administration and installation of packages was supposed to be easier, and the system upgrade was possible remotely. Well, I had to learn the Debian packaging stuff (I was used to the RPM) and I was using Debian at work since 2002, so let's try it. The installation was easy and worked well. Again, the KDE environment, the keyboard layouts, system behavior became different compared to Mandrake 10.2, but after some adaptations it worked (still it takes you some time). Then, came the problems. I needed specific Debian packages and they were in the 'testing' category. I installed them and of course broke some package dependencies. That's a nightmare and always a hassle,... but with some time and Debian administration knowledge I've managed to solve the problems.

    Meanwhile, the Ubuntu distribution came in. The big difference with Debian is that the distribution contains packages which are updated more regularly, work well together and appears to be much more tested and stable over time. The first computer I've switch was my Linux router and I have installed the Ubuntu Server distribution (the server was running a heavily stripped-down Mandrake 10 distribution). The Ubuntu Server distribution was installed easily and it contained quite all the packages I needed. I also installed a few specific packages for monitoring, backup system and everything went well. Upgrading the packages is similar to Debian (since it's the same package tools) but it appears to me it's a little bit more stable.

    Then, I decided to switch my desktop station to use the Ubuntu Desktop distribution. It's probably the easiest installation I have ever seen. It worked quite out of the box without having to specify or tell any funcky parameters. But it was too simple: not enough utility packages, no development package, no KDE, the printer setting was a nightmare (USB printer). Anyway, the system works now but it required quite a lot of Debian/Linux administration knowledge to set it up. Basically, the problem lies on the Gnome, KDE, X11 and user applications that come with the Ubuntu Desktop.

    Despite the Ubuntu Desktop installation issues, I'm very happy to have switch to Ubuntu. I guess this is the Linux distribution that will give me less problems in the future, will allow me to stay reasonably up to date with new packages and security fixes. The Ubuntu community has still some progress to make to package the distribution in a way that non-Linux users can use it. That's their challenge.

    • Page 1