Java 2 Ada - Tag bacula2013-06-30T08:09:18+00:00Stephane Carrezurn:md5:d12e23c53b2436d6becce3d51ddbdf38AWABacula database cleanupurn:md5:d2b6c0ab9c36d262ff024e62c1c8dd752013-06-30T08:09:18+00:002013-06-30T08:09:18+00:00Stephane CarrezReadyNASbaculamysql
<div class="post-text"><p><a href="http://www.bacula.org/en/">Bacula</a> maintains a catalog of files in a database. Over time, the database grows and despite some automatic purge and job cleanup, some information remains that is no longer necessary. This article explains how to remove some dead records from the Bacula catalog.</p><p>Bacula maintains a list of backup jobs that have been executed in the <code>job</code> table. For each job, it keeps the list of files that have been saved in the <code>file</code> table. When you do a restore, you somehow select the job to restore and pick files from that job. There should not exist any <code>file</code> entry associated with a non existing job. Unfortunately this is not the case. I've found that some files (more than 2 millions entries) were pointing to some job that did not exist.</p><h2>Discovering dead jobs still referenced</h2><p>The first step is to find out which job has been deleted and is still referenced by the <code>file</code> table. First, let's create a temporary table that will hold the job ids associated with the files.</p><pre><code class="lang-sql">mysql> create temporary table job_files (id bigint);
</code></pre><p>The use of a temporary table was necessary in my case because the <code>file</code> table is so big and the <a href="http://www.readynas.com/?p=1514">ReadyNAS</a> so slow that scanning the database takes too much time.</p><p>Now, we can populate the temporary table with the job ids:</p><pre><code class="lang-sql">mysql> insert into job_files select distinct file.jobid from file;
Query OK, 350 rows affected (8 min 53.26 sec)
Records: 350 Duplicates: 0 Warnings: 0
</code></pre><p>The list of jobs that have been removed but are still referenced by a file is obtained by:</p><pre><code class="lang-sql">mysql> select job_files.id from job_files
left join job on job_files.id = job.jobid
where job.jobid is null;
+------+
| id |
+------+
| 2254 |
| 2806 |
+------+
2 rows in set (0.05 sec)
</code></pre><h2>Deleting Dead Files</h2><p>Deleting all the file records in one blow was not possible for me because there was too many files to delete and the mysql server did not have enough resources on the ReadyNAS to do it. I had to delete these records in batch of 100000 files, the process was repeated several times (each delete query took more than 2mn!!!).</p><pre><code class="lang-sql">mysql> delete from file where jobid = 2254 limit 100000;
</code></pre><h2>Conclusion</h2><p>This cleanup process allowed me to reduce the size of the <code>file</code> table from 10 millions entries to 7 millions. This improves the database performance and speeds up the <a href="http://www.bacula.org/en/dev-manual/main/main/Catalog_Maintenance.html">Bacula catalog backup</a> process.</p></div> Connecting to a ReadyNAS duo using SSHurn:md5:e74100b4fe2d282be2e429c7be5fdad22010-09-18T20:20:07+00:002010-09-18T20:20:07+00:00Stephane CarrezLinuxReadyNASbaculassh
<div class="post-text"><p>Before you start, you must be aware that there is a risk that you break your ReadyNAS. You should not do this unless you really understand what it is doing.</p><h2>Installing the EnableRootSSH extension</h2><p>The first step is to install the addon which allows you to connect to your ReadyNAS using ssh:</p><ol><li>Download the EnableRootSSH extension. I've used the following link:</li></ol><p><a href="http://www.readynas.com/download/addons/4.00/EnableRootSSH_1.0.bin">http://www.readynas.com/download/addons/4.00/EnableRootSSH_1.0.bin</a></p><ol><li>Go in the ReadyNAS FrontView with your browser and go to <i>System -> Update -> Local Update</i></li><li>Upload the EnableRootSSH binary file. The ReadyNAS verifies that content and if it is correct it displays a description of the addon.</li><li>Acknowledge the installation of the addon</li></ol><p>After installation, the ReadyNAS must be restarted. Shortly after, you will receive an email:</p><pre><code>Subject: Addon Package Progress (nas-XX-XX-XX)
Successfully enabled root SSH access. The root password is now the same as your admin password.
</code></pre><h3>Connecting to the ReadyNAS using ssh</h3><p>With the EnableRootSSH extension in place, you can easily connect using <a href="https://help.ubuntu.com/community/SSHHowto">ssh</a>. The RSA key fingerprint of your ReadyNAS is prompted and you must accept it in your <i>known_hosts</i>.</p><pre><code>$ ssh -l root pollux
The authenticity of host 'pollux (192.168.1.6)' can't be established.
RSA key fingerprint is 01:c8:00:b4:56:5a:f9:fe:2d:73:9a:b0:55:a1:31:2f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'pollux,192.168.1.6' (RSA) to the list of known hosts.
root@pollux's password:
Linux nas-D2-24-F2 2.6.17.8ReadyNAS #1 Fri Sep 19 15:04:06 PDT 2008 padre unknown
nas-D2-24-F2:~#
</code></pre><h3>Exploring the ReadyNAS</h3><p>Since the ReadyNAS runs a <a href="http://www.debian.org/releases/sarge/">Debian Sarge</a> with a GNU/Linux 2.6.17 kernel, you can easily explore the system.</p><h4>CPU and Memory</h4><p>The CPU is a Sparc-V8 (<a href="http://en.wikipedia.org/wiki/LEON">LEON</a>) that Infrant has optimized for their needs. It integrates hardware RAID, the gigabit Ethernet and 4 SATA channels, a 64-bit DDR SRAM controller, a DMA, a 3 DES engine and a PCI/USB interface.</p><pre><code>nas-D2-24-F2:~# cat /proc/cpuinfo
cpu : Infrant Technologics, Inc. - neon version: 0
fpu : Softfpu
ncpus probed : 1
ncpus active : 1
BogoMips : 186.36
MMU : version: 0
LP : HW.FW version: 0.1
FPGA : fpga000000-0 Configuration: 0
AHB arbitraion : 7
CPU id : 0
Switch : 0
ASIC : IT3107
</code></pre><p>And the memory:</p><pre><code>nas-D2-24-F2:~# cat /proc/meminfo
MemTotal: 226384 kB
MemFree: 146560 kB
Buffers: 15440 kB
Cached: 42352 kB
SwapCached: 0 kB
Active: 61776 kB
Inactive: 22944 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 226384 kB
LowFree: 146560 kB
SwapTotal: 255968 kB
SwapFree: 255968 kB
Dirty: 0 kB
Writeback: 0 kB
Mapped: 39712 kB
Slab: 5488 kB
CommitLimit: 391792 kB
Committed_AS: 72048 kB
PageTables: 0 kB
VmallocTotal: 131008 kB
VmallocUsed: 1120 kB
VmallocChunk: 129408 kB
</code></pre><h4>Disks</h4><p>The system is installed on the hard disk. It appears to use arround 1.9G from my 1Tb disk.</p><pre><code>nas-D2-24-F2:/usr# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hdc1 1.9G 224M 1.7G 12% /
tmpfs 16k 0 16k 0% /USB
/dev/c/c 925G 543M 924G 1% /c
</code></pre><h4>Other interesting commands:</h4><p>The following commands are interesting to explore the system and learn more about it:</p><pre><code>dpkg -l List of installed packages
netstat -ln Network open ports (TCP/UDP/Unix)
top Top running processes
ps aux List all running processes
</code></pre><h2>Next step</h2><p>The next step for me is to see if I can install the <a href="http://www.bacula.org/en/dev-manual/Storage_Daemon_Configuratio.html">Bacula Storage Daemon</a> and see if my bacula server is able to connect to it directly.</p></div> Installing Mysql server on a ReadyNAS duourn:md5:c5b6e17aad7863cc8e37de526f92a1d02010-09-18T20:20:06+00:002010-09-18T20:20:06+00:00Stephane CarrezLinuxMysqlReadyNASbacula
<div class="post-text"><h2>Installing the APT extension</h2><p>The ReadyNAS duo runs a Debian Sarge distribution. The apt-get commands are not available and we must install the APT addon:</p><ol><li>Download the APT extension. I've used the following link:<br></li></ol><p><a href="http://www.readynas.com/download/addons/4.00/APT_1.0.bin">http://www.readynas.com/download/addons/4.00/APT_1.0.bin</a></p><ol><li>Go in the ReadyNAS FrontView with your browser and go to <i>System -> Update -> Local Update</i></li><li>Upload the APT binary file. The ReadyNAS verifies that content and if it is correct it displays a description of the addon.</li><li>Acknowledge the installation of the addon</li></ol><p>After installation, the ReadyNAS must be restarted. Shortly after, you will receive an email:</p><pre><code>Subject: Addon Package Progress (nas-XX-XX-XX)
Successfully installed APT.
</code></pre><h3>Debian Sarge Package update</h3><p>It may be good to check the debian packages. <a href="/index.php?post/2009/03/21/Connecting-to-a-ReadyNAS-duo-using-SSH%3A">Connect to the ReadyNAS using ssh</a> and run the following commands:</p><pre><code>apt-get update
</code></pre><h2>Installation of Mysql server</h2><p>The mysql-server-5.0 is supposed to be installed according to <code>dpkg -i</code> command. However, the files are not there and they have probably be removed. This is also the case for some utilities which are used by some installation scripts.</p><h4>Preparation</h4><p>The <code>/usr/bin/chfn</code> utility is missing and we need it for Mysql installation. We must re-install the <a href="ftp://archive.debian.org/debian-archive/debian/pool/main/s/shadow/passwd_4.0.3-31sarge9_sparc.deb">passwd</a> package. Download it and install it as follows;</p><pre><code># dpkg -i passwd_4.0.3-31sarge9_sparc.deb
</code></pre><p>The <code>/usr/bin/logger</code> is also missing. We must re-install the <a href="ftp://archive.debian.org/debian-archive/debian/pool/main/u/util-linux/bsdutils_2.12p-4sarge2_sparc.deb">bsdutils_2.12p-4sarge2_sparc.deb</a> package. Download it and install it as follows:</p><pre><code># dpkg -i bsdutils_2.12p-4sarge2_sparc.deb
</code></pre><p>Edit the <code>/etc/mysql/my.cnf</code> file and change the line:</p><pre><code>user = admin
</code></pre><p>into</p><pre><code>user = mysql
</code></pre><h4>Additional packages</h4><p>As reported by RoB (see comments), the following packages are also necessary for mysql-server:</p><pre><code># apt-get install libreadline5 libdbi-perl
</code></pre><h4>Get Mysql packages</h4><p>The mysql packages are part of backports.org (backport from Etch). You should download the following:</p><ul><li><a href="ftp://archive.debian.org/debian-archive/backports.org/pool/main/m/mysql-dfsg-5.0/libmysqlclient15off_5.0.32-7etch5~bpo31+1_sparc.deb">libmysqlclient15off_5.0.32-7etch5~bpo31+1_sparc.deb</a></li><li><a href="ftp://archive.debian.org/debian-archive/backports.org/pool/main/m/mysql-dfsg-5.0/mysql-common_5.0.32-7etch5~bpo31+1_all.deb">mysql-common_5.0.32-7etch5~bpo31+1_all.deb</a></li><li><a href="ftp://archive.debian.org/debian-archive/backports.org/pool/main/m/mysql-dfsg-5.0/mysql-server-5.0_5.0.32-7etch5~bpo31+1_sparc.deb">mysql-server-5.0_5.0.32-7etch5~bpo31+1_sparc.deb</a></li><li><a href="ftp://archive.debian.org/debian-archive/backports.org/pool/main/m/mysql-dfsg-5.0/mysql-client-5.0_5.0.32-7etch5~bpo31+1_sparc.deb">mysql-client-5.0_5.0.32-7etch5~bpo31+1_sparc.deb</a></li><li><a href="ftp://archive.debian.org/debian-archive/backports.org/pool/main/m/mysql-dfsg-5.0/mysql-client_5.0.32-7etch5~bpo31+1_all.deb">mysql-client_5.0.32-7etch5~bpo31+1_all.deb</a></li></ul><ul><li></li></ul><h4>Mysql installation</h4><p>Install the Mysql server debian packages:</p><pre><code># dpkg -i libmysqlclient15off_5.0.32-7etch5~bpo31+1_sparc.deb \
mysql-common_5.0.32-7etch5~bpo31+1_all.deb \
mysql-server-5.0_5.0.32-7etch5~bpo31+1_sparc.deb \
mysql-client-5.0_5.0.32-7etch5~bpo31+1_sparc.deb \
mysql-client_5.0.32-7etch5~bpo31+1_all.deb
</code></pre><h5>Installation Problem and Solution</h5><p>If in the installation process you see a message '/etc/init.d/mysql' not found, force the extraction of mysql-server files without the execution of the installation script:</p><pre><code># dpkg --extract mysql-server-5.0_5.0.32-7etch5~bpo31+1_sparc.deb /
</code></pre><p>After that, redo the <code>dpkg -i</code> command.</p><p>If you see some errors in the logs:</p><pre><code>[ERROR] /usr/sbin/mysqld: Can't find file: './mysql/host.frm' (errno: 13)
</code></pre><p>Fix ownership of some existing directories:</p><pre><code># chown mysql /var/lib/mysql
# chown mysql /var/lib/mysql/mysql
</code></pre><p>and restart mysql</p><h5>Fix /etc/mysql/debian.cnf</h5><p>After installation, the debian.cnf file used by mysqlcheck uses a user that does not exist. You can either create the <code>debian-sys-maint</code> user in mysql or change it to 'root' by editing the file <code>/etc/mysql/debian.cnf</code> and change user and password to use 'root'.</p><h5>Fix startup scripts</h5><p>Remove the following two startup scripts, they are not necessary for us:</p><ol><li>rm /etc/rc2.d/S18mysql-ndb</li><li>rm /etc/rc2.d/S17mysql-ndb-mgm</li></ol><h3>Testing the database</h3><p>Verify that the Mysql database is running:</p><pre><code>nas-D2-24-F2:/var/log# mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 6
Server version: 5.0.32-Debian_7etch5~bpo31+1 Debian etch distribution
Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
mysql>
</code></pre></div> One year of data backup with Bacula on a ReadyNAS duourn:md5:dd98193638e3ccebbe33821d4851c2c02010-09-18T20:20:06+00:002010-09-18T20:20:06+00:00Stephane CarrezReadyNASbackupbacula
<div class="post-text"><p>After one year of daily and weekly backup using Bacula on a ReadyNAS duo, I wanted to share information about this success story. <a href="http://www.bacula.org">Bacula</a> is a network backup solution that I installed on a <a href="http://www.readynas.com">ReadyNAS duo</a>. Bacula allows to make full as well as incremental backups of remote machines. It uses a MySQL database that also runs on the ReadyNAS (see <a href="/index.php?post/2009/03/22/Installing-Mysql-server-on-a-ReadyNAS-duo">Installing Mysql server on a ReadyNAS duo</a>) and it stores backups on media such as tapes, CDs, DVDs or files.</p><h2>Backup Architecture</h2><p>The Bacula software is running directly on the ReadyNAS duo. The backup is configured to backup my desktop which is accessed locally, and it also backups a server running on the Internet (vacs.fr). Since the ReadyNAS is behind my Livebox, it connects to the Internet server by using a secure tunnel with OpenVPN.</p><p><img src="/images/PC/bacula-readynas-backup.png" longdesc="Network Backup with Bacula on a ReadyNAS, avr. 2010" alt="Network Backup with Bacula on a ReadyNAS"></img></p><p>The ReadyNAS duo has two 1To hard disks configured as RAID 1 mirrors.</p><ul><li>Bacula director and bacular storage daemons are running on the ReadyNAS duo</li><li>Bacula client is running on each machine that must be backed up (<i>Desktop</i> and <i>Remote Server</i>).</li></ul><h2>Backup Pools and Strategy</h2><p>Bacula is configured to create backups on file tapes. Each tape is a flat file stored on the ReadyNAS duo in some directory. I've configured file tapes so that they do not extend 4.3G (so that copying and burning DVDs could be possible).</p><p>File tapes are grouped in several pools. Each pool represent a class of backup. My primary backup strategy is split in 3 backup grades:</p><p><b>A-Grade</b> backups represent critical files that must not be lost at all. They represent the files that I really care and for which I want to have one year of backup. The retention policy is set to one year with one full backup per month. In short, it means I can restore the data I had anytime during the last year. Basically it contains my full desktop home directory as well as specific directories (private photos and so on).</p><p><b>B-Grade</b> backups represent less critical files for which I may not need to restore an old version. The retention policy is 180 days. This backup grade is used for software or files that I download from Internet.</p><p><b>C-Grade</b> backups have a 65-days retention policy and they are used for the system. Basically, re-installation of a server or desktop from scratch is always possible but keeping the configuration files in the backup is very helpful.</p><p>A Pool is defined for each of these grades:</p><pre><code># A-Grade pool: 1 year retention, 12 full backups (1 full bkp/month)
Pool {
Name = A-Full-Pool
Pool Type = Backup
# Bacula can automatically recycle Volumes
Recycle = yes
AutoPrune = yes # Prune expired volumes
Volume Retention = 360 days
Label Format=A-Full-
# 100 volumes of 4G (expecting 8 volumes/full backup)
Maximum Volumes=100
}
# B-Grade pool: 6 months retention, 3 full backups (1 full bkp/2 months)
Pool {
Name = B-Full-Pool
Pool Type = Backup
Recycle = yes
AutoPrune = yes # Prune expired volumes
Volume Retention = 180 days
Label Format=B-Full-
Maximum Volumes=40
}
# C-Grade pool: 2 months retention, 2 full backups (1 full bkp/45 day)
Pool {
Name = C-Full-Pool
Pool Type = Backup
Recycle = yes
AutoPrune = yes
Volume Retention = 65 days # 2 months
Label Format=C-Full-
# 5 volumes of 4G (expecting 2 volumes/full backup)
Maximum Volumes=5
}
</code></pre><p>In addition to these pools, an incremental and a differential pool must be defined.</p><h2>Bacula FileSet</h2><p>The Bacula FileSet represent the file patterns that have to be backed up. I have defined one FileSet for each machine and backup grade combination. The filesets are compressed. Files matching some patterns are excluded (<code>*.o, *.log, *.bak, *~</code>). The FileSet below is for my desktop and for the <b>A-Grade</b> backup. Directories <code>/home</code>, <code>/data</code> and <code>/photos</code> will be taken into account in the backup.</p><pre><code># List of files to be backed up
FileSet {
Name = "Zebulon A-Grade"
Include {
Options {
signature=SHA1
compression=GZIP
verify = pins1
onefs = yes
WildFile = "*~"
WildFile = "*.bak"
WildFile = "*.log"
WildFile = "*.o"
Exclude = yes
}
File = /home
File = /data
File = /photos
}
}
</code></pre><p>Other FileSets are defined for the same machine but for different files. They will be used for other backup grades.</p><h2>Backup Schedule</h2><p>The schedule defines when the backup has to be executed. Each backup grade has its own schedule. This allows to run B-Grade and C-Grade backups less frequently than <b>A-Grade</b>.</p><p>The <b>A-Grade</b> backups have a full backup schedule the first Saturday of each month. A full backup of the desktop takes arround 5 hours and uses 57Go (compressed). A differential backup takes arround 2 hours and uses 10Go (compressed). The incremental backup uses 2-4Go (compressed) and 5 to 15 minutes. (these numbers depend on what is being backed up). The schedule hours are defined according to this.</p><pre><code>Schedule {
Name = "Weekly-A-Grade"
Run = Full 1st sat at 23:05
Run = Differential 2nd-5th sun at 22:10
Run = Incremental sun-fri at 22:10
}
Schedule {
Name = "Weekly-B-Grade"
Run = Full jan 1st sat at 23:05
Run = Full mar 1st sat at 23:05
Run = Full may 1st sat at 23:05
Run = Full jul 1st sat at 23:05
Run = Full sep 1st sat at 23:05
Run = Full nov 1st sat at 23:05
Run = Differential 2nd-5th sun at 22:10
Run = Incremental wed at 22:10
}
Schedule {
Name = "Weekly-C-Grade"
Run = Full jan 1st sat at 2:05
Run = Full mar 1st sat at 2:05
Run = Full may 1st sat at 2:05
Run = Full jul 1st sat at 2:05
Run = Full sep 1st sat at 2:05
Run = Full nov 1st sat at 2:05
Run = Differential 2nd-5th sat at 2:10
Run = Incremental sat at 2:10
}
</code></pre><h2>Bacula Job</h2><p>The Bacula Job describes what must be backed up (FileSets), when (Schedule) and where (Pools). There is one job definition for each fileset.</p><pre><code>Job {
Name = "Zebulon-A"
Type = Backup
Client = zebulon-fd
FileSet = "Zebulon A-Grade"
Schedule = "Weekly-A-Grade"
Storage = File
Messages = Standard
Pool = Default
Full Backup Pool = A-Full-Pool
Incremental Backup Pool = Incr-Pool
Differential Backup Pool = Diff-Pool
Priority = 8
}
</code></pre><h2>Some Statistics</h2><p>After more than one year of backups, the total storage space used is now 599G, each tape is 4.3G. The storage space used by file pools is as follows:</p><pre><code>A Grade Full Tapes 73 313Go
B Grade Full Tapes 28 120Go
C Grade Full Tapes 4 17Go
Differential tapes 22 94Go
Incremental tapes 13 55Go
</code></pre><p>The MySQL database has grown a lot and is quite large. The InnoDB database file only contains the bacula database and it has grown up to 2Go now. The <code>filename</code> table references 885527 records and the <code>path</code> table references 546784 rows.</p><h2>Conclusion</h2><p>Bacula is not easy to configure but when you do it right it provides a performant backup solution. To learn more about the configuration, have a look at <a href="http://www.bacula.org/2.4.x-manuals/en/main/index.html">Bacula Documentation</a>. Installed on a ReadyNAS duo, it proved to be a robust solution for a backup of a small set of machines. You cannot expect big performances during backup or restore. The performance bottleneck is the MySQL database which runs on the ReadyNAS.</p><p>Restoring files from the backup is quite easy but this is another story...</p></div> Solving Linux system lock up when intensive disk I/O are performedurn:md5:bd34a9db0f0442fe4bc9f607a69e8f8b2010-08-28T08:02:43+00:002010-08-28T08:02:43+00:00Stephane CarrezLinuxbackupbaculaperformance
<div class="post-text"><p>When a system lock up occurs, we often blame applications but when you look carefully you may see that despite your multi-core CPU, your applications are sleeping! No cpu activity! So what happens then? Check the I/Os, it could be the root cause!</p><p>With Ubuntu 10.04, my desktop computer was freezing when the <a href="/index.php?post/2010/04/09/One-year-of-data-backup-with-Bacula-on-a-ReadyNAS-duo">ReadyNAS Bacula backup</a> was running. Indeed, the Bacula daemon was performing intensive disk operations (on a fast SATA hard disk). The situation was such that it was impossible to use the system, the interface was freezing for a several seconds then working for a few seconds and freezing again.</p><h2>Linux I/O Scheduler</h2><p>The I/O scheduler is responsible for organizing the order in which disk operations are performed. Some algorithms allow to minimize the disk head moves, other algorithms tend to anticipate read operations,</p><p>When I/O operations are not scheduled correctly, an interactive application such as a desktop or a browser can be blocked until its I/O operations are scheduled and executed (the situation can be even worse for those applications that use the <code>O_SYNC</code> writing mode).</p><p>By default, the Linux kernel is configured to use the <a href="http://en.wikipedia.org/wiki/CFQ">Completely Fair Queuing</a> scheduler. This I/O scheduler does not provide any time guarantee but it gives in general good performances. Linux provides other I/O schedulers such as the <a href="http://en.wikipedia.org/wiki/Noop_scheduler">Noop scheduler</a>, the <a href="http://en.wikipedia.org/wiki/Anticipatory_scheduling">Anticipatory scheduler</a> and the <a href="http://en.wikipedia.org/wiki/Deadline_scheduler">Deadline scheduler</a>.</p><p>The deadline scheduler puts an execution time limit to requests to make sure the I/O operation is executed before an expiration time. Typically, a read operation will wait at most 500 ms. This is the I/O scheduler we need to avoid the system lock up.</p><h2>Checking the I/O Scheduler</h2><p>To check which I/O scheduler you are using, you can use the following command:</p><pre><code>$ cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]
</code></pre><p>where <code>sda</code> is the device name of your hard disk (or try <code>hda</code>).</p><p>The result indicates the list of supported I/O scheduler as well as the current scheduler used (here the <a href="http://en.wikipedia.org/wiki/CFQ">Completely Fair Queuing</a>).</p><h3>Changing the I/O Scheduler</h3><p>To change the scheduler, you can <code>echo</code> the desired scheduler name to activate it (you must be root):</p><pre><code># echo deadline > /sys/block/sda/queue/scheduler
</code></pre><p>To make sure the I/O scheduler is configured after each system startup, you can add the following lines to your <code>/etc/rc.local</code> startup script:</p><pre><code>test -f /sys/block/sda/queue/scheduler &&
echo deadline > /sys/block/sda/queue/scheduler
test -f /sys/block/sdb/queue/scheduler &&
echo deadline > /sys/block/sdb/queue/scheduler
test -f /sys/block/hda/queue/scheduler &&
echo deadline > /sys/block/hda/queue/scheduler
</code></pre><p>You may have to change the <code>sda</code> and <code>sdb</code> into <code>hda</code> and <code>hdb</code> if you have an IDE hard disk.</p><h2>Conclusion</h2><p>After changing the I/O scheduler to use the <a href="http://en.wikipedia.org/wiki/Deadline_scheduler">Deadline</a> scheduler, the desktop was not freezing any more when backups are running.</p></div> Restoring a complete system after a hard disk failure: bacula to the rescue!!!urn:md5:b455e5b3ffca5ceb8df9c4a11eb3503c2008-09-08T20:32:34+00:002008-09-08T20:32:34+00:00Stephane CarrezLinuxUbuntubackupbaculacrashrecovery
<div class="post-text"><p><b>Step 1: Boot on your Ubuntu 8.04 CD</b></p><p>Since the disk that crashed contained the system, my computer was not even able to boot. A first step for me was to boot on the Ubuntu CDrom without installing Ubuntu again. After booting I was able to check my other disks, look at the kernel logs to realize that the disk was really completely dead without any hope to recover anything. By looking at my second hard disk, I was able to evaluate what was lost and needed to be recovered. If you have no other disk, you have to setup a new disk to proceed. Booting on the CD also helped me discover some room on my second disk where I would install a new system.</p><p><b>Step 2: Install the system</b></p><p>If the system has gone, you may have to re-install it from scratch. This is what I had to do. Having found an old debian partition on my second hard disk, I decided to install <a href="http://www.ubuntu.com/getubuntu/download" lang="en">Ubuntu 8.0.4 Desktop</a> on it. After 15 minutes, my computer was working again, running Ubuntu 8.0.4 as before. Still, my data were lost.</p><p><b>Step 3: Restore with bacula</b></p><p><a href="http://www.bacula.org/en/" lang="en">Bacula</a> is a great network backup solution that I put in place 2 years ago. Every night my bacula server is creating an incremental, differential or full backup of my computer (<i>zebulon</i>). It is the first time thought that I had to recover a full content. For the recovery, you have to use the <a href="http://www.bacula.org/en/dev-manual/Bacula_Console.html" lang="en">Bacula Console</a> and use the <b>restore</b> command.</p><p>ciceron $ <b>bconsole</b></p><p>Every action made in bacula creates a job that is recorded in the database. The first thing is to identify those jobs that did the full, differential and incremental backups.</p><pre><code>* __list jobs__
</code></pre><pre><code> | JobId | Name | StartTime | Type | Level | JobFiles | JobBytes | JobStatus |
| 877 | Zebulon | 2007-12-02 02:22:27 | B | F | 1,245,258 | 31,026,036,274 | T |
| 1,067 | Zebulon | 2008-02-03 00:52:18 | B | F | 0 | 0 | f |
| 1,319 | Zebulon | 2008-04-26 22:28:29 | B | D | 207,801 | 6,048,511,830 | T |
| 1,328 | Zebulon | 2008-04-29 22:17:04 | B | I | 0 | 0 | E |
| 1,331 | Zebulon | 2008-04-30 22:17:04 | B | I | 1,025 | 761,323,545 | T |
| 1,511 | Zebulon | 2008-06-29 22:47:57 | B | I | 77,997 | 9,050,108,256 | T |
| 1,514 | Zebulon | 2008-06-30 22:16:40 | B | I | 968 | 613,957,318 | T |
| 1,517 | Zebulon | 2008-07-01 22:16:38 | B | I | 16,710 | 866,232,575 | T |
| 1,520 | Zebulon | 2008-07-02 22:17:00 | B | I | 11,530 | 887,021,057 | T |
</code></pre><p>In result above is just an extract of the list command. Job 877 is a full backup (level <b>F</b>) and I had no other recent full backups than this one. It must be restored first. Since bacula has pruned the files, it has lost all the information about its contain (my backup could have been improved). Anyway, it is possible to restore completely this full backup. Jobs 1067 and 1328 cannot be used because they were in errors (I had many of them because the computer is off when the daily backup is started or for some other reasons). This is not a problem, bacula just ignores those jobs for the restore. To restore the full backup use the <b>restore</b> command:</p><pre><code> * __restore__
First you select one or more JobIds that contain files
to be restored. You will be presented several methods
of specifying the JobIds. Then you will be allowed to
select which files from those JobIds are to be restored.
</code></pre><p>After this, the bacula restore command prompts for a restore method. You can restore a files selectively, find files or restore a complete job or complete client. For me, I had to restore the full backup (job 877) so I selected the <b>Enter list of comma separated JobIds to select</b> method with my full backup job id:</p><pre><code> To select the JobIds, you have the following choices:
1: List last 20 Jobs run
2: List Jobs where a given File is saved
3: Enter list of comma separated JobIds to select
4: Enter SQL list command
5: Select the most recent backup for a client
6: Select backup for a client before a specified time
7: Enter a list of files to restore
8: Enter a list of files to restore before a specified time
9: Find the JobIds of the most recent backup for a client
10: Find the JobIds for a backup for a client before a specified time
11: Enter a list of directories to restore for found JobIds
12: Cancel
Select item: (1-12): __3__
Enter JobId(s), comma separated, to restore: __877__
You have selected the following JobId: 877
Building directory tree for JobId 877 ...
There were no files inserted into the tree, so file selection
is not possible.Most likely your retention policy pruned the files
Do you want to restore all the files? (yes|no): __yes__
</code></pre><p>After this step, bacula searches which volumes (backup files, DVD, tapes) contain the backup:</p><pre><code> Bootstrap records written to /var/lib/bacula/janus-dir.restore.12.bsr
The job will require the following
Volume(s) Storage(s) SD Device(s)
===================================================
Full-0013 File FileStorage
Full-0014 File FileStorage
Full-0015 File FileStorage
Full-0016 File FileStorage
Full-0017 File FileStorage
Full-0035 File FileStorage
Full-0036 File FileStorage
Full-0037 File FileStorage
1,245,258 files selected to be restored.
</code></pre><p>Now, I had to choose the client for the restore. For some reasons, I had to choose my crashed computer (zebulon):</p><pre><code> Defined Clients:
1: janus-fd
2: zebulon-fd
Select the Client (1-2): __2__
</code></pre><p>Bacula describes the restore job and you have a chance to change some parameters. In general, the restore process is made by the bacula daemon on the computer that you want to restore (ie, the client). This is natural, your computer X crashed and you want to recover on it. In my case, I wanted to recover on bacula server (called janus).</p><pre><code> Run Restore job
JobName: RestoreFiles
Bootstrap: /var/lib/bacula/janus-dir.restore.13.bsr
Where: /tmp/bacula-restores
Replace: always
FileSet: Janus Files
Backup Client: zebulon-fd
Restore Client: zebulon-fd
Storage: File
When: 2008-07-05 14:16:28
Catalog: MyCatalog
Priority: 10
OK to run? (yes/mod/no): __mod__
Parameters to modify:
1: Level
2: Storage
3: Job
4: FileSet
5: Restore Client
6: When
7: Priority
8: Bootstrap
9: Where
10: File Relocation
11: Replace
12: JobId
Select parameter to modify (1-12): __5__
The defined Client resources are:
1: janus-fd
2: zebulon-fd
Select Client (File daemon) resource (1-2):__ 1__
Run Restore job
JobName: RestoreFiles
Bootstrap: /var/lib/bacula/janus-dir.restore.13.bsr
Where: /tmp/bacula-restores
Replace: always
FileSet: Janus Files
Backup Client: zebulon-fd
Restore Client: janus-fd
Storage: File
When: 2008-07-05 14:16:28
Catalog: MyCatalog
Priority: 10
OK to run? (yes/mod/no): __yes__
</code></pre><p>The restore process runs in background and a message and an email are sent after the restore job has finished. In my case, the files were restored on my bacula server in a <b>/tmp/bacula-restores</b> directory. When the restore process finished, that directory contained all my files.... back in December 2007. The differential backup was restored in the same say because the files were pruned too. Other jobs were restored as follows, using the same <b>restore</b> command:</p><pre><code> * __restore__
First you select one or more JobIds that contain files
to be restored. You will be presented several methods
of specifying the JobIds. Then you will be allowed to
select which files from those JobIds are to be restored.
To select the JobIds, you have the following choices:
1: List last 20 Jobs run
2: List Jobs where a given File is saved
3: Enter list of comma separated JobIds to select
4: Enter SQL list command
5: Select the most recent backup for a client
6: Select backup for a client before a specified time
7: Enter a list of files to restore
8: Enter a list of files to restore before a specified time
9: Find the JobIds of the most recent backup for a client
10: Find the JobIds for a backup for a client before a specified time
11: Enter a list of directories to restore for found JobIds
12: Cancel
Select item: (1-12):__ 3__
Enter JobId(s), comma separated, to restore: __1331,1511,1514,1517,1520__
You have selected the following JobIds: 1331,1511,1514,1517,1520
Building directory tree for JobId 1331 ...
Building directory tree for JobId 1511 ... +++++++++++++++++++++++++++++++++
Building directory tree for JobId 1517 ... +++++++++++++++++++++++++
Building directory tree for JobId 1520 ... +++++++++++++++++++++++++++++
5 Jobs, 75,552 files inserted into the tree.
You are now entering file selection mode where you add (mark) and
remove (unmark) files to be restored. No files are initially added, unless
you used the "all" keyword on the command line.
Enter "done" to leave this mode.
cwd is: /
$ __mark *__
79,536 files marked.
$ __done__
Bootstrap records written to /var/lib/bacula/janus-dir.restore.14.bsr
The job will require the following
Volume(s) Storage(s) SD Device(s)
======================================================
Incr-0002 File FileStorage
Incr-0005 File FileStorage
Incr-0001 File FileStorage
Incr-0006 File FileStorage
79,536 files selected to be restored.
</code></pre><p>After the restore jobs finished, all my files were restored back to July 2nd 2008.</p><p><b>Lesson learned and conclusion</b></p><ol><li>Backup is vital in computer world. You don't want to loose your photos, emails and documents. When you loose one of them, you just cry. When you loose everything, you....die.</li><li>My bacula configuration is not perfect. In particular it should do a full backup every 3 or 6 months. In the past I only used some file recovery but I've never tested a full recovery. This was an error (without bad consequences hopefully). Every change in bacula configuration must be followed by a full recovery test.</li><li>The system partitions (/ and /usr) were not backup. Even if we can restore them with an installation, this may not be a good idea. You loose the configuration files and the knowledge of all the packages you have installed. Loosing this is not a big deal but it is a matter of time.</li><li>It is necessary to test on a regular basis that we can recover from the backup. The problem is absolutely not the software itself. The problem is the backup configuration and backup needs that change over the time.</li></ol><p>I am very thankful to the Bacula development team for their software. It is really a professional backup solution. I knew that for sure but now I can say I tested it in real situation. The hard disk failure only costs me time: time to install, time to recover the backup and time to write this story....</p></div>