Java 2 Ada

Bacula database cleanup

By Stephane Carrez

Bacula maintains a catalog of files in a database. Over time, the database grows and despite some automatic purge and job cleanup, some information remains that is no longer necessary. This article explains how to remove some dead records from the Bacula catalog.

Bacula maintains a list of backup jobs that have been executed in the job table. For each job, it keeps the list of files that have been saved in the file table. When you do a restore, you somehow select the job to restore and pick files from that job. There should not exist any file entry associated with a non existing job. Unfortunately this is not the case. I've found that some files (more than 2 millions entries) were pointing to some job that did not exist.

Discovering dead jobs still referenced

The first step is to find out which job has been deleted and is still referenced by the file table. First, let's create a temporary table that will hold the job ids associated with the files.


mysql> create temporary table job_files (id bigint);

The use of a temporary table was necessary in my case because the file table is so big and the ReadyNAS so slow that scanning the database takes too much time.

Now, we can populate the temporary table with the job ids:


mysql> insert into job_files select distinct file.jobid from file;
Query OK, 350 rows affected (8 min 53.26 sec)
Records: 350  Duplicates: 0  Warnings: 0

The list of jobs that have been removed but are still referenced by a file is obtained by:


mysql> select job_files.id from job_files
 left join job on job_files.id = job.jobid
 where job.jobid is null;
+------+
| id   |
+------+
| 2254 | 
| 2806 | 
+------+
2 rows in set (0.05 sec)

Deleting Dead Files

Deleting all the file records in one blow was not possible for me because there was too many files to delete and the mysql server did not have enough resources on the ReadyNAS to do it. I had to delete these records in batch of 100000 files, the process was repeated several times (each delete query took more than 2mn!!!).


mysql> delete from file where jobid = 2254 limit 100000;

Conclusion

This cleanup process allowed me to reduce the size of the file table from 10 millions entries to 7 millions. This improves the database performance and speeds up the Bacula catalog backup process.

Connecting to a ReadyNAS duo using SSH

By Stephane Carrez 6 comments

Having acquired a ReadyNAS duo for my new backup system, I wanted to explore the system that runs on it and see if I could run more services on it. There is nothing terrific in this article as many people have already done

Read more
6 comments
To add a comment, you must be connected. Login to add a comment

Installing Mysql server on a ReadyNAS duo

By Stephane Carrez 11 comments

Being able to connect to my ReadyNAS duo using SSH (See Connecting to a ReadyNAS duo using SSH), the next step for setting up a Bacula backup solution was to setup a MySQL server. Th

Read more
11 comments
To add a comment, you must be connected. Login to add a comment

One year of data backup with Bacula on a ReadyNAS duo

By Stephane Carrez

After one year of daily and weekly backup using Bacula on a ReadyNAS duo, I wanted to share information about this success story. Bacula is a network backup solution that I installed on a ReadyNAS duo. Bacula allows to make full as well as incremental backups of remote machines. It uses a MySQL database that also runs on the ReadyNAS (see Installing Mysql server on a ReadyNAS duo) and it stores backups on media such as tapes, CDs, DVDs or files.

Backup Architecture

The Bacula software is running directly on the ReadyNAS duo. The backup is configured to backup my desktop which is accessed locally, and it also backups a server running on the Internet (vacs.fr). Since the ReadyNAS is behind my Livebox, it connects to the Internet server by using a secure tunnel with OpenVPN.

Network Backup with Bacula on a ReadyNAS

The ReadyNAS duo has two 1To hard disks configured as RAID 1 mirrors.

  • Bacula director and bacular storage daemons are running on the ReadyNAS duo
  • Bacula client is running on each machine that must be backed up (Desktop and Remote Server).

Backup Pools and Strategy

Bacula is configured to create backups on file tapes. Each tape is a flat file stored on the ReadyNAS duo in some directory. I've configured file tapes so that they do not extend 4.3G (so that copying and burning DVDs could be possible).

File tapes are grouped in several pools. Each pool represent a class of backup. My primary backup strategy is split in 3 backup grades:

A-Grade backups represent critical files that must not be lost at all. They represent the files that I really care and for which I want to have one year of backup. The retention policy is set to one year with one full backup per month. In short, it means I can restore the data I had anytime during the last year. Basically it contains my full desktop home directory as well as specific directories (private photos and so on).

B-Grade backups represent less critical files for which I may not need to restore an old version. The retention policy is 180 days. This backup grade is used for software or files that I download from Internet.

C-Grade backups have a 65-days retention policy and they are used for the system. Basically, re-installation of a server or desktop from scratch is always possible but keeping the configuration files in the backup is very helpful.

A Pool is defined for each of these grades:

# A-Grade pool: 1 year retention, 12 full backups (1 full bkp/month)
Pool {
  Name = A-Full-Pool
  Pool Type = Backup
  # Bacula can automatically recycle Volumes
  Recycle = yes 
  AutoPrune = yes # Prune expired volumes
  Volume Retention = 360 days
  Label Format=A-Full-
  # 100 volumes of 4G (expecting 8 volumes/full backup)
  Maximum Volumes=100
}

# B-Grade pool: 6 months retention, 3 full backups (1 full bkp/2 months)
Pool {
  Name = B-Full-Pool
  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes # Prune expired volumes
  Volume Retention = 180 days
  Label Format=B-Full-
  Maximum Volumes=40
}

# C-Grade pool: 2 months retention, 2 full backups (1 full bkp/45 day)
Pool {
  Name = C-Full-Pool
  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 65 days # 2 months
  Label Format=C-Full-
  # 5 volumes of 4G (expecting 2 volumes/full backup)
  Maximum Volumes=5
}

In addition to these pools, an incremental and a differential pool must be defined.

Bacula FileSet

The Bacula FileSet represent the file patterns that have to be backed up. I have defined one FileSet for each machine and backup grade combination. The filesets are compressed. Files matching some patterns are excluded (

  • .o, *.log, *.bak, *~). The FileSet below is for my desktop and for the A-Grade backup. Directories /home, /data and /photos will be taken into account in the backup.
# List of files to be backed up
FileSet {
  Name = "Zebulon A-Grade"     
  Include {                    
  Options {                  
      signature=SHA1           
      compression=GZIP         
      verify = pins1           
      onefs = yes              
      WildFile = "*~"          
      WildFile = "*.bak"       
      WildFile = "*.log"       
      WildFile = "*.o"         
      Exclude = yes            
    }
    File = /home
    File = /data
    File = /photos
  }
}

Other FileSets are defined for the same machine but for different files. They will be used for other backup grades.

Backup Schedule

The schedule defines when the backup has to be executed. Each backup grade has its own schedule. This allows to run B-Grade and C-Grade backups less frequently than A-Grade.

The A-Grade backups have a full backup schedule the first Saturday of each month. A full backup of the desktop takes arround 5 hours and uses 57Go (compressed). A differential backup takes arround 2 hours and uses 10Go (compressed). The incremental backup uses 2-4Go (compressed) and 5 to 15 minutes. (these numbers depend on what is being backed up). The schedule hours are defined according to this.

Schedule {
  Name = "Weekly-A-Grade"
  Run = Full 1st sat at 23:05
  Run = Differential 2nd-5th sun at 22:10
  Run = Incremental sun-fri at 22:10
}
Schedule {
  Name = "Weekly-B-Grade"
  Run = Full jan 1st sat at 23:05
  Run = Full mar 1st sat at 23:05
  Run = Full may 1st sat at 23:05
  Run = Full jul 1st sat at 23:05
  Run = Full sep 1st sat at 23:05
  Run = Full nov 1st sat at 23:05
  Run = Differential 2nd-5th sun at 22:10
  Run = Incremental wed at 22:10
}
Schedule {
  Name = "Weekly-C-Grade"
  Run = Full jan 1st sat at 2:05
  Run = Full mar 1st sat at 2:05
  Run = Full may 1st sat at 2:05
  Run = Full jul 1st sat at 2:05
  Run = Full sep 1st sat at 2:05
  Run = Full nov 1st sat at 2:05
  Run = Differential 2nd-5th sat at 2:10
  Run = Incremental sat at 2:10
}

Bacula Job

The Bacula Job describes what must be backed up (FileSets), when (Schedule) and where (Pools). There is one job definition for each fileset.

Job {
  Name = "Zebulon-A"
  Type = Backup
  Client = zebulon-fd
  FileSet = "Zebulon A-Grade"
  Schedule = "Weekly-A-Grade"
  Storage = File
  Messages = Standard
  Pool = Default
  Full Backup Pool = A-Full-Pool
  Incremental Backup Pool = Incr-Pool
  Differential Backup Pool = Diff-Pool
  Priority = 8
}

Some Statistics

After more than one year of backups, the total storage space used is now 599G, each tape is 4.3G. The storage space used by file pools is as follows:

A Grade Full Tapes  73  313Go
B Grade Full Tapes  28  120Go
C Grade Full Tapes   4   17Go
Differential tapes  22   94Go
Incremental tapes   13   55Go

The MySQL database has grown a lot and is quite large. The InnoDB database file only contains the bacula database and it has grown up to 2Go now. The filename table references 885527 records and the path table references 546784 rows.

Conclusion

Bacula is not easy to configure but when you do it right it provides a performant backup solution. To learn more about the configuration, have a look at Bacula Documentation. Installed on a ReadyNAS duo, it proved to be a robust solution for a backup of a small set of machines. You cannot expect big performances during backup or restore. The performance bottleneck is the MySQL database which runs on the ReadyNAS.

Restoring files from the backup is quite easy but this is another story...

To add a comment, you must be connected. Login to add a comment

Solving Linux system lock up when intensive disk I/O are performed

By Stephane Carrez

When a system lock up occurs, we often blame applications but when you look carefully you may see that despite your multi-core CPU, your applications are sleeping! No cpu activity! So what happens then? Check the I/Os, it could be the root cause!

With Ubuntu 10.04, my desktop computer was freezing when the ReadyNAS Bacula backup was running. Indeed, the Bacula daemon was performing intensive disk operations (on a fast SATA hard disk). The situation was such that it was impossible to use the system, the interface was freezing for a several seconds then working for a few seconds and freezing again.

Linux I/O Scheduler

The I/O scheduler is responsible for organizing the order in which disk operations are performed. Some algorithms allow to minimize the disk head moves, other algorithms tend to anticipate read operations,

When I/O operations are not scheduled correctly, an interactive application such as a desktop or a browser can be blocked until its I/O operations are scheduled and executed (the situation can be even worse for those applications that use the O_SYNC writing mode).

By default, the Linux kernel is configured to use the Completely Fair Queuing scheduler. This I/O scheduler does not provide any time guarantee but it gives in general good performances. Linux provides other I/O schedulers such as the Noop scheduler, the Anticipatory scheduler and the Deadline scheduler.

The deadline scheduler puts an execution time limit to requests to make sure the I/O operation is executed before an expiration time. Typically, a read operation will wait at most 500 ms. This is the I/O scheduler we need to avoid the system lock up.

Checking the I/O Scheduler

To check which I/O scheduler you are using, you can use the following command:

$ cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]

where sda is the device name of your hard disk (or try hda).

The result indicates the list of supported I/O scheduler as well as the current scheduler used (here the Completely Fair Queuing).

Changing the I/O Scheduler

To change the scheduler, you can echo the desired scheduler name to activate it (you must be root):

# echo deadline >  /sys/block/sda/queue/scheduler

To make sure the I/O scheduler is configured after each system startup, you can add the following lines to your /etc/rc.local startup script:

test -f /sys/block/sda/queue/scheduler &&
  echo deadline > /sys/block/sda/queue/scheduler

test -f /sys/block/sdb/queue/scheduler &&
   echo deadline > /sys/block/sdb/queue/scheduler

test -f /sys/block/hda/queue/scheduler &&
   echo deadline > /sys/block/hda/queue/scheduler

You may have to change the sda and sdb into hda and hdb if you have an IDE hard disk.

Conclusion

After changing the I/O scheduler to use the Deadline scheduler, the desktop was not freezing any more when backups are running.

To add a comment, you must be connected. Login to add a comment

Restoring a complete system after a hard disk failure: bacula to the rescue!!!

By Stephane Carrez

Last day the main disk of by computer stopped to work. My Western Digital 150Gb raptor hard disk was no longer recognized by the system: it was simply dead after one year of work. The 10000 rpm di

Read more