Tracking changes in a server configuration can be critical to understand problems, identify security breaches and repair a server. When several people are in charge of administering one or several servers, sharing the configuration changes is helpful to i

The overall idea is to put the server configuration files stored in /etc directory under a version control system: subversion. The VCS is configured to send an email to the system administrators. The email contains the differences with a previous version. A cron script is executed every day to automatically commit the changes, thus triggering the email.

The best practice is of course that each system administrator commits their changes after they validated the new running configuration. If they do so, they are able to specify a comment which is helpful to understand what was done.

Install subversion

First, you should install subversion with its tools.

 sudo apt-get install -y subversion subversion-tools

Mail notification

For the mail notification, you may use postfix, exim or sendmail. But to avoid to setup a complete mail system, you may just use a simple mail client. For this, you can use the combination of esmtp and procmail.

 sudo apt-get install -y procmail esmtp

Create the subversion repository

The subversion repository will contain all the version and history of your /etc. It must be protected carefully because it contains sensitive information.

 sudo mkdir /home/svn
 sudo svnadmin create /home/svn/repos
 sudo chmod 700 /home/svn
 sudo chmod 700 /home/svn/repos

Now, setup the subversion repository to send an email for each commit. For this, copy or rename the post-commit.tmpl file and edit it to specify to whom you want the email to be sent:

 sudo cp /home/svn/repos/hooks/post-commit.tmpl  \
           /home/svn/repos/hooks/post-commit

and change the last line to something like (with your email address)

 /usr/share/subversion/hook-scripts/commit-email.pl \
  --from yoda+mercure@alliance.com \
  "$REPOS" "$REV" yoda@alliance.com

Initial import

To initialize the repository, we can use the svn import command:

 sudo svn import -m 'Initial import of /etc' \
               /etc file:///home/svn/repos/etc

Subversion repository setup in /etc

Now the hard stuff is to turn /etc into a subversion environment without breaking the server. For this, we extract the subversion /etc repository somewhere and copy only the subversion files in /etc.

 sudo mkdir /home/svn/last
 sudo sh -c "cd /home/svn/last && svn co file:///home/svn/repos/etc"
 sudo sh -c "cd /home/svn/last/etc && tar cf - `find . -name .svn` | (cd /etc && tar xvf -)"

At this step, everything is ready. You can go in /etc directory and use all the subversion commands. Example:

 sudo svn log /etc/hosts

to see the changes in the hosts file.

Auto-commit and detection of changes

The goal now is to detect every day the changes that were made and send a mail with the changes to the supervisor. For this, you create a cron script that you put in /etc/cron.daily. The script will be executed every day at 6:25am. It will commit the changes that were made and send an email for the new files.

 #!/bin/sh
 SVN_ETC=/etc
 HOST=`hostname`
 # Commit those changes
 cd $SVN_ETC && svn commit -m "Saving changes in /etc on $HOST"
 # Email address to which changes are sent
 EMAIL_TO="TO_EMAIL"
 STATUS=`cd $SVN_ETC && svn status`
 if test "T$STATUS" != "T"; then
   (echo "Subject: New files in /etc on $HOST";
    echo "To: $EMAIL_TO";
    echo "The following files are new and should be checked in:";
    echo "$STATUS") | sendmail -f'FROM_EMAIL' $EMAIL_TO
 fi

In this script you will replace TO_EMAIL and FROM_EMAIL by real email addresses.

Complete setup script

To help setup and configure all this easily, I'm now using a script that configures everything. You can download it: mk-etc-repository. The usage of the script is really simple, you just need to specify the email address for the notification:

 sudo sh mk-etc-repository sysadmin@mycompany.com