A few words about Continuous Deployment
Continuous Deployment is the deployment or release of code to Production as soon as it is ready. (...) The automated process is key because it should be able to be performed by anyone in a matter of minutes (preferably by the press of a button).
Once you have moved to a Continuous Deployment process, you will have to have several pieces of automation in place. You must automate your Continuous Integration Build Server and Continuous Delivery to Staging, as well as have the ability to automatically deploy to Production.
Read full blog post
...we're on the way
Because every manual step implies a risk for failure, our goal is to minimize such risks as well as our amount of work by automating our processes of software delivery. In several of our applications we already use a deployment script for automatic deployment of Tomcat applications.
For some applications we even use a continuous deployment script triggered by crontab (e.g. every hour) to check if Nexus has a new version of the application and if so to fetch and deploy it automatically.
However the backup process was still manual - although you always perform the same steps. This is a perfect opportunity for automation. And a perfect opportunity for me to leave the Java world for a while and to learn more about shell scripting.
The typical backup steps before a deployment are:
- saving war file resp. information about the current deployed version
- saving database data in a dump
- saving directories and/or files, e.g. generated error reports
This blog post explains the configuration and the most important steps of the backup script. You can find the entire backup script and the configuration files on Github.
Configuration for mysqldump
mysqldump client is the usual command line tool to dump a database for backup or for transferring the database data. The created dump contains SQL statements to create and/or to populate table(s). To be able to execute
mysqldump in an automatic process with nobody around to enter the password, you have to provide the access data in a file. Create a
.ini-style file named
.my.cnf providing user and password information. Later in the script,
mysqldump will read this configuration and will automatically use the supplied user name and password.
If you have only one database to be dumped, following lines are enough:
[mysqldump] user = myUser password = xxx
If you have more than one database to be dumped needing different access data, you have to specify an access data section for every database.
For example, if you want the databases "foo" and "bar" to be dumped, you'll need two sections with the corresponding database name as suffix:
[mysqldumpfoo] user = fooUser password = xxx [mysqldumpbar] user = barUser password = xxx
Caution: In this case there must not be a
[mysqldump] section since the more specific sections with database name as suffix would be ignored.
If you want to prevent other users from reading your
.my.cnf, change the file permission so that only the owner has full read and write permissions.
chmod 600 ~/.my.cnf
Configuration for Backup script
The backup script will depend on a configuration file containing details for the backup process:
- the absolute path of the deployed webapp
- the absolute path of the parent directory where the backups should be stored
- the names of the databases to be dumped
- the absolute path of the mysqldump config file with access data
- if the deployed war file itself or if only the information about the current deployed version should be saved
- if there are files and/or directories that should be moved or copied to the backup directory
# absolute path of the deployed webapp WEBAPP_PATH="$HOME/tomcat/webapps/ROOT" # absolute path where the backups should be stored BACKUP_PATH="$HOME/backup" # names of the databases to be dumped # separate with whitespace DB_NAME="db1 db2 db3" # absolute path of the mysql config file with user data # for the above specified databases MYSQL_CONF="$HOME/.my.cnf" # decide if current deployed war should be saved in backup dir # or if only the information about the version that is deployed # should be saved # # 0: save war only if the current deployed version is a snapshot # 1: always save war # # if you specify nothing or something that is not 0 or 1, # current deployed war file won't be saved WAR=1 # specifiy folders and/or files (absolute path!) # that should be copied resp. moved into backup dir # separate with whitespace CONTENT_TO_BE_COPIED="$HOME/file $HOME/folder" CONTENT_TO_BE_MOVED="$HOME/folder/file $HOME/folder/*"
The backup script
Now let's have a look at the actual backup script and the most important steps of it.
Get configuration file
You can call the script with the option -c to specify which configuration file should be used.
./backup.sh -c config/custom.conf
If you call the script without the -c option, the script tries to get a file named
backup.conf in the directory the script is located.
# default config file CONF_FILE="$(dirname "$0")/backup.conf" # a colon after the option means that the option # expects an argument while getopts "hc:" OPTION; do case "$OPTION" in c) CONF_FILE=$OPTARG ;; esac done
Read and validate configuration file
The script makes sure that the configuration file exists and contains all the required parameters. Otherwise it will fail with an appropriate error message.
# variable is empty if [ -z "$CONF_FILE" ]; then echo "CONF_FILE not set" exit 1 fi # no file found for the given variable if [ ! -f "$CONF_FILE" ]; then echo "No configuration file found" exit 1 fi # read in variables of conf file . "$CONF_FILE" # make sure that conf file contains all required variables if [ -z "$WEBAPP_PATH" ] || [ -z "$BACKUP_PATH" ] || [ -z "$DB_NAME" ] || [ -z "$MYSQL_CONF" ]; then echo "Configuration file seems to be incorrect: required variables missing. Please check your config file: $(readlink -f "$CONF_FILE")" exit 1 fi # make sure that WEBAPP_PATH exists if [ ! -d "$WEBAPP_PATH" ]; then echo "The given webapp path doesn't exist. Please check your config file: $(readlink -f "$CONF_FILE")" exit 1 fi # make sure that mysql config file exists if [ ! -f "$MYSQL_CONF" ]; then echo "The given mysql config file doesn't exist. Please check your config file: $(readlink -f "$CONF_FILE")" exit 1 fi
Creating the backup directories
The script checks if the specified parent backup directory already exists - and will create it if it doesn't. For every processed backup a time stamped folder (with the pattern yyyy-MM-dd_hh-mm) is created in this directory containing the backup data. Allowed is only one backup per minute. If you try to run the script and there is already a backup folder for the current minute, the script will fail with an error message.
CURRENT_DATE="$(date +%Y-%m-%d_%H-%M)" FULL_BACKUP_PATH="$BACKUP_PATH/$CURRENT_DATE" # check if backup dir exists, if not create it if [ ! -d "$BACKUP_PATH" ]; then mkdir "$BACKUP_PATH" echo "Create parent backup directory $(readlink -f "$BACKUP_PATH")" fi # check if there is already a dir for current date, if not create it if [ ! -d "$FULL_BACKUP_PATH" ]; then mkdir "$FULL_BACKUP_PATH" echo "Create backup directory $(readlink -f "$FULL_BACKUP_PATH")" else echo "$(readlink -f "$FULL_BACKUP_PATH") already exists" echo "Wait until $(readlink -f "$BACKUP_PATH/$(date -d '+1min' +%Y-%m-%d_%H-%M)") can be created" exit 1 fi
Save the information about the current deployed version
You find the information about the current deployed version in the
It looks like this:
#Generated by Maven #Sat Mar 30 02:03:52 CET 2013 version=1.1.10-SNAPSHOT groupId=org.synyx artifactId=foo
This .properties file contains all the relevant information we need if we'd like to rollback to the backup state. The Nexus deploy script takes the group id, artifact id and version as parameters:
./nexusdeploy.sh -i org.synyx:foo:1.1.10-SNAPSHOT -w ROOT
So this properties file is copied to the backup directory.
PROPS_NAME="pom.properties" APP_PROPS="$(find "$WEBAPP_PATH" -name "$PROPS_NAME")" # more than one file matching the criterias found if [ ! $(find "$WEBAPP_PATH" -name "$PROPS_NAME" | wc -l) -eq 1 ]; then echo "No or more than one $PROPS_NAME was found under the webapp path $(readlink -f "$WEBAPP_PATH")" exit 1 fi # copy the information about the current deployed version cp "$APP_PROPS" "$FULL_BACKUP_PATH"
Save the war file itself if configured
We remember, in the configuration file there are three possibilities what to do with the war file during the backup process:
- always save the war file
- never save the war file
- save the war file only if the current deployed version is a snapshot
The first two states are self-explanatory: either the war file is copied to the backup directory or not. If the war file should be only saved if it is a snapshot, the
pom.properties file has to be read in and the variable
$version has to be checked if it contains the string 'SNAPSHOT'.
# variable isn't empty if [ ! -z "$WAR" ]; then SAVE_WAR=0 # decide if war should be saved # if conf = save war only if current deployed version is a snapshot if [ "$WAR" -eq 0 ]; then # check if current deployed version is a snapshot # read in properties . "$APP_PROPS" # $version has information about current deployed version case "$version" in *SNAPSHOT) SAVE_WAR=1 ;; esac # if conf = save war always elif [ "$WAR" -eq 1 ]; then SAVE_WAR=1 fi if [ "$SAVE_WAR" -eq 1 ]; then echo "Backup war file: " cp -v "$WEBAPP_PATH"/../*.war "$FULL_BACKUP_PATH" fi fi
Save the database data
For every element in
$DB_NAME a dump is created.
Due to the existing
.my.cnf with access data information,
mysqldump can be called without the parameters
password. However it needs the following information:
- the path to the config file with the access data (
- the suffix (database name) which access data section of the config file should be used (
- the database name to know which database should be dumped
If something goes wrong during database dump (e.g. access is denied), the error is redirected to
/dev/null because the script has its own error handling. It checks if
$? is 0 or not.
$? gives the exit status of the last executed command. A status not 0 is an error code, i.e. the last command wasn't successful.
for db in $DB_NAME do # create dump and hide mysqldump errors due to own error handling mysqldump --defaults-file="$MYSQL_CONF" --defaults-group-suffix="$db" "$db" > "$FULL_BACKUP_PATH/$db-$CURRENT_DATE.dump.sql" 2>/dev/null # creating dump was successful, so return value is 0 if [ "$?" -eq 0 ]; then echo "Create dump for database $db: $(readlink -f "$FULL_BACKUP_PATH/$db-$CURRENT_DATE.dump.sql")" # something went wrong while trying to create dump (e.g. access denied), so return value is not 0 but any other number else echo "Problems encountered while trying to create dump for database $db" fi done
Save specified files and/or directories
To move the specified files and/or directories to the backup directory, following steps are performed:
- make sure that the variable
- execute the
moveoperation for each element
- if an element doesn't exist, print an error message
# make sure that variable $CONTENT_TO_BE_MOVED isn't empty if [ ! -z "$CONTENT_TO_BE_MOVED" ]; then for i in $CONTENT_TO_BE_MOVED do # if the given content is a dir or a file, process else throw an error if [ -d "$i" ] || [ -f "$i" ]; then echo "Move $(readlink -f "$i") into backup dir" mv "$i" "$FULL_BACKUP_PATH" else echo "Can't move $i: doesn't exist"; fi done fi
These steps are similar for the files/directories to be copied, only the action (
copy) and the parameter (
$CONTENT_TO_BE_COPIED) are different.
Don't forget to make your script executable
chmod +x backup.sh
Now it's no longer required to do the backup manually. And even better: if this script is integrated into our existing automatic deployment script, every time a deployment is performed, a backup will be performed first.