Road to Continuous Delivery Part 2


In Part 2 of the series we'll focus on setting up a working backup of our devel-stack we set up in Part 1.


No server is complete without a backup. For backups my tools of choice are borg and rsync. Borg is a deduplication backup program, which supports compression and authenticated encryption.

To get started I created a bash script, which can be found here. Currently the script creates a local borg repo and pushes the borg repo to a off-site server. This is not optimal but we'll change the script later to create borg remote repositories. The script also generates a prom file with metrics, so we'll be able to monitor the backup process with prometheus later.

The script covers the following:

  • creates a new gitlab backup
  • rsync /data/docker to /data/backup/temp/docker
  • stopping gitlab & nexus
  • borg backup
  • starting gitlab & nexus
  • rsync to remote location
# upload script and exclude file
scp -r -P 2605"/data/config/scripts/"  
scp -r -P 2605 exclude.txt"/data/config/scripts/exclude.txt"

# login to server
ssh -p 2605

chmod 0600 /data/config/scripts/

# execute backup script for the first time

# add backup script to run every day at 2 o'clock 
echo "0 2 * * *   root  /data/config/scripts/ > /data/logs/cron/backup" > /etc/cron.d/backup  
echo "" >> /etc/cron.d/backup  


After we ensured that the backup works (checking remote location) it's time to test the backup:

# vars
BACKUP_RSYNC_SSH="ssh -p 2605 -T -o Compression=no -x -c aes128-ctr -i $BACKUP_SAVE/config/ssh/"  

# let's stop and remove all containers and images
docker stop $(docker ps -a -q)  
docker rm $(docker ps -a -q)

service docker stop  
mkdir $BACKUP_SAVE  
mv /data/* $BACKUP_SAVE

# fetch backup from remote location
mkdir $BACKUP_DIR -p  
rsync -aHAXxv --numeric-ids --delete --progress -e "$BACKUP_RSYNC_SSH" -r -P $BACKUP_RSYNC_REMOTE/borg $BACKUP_DIR

# create basic folder structure
mkdir -p /data/backup       > /dev/null 2>&1  
mkdir -p /data/logs/cron    > /dev/null 2>&1  
mkdir -p $RESTORE_MOUNT     > /dev/null 2>&1

# get last reponame 
export BORG_REPO=$BACKUP_DIR/borg/repo  
REPO_NAME=$(borg list --short | tail -1)

# mount borg repo
echo "mounting $REPO_NAME"  

# check contents of mounted borg archive

# mv data back
rsync -av --delete --progress -r -P $RESTORE_MOUNT/data/config /data  
rsync -av --delete --progress -r -P $RESTORE_MOUNT/data/www /data  
rsync -av --delete --progress -r -P $RESTORE_MOUNT/data/backup/temp/docker /data

# umount
borg umount $RESTORE_MOUNT

# restart our services
service docker start  
cd /data/config/nginx  && \  
    docker-compose pull && \
    docker-compose up -d && \
    cd /data/config/gitlab  && \
    docker-compose pull && \
    docker-compose up -d && \
    cd /data/config/gitlab-runner && \
    docker-compose pull && \
    docker-compose up -d && \
    cd /data/config/nexus && \
    docker-compose pull && \
    docker-compose up -d

If everything worked our services should be up and running in a few minutes.

Files are available on github and gitlab: