Backup OpenStack Service and Ceph Configurations

This guide will explain how to make backups of OpenStack and
Ceph configurations.

Background Information

Our Private Cloud Core deployments make use of Kolla Ansible and Ceph Ansible
to deploy the OpenStack services that comprise your cloud and the storage
backend, Ceph.

OpenStack service configurations are kept within MariaDB databases as well as
within the filesystem. An OpenStack service would be nova, cinder, or
glance, to name a few examples.

Backup OpenStack services using Kolla Ansible

Kolla Ansible allows you to back up the MariaDB databases associated with
OpenStack services by using MariaDB’s mariabackup function.

NOTE! – A good backup policy is to not store backups in the same location
where production data lives, but rather to store backups offsite.

For an authoritative source of information on Kolla Ansible and using
mariabackup see these documentation links:

The MariaDB database backup and restore guide goes into full detail on how to
enable backups, create them, and restore the backups.

Using Kolla Ansible

To start using Kolla Ansible, an environment needs to be created. This section
explains the steps needed to create that environment.

To get a quick idea of what is required, here is a high-level overview of the

# Copy Kolla Ansible configuration from FM-Deploy Docker container
$ docker cp fm-deploy:/opt/kolla-ansible /opt/kolla-ansible

# Navigate to /opt/kolla-ansible
$ cd /opt/kolla-ansible

# Initialize a Python virtual environment
$ virtualenv .venv

# Activate the virtual environment
$ source .venv/bin/activate

# Install kolla-ansible using requirements.txt
$ pip install -r requirements.txt

# Set the SSH private key so kolla-ansible can connect to each host
$ export EXTRA_OPTS="--private-key /root/.ssh/fm-deploy"

The above takes care of preparing the Kolla Ansible environment.

These steps are explained in more detail below.

Prepare and use Kolla Ansible

Follow these steps to learn how to prepare and use Kolla Ansible.

Relevant files

  • Globals file: /etc/kolla/globals.yml
  • Iventory file: /etc/fm-deploy/kolla-ansible-inventory

Note! — The above files may not be present on each host. Inspect each
host until you find the above, and perform the Kolla Ansible run from this


Step 1 – Prepare environment

From the Docker container called fm-deploy, copy /opt/kolla-ansible to
the local filesystem:

$ docker cp fm-deploy:/opt/kolla-ansible /opt/kolla-ansible

Step 2 — Prepare Python virtual environment and install requirements

Create a Python virtual environment from which Kolla Ansible will be used:

$ cd /opt/kolla-ansible
$ virtualenv .venv
$ source .venv/bin/activate
(.venv) $ pip install -r requirements.txt

Step 3 — Export the private key

In order for Kolla Ansible to make changes to each host, the SSH private key
path needs to be set as an environment variable:

$ export EXTRA_OPTS="--private-key /root/.ssh/fm-deploy"

Backup Kolla Ansible Files

There are important files related to Kolla Ansible that should be backed up.

These files are:

  • /etc/kolla/globals.yml
  • /etc/kolla/passwords.yml

Should they be lost, it will not be possible to manage the OpenStack cloud.

Create copies of these files and store them in an offsite backup location.

NOTE! — Due to the sensitive nature of these files, ensure the files are
transported using an encrypted connection and are stored in a secure, private

Create OpenStack Database Backups

With Kolla Ansible prepared, you can create a full backup of all OpenStack
service databases using mariabackup.

To create backups of the OpenStack databases, use:

$ kolla-ansible -i /etc/fm-deploy/kolla-ansible-inventory <subcommand>

Note that <subcommand> is a placeholder for the Kolla-ansible subcommand to run
and in this case is mariadb_backup.

The full command to create database backups is:

$ kolla-ansible -i /etc/fm-deploy/kolla-ansible-inventory mariadb_backup

This will create a Docker volume called mariadb_backup that can be used to
restore OpenStack databases.

The output of the Kolla Ansible run will let you know to what host the
backups were sent.

To see the Docker volume, SSH into the host where the backups were made, and

# docker volume ls | grep mariadb_backup
local               mariadb_backup

Restore OpenStack Database Backups

This procedure outlines how to perform a full database restore. An
incremental restoration can also be completed.

See MariaDB’s documentation
for information on how to use mariabackup to make incremental backups.

The previous Kolla Ansible run created a Docker volume called
mariadb_backup. This volume can be used to restore the OpenStack service

Step 1 — Create Docker container

To restore this backup, create a new Docker container using the previously
created volume:

# docker run --rm -it --volumes-from mariadb --name dbrestore 
--volume mariadb_backup:/backup kolla/centos-binary-mariadb:train-centos8 

Step 2 — Perform full backup of databases

Once in that container, these series of commands can be run to perform a full
backup of all OpenStack service databases:

$ cd /backup/
$ rm -rf /backup/restore/
$ mkdir -p /backup/restore/full
$ gunzip mysqlbackup-10-08-2020-1597091449.qp.xbc.xbs.gz
$ mbstream -x -C /backup/restore/full < mysqlbackup-10-08-2020-1597091449.qp.xbc.xbs
$ mariabackup --prepare --target-dir /backup/restore/full

Step 3 — Stop MariaDB

Once the above commands are successfully run, stop the MariaDB Docker
instance using:

# docker stop mariadb

Step 4 — Restore database backups

Navigate back into the container with the mariabackup volume mounted and either move
or remove the contents of /var/lib/mysql and then move the contents of
/backup/restore/full into /var/lib/mysql:

$ mkdir /backup/mariadb_original/
$ mv -v /var/lib/mysql/{*,.[^.]*} /backup/mariadb_original/
$ mv -v /backup/restore/full/{*,.[^.]*} /var/lib/mysql/

Step 5 — Start MariaDB

Finally, start the MariaDB Docker container:

# docker start mariadb

OpenStack Configuration Backups

In addition to backing up the MariaDB OpenStack databases, it is
imperative the OpenStack service files are backed up. Services include
nova, swift, and glance, to name a few.

All service configuration files are stored on the hardware nodes within the
folder /etc/kolla/.

To backup the service’s configuration files, copy /etc/kolla to an offsite
backup location.

Ceph Configuration Backups

The configuration files for Ceph are stored on each hardware node within the
folder /etc/ceph. Backing up this folder is sufficient for backing up
Ceph’s configuration.

To backup Ceph’s configuration, copy /etc/ceph to an offsite backup

Nick West Systems Engineer

Nick is an avid aggressive inline skater, nature enthusiast, and loves working with open source software in a Linux environment.

More Articles by Nick

Was this article helpful? Let us know!