Introduction to CI/CD for databases

Introduction

In this guide, we outline the steps required to bring databases into a Continuous Integration/Continuous Delivery (CI/CD) pipeline. The information here is not about simply wrapping a bunch of scripts and running it from a “CI/CD” automation tool, but rather about truly leveraging the practices of CI/CD to significantly improve the flow of database changes through your delivery pipeline with all of the quality and control that are intrinsic to a well-architected modern pipeline.

CI/CD primer

The pattern for setting up a CI/CD pipeline is well established. Continuous Integration (CI) as a named engineering practice is about 20 years old and Continuous Delivery (CD) is solidly over 10. The combination of the two as a unified topic has been a central driving practice of the DevOps movement over the past decade. However, that conversation has been focused primarily on flowing application code and, to a lesser extent, infrastructure definitions through pipelines to rapidly deliver new functionality to application systems. Conspicuously absent from most of the conversations are the changes to the databases that underlie the applications.

Why is the database left out?

There are a lot of theories for why databases got left out of the CI/CD discussion. Most of them revolve around the notion that databases are stateful and, by nature, persistent while CI/CD is focused on the more ephemeral components of the application stack and is therefore not appropriate for, or even applicable to, databases. 

The problem with this perspective is that there is nothing in the underlying principles of CI/CD that are not applicable to database change management.

CI/CD for databases

As a mature set of engineering practices, CI/CD has a well defined and understood set of core principles. These principles are all just as applicable to database changes as they are to application changes. 

• “Shift Left” to build quality in
• Work with small, atomic changes
• Drive out the toil of repetitive tasks
• Measure and adjust to continuously improve
• The whole team owns the outcome

To be fair, the implementation of CI/CD for database changes does look very different and it can be hard to find clear guidance on how to overcome those differences. The purpose of this guide is to provide the missing knowledge and guidance to help teams begin solving these problems for their situations. The approach will be to break down a typical pipeline into primary phases, move systematically from “left” (change creation) to “right” (production), and systematically address the main challenges you will encounter.

Ready to dive into all 12 parts of the complete CI/CD for databases guide? It covers:

1. The goals & challenges of designing CI/CD pipelines
2. Batching database changes
3. Automating database “builds”
4. Ensuring quality database changes
5. Reliable and predictable database deployments
6. Trustworthy database environments
7. Sandbox databases
8. Database governance for consistency, quality, and security
9. Observability for database CI/CD
10. Measuring database CI/CD pipeline performance
11. Handling and avoiding problems with database changes
12. Bring CI/CD to the database with Liquibase