Observability for database CI/CD: Visibility into the pipeline

This is one section of our complete 12-part guide to CI/CD for databases - get your copy.

Collaboration is central to DevOps practices such as CI/CD. Collaboration is so important within DevOps that it is regarded as one of the core ‘pillars’ of DevOps as embodied by the CALMS acronym (Collaboration, Automation, Lean, Measurement, Sharing). The goal is to achieve a common understanding of a problem and a common framework for sharing solutions while efficiently delivering valuable changes through the pipeline to Production.

Maintaining awareness of changes across the team or even the larger organization it serves can be difficult without considering how to monitor change operations. Further, an important part of that awareness is associated with maintaining an analytics structure around the pipeline to measure flow, track changes, and identify bottlenecks as discussed in the previous section. Both aspects – monitoring change operations and providing pipeline analytics –  should be built into the automation layer of the CI/CD pipeline.

Database observability insights provide the feedback teams need to improve CI/CD pipeline performance, consistency, efficiency, and agility as databases change over time. Observability can also:

• Improve security and compliance
• Enable trend analysis
• Reduce remediation time
• Anticipate issues before they occur
• Identify drifts in database state 

Transparent workflows

Some team cultures can chafe at the idea of constant monitoring of change operations and reporting on pipeline analytics. It can be perceived as intrusive. However, without it, the overall organization cannot achieve a critical mass of information about its systems and no one will have a holistic view of how things are actually working. This will inevitably lead to inefficiencies, problems, and frustration. So, it is important to focus on the fundamental importance of the practices and how situational awareness will make everyone’s life easier by reducing the amount of uncertainty while doing the work.

Consider this truth: Even the smartest people will make the wrong decisions if they have incomplete information. It will be the absolutely correct and smartest decision as far as they know, of course. But, the key part of the statement is ‘as far as they know’. 

Actionable visibility

Transparency makes everything visible, but the goal of enabling observability into processes is to give stakeholders the ability to take action. This means that beyond just seeing things in an environment, they have to be able to put those things in an understandable context and understand their implications on future outcomes.

In database CI/CD terms, this is all about getting insight into the following:

• Identifying the database changes in the pipeline
• Understanding the current status of those changes
• Understanding which activities have happened within the pipeline

Identifying the changes

Before we discuss what information we want to know about a change within a pipeline, we first have to address the basic challenge of identifying individual changes so that we can track them. That implies both the technical specification of the change in question as well as the metadata describing how and why the change came to exist in the first place.

In order to do that, we need to be able to determine and present enough information about each change at all times. As a simple rubric, we should always be able to answer some basic questions:

• Where did the change come from?
• What batch is the change a part of?
• What feature is the change associated with? 
• When was the change created?
• Who created the change?

Understanding change status

Next, we have to establish the status of changes within the pipeline. This is fundamental to tracking the changes as they progress. For each change, we should know its status relative to each environment in the pipeline. 

That means that for each and every environment, we must be able to determine and list:

• Which changes have been applied to the environment?
• Which changes, if any, failed when attempting to apply them to the environment?some text
  • If any did fail, what were the errors?
• What changes are currently undergoing a validation cycle in the environment?
• What were the outcomes of each validation cycle for each change in the environment?

Pipeline activity

Finally, we have to be able to display overall information about how the pipeline and its changes have reached their current situation. 

This means that we have to summarize all of the events that have happened to both the changes and the environments across the whole pipeline. For this, we need to know the following:

• When was each change applied to each environment?
• What were the failure events when changes were applied to each environment?
• Who applied each of those changes in each environment?
• How were the changes applied in each environment?
• When was each environment refresh event?
• Which changes required more than one attempt to get them successfully applied to any environment? 
• When did those failures occur?

Observability is at the heart of effective collaboration regardless of the process, but it is central to DevOps practices such as CI/CD. It does not happen naturally and requires deliberate effort when the process is being designed. Done correctly, transparent processes provide teams good situational awareness and empower people to make better decisions and drive toward goals more efficiently and effectively.

Observability also extends to the performance of the database change workflow itself. With automation, tracking, governance, and observability enacted throughout, we can measure database CI/CD pipeline performance to identify problems and empower continuous workflow optimization. 

Ready to dive into all 12 parts of the complete CI/CD for databases guide? It covers:

1. The goals & challenges of designing CI/CD pipelines
2. Batching database changes
3. Automating database “builds”
4. Ensuring quality database changes
5. Reliable and predictable database deployments
6. Trustworthy database environments
7. Sandbox databases
8. Database governance for consistency, quality, and security
9. Observability for database CI/CD
10. Measuring database CI/CD pipeline performance
11. Handling and avoiding problems with database changes
12. Bring CI/CD to the database with Liquibase

‍