Database rollbacks: The DevOps approach to rolling back and fixing forward

In the course of database administration and change management, there are times when a change is deployed that could or does break the application, push a feature before it’s ready, degrade data integrity, run afoul of compliance regulations, or simply not fulfill expectations. 

In those times, the team can be up in arms because whatever error just occurred could bring down the business. Or they can be calm, cool, and collected because they’re easily able to perform a database rollback. 

Of course, the situation is quite a bit more nuanced than that, as you’ll learn below. When problems arise in database changes that have already been deployed, undoing that change is referred to as “rolling back” the change, hence the concept of a “database rollback.” 

Here’s what a database rollback is, why it’s used, the risks involved, and how automation can simplify the process. 

What is a database rollback?

A database rollback is, to some degree, like an “undo” button for database changes. To perform a database rollback, the database is reverted to a previous state, eliminating the problematic change and the issues it caused for the infrastructure, application, data, and end users. A database rollback might completely undo a change or roll it back to earlier stages in the pipeline to be revised and re-tested. A database rollback requires precision and collaboration to pull off successfully – effectively erasing the impact as if it had never been made. 

In database change management, rollbacks provide a controlled method for reversing changes without upsetting the stability or availability of the database. They’re most often deployed in response to data correction, testing outcomes, patches, or vulnerabilities. 

Done right, a database rollback can protect data integrity and consistency after unintentional changes, human error, faulty integrations, broken code, or other issues. This action is crucial to maintain data integrity and correct errors after they have been introduced to the live environment.

It’s not as easy as rolling back application code

When changes need to be rolled back in application code, it’s not easy per se, but it is pretty straightforward – applications are stateless. Their operations don’t inherently change the underlying data within their environment. If a new version of an application introduces a bug or an issue, you can often redeploy the previous version of the code, since version control is built into the application development process.  

Databases have state, so when you make changes to a database, the underlying structure and content itself changes. If you have to roll back changes in a database, you can't just "redeploy" the old database code because the actual data might have changed since that code was last deployed. A rollback in a database must be careful to undo the changes to the schema or data entries themselves without losing the new data that has been added since the problematic change was deployed.

Roll back vs fix forward

There is a remediation approach similar to how application code is treated. By “fixing forward,” the goal is not to revert to a previous working state but to deploy changes that get to a properly functioning state, inclusive of any data collected, removed, or changed in the meantime. DBAs then focus on writing code that’s easier to fix in the future.

The fix-forward method works best when database changes are broken into small chunks that are deployed independently and automatically. If you’re starting from a software development environment where you have one year’s worth of work about to deploy, this may not be the approach you adopt right now. However, there are tools that can help you break up your database scripts and schema changes into small, trackable chunks that make this approach much more accessible to companies that are ready to try this out.

Implementing a "fix-forward" strategy effectively requires a blend of technical capability and organizational culture. Teams need robust monitoring and alerting systems to detect issues swiftly, plus a team skilled in quick, impactful fixes and strong collaboration across development, database, and operations teams. Automated testing is vital to validate fixes without introducing new issues, and meticulous versioning and documentation are crucial for tracking changes and enabling easy troubleshooting.

Teams also need a risk assessment strategy that helps determine the feasibility and impact of immediate fixes versus rollbacks, including a fallback plan for unexpected complications. To collaborate productively, a cultural shift towards embracing failures as opportunities for improvement prioritizes learning and process enhancement. These factors underscore the complexity of choosing between fixing forward and rolling back, highlighting the importance of a balanced, informed, and automated approach to managing database changes and system updates.

Database change automation enables a viable fix-forward approach since it deploys database changes as code in frequent, small updates in line with DevOps principles. With more granular change deployments, the forward fix can focus on a smaller, more manageable bug than deploying weeks' or months’ worth of database changes at once. 

While fixing forward is the right solution in many cases, it’s not a cure-all. Teams still need to be able to actually revert the database to a previous state in a manageable, efficient way. 

Why would you need to run a database rollback?

Changes to databases, even after going through rigorous testing stages, still have the potential to break the application – or if not break, cause an error to the software, data, or user experience that requires it be undone. Sometimes, databases experience unintentional drift, in which the actual database state does not match what it’s supposed to be. Other times, the volume of change requests coming through leads to overlapping issues – database change that didn’t cause trouble on its own now does, once combined with other changes in the pipeline. 

It might be an immediate discovery, if an error is detected during the change management process itself, or after the fact if anissue is discovered later and can be traced back to a specific change. Sometimes, no one knows there’s a problem with the database until a customer or user starts having a bad experience or encounters bad data. 

Situations requiring a database rollback can include:

• Human error, such as a developer accidentally dropping a critical table or inserting incorrect data 
• Software bugs, such as those in newly developed applications that elude testing but corrupt data and degrade user experiences
• Failed deployments, such as during a feature release or system update, in which the deployment fails at production and must be sent back to testing and development
• Data corruption, such as when improper data collection occurred or external forces lead to degradation of data integrity, and reverting the database helps contain the issue
• Repeated testing, such as when a change is known to need multiple rounds of testing before production, and the team wants to avoid a manual reset
• Compliance violations, such as when improper access is granted in the midst of deploying a database change
• Security breaches, such as when vulnerabilities must be patched or corrupted data must be removed
• Performance issues, such as when a database change leads to an unintended impact on query speed and application connections
• Dependency issues, such as when a change ends up breaking a downstream element of the pipeline, and they both need to be updated before redeployment

Automation can greatly simplify rollbacks, minimizing the risks and downtime associated with manual interventions. Rollback automation can be enabled with database change management automation, which treats database change as code and includes database version control, just as an application team would use versioning in their own development workflow. As database change scripts are committed to version control, the automation tool can automatically generate a rollback script for the change so it goes live with the built-in ability to be easily rolled back. This proactive, DevOps-aligned approach to rollbacks ensures database resiliency and reliability.

Automated rollbacks, especially when paired with targeted rollback capabilities, ensure that only the erroneous changes are discarded, leaving all subsequent valid changes intact and operational. 

Automating database rollbacks

In the foundational sense of the term, a database rollback will undo all the changes back to the problematic deployment. This reverts the database to a state prior to the problematic change, yet also without any subsequent changes applied. Changes between the problematic change and the current state will need to be redeployed. Once the problematic change is rectified, it, too, can be redeployed. All or most of this happening manually means the database team burns a lot of time undoing progress instead of processing new changes or innovating the database pipeline. 

With Liquibase Open Source, database change management automation includes the ability to Rollback any change deployed through Liquibase easily. Just as Liquibase makes it quick and streamlined to push database changes through an automated CI/CD pipeline, it can roll those changes back with a simple command. That’s because it was packaged with a rollback script after is was committed, whether that rollback would ever be used or not.  

For teams that need more precision and control, Liquibase Pro can target specific changes and roll back just that one, avoiding the redeployment stage. 

Targeted database rollbacks

Targeted Rollbacks elevate the level of granularity and precision teams have when executing database rollbacks. This advanced capability in Liquibase Pro allows users to cherry-pick a specific database change and pluck it out of the database’s current state, without upsetting any changes that have happened since. Plus, rollback testing ensures confidence in the remediation plans and the amended database so teams can move on quickly. Targeted Rollbacks can be combined with Quality Checks to build in development rules for automatic rollback validation, too. 

Targeted Rollbacks can also be set up to trigger in the event of a change deployment failure. Liquibase Pro’s Rollback on Error capability simply removes the change from the database state and kicks it back into the testing and development workflow 

With less powerful database change management solutions, a standard, tedious, manual rollback might be the only option. That means undoing hours, days, even weeks (or more!) worth of changes just to undo one faulty chunk of change code. 

With Automated Rollback capability, teams can undo problematic changes in an instant. With Targeted Rollbacks, that instant undoing is more of a cherry-picking, plucking out the problematic change without affecting any prior or subsequent changes. 

Automating database rollbacks brings teams the speed and agility they need at the database change workflow. Alongside complete database change management automation, it empowers database DevOps and completes the final stretch of organizations’ CI/CD pipelines.

Learn more about Targeted Rollbacks with Liquibase’s change management automation solution. Find out how they fit into a modern database DevOps workflow that empowers speed and agility.