New Test Automation Framework for Liquibase

Our vision at Liquibase is to be the easiest, safest, and most powerful community-led database change management solution. In order to make this a reality for a large community that uses dozens of database platforms (in the cloud and on-prem) with over 50 different types of changesets, you need a fast, scalable, and extensible test automation framework.

Introducing our new test automation framework

I’m excited to announce that we have released the Liquibase Test Harness. This new framework benefits the entire Liquibase community in a few different ways:

• It’s now much easier to validate support for all the change types. This means that we can communicate in detail with users about which versions and change types work for each database.
• More members of the Liquibase community can add tests when they come across issues in the open source code.
• Liquibase extension creators can use a supplied framework and set of test cases to validate against when they extend core Liquibase functionality.

Finally, it’s simple for everyone to use. I really wanted to lower the barrier to entry so that everyone in the community can participate. This new framework is designed to be a harness style of tests—meaning the test classes themselves do not need much alteration.

How it works

The harness ingests input files, the test class invokes Liquibase methods, generates outputs, and the test compares the generated result to the expected result. If the results match, then the test passes. In addition, since you don’t need to create new test classes or modify existing tests, users do not need to know much Groovy or Java. The tests are written in Spock, which is a Behaviour-Driven Development (BDD) style of writing tests.

How we got here

Before we created this new test harness, we had three different automated test suites that validated database support via unit, integration, and functional level tests. The frameworks were older and not very performant, especially when it came to the integration tests. It underscored the fact that the integration test suite was where we really needed to make vast improvements so that we could test at scale as a leading provider of database migration software.

As I got started, I looked around at a few other OSS projects to see if I could get some inspiration. I was surprised to find that the frameworks weren’t as “simple” as I thought they would be. It could take hours to get acquainted with what I needed to do before ever writing my first test. This is exactly what we want to avoid with our new framework. I knew that our new framework needed to be easy to use with as little ramp up time as possible so that the whole Liquibase community can submit test cases for issues or extensions.

Keeping it simple

With tests especially, there is sometimes a desire to make them more complicated and intricate than they need to be. Sometimes a complex test suite is warranted. However, if you want to build something for a community where you may not really understand all your audience’s needs, it is best to start simple. I kept the KISS principle in mind while coming up with the test structure, the stack, and even the first test.

Building the new test automation framework

Building this new test framework was simultaneously easy and complicated. The first test class was purely about change object support and it was fairly simple to run that test class with varied change type inputs against different database platforms and different versions (in the cloud and on-prem).

The hard part was ensuring this new harness validates functionality for Liquibase extensions (such as Snowflake, SAP-HANA, MongoDB, Hibernate and many more). Writing a harness-style test for Liquibase extensions like Hibernate is very different because it’s not just another database, it’s an ORM. We’re continuing to build out this harness so that it allows the whole community to validate both native Liquibase functionality and also extended Liquibase functionality.

None of this framework would exist without the help of one of our developers, Oleh K. He helped put the pieces together and went through multiple iterations to even get the first set of tests working correctly. Every time there were changes or improvements I needed him to make, he quickly found solutions and even improved upon initial suggestions.

I also had the steady support of our CTO, Robert Reeves. Right from the start when I would talk about the need to elevate the level and quality of our community tests, Robert was staunchly in favor. When he experienced the annoyance of not having a test framework that he could use himself, he went from championing the cause, to being an early user, to adding test cases himself.

More to come

The test harness is still early in its evolution. The first test class focuses on change type support — the inner workings of the liquibase update operation. Here are some iterations that will be released over the next several months:

• Subsequent test classes will include snapshot generation validation as it relates to the liquibase generateChangelog operation

• Capability for the test harness to execute tests against Liquibase extensions and not just native Liquibase core capabilities. We are currently working on a test class that allows us to validate support for a Liquibase extension.

• More advanced reporting available through the framework

I will also be looking to see what kind of feedback we get from the community to help drive further development ideas. Let us know what you think! Submit issues and requests via GitHub.