MongoDB schema migration

As systems evolve and requirements change, schema migrations enable databases to adapt cohesively without compromising existing data or system functionalities.

So, how does it work with a “schemaless” platform like MongoDB? This post explains the basics of schema migration, MongoDB’s unique approach, its benefits, and challenges. 

What is schema migration?

Database schema migration is the process of managing and applying changes to a database’s structural framework or schema. Sometimes referred to as simply “database migrations,” they allow the schema to evolve alongside the application.

This often involves tasks such as adding or removing tables, columns, or indexes, as well as modifying data types or constraints.

How does schema migration work?

As applications advance, database schemas must adapt swiftly, maintaining or improving performance, flexibility, and governance. Schema migration allows for changes that meet evolving business and technical needs while maintaining data integrity. Keeping the database and application in sync is important to remove bugs or errors caused by data differences, making sure each deployment is accurate and reliable.

There are two basic approaches to schema migration:

• State-based database deployments: This method focuses on the end goal by comparing the current database state with a predefined ideal state, using tools to generate scripts that bridge this gap. It's favored by DBAs for its ability to closely manage and monitor changes, ensuring a controlled update process.

• Artifact-based deployments: Emphasizing the process, this approach captures individual change scripts during development, appealing to developers for its simplicity and control. Migration tools allow for customization of scripts, facilitating quick feedback and faster deployments, akin to how application code is developed.

Example: schema migration in action

i360  builds thousands of PostgreSQL databases in Amazon Aurora. They needed an easy way for developers to migrate schemas while keeping them in sync throughout the CI/CD process.

Here is the schema migration workflow they created using Liquibase:

• Developers create database scripts, rollback scripts, and add comments about the bug or feature ID in their Liquibase changeset that match what they have in TFS for tracking.
• Developers run both their migration and rollback script through Liquibase to verify they are valid, run successfully, and are idempotent.
• Developers check the script and Liquibase changelog into GitLab source control.
• Includes the author name as their email address for notifications for script success and failure.
• A record is added to the approval table in Postgres. 
• A copy of the script and changelogs are put into S3 so that Fargate can have access to running those scripts. 
• The migration runs and kicks off a build.

Additionally, they use drift detection tools to automatically detect differences between database schemas using JSON output

MongoDB and (schemaless) schema migration

MongoDB is a non-relational, schemaless database. It does not require a predefined schema set up before data insertion. This allows documents within the same collection to have different structures with varying fields and data types. Here are some key aspects of this characteristic:

• Flexibility in data representation: You can store documents in a collection that do not all have the same set of fields or structure, making MongoDB very flexible in accommodating the varied data of real-world applications.
• No fixed schema requirement: The lack of a fixed schema allows developers to iterate on the data model quickly and adapt more readily to changes in the application's requirements.
• Schema evolution: Over time, the schema can evolve without the need to modify all existing data. New fields can be added to documents without affecting existing documents.

While this schemaless nature offers significant flexibility, MongoDB also supports schema validation rules that can be applied to ensure data consistency as needed.

The challenge with typical MongoDB migration

Migrating schema in MongoDB, despite its schemaless nature, can present several challenges and limitations. Here are some key issues to consider:

• Data consistency: Without a fixed schema, ensuring consistency across documents can be challenging. As applications evolve, documents that were not updated to the new schema format can lead to inconsistencies.
• Performance impact: Applying changes to a large dataset can be resource-intensive and slow, especially if the changes require updating many documents or rebuilding indexes.
• Complexity in querying: A varied document structure can make querying more complex and less predictable, particularly when dealing with documents that lack certain fields expected by the query.
• Data validation: Enforcing data integrity and validation becomes more challenging as the schema evolves. MongoDB provides schema validation tools, but implementing these in a dynamic schema environment requires careful planning and testing.
• Dependency on application logic: Much of the responsibility for maintaining schema consistency might shift to application logic, increasing complexity and potential for bugs in application code.
• Migration scripting: Writing scripts to migrate data to a new schema can be error-prone and requires thorough testing. This process can be complicated if the changes are substantial.

These challenges underline the importance of careful schema design and planning, even in a flexible environment like MongoDB. While MongoDB offers tools and features to manage schema evolution, leveraging these effectively requires a strategic approach to database management and application development.

MongoDB schema migration automation

Migrating schemas in MongoDB (and other NoSQL databases) is challenging, given their flexibility and rapid evolution. The ideal way to simplify schema migration is to integrate an automated change management workflow into existing pipelines. 

Liquibase’s MongoDB database DevOps solution provides a structured, automated approach to schema migrations that addresses these issues by integrating version control, rollback capabilities, and compliance features directly within MongoDB pipelines. 

This approach gives teams confidence in managing MongoDB schema changes safely, supporting consistent deployments, governance, and visibility across development and production workflows. 

Start automating your database change process

Dive into database CI/CD with Liquibase and discover how database schema change automation works. To get your broader team on board with the cultural and technology investments needed for successful database DevOps, learn the foundations of database change management.