November 7, 2023

Solving NoSQL database governance and compliance challenges with CI/CD

See Liquibase in Action

Accelerate database changes, reduce failures, and enforce governance across your pipelines.

Are you struggling to get a handle on compliance and governance for your NoSQL databases?

When enterprises adopt NoSQL (nonrelational, unstructured) databases, they often neglect to apply the same level of governance as they do to traditional SQL (relational) databases. As NoSQL databases gain traction in business-critical applications and become subject to the same regulatory, security, and general compliance requirements, the relative lack of governance is concerning, to say the least.

By understanding NoSQL databases and their unique governance challenges, you can overcome them through database DevOps – specifically, CI/CD and automation in the database change process. Keep all the benefits of NoSQL without exposing your business to compliance, security, or regulatory risks.

In this article

The rise of NoSQL databases

To understand NoSQL databases, it helps first to acknowledge the explosion in the types and sources of data common across the world today – things like social media, documents, IoT sensors, audio/visual content, and others sprouting from our digitally native world. Modern data requires modern solutions – especially when this newer data doesn’t easily fall into the traditional structured SQL databases.

There is simply so much more data from so many sources that relational databases cannot scale to manage it. To perform at scale with varied data can only be achieved with more specialized technology. Thus, a similarly explosive influx of specialized nonrelational database engines in the marketplace. NoSQL databases meet the technical and business needs of modern data with enough raw computational capacity available at such a relatively low cost that they’ve actually become affordable, too.

Finally empowered with a more flexible and scalable database option, NoSQL database developers are freed from the overhead of force-fitting data into ill-suited relational models. They’ve rapidly released new and innovative solutions that show the potential of NoSQL databases, such as:

Using key-value pair databases for better real-time inventory tracking in ecommerce
Improving multimedia content management systems with document databases
Powering recommendation engines with graph databases
Enabling very high-throughput data warehousing capabilities with a wide-column engine

These capabilities all serve to deliver enhanced or net-new digital experiences for businesses worldwide.

Benefits of NoSQL databases

Since NoSQL databases are workload-specific, they have a natural affinity for the type of data being managed. That means developers don’t have to waste time writing code that crams nonrelational data into a relational model. Productivity improves while the risk of bugs due to code complexity decreases.

Specialized NoSQL databases can also speed up solution delivery. Thanks to APIs, they return data in a format that requires less effort from developers to parse in their application code. Analysts and other consumers of the data receive it in a format that requires less conversion effort, so they can spend more time analyzing.

And in terms of scalability and performance, NoSQL databases outperform traditional databases of the same cost.

Types of NoSQL databases

There could be an unending list of NoSQL database engines in the world to deal with any kind of specific workload – time-series, geo-spatial, et cetera – but most fall into four common types.

Document

Document databases store data as a self-contained unit of data that can include multiple media types (text, audio, images, etc.).
Common uses: Content management systems (CMS), ecommerce, mobile apps, and real-time/IoT scenarios
Examples: MongoDB, Couchbase

Key-value pair (KVP)

KVP databases store data in the form of a unique identifier, or key, that associates to some data, or value. They’re often chosen for speed and scalability.
Common uses: Caching, session management, real-time data, gaming data, IoT data
Examples: Redis, Riak

Graph

Graph databases store data as a combination of nodes and relationships. A node represents an entity (e.g., a person) and a relationship connects two nodes while also describing the nature of the relationship.
Common uses: Social network analysis, master data management, fraud detection, recommendation or configurator systems, chatbot knowledge graphs
Examples: Neo4J, Flock

Wide-column

Wide-column databases store data as a long list of columns rather than as rows within a table. This makes them good for aggregation-type queries.
Common uses: Real-time analytics, log management, data warehouses, business intelligence
Examples: Cassandra, Hbase

NoSQL database workloads are growing rapidly

During the pandemic era, enterprises fast-tracked tech modernization and digital transformation, investing in advanced infrastructure for flexibility, scalability, and enhanced customer experiences. The adoption of NoSQL databases rose in this wave, powering new projects that have since turned into production-grade applications. What started for many large businesses as experiments in NoSQL databases quickly turned into business-critical tech stacks dependent on NoSQL databases.

The growth in the adoption of NoSQL databases can be clearly seen in the rise of the most prominent and widely adopted player, MongoDB (45% market share). The company, which shared its Q2 report, shows strong increases overall and at the enterprise scale ($100k+) specifically.

Compliance and governance challenges of NoSQL databases

The haste and agility of pandemic-era digital infrastructure transformations often made NoSQL database compliance an afterthought in initial rollouts. Pushing compliance further out of focus was the exciting flexibility of NoSQL databases, which don’t require the same level of complexity in change management as relational databases.

But while NoSQL databases enable faster delivery and lower overhead, they’re still subject to compliance regulations.

Systems using NoSQL databases are becoming more common in business-critical situations, so now they’re dealing with massive volumes of sensitive data including PII, medical, and financial data. While enterprises are familiar with managing HIPPA, PCI, GDPR, SOX, SOC-2, and other compliance requirements on their traditional relational databases, new NoSQL platforms lagged behind.

Every compliance standard bears its own requirements, and while some overlap, there’s no one-size-fits-all approach. SOX, for example, requires companies to prove they have good IT controls for items such as:

Access control
Security and cybersecurity
Segregation of duties
Change management

While not an exhaustive list, this example shows how some typical controls might get bypassed in experimental systems, such as fledgling NoSQL database applications. During experimentation, teams want a high degree of access to perform maintenance, make the inevitable ‘tweaks’, or perform quick data manipulation to support forward feature development.

But when the new nonrelational databases go live on public-facing and business-critical applications, compliance suddenly becomes a priority. The challenge, then, is to manage one or multiple compliance guidelines in the relatively uncharted world of NoSQL governance, while maintaining the benefits of flexibility, speed, and scalability.

CI/CD automation for NoSQL database compliance and governance

To quickly ramp up compliance for NoSQL databases, turn to database DevOps. Specifically, the DevOps practice of Continuous Integration / Continuous Delivery (CI/CD). The CI/CD for database compliance practice is really two in one:

Continuous integration (CI)

CI practices declare that people making changes to a system should merge those changes with the work of their peer change-makers as often as possible in order to reduce the amount of work required to integrate everyone’s changes into a coherent whole.

Continuous delivery (CD)

CD practices declare that the integrated changes should be delivered as frequently as possible to Production in order to keep the actual number of changes applied in each batch to a minimum. This reduces the number of potential problems per batch and makes it easier to find problems during testing or troubleshoot problems should they occur.

Enacting CI/CD for NoSQL database change management provides the framework for governance that allows compliance with any necessary regulations. It does so by treating database change as code, automating changes, and shortening feedback loops.

Database change as code

Central to CI/CD practices is the idea that infrastructure and maintenance instructions are code and should be managed as such. This enables teams to use their SCM and code governance mechanisms across their entire application stack. This greatly eliminates a lot of duplicate effort to reconcile which changes are related to each other and creates transparency for understanding the changes themselves.

And when database change is treated as code, it can be version-controlled and automated.

Automating database change execution

Related to the “as-code” approach is a heavy focus on automating all database change executions. In other words, running the code-based changes programmatically. This allows for numerous benefits to governance and compliance, such as consistency, security, separation of duties, observability, and reporting.

Consistency

Automated processes happen the same way every time and can, themselves, be managed using standard SCM code governance methods. That means it is easy to track and confirm both the process and the change definition when the change was executed by the automation system.

Security

By using automation instead of a manual change, organizations can limit or even eliminate the need for humans to directly access databases to perform routine tasks. All executions – and therefore accesses – are logged by the automation system. While there will certainly be exceptions, making those as rare as possible means that compliance becomes much easier to prove in an audit.

Separation of duties

The simple definition of separation of duties is that no one person should have enough privileges to misuse a system on their own. This is easily achieved by ensuring that the database CI/CD automation system itself is managed by a separate team from the developers.

Observability

Automated processes also provide consistent output which can be routed to observability platforms. This provides a high degree of visibility and situational awareness that changes are happening to a database and makes it easier for all concerned to quickly identify anomalies. (Check out this on-demand webinar for more on database observability.)

Reporting

Automation systems are better at recording what they have done – and in a consistent format – than humans. This is especially true of tedious tasks, such as filling out change management tickets. Using DevOps-style automation to make changes and update systems of record in a consistent manner means that reconciling records during an audit becomes very simple.

Shift left

To shorten the feedback loops for people making changes, database CI/CD moves tasks, processes, and testing earlier in the development lifecycle. The idea is to catch issues as soon as possible, ideally when they are easier and less costly to fix.

By building tests and other checks into the automated processes, teams can ensure that their change creators get immediate and consistent feedback on their changes along with recommended remedies. The key is to use automation to quickly identify common errors or forbidden patterns without making a developer wait for an expert to manually evaluate the change.

This provides greater productivity for the developers by allowing them to adjust and test their own work without context switching. It also ensures that the database experts do not lose productivity answering routine, basic questions. Exceptions can be tracked and systematically reduced by reviewing and improving the automated checks.

How to bring CI/CD automation to your NoSQL databases

The use of modern, nonrelational, NoSQL database engines in enterprise production workloads does not free companies from the regulatory requirements associated with such workloads. However, that does not mean that they need to use old-fashioned, manual processes to ensure that the new workloads meet those compliance standards.

Instead, NoSQL databases should be governed using database DevOps techniques, such as CI/CD and automation, to ensure that the productivity gains associated with the specialized NoSQL databases are not lost by saddling them with legacy governance and compliance processes.

Many enterprises – including those in highly regulated industries like finance and healthcare – look to the leading database change automation tool to enable governance that meets various compliance standards.

Learn more about how Liquibase brings CI/CD, automation, and observability to MongoDB, the leading NoSQL database with these additional resources:

Not on MongoDB? Explore our full list of 59+ supported databases, including AWS DocumentDB, Cassandra, and others.

When you’re ready to talk with an expert about solving NoSQL database compliance and governance challenges with CI/CD, get started with a demo, trial, or call with a database DevOps expert.