Understand & use graph databases: Automate schemaless deployments with Neo4j & Liquibase

In environments dealing with large volumes of data, graph databases simplify the process of exploring and understanding the relationships among billions of data points. They make it easier and more efficient to see how different data points influence each other.

What is a graph database?

A graph database is a type of NoSQL database designed to effectively manage and represent data through connections known as nodes, edges, and properties. These databases are particularly good at handling data that involve complex relationships and interconnections between different data elements. They are ideal for situations where the relationships between data points are just as crucial as the data itself.

Graph databases are based on graph theory, a branch of computer science and database management. In these databases, data is organized in a nonlinear format using vertices (also called nodes) and edges. This setup is not only versatile but also provides clear visual representations of data and its connections. This clarity is especially useful for simplifying and understanding complex relationships between data, offering a significant advantage over traditional, linear data structures.

Comparing graph to relational databases (RDMS)

Relational Database Management Systems (RDBMS) are a traditional form of database that stores data in structured formats known as tables. Although data in these tables can be linked, creating these links typically involves matching unique identifiers across tables using JOIN operations or lookups. This means that the data itself does not inherently show its connections. Additional operations are necessary to reveal and utilize relationships between different data and tables.

While relational databases are effective for numerous applications, they often encounter difficulties when dealing with complex relationships among data elements. On the other hand, graph databases are specifically designed with flexibility to manage relationships more efficiently. This design allows graph databases to introduce new types of relationships or nodes without needing to change the underlying database schema, a common challenge in relational systems.

Components of a graph database

Essentially, a graph database contains and explains relationships among various independent pieces of data through nodes (the specific data units), edges (the connections between data points), and properties (information contextualizing the nodes and edges). 

• Nodes represent entities or instances such as people, businesses, accounts, or any other item to track
• Edges represent the connections or interactions between these entities, which can be directional (indicating a one-way relationship from one node to another) or non-directional
• Properties provide contextual information attached to both nodes (for example, names, addresses, descriptions) and edges (such as the type or strength of the connection)

Additionally, properties can assign weights to edges, which is particularly valuable in algorithms designed for pathfinding or optimizing networks.

The detailed and dynamic representation of relationships in a graph database makes it clearer and more effective than traditional database models. The inherent flexibility of graph databases proves especially beneficial in various applications where dynamic relationship mapping is crucial.

Neo4j, a leading graph database and Liquibase technology partner, offers a wonderfully concise explanation in their Graph Databases in 60 seconds video. Neo4j also uses labels, for additional context.

If you only take away one thing, make it this: 

“In essence, a relational database does Joins on read, whereas a graph database does Joins on write.”

Graph databases treat data connections as inherent parts – every piece of data is connected to at least one other point somehow, and the nature of that connection can be as important as the two data points themselves.

Imagine a network of individuals, such as an organizational chart or a family tree – it’s a simple and familiar application of graph data structure. 

This kind of database structure is perfectly suited for any scenario in which relationships between data are just as informative or essential to understanding. Some questions lend themselves to graph databases:

• Could you travel somewhere without knowing the route? 
• Could you hunt down a malicious individual without a trail of evidence? 
• Could you find lost relatives without knowing how they’re related to you?
• Could you recommend a new show to watch based on others they like?

Neo4j refers to these as “highly connected data questions,” and they’re more prominent than you might expect. 

Why and how are graph databases used?

Graph databases are specifically designed to manage "highly connected data questions." These databases allow you to not only retrieve a single data point but also understand its important relationships. This feature is particularly valuable because it enables you to see both the direct information and deeper links and insights.

Graph databases are excellent for mapping various types of relationships, such as those between customers, merchants, or products, which can enhance customer experiences and refine business strategies. In the banking industry, for instance, graph databases play a crucial role in fraud detection. They help trace and analyze strong connections among different transactions, enabling banks to identify and prevent fraudulent activities effectively.

This makes graph databases very useful for applications with highly interconnected data like:

• Social networking platforms, where they manage complex social graphs like friendships, group memberships, and shared content.
• Recommendation engines, where they create personalized suggestions by analyzing relationships between users, their preferences, and products.
• Fraud and pattern detection systems, where they identify and study patterns of fraud by looking at transactions and connections between accounts.
• Crime investigation, where they are used to analyze and visualize relationships, networks, locations, and other evidence to uncover crime rings and speed up forensic investigations
• Healthcare data management, where they are used for complex patient records, public health data, provider information, and other sensitive data that can be leveraged for research and treatment.
• Knowledge graphs, where they are used to build and query large networks of interconnected data, useful in search engines, artificial intelligence, and semantic web technologies.
• IT network optimization, where they help determine the most efficient data paths, enhancing speed and reducing latency.
• Bioinformatics and genomics, where they are used by scientists to study genetic markers, their relationships, and biological network pathways to conduct research and develop therapies
• Mapping, logistics, and supply chain management, where they optimize routes, storage, and overall visibility, also enabling simulations of supply chains.

Graph databases excel in managing complex, interconnected data – enabling quick navigation through large networks of data and improving the accuracy and efficiency of queries – perfect for large, complex datasets.

Plus, graph databases scale effectively as data increases — helping to manage costs and workload efficiently. For those dealing with big data challenges, graph databases offer a smart, scalable, and flexible solution.

What is a graph API?

A graph API is a tool that allows you to interact with a graph database programmatically. It stands for application programming interface (API) and offers functions to create, read, update, and delete (known as CRUD operations) nodes, edges, and properties in a graph database. 

Additionally, a graph API supports complex graph-specific tasks such as finding the shortest path, detecting communities, and implementing other graph algorithms essential for making the most of the graph structure of the data. This API helps developers retrieve data in ways that are meaningful and semantically appropriate.

The graph API acts as a link between client applications and the underlying graph database. It provides developers with a predefined set of operations to interact with the database, simplifying the handling of graph data structures. This lets developers concentrate on the application logic rather than the complexities of data storage and retrieval.

Features of graph APIs include:

• Easy data manipulation: Developers can add or remove nodes and edges without complex queries or deep knowledge of the database schema.
• Traversal operations: These operations allow efficient navigation through the graph to examine relationships, patterns, and nodes at different levels, and can include filters to refine the traversal.
• Support for query languages: Many graph databases have their own query languages, like Cypher for Neo4j, Gremlin for Apache TinkerPop-enabled databases, and GQL, which is an emerging ISO standard.
• Integration with data pipelines: Graph APIs make it easier to integrate the graph database with existing data pipelines in an enterprise setting, covering data ingestion, transformation, and output processes.

It provides the necessary capabilities to manage and analyze highly connected data effectively. It plays a fundamental role in developing applications that require deep insights into complex relationships, ensuring efficiency and scalability in managing network-centric data.

Graph APIs integrate smoothly into data pipelines, helping developers incorporate graph database functionalities into broader applications and monitor workflow performance for continuous improvements.

Neo4j + Liquibase: graph database DevOps

Neo4j is a leading provider of graph databases that offers a native graph database and management system, a unique query language, and its own graph API for handling, querying, and analyzing data. It is available in both Community (open source) and Enterprise editions, along with AuraDB, which is Neo4j's managed SaaS option.

The query language used by Neo4j, known as Cypher, is explicitly designed for graph databases and shares similarities with SQL. Neo4j is particularly efficient in storing and representing graph data, which enables it to perform constant time traversals. This efficiency ensures that as the database size increases, access remains quick and easy. For users managing large and complex networks, such as social media analysis, recommendation systems, and other big data applications, Neo4j offers a scalable solution that maintains high performance without requiring significant infrastructure investments.

Neo4j supports popular programming languages like Java and JavaScript. Its flexible property graph schema allows for easy modifications over time to update relationships. These database modifications can be implemented swiftly and automatically, bypassing the usual delays associated with manual DBA reviews, especially when using an automated database DevOps tool like Liquibase.

Liquibase supports over 60 different database types, helping to unify and simplify change management for teams and applications that use diverse environments. It focuses on managing database changes and offers a comprehensive set of tools for graph database pipelines. As a complete database DevOps solution, Liquibase enables:

• Automated change management from development through to deployment, facilitating self-service deployments
• Governed change management with automatic, custom-defined rules, policies, and safety measures for security, compliance, and quality
• Detailed observation and tracking of change management, providing metadata that supports analytics, monitoring, and auditing

Liquibase connects with the entire application development process, linking database changes as code from a developer's initial request to source control, CI/CD automation tools, observability platforms, and finally, the production environment.

When used with Neo4j, Liquibase enhances the CI/CD pipeline with graph database capabilities. This integration utilizes deployment tools enhanced by governance and observability features, making the advantages of flexible graph database technology accessible to all teams.

Explore the capabilities of graph database DevOp, including automation, governance, and observability in our on-demand webinar: Automating schemaless database migrations on Neo4j with Liquibase. You’ll learn from the lead developer of Neo4j as he discusses the benefits of using Liquibase for graph databases and demonstrates the automated process.

‍