What is CI/CD?
Let’s break it down into pieces.
What is database continuous integration (database CI)?
Database continuous integration (CI) is the rapid integration of database schema and logic changes into application development efforts and to provide immediate feedback to developers on any issues that arise. As a further evolution, database continuous delivery (CD) seeks to produce releasable database code in short, incremental cycles.
What is database continuous delivery?
Database continuous delivery (CD) is a process that focuses on getting changes for new features, updates, and bug fixes into production and to the users as safely and quickly as possible. This is achieved by streamlining the development process and keeping application code and database code in a deployable state.
The goal of continuous integration and continuous delivery (CI/CD) is to reduce time-to-market and create a steady stream of end-user value with frequent, high-quality software releases.
To deliver new software experiences more quickly to market, companies have spent the last decade trying to modernize the way they build and deliver software. In their efforts to modernize and improve the continuous delivery pipeline, many tools have focused on bringing CI and CD to application code. However, the same has not happened for database code. Many organizations still rely on a shared service database team that manually reviews and deploys DB code changes. Given that the end-user experience is incomplete without the full software stack, which includes both the application and the database, there has been growing interest and demand in continuous integration and delivery tools for the database as well.
How do database CI tools impact database deployments?
The aim of database CI tools is to bring the same integration and deployment best practices to the database and enable SQL code to flow through the software release pipeline, synchronized with application code. By aligning database deployments with application deployments, teams see a much better return on their investments in the tooling and process updates. This, in turn, helps teams bring new innovations to the market faster and with higher quality.
Continuous Integration and Database Deployment Automation Tools
In order to build a good foundation and implement CI/CD for the database, you need to have the correct database integration and deployment tools. There are four categories of database deployment tools you should have in your automation framework to achieve database continuous integration:
Database Development Tools
When you think of database deployment tools, it’s unlikely you’ll consider application development tools as part of that category. However, all your tooling should work together to achieve database continuous integration. Just as application development tools have grown to integrate with CI solutions, the same is true for database development tools as well.
Quest Software’s Toad has some great features for supporting Agile database development. Databases are different than the application and have state, so having a database instance to develop against as part of a team is key. Toad’s Team Coding allows users to leverage existing Source Code Control (like Git, Subversion, and others), and a live database to support database developers. Not only can they check their SQL scripts in and out, but they can do so with the database objects like tables and stored procedures, as well.
Database Release Automation Tools
Beyond an appropriate development tool, database CI requires a DB release automation solution. Database release automation tools like Liquibase deliver automated validation, build, test, and deployment of database changes. These core capabilities ensure that any database code pushed to source code control is pulled out into a CI process that can provide developers with near-immediate feedback. With a database release automation solution, software teams can consistently deliver a continuous stream of value to end-users without getting slowed down by database deployments. Plus, Liquibase integrates with DevOps tools your team is already using, like Jenkins.
Application Release Automation Tools
As organizations add new features and enhance existing software, the number of components and the complexity of the software stack have only grown. As new trends emerge that allow functional isolation and avoid single points of failure, orchestrating and aligning the release of all the necessary components requires application release automation (ARA). Tools like CA’s Automic, IBM’s UrbanCode Deploy, Serena’s Deployment Automation from MicroFocus, and digital.ai’s XL Deploy integrate with Database Release Automation solutions like Liquibase Enterprise to enable Continuous Integration for the full software stack.
Test Data Management Tools
A key trend in software development is test-driven development, in which the CI process is enhanced with production-quality test data. It’s common for applications to change functionality based on the data stored in the database. Consequently, providing data that mirrors (or matches) production data is essential to a database continuous integration process that can generate high-quality test output.
DevOps Tools like CA’s TDM, Delphix Data Virtualization, and IBM’s Optim, populate test databases with data. Some solutions, such as Delphix, allow users to request masked self-service copies of production databases for integrated testing environments.
Continuous Integration and Continuous Delivery are best practices for accelerating the speed and quality of application code changes. Similarly, database Continuous Integration is important in accelerating the database release process while reducing risk. By including a database release automation solution and adding database CI to your existing application delivery toolchain and process, you can increase the pace and quality at which the entire software stack – including both the application and database – can be delivered to the market.
Why Continuous Delivery for the Database Matters
Automating database deployments is an intimidating task as companies are releasing application updates with higher frequency. To meet this demand for increased performance, it is critical to focus on both application code and database code. Teams need to fill the gap in DevOps with Database continuous delivery. This involves improving processes and closing the DevOps feedback loop. Put simply, application teams need to create an end-to-end process that covers application and database development for efficient CD.
Today’s businesses are struggling to keep pace with consumer demand. Not so much the demand for goods and services, but demand for the ideal customer experience. Consumers want to interact with their favorite companies on their preferred platform. And if they don’t receive the experience they’re looking for, the competition is only a click away. Chances to capitalize on market opportunities appear to be dwindling. This is because of customer expectations, fierce competition, and rapid innovation.
In response, businesses increase the pressure on IT teams to step up. This demands that IT deliver products and services faster and more often. IT teams developed and now leverage practices like Agile Development, Continuous Delivery, and DevOps.
Teams and organizations are adopting database automation and application release automation tools to support these practices. Tools that aim to help IT deliver stable high-quality software faster. These include source control systems, continuous integration servers, automated testing infrastructure, and more.
Continuous Delivery, as a philosophy, is gaining rapid adoption among today’s enterprises. It borrows lean manufacturing concepts and applies them to software creation and delivery. It coaches us to view the entire SDLC as one process and map it out to understand how value flows through the system. Once understood, Continuous Delivery offers clear advantages. It allows us to identify inefficiencies and leverage standardizing and automating workflows.
Standardizing a workflow makes it repeatable. Once a workflow is repeatable, it becomes a good candidate for automation. With automation, workflows become faster, and human error decreases. Continuous Delivery aims to automate the process from development, test, and even production. By leveraging automation, a highly-efficient pipeline is achievable.
Continuous Delivery works. With correct implementation, IT organizations can deliver features faster and with fewer errors. Organizations can go to market quicker and reduce operating expenses. This gives businesses a competitive advantage.
This phenomenon is now causing a problem for the database team. They aim to provide a safe, high-performance environment for the company’s data. That mandate is being threatened by the speed of innovation in development. DBAs must sort how to maintain the pace of application release without hurting the data.
That is the challenge. That is why Continuous Delivery for the Database matters.
The State of DevOps Report noted databases are often a major source of risk and delay in application release cycles and teams that do well at continuous delivery store database changes as scripts in version control and manage these changes in the same way as production application changes.
5 Reasons to Create the Continuous Delivery Database
“Continuous Delivery is the ability to get changes of all types into production, or into the hands of users, safely and quickly in a sustainable way.” – continousdelivery.com
When I ask how organizations support continuous delivery, I often hear: “We are doing Continuous Delivery!” Upon further examination, this is really not the case. “But we create tickets” is NOT continuous delivery for the database. A ticket to an external service team does not provide safety, speed, or sustainability.
Here are 5 reasons your team should consider a database continuous delivery processes:
1. Align the speed of change for the database and the application
You’re moving fast with your application deployments. You’re using build automation and application release automation (ARA) to deploy your apps. Sadly, when you hand off the database changes to another team, you now have a two-track process for releasing your entire application. This causes a cadence mismatch and results in slower releases because you can only release your app as fast as your database releases.
2. Enable Digital Transformation
Digital Transformation involves accelerating the adoption of technology to bring agility to your business. Thus, your company will be able to quickly adapt to changing market conditions. (In other words, be more like Uber and less like Sears.)
You need to provide continuous delivery for all components in applications. Otherwise, your venture will not reach digital transformation. Your company will miss market opportunities your competitors will not.
3. Enable Self-Service for Development
There is absolutely no reason for a development team to wait seven (or more) days for a database change. In a two-week sprint cycle, that’s half a sprint. Unacceptable. With database continuous delivery, your dev teams will be able to release and improve applications fast. However, safety is an absolute must when you distribute responsibilities for changes. Thus, you will need both speed and safety to support self-service.
4. Enhance Data Security
Database release automation (DRA) ensures better database code.
This is accomplished by eliminating errors that can cause performance issues or downtime. Better database code also means better data security. Common mistakes that make databases more vulnerable to breaches and data theft are eliminated. These include misplaced GRANT statements, application misconfigurations, and more.
5. Your investment in DevOps and Application Release Automation will now return value
When describing their DevOps implementation experience, customers often wish they had done the database first. Though ARA is a necessary component in the DevOps toolchain, DRA is just as important. However, prioritizing the database over the application will return far more dividends than ARA. You must solve the hard problems first to make the impossible possible.
You can achieve database continuous integration with database DevOps. Contact us to learn how we can help your team.
Learn more about continuous delivery for the database by downloading our white paper:
CI/CD for Databases: A Guide for Bringing Database Changes into Your CI/CD Pipeline
Feedback Loops, DevOps and Database Continuous Delivery
Source control is your single source of the truth. Using a continuous build server with source code control is the first step in Agile Development, Continuous Integration, and Continuous Delivery. When a build happens, source code is transformed from human-readable text to machine instructions that can be deployed. The source is tagged so that the code is forever tied to a specific application release. You can recreate it at any future time.
However, database source control is missing from the database change process today. Sure, maybe we are checking in our DB change scripts. Maybe we use Toad Team Coding to check-in and check-out DB objects for us during development. Once that build is performed, you have an atomic application binary ready for automated deployment and a bunch of SQL scripts. But that will still not speed your deployment process. You are still tied to a manual database deployment process.
The big challenge is that you cannot determine if a change has been persisted to a database without a manual review. If you need manual DBA intervention for every change to a database, you will fail.
The best way to guarantee safety is by using the same source control repository for both application code and DB code changes. This creates database source control: a single source of the truth for what database changes are part of a release. This helps teams know what application version goes with what database schema version. But this can be a lofty goal for many organizations.
Database Release Automation (DRA) solves this problem. Just like Application Release Automation orchestrates and automates previously manual tasks, database automation enables full stack automation by changing our database schemas and stored procedures safely and quickly without the need for human intervention. Of course, databases are different than applications. We cannot simply blow away our databases and recreate them with a new version of the schema. We would lose our data. Databases have state and evolve over time.
Balancing safety and speed is key; you must have both with Database Release Automation. As part of every Database Release Automation tool evaluation, you should have use cases that test if you can perform a completely automated database deployment. Review your past year’s schema changes and choose the difficult ones to automate. Renaming tables is always a good one. Always make sure that database standards are being met. Also, attempt to enforce naming conventions and the like with changes. Look for ones that you know should fail and make sure your DRA tool will catch them.
Updating data and database schema with application releases is already a time-consuming task. It is no longer effective to rely on old methodologies and adding resources to a process that is already inefficient. With the rise in demand for quick application releases and updates, organizations need to focus on leveraging database DevOps and database continuous delivery.
A reliable DB change process should support lights-out automation.
You need a solution that provides more than source control for the database.
Filling the Data Gap in DevOps with Database Continuous Delivery
Many IT teams have successfully automated many manual processes in software development. The same has not held true for the data underpinning the system.
Provisioning, versioning, and aligning data to database code is still a largely manual process. It adds considerable friction to application projects, slowing the pace of innovation.
As companies adopt application release automation, digital transformation is constrained by data. Data is oftentimes neglected and the weakest link in the toolchain. This is why leveraging continuous delivery database matters. Even now, database provisioning times are still counted in days, weeks, or even months. Most companies – even those with the latest DevOps tools- still manage and deploy DB changes manually. This only serves to anchor and delay development teams. In other words, most organizations aim to move to agile dev environments. However, they want to achieve this with old data management processes and procedures. The result is like mounting a modern Ferrari on wagon wheels. It is a great way to get nowhere fast. There are several advantages to creating a continuous delivery database. One of them being a faster, streamlined integration, delivery, and deployment pipeline.
It’s time to focus on the database
Keeping data and database schema changes current with application releases is a challenging task. It is no longer adequate to add more people and infrastructure to solve these challenges. There is an increasing need to bring the agile methods and DevOps database automation tools to the database, allowing teams to get access to high-fidelity, usable data in a fast and repeatable manner.
It’s time to focus automation on the data and the database and enable database continuous delivery.
It is important to be able to provision, refresh and tear down environments as fast as possible. This provides a quick, simple, and non-disruptive way to pave the way for DevOps and digital transformation. By taking a data-first strategy, companies can break the log-jam in development. With this approach, companies can achieve faster time-to-market.
In parallel, database releases should be automated and managed by development teams in the same way the application code is managed. This will transform schema management into a reliable database continuous delivery process.
Two is usually better than one
Sometimes a providential pairing delivers outstanding benefits. That certainly happened when Harry Reese first tinkered with combining chocolate and peanut butter in the 1920s. And it happened again with a combination of solutions from Liquibase and Delphix.
Delphix plus Liquibase unlocks the full potential of DevOps. Leveraging both solutions enables database continuous delivery.
Delphix simplifies the creation of masked and unmasked copies of data. Liquibase enables the seamless integration of schema changes across those data copies.
Using Liquibase and Delphix allows for acceleration of data delivery, as well as the automation of database releases, increasing the speed and quality of the total build environment process. Delphix and Liquibase provide self-service controls through a continuous integration layer. This enables developers to use tools, such as Jenkins, to version data and database schema changes like code. Meaning, there is more autonomy, less dependence on infrastructure teams, and faster releases.
Fill the Data Gap
There are many reasons to address the automation of data and database schema management. For organizations considering continuous integration and delivery, it is crucial to fill the data gap. Even for companies that are new to DevOps, it can be beneficial to start with data. Changing how developers code and deploy, manage builds, and perform migrations is profitable. However, there is the initial hurdle of changing an existing process. Solving for data and database schema management with Delphix and Liquibase is the opposite. IT teams can transform a highly manual process without disruption, unlocking immediate value.
Learn more about DevOps databases. Read our white paper: Filling the Data Gap in DevOps.
Database Deployment and Database Continuous Integration Best Practices
Implementing and following continuous integration and continuous delivery best practices for databases is easier than you might think. Many of the best practices that apply to application CI and delivery tools readily apply to database CI as well. These best practices include:
1. Tracking Database Code Changes
Database code changes should be tracked in the same source or version control system as application code. Database code should not be treated separately or tracked in an entirely different system. A separate system for database code leads to redundant effort, lack of visibility, confusion, and errors. Application and database code will also begin to drift on a separate system and get misaligned. Automated DB deployment tools like Liquibase allow teams to push database code into the source or version control solution that is already in place for application code.
2. Automated Database Code Validation and Feedback
Once the application code is checked in, a series of automated tests are immediately triggered. These tests assess if there are any issues in the code that warrant rework. Unfortunately, the same isn’t true for database code at most companies. One of the biggest challenges in accelerating database deployments is getting rid of the manual SQL code review that DBAs must perform.
This tedious manual effort can and should be largely eliminated by intelligent automation. That way, developers can get immediate feedback on SQL code – just as they do with application code – and avoid a long wait state in which they move on to a different task. This inefficiency in feedback causes large delays in database deployments and contributes to poor quality. It also forces developers to context switch to make fixes on SQL code changes that they wrote days or weeks ago.
To properly automate the validation of database code, a database deployment automation solution must have an object model of the proposed SQL code change. Otherwise, functional rules, such as ensuring that all tables having a primary key or unique constraint, cannot easily be validated. This means a DBA will need to give the code manual attention – which also means developers don’t get immediate feedback on changes.
The Liquibase Enterprise Dynamic Rules Engine is unique in that it is an object-based rules engine that can be easily extended. The Dynamic Rules Engine gets rid of much of the tedious DBA review otherwise required.
Stay away from simple, regular expression-based rules systems. It’s impossible for these systems to functionally validate the common organizational rules and standards that DBAs end up spending much of their time and energy on, instead of making progress on more critical value-add projects such as performance tuning, data architecture, high availability strategy, system upgrade planning and more.
3. Packaging Database Code
An important DevOps mantra is to “build once, deploy often.” Effectively all continuous integration tools allow application code to be built into an immutable package for consistent, repeatable, and predictable downstream deployment. Should anything go wrong, its errors can immediately be traced either to the application code or the environment when working with an immutable artifact.
Database continuous integration tools bring this same advancement to database code. As an example, the Liquibase Enterprise Database Code Packager creates an immutable artifact from validated database code so automated DB deployments can enjoy the same consistency, repeatability, and predictability as application code releases.
4. Providing Visibility into Database State
Another key DevOps tenet is to “amplify feedback.” Continuous integration solutions focus on providing visibility and feedback readily and immediately. As such, a database continuous integration solution should have an accessible web interface. This interface should allow all stakeholders to quickly understand the status of every database. Furthermore, it’s important for the solution to integrate with ticketing systems such as JIRA and TFS, and to support parallel development strategies commonly found across enterprise development teams.
DB deployment automation tools like Liquibase Enterprise provide a web interface and have labeling systems that integrate with ticketing solutions to provide necessary visibility and feedback to appropriate stakeholders. With Liquibase, all DB code changes can be traced back to source code control. This quickly links the changes to the business value that they are meant to deliver.
