Using Liquibase for Kubernetes database migrations with init containers

Kubernetes (AKA K8s) is a go-to choice for deploying and managing containerized applications since it supports flexible, scalable, reliable, and efficient environments. It makes nearly every aspect of management and deployment easier thanks to features like:

• Automated rollouts and rollbacks
• Service discovery
• Load balancing
• Storage orchestration
• Self-healing

However: database migrations still present unique challenges, namely ensuring schema consistency and preventing conflicts during updates.

That’s where Liquibase comes in. By integrating Kubernetes and Liquibase, teams overcome these challenges to enact a simpler approach to database migrations that keeps schemas updated and aligned with relevant application releases. 

Embedding Liquibase into the startup process of an application is a very common pattern for good reason. In this post, you’ll learn how to integrate Liquibase with Kubernetes, so you can:

• Streamline and automate database migrations
• Track database changes
• Use init containers to avoid stuck database change locks that lead to incomplete or problematic database migrations before application startup

Leveraging Kubernetes’ init containers to run prior to main containers avoids the issue of stuck locks, enabling a streamlined workflow within the Kubernetes infrastructure. 

Once you set up Liquibase to deploy on app startup with init containers, your database state will always match what your app code expects. Liquibase even ships with built-in support for this with the Spring Boot and Servlet integrations.

New to Liquibase? Learn more about how Liquibase provides a complete database DevOps solution.

Kubernetes database migrations

Database migrations are crucial for maintaining database state, especially in dynamic environments leveraging Kubernetes. Migrations ensure that your database schema evolves with application releases, preventing discrepancies that can lead to errors and downtime. 

Without regular migrations, application and database teams risk misalignments between application and database code, which can cause issues with data integrity, scalability, and system availability. By keeping schema consistent and up-to-date, application pipelines can ensure smooth operations, reliable data access, and seamless integration between the app and database layers, supporting current functionality and enabling innovations.

Kubernetes excels in managing containerized applications, but database migrations within this environment introduce several complexities:

• Ensuring database schema consistency across all instances of the application during updates
• Avoiding conflicts and data corruption during simultaneous updates from multiple instances
• Coordinating schema changes across interdependent microservices 
• Minimizing downtime or other disruptions to application availability or performance
• Dealing with restarting pods, stuck locks, and incomplete migrations

Liquibase’s capabilities for tracking, versioning, and deploying database changes work within Kubernetes’ automated and containerized environment, enabling Continuous Integration and Continuous Deployment (CI/CD) for database changes. This integration helps maintain consistency, reliability, and efficiency in managing database updates in a scalable, containerized infrastructure.

Database change lock problems

Another way Liquibase supports Kubernetes integration is with database locks. Locking ensures that if multiple servers start simultaneously, they won't run into problems when trying to apply the same database changes. 

Using Liquibase’s database locks is important when deploying applications that involve multiple instances or pods, especially in environments like Kubernetes. These locks ensure that only one instance performs schema changes at a time, maintaining consistency and preventing race conditions. Locks are particularly crucial during deployment and scaling operations when multiple instances might start up and attempt to apply migrations concurrently.

Liquibase’s locking system uses a DATABASECHANGELOGLOCK table as the synchronization point, with the service setting the LOCKED column to 1 when it takes the lock and setting it to 0 when it finishes. If the lock is active, the pod waits until it is released. This mechanism ensures that schema changes are applied in a controlled manner, even when multiple pods are deployed or restarted simultaneously.

Stuck locks: "When in doubt, kill the process"

As long as the process that has set the LOCKED column to 1 isn't killed before it has a chance to set back to 0, this works fine. When it's not working fine, all of the other Liquibase processes (including a newly restarted process on the same machine) will just continue to wait for a 0 value which will never come. The only way to recover from this scenario is by running the Liquibase unlock command or updating the DATABASECHANGELOGLOCK table manually.

Historically, the stuck locks have not been a problem because the Liquibase process was rarely killed. If it was killed, it was done manually and easier to recover from. Tools like Kubernetes have taken a philosophy of "when in doubt, kill the process" and this causes problems.

Because Kubernetes emphasizes stability and reliability, it often resorts to restarting pods that exhibit issues or become unresponsive. This approach helps ensure that applications continue running smoothly, but it can lead to complexities with operations like database migrations, where processes may need to complete specific tasks before termination.

Kubernetes’ tendency to restart pods can result in stuck locks if a pod is killed before releasing the lock. This scenario can halt further schema updates, requiring manual intervention to unlock the database. Implementing robust monitoring and automated unlocking mechanisms can help mitigate these issues, ensuring smooth operation and minimizing downtime.

At the heart of this issue is Kubernetes' expectations of pods starting quickly. Rightly so. You WANT to deal with slow startup issues quickly. However, you also need to give your pods the time they need to do potentially time-consuming initialization work, such as migrating your database.

The Solution: Init Containers

The best practice to run Liquibase in Kubernetes is to use an init container in Kubernetes.

To do so, create a Pod that includes the Liquibase init container and your main application container. The init container will run Liquibase to update the database schema before the main application container starts.

Here is an example of how to configure a Pod with a Liquibase init container:

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  initContainers:
  - name: liquibase
    image: liquibase/liquibase:latest
    command: ["liquibase", "update", "--changeLogFile=/liquibase/changelog/changelog.xml"]
    env:
    - name: LIQUIBASE_URL
      value: "jdbc:postgresql://postgres:5432/mydb"
    - name: LIQUIBASE_USERNAME
      value: "myuser"
    - name: LIQUIBASE_PASSWORD
      value: "mypassword"
    volumeMounts:
    - name: liquibase-changelog-volume
      mountPath: /liquibase/changelog
  containers:
  - name: my-app
    image: my-app:latest
    env:
    - name: DATABASE_URL
      value: "jdbc:postgresql://postgres:5432/mydb"
    - name: DATABASE_USERNAME
      value: "myuser"
    - name: DATABASE_PASSWORD
      value: "mypassword"
    ports:
    - containerPort: 8080
  volumes:
  - name: liquibase-changelog-volume
    configMap:
      name: liquibase-changelog

In this example, the init container is named liquibase and uses the liquibase/liquibase:latest image. The command to run is liquibase update --changeLogFile=/liquibase/changelog/changelog.xml. The environment variables are set to configure the connection to the database. These can also be configured using Kubernetes Secrets.

The main container is my-app, which uses the my-app:latest image and also has environment variables set to configure the connection to the database.

You should store your Liquibase ChangeLog in Kubernetes using a ConfigMap. This will allow you to mount the changelog file as a volume and allow the init container to access the changelog file.

Create a ConfigMap using the following command:

kubectl create configmap changelog-volume --from-file=changelog.xml

Once you have the pod definition in a file, you can create the pod using

kubectl apply -f my-pod-definition.yaml

It's important to note that the init container will run before the main container starts and will exit after the update is done, if the update fails the pod will fail too and you can check the logs of the init-container to see what went wrong.

Summary

By taking advantage of Kubernetes' init phase, you can sidestep the cause of stuck Liquibase locks and better fit into the infrastructure Kubernetes provides. Win-win.

That means you have a clear path to the automation, governance, compliance, security, and observability benefits Liquibase brings as a complete database DeVOps solution.

Updated June 2024; originally published February 2023.