February 20, 2023

Using Liquibase in Kubernetes

Embedding Liquibase into the startup process of an application is a very common pattern for good reason. Once you set up Liquibase to deploy on app startup, your database state will always match what your code expects. Liquibase even ships with built-in support for this with the Spring Boot and Servlet integrations.

Database Change Lock

Another way Liquibase supports this pattern is to include a locking mechanism. This lock ensures that if multiple servers start simultaneously, they won't run into problems as they both try to apply the same database changes. 

The locking system uses a DATABASECHANGELOGLOCK table as the synchronization point, with the service setting the LOCKED column to 1 when it takes the lock and setting it to 0 when it finishes. 

As long as the process that has set the LOCKED column to 1 isn't killed before it has a chance to set back to 0, this works fine. When it's not working fine, all of the other Liquibase processes (including a newly restarted process on the same machine) will just continue to wait for a 0 value which will never come. The only way to recover from this scenario is by running the Liquibase unlock command or updating the DATABASECHANGELOGLOCK table manually.

Kubernetes: "When in doubt, kill the process"

Historically, the stuck locks have not been a problem because the Liquibase process was rarely killed. If it was killed, it was done manually and easier to recover from. Tools like Kubernetes have taken a philosophy of "when in doubt, kill the process" and this causes problems.

While we are working on better handling killed processes, Kubernetes provides additional options to better fit Liquibase into the application startup process.

At the heart of this issue is Kubernetes' expectations of pods starting quickly. Rightly so. You WANT to deal with slow startup issues quickly. However, you also need to give your pods the time they need to do potentially time-consuming initialization work, such as migrating your database.

The Solution: Init Containers

The best practice to run Liquibase in Kubernetes is to use an init container in Kubernetes. To do so, create a Pod that includes the Liquibase init container and your main application container. The init container will run Liquibase to update the database schema before the main application container starts.

Here is an example of how to configure a Pod with a Liquibase init container:

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
spec:
  initContainers:
  - name: liquibase
    image: liquibase/liquibase:latest
    command: ["liquibase", "update", "--changeLogFile=/liquibase/changelog/changelog.xml"]
    env:
    - name: LIQUIBASE_URL
      value: "jdbc:postgresql://postgres:5432/mydb"
    - name: LIQUIBASE_USERNAME
      value: "myuser"
    - name: LIQUIBASE_PASSWORD
      value: "mypassword"
    volumeMounts:
    - name: liquibase-changelog-volume
      mountPath: /liquibase/changelog
  containers:
  - name: my-app
    image: my-app:latest
    env:
    - name: DATABASE_URL
      value: "jdbc:postgresql://postgres:5432/mydb"
    - name: DATABASE_USERNAME
      value: "myuser"
    - name: DATABASE_PASSWORD
      value: "mypassword"
    ports:
    - containerPort: 8080
  volumes:
  - name: liquibase-changelog-volume
    configMap:
      name: liquibase-changelog

In this example, the init container is named liquibase and uses the liquibase/liquibase:latest image. The command to run is liquibase update --changeLogFile=/liquibase/changelog/changelog.xml. The environment variables are set to configure the connection to the database. These can also be configured using Kubernetes Secrets.

The main container is my-app, which uses the my-app:latest image and also has environment variables set to configure the connection to the database.

You should store your Liquibase ChangeLog in Kubernetes using a ConfigMap. This will allow you to mount the changelog file as a volume and allow the init container to access the changelog file.

Create a ConfigMap using the following command: 

kubectl create configmap changelog-volume --from-file=changelog.xml

Once you have the pod definition in a file, you can create the pod using

kubectl apply -f my-pod-definition.yaml

It's important to note that the init container will run before the main container starts and will exit after the update is done, if the update fails the pod will fail too and you can check the logs of the init-container to see what went wrong.

Summary

By taking advantage of Kubernetes' init phase, you can sidestep the cause of stuck Liquibase locks AND better fit into the infrastructure Kubernetes provides. Win-win. Give it a try and let us know what you think or if we should add any other details that will be helpful for other users.

Nathan Voxland
Nathan Voxland
Share on: