Using Liquibase in Kubernetes
Embedding Liquibase into the startup process of an application is a very common pattern for good reason. Once you set up Liquibase to deploy on app startup, your database state will always match what your code expects. Liquibase even ships with built-in support for this with the Spring Boot and Servlet integrations.
Database Change Lock
Another way Liquibase supports this pattern is to include a locking mechanism. This lock ensures that if multiple servers start simultaneously, they won’t run into problems as they both try to apply the same database changes.
The locking system uses a DATABASECHANGELOGLOCK table as the synchronization point, with the service setting the LOCKED column to 1 when it takes the lock and setting it to 0 when it finishes.
As long as the process that has set the LOCKED column to 1 isn’t killed before it has a chance to set back to 0, this works fine. When it’s not working fine, all of the other Liquibase processes (including a newly restarted process on the same machine) will just continue to wait for a 0 value which will never come. The only way to recover from this scenario is by running the Liquibase unlock command or updating the DATABASECHANGELOGLOCK table manually.
Kubernetes: “When in doubt, kill the process”
Historically, the stuck locks have not been a problem because the Liquibase process was rarely killed. If it was killed, it was done manually and easier to recover from. Tools like Kubernetes have taken a philosophy of “when in doubt, kill the process” and this causes problems.
While we are working on better handling killed processes, Kubernetes provides additional options to better fit Liquibase into the application startup process.
At the heart of this issue is Kubernetes’ expectations of pods starting quickly. Rightly so. You WANT to deal with slow startup issues quickly. However, you also need to give your pods the time they need to do potentially time-consuming initialization work, such as migrating your database.
The Solution: Init Containers
The best practice to run Liquibase in Kubernetes is to use an init container in Kubernetes. To do so, create a Pod that includes the Liquibase init container and your main application container. The init container will run Liquibase to update the database schema before the main application container starts.
Here is an example of how to configure a Pod with a Liquibase init container:
apiVersion: v1 kind: Pod metadata: name: my-app-pod spec: initContainers: - name: liquibase image: liquibase/liquibase:latest command: ["liquibase", "update", "--changeLogFile=/liquibase/changelog/changelog.xml"] env: - name: LIQUIBASE_URL value: "jdbc:postgresql://postgres:5432/mydb" - name: LIQUIBASE_USERNAME value: "myuser" - name: LIQUIBASE_PASSWORD value: "mypassword" volumeMounts: - name: liquibase-changelog-volume mountPath: /liquibase/changelog containers: - name: my-app image: my-app:latest env: - name: DATABASE_URL value: "jdbc:postgresql://postgres:5432/mydb" - name: DATABASE_USERNAME value: "myuser" - name: DATABASE_PASSWORD value: "mypassword" ports: - containerPort: 8080 volumes: - name: liquibase-changelog-volume configMap: name: liquibase-changelog
In this example, the init container is named
liquibase and uses the
liquibase/liquibase:latest image. The command to run is
liquibase update --changeLogFile=/liquibase/changelog/changelog.xml. The environment variables are set to configure the connection to the database. These can also be configured using Kubernetes Secrets.
The main container is
my-app, which uses the
my-app:latest image and also has environment variables set to configure the connection to the database.
You should store your Liquibase ChangeLog in Kubernetes using a ConfigMap. This will allow you to mount the changelog file as a volume and allow the init container to access the changelog file.
Create a ConfigMap using the following command:
kubectl create configmap changelog-volume --from-file=changelog.xml
Once you have the pod definition in a file, you can create the pod using
kubectl apply -f my-pod-definition.yaml
It’s important to note that the init container will run before the main container starts and will exit after the update is done, if the update fails the pod will fail too and you can check the logs of the init-container to see what went wrong.
By taking advantage of Kubernetes’ init phase, you can sidestep the cause of stuck Liquibase locks AND better fit into the infrastructure Kubernetes provides. Win-win. Give it a try and let us know what you think or if we should add any other details that will be helpful for other users.