November 22, 2021

Database DevOps is Difficult^2 — The Journey that Brought Me to Liquibase

Prior to my first official 100% tech job, I was always the technical person in a non-technical department. I spent a lot of time generating, recording, manipulating, and reporting on data. This helped me understand just enough to follow along and ask some pretty annoying questions once I made the pivot to being a full-time software professional.

The Old Software Silos

I discovered while working at a small CRM company that the hidden blessing in my ignorance was that I’d never experienced the siloed roles that had developed inside the software factory. We were split into three main divisions: development/engineering, system administration, and database administration. To me, they were all “programmers” — they had the skills I didn’t to deliver the user experience I wanted to deliver.

I quickly learned that these “programmers” did not see themselves as all being the same.

Developers built the new features and fixed product bugs
Sys admins made sure the Product team had the machines and systems needed to run effectively, and securely (often by resisting changes submitted by the developers).
Database administrators (DBAs) were magical oracles who worked in some mythical walled land and spoke a secret language. (There was an unspoken rule in solution design to do everything possible to avoid having to submit a change to them.)

DBAs: Protectors of the Database

This description of DBAs is perhaps a tad unfair — it makes it seem as if they were scary people. They were not, at all! I loved working with them and learning from them. They were merely doing their job and doing it extremely well: Protect the database at all costs. While an error in code from a developer might cause a bad user experience, a bug, or a security vulnerability, an error in the database can literally destroy an entire business. Even worse, the database could also destroy the efforts of our client organizations doing important things like fighting for marriage equality, stopping Ebola, and preventing animal abuse (to name just a few).

Today: Automation Everywhere

Fast forward 10+ years and the industry has definitely changed. Developers and system administrators have made major strides in working together. They now recognize that delivering quality products to users more quickly requires their combined skills and knowledge. These are the various practices (and culture and tools) that now make up the elusive world of DevOps. We are all buzzing around talking about ephemeral this and gitops that. It’s great! Automation everywhere!

The Database is Still Left Behind

And then there’s the database. After working at the small CRM company, I was fortunate to spend several years working at CloudBees, where I often helped teams map out the current and future software delivery pipelines. We talked through code commit, unit testing, integration testing, security scanning, artifact storage, release management, deployment, infrastructure, and observability. But when I asked these teams, “what are you doing about the databases” the room either went silent or whoever was in charge quickly dismissed the topic as one they just weren’t able to handle at the time. It was just too messy — technically and politically. I didn’t have a good answer for them either.

The reality is, no one wants to mess up the data. While automating your integration testing might result in accidentally sending an email to your whole user base announcing “Integration Test Email #1” (#hugops to HBO Max), automating your database changes and deployments could result in irreversible, permanent, very, very serious damage. DevOps is difficult. Databases are difficult. Q.E.D., Database DevOps is difficult^2.

Why I Joined Liquibase

Database DevOps is difficult — this is exactly why I decided to come work at Liquibase. We have come too far in the journey for continuous deployment to continue to avoid the database bottleneck. The fact that it is not an easy problem to solve is evidence that it is exactly what we should focus on. I am very much looking forward to putting our heads together, rolling up our sleeves, leaving our egos and silos at the door, and taking this movement into its next evolution (iteratively, safely, and with the ability to rollback when we screw up!)

Help make database DevOps a bit easier: