Guardrails for CI/CD

This is the ninth post in a series we’re calling CI/CD:DB. Read from the beginning.

The notion that database changes can be handled with safety and minimal intervention is crucial to maintaining the flow of changes through a CI/CD pipeline. In order to do this, you have to plan ahead for how you ensure that the changes stay within known parameters so that they don’t crash, bump into things, bump into each other, or misbehave in some way that will slow the overall flow down. That requires constructs within the pipeline itself that channel the flow without constricting it. A popular metaphor for those constructs is that of “Guardrails”.

Guardrails, not gates

The Guardrails metaphor is useful because it teaches us about the core principles of what we want to achieve. We need something perceived as an actual barrier when hit. A guardrail along a cliffside is vastly more effective and reassuring than a painted “guideline” in the same place. We need something that is visible to everyone so that we know which way to go. Guardrails are large and often have reflectors on them. We need something that does not slow or constrain progress. Guardrails run alongside the road, not across it. You can actually continue the guardrail metaphor in a number of ways, but one of the most important is that it serves as a contrast to another approach to safety — gates.

A lot of classic process and project management approaches rely on gates between stages. Gates have their uses but are problematic when you are trying to achieve a smooth flow. Gates definitely have strength. They are visible and frequently decorated to be special, ceremonial, and impressive. They sit right across a road, so they meter and control progress. Of course, gates are designed to keep things from passing unless they have permission. The problem with that approach is that everything that wants to go through must be inspected before permission to pass is granted. That leads to waiting, traffic jams, etc. as work sits idle waiting for permission to continue through the bottleneck of the gate.

Setting up guardrails

The trick with setting up guardrails is to know where your delivery process needs them and where it does not. The metaphor provides some conceptual guidance here. For example, a long straight section of road through a wide, flat desert is probably not a high priority for a guardrail, but, per the example above, a tight turn next to a cliff face probably is. Given the variability among apps, architectures, technology stacks, businesses, and industries, you will have to do some analysis work to figure out where the equivalent to those straights and curves are within your CI/CD pipeline. To that end, we can talk about some general concepts and look at some examples from this blog series so far.

Types of CI/CD guardrails

We can group Guardrails into two broad categories: Path Markers and Problem Stoppers. Path Markers provide direction and create a clear delineation of the smoothest road toward successful change delivery. Problem Stoppers serve as safety points that ensure that if you lose control, the worst that will happen is that you stop suddenly with some wrinkled bodywork (as opposed to plunging to your death over that cliff).

Path Markers

In CI/CD terms, Path Markers will take the form of well-defined automation points or the handoff points between automation systems. For example, having only one source code branch that feeds the CI job that produces deployable artifacts in the rest of the pipeline. These should be clear and very simple to explain.

Examples of CI/CD Path Markers:

• If your changes are ready to be part of a batch progressing toward PRODUCTION, put them in this place.
• If you do not put your changes in this place, they will never become part of a batch.
• This is what the CI system will do with your change.
• This is how you identify the post-CI artifact that includes your change.

With a clear set of statements, it is easy to communicate tasks and expectations — both of which are bounded by a set of procedural and technological constructs. Further, if the procedural or technological constructs result in “traffic jams” or slowdowns, we know where the Path Markers are and can adjust them accordingly. Maybe we “widen the path”, raise the “speed limit”, or eliminate a “toll booth”. Regardless of the analogy, the key point is that because we know where the path is and we have structures to help people stay on the path, we can have efficient conversations about how to improve the flow.

Problem Stoppers

Problem Stoppers are the guardrails in our CI/CD process that take a more active role along the path. Since their job is not so much to define the path as to ensure that people stay on it (particularly when the path is perilous) they are the things people hit and get knocked back a bit. While that may not be fun, we have to remember that as unpleasant as wrinkled cars can be, they are a lot better than a plunge over a cliff.

Examples of CI/CD Problem Stoppers:

• Code scans or other quality inspections during the CI process
• Validating that batches of changes work before creating a deployment artifact for them
• Verifying the checksums of deployment artifacts before deploying them
• Actively refreshing environments to make sure they are representative of PRODUCTION

Problem Stoppers, largely because they are active, can be more visible and therefore perceived as more important in the pipeline than Path Markers. However, the truth is that people should almost never hit them. In fact, if someone does hit them, it can interrupt the flow while the impact is cleaned up. Consider that you don’t think of a guardrail on the road as something you deliberately bounce off of every time you want to make a turn. In fact, if people are frequently hitting a guardrail, the proper solution is to re-engineer the road so that the frequency goes down or, ideally, the guardrail is simply no longer needed. So, it is actually better to look at Problem Stoppers as expensive necessities.

Wrapping up

The metaphor of guardrails as a way to understand where to place protective constructs along your CI/CD pipeline is a powerful mental model. You can use it to build a smooth path, from change creation to eventual deployment into production, by carefully considering how to guide the changes down a safe path that avoids trouble whenever possible while having strong measures to ensure that when trouble inevitably happens, it does not get out of hand. In the next post, we’ll cover visibility into the CI/CD pipeline.