arnau díaz

this is ci

2025 / 10 / 21  ·  trust  ·  3 min

A pipeline can pass every test and still miss what broke. I have watched it happen more than once: bump a library version, change an environment variable, edit a path in a workflow. Everything green, and none of it covered the thing I actually did.

That is what changed how I think about CI. I used to treat it as running the tests on GitHub. Now I treat it as noticing when something changed. Tests tell me the code is right; CI is that, made continuous.

Everyone having the same code, tagged and released, on a setup that reproduces, are consequences of CI. I like them. They are not the point.

What I try to test

CI starts working for me when every change makes a test fail. Pipelines, environments and tooling are how I get there, not the thing I am after.

So I test everything I can: code, operations, environments, migrations and rollbacks. GitHub itself, the releases and tags and workflows. Observability, the logs and the metrics. Binary size.

I keep linting and formatting inside the test rather than beside it. Same with benchmarks, so a performance regression fails like anything else. I test the Docker setup. I test the pipelines, so that a change to a script or an action gets caught by the thing it changed.

If it can break my workflow, I want it tested.

What I am aiming at:

  • move from AWS to Azure, a test fails
  • bump a library and something breaks, a test fails
  • change a path in a pipeline, a test fails
  • run a migration and forget a prerequisite, a test fails
  • emit a log or metric I did not expect, a test fails

My limit is time and how good I am at this. Yours will be somewhere else.

Detecting changes in practice

So how do I make a test fail for code that has no test yet? Coverage.

Coverage does not measure correctness, and it does not tell me how complete my testing is. It tells me which lines the tests execute. That turns out to be enough, because when I add or remove code the number moves. I fail the build when it drops below a threshold, and then a change has to be dealt with instead of merged quietly. Google’s risk-driven testing points the same instinct at where to spend the effort.

The same trick works for removing or changing behavior. What I want is for the change to be visible.

This is CI

If something changes, a test must fail or force me to add a check. I have read that sentence back to myself a lot. For me, that is CI.