How CI/CD works,
on one workbench

Everything starts with a change pushed to the shared repository. The push is an event, and the CI system is subscribed to it: the repository host emits an event (a webhook, for an external CI server), and a pipeline run starts for that exact commit.

In most teams the first run happens before the merge. Opening a pull request triggers the build and tests on the proposed change, and branch protection refuses the merge until those required checks are green. The run on main after the merge then carries the change the rest of the way down the line.

On the line, each blank in the magazine is a commit. The gate lifts, one blank drops onto the belt, and the run gets a number and the commit's short hash. That hash travels with the part all the way down the line, so every result can be traced back to the source that produced it.

In real pipelinesgit push → a push or pull_request event → the CI system queues a run for commit a1f3c9e. Required status checks and branch protection rules gate the merge.

Small, frequent commits keep each run fast and make a failure easy to pin on one change. That is the “integration” in continuous integration: everyone merges into main often (trunk-based development), and the machine checks every merge.

The build turns source into something that can run: it fetches dependencies, compiles, bundles and minifies. Like a press, it should be repeatable. The same blank under the same die always gives the same part. (Bit-for-bit reproducible builds go further and also pin timestamps, toolchains and file ordering. Most teams aim for repeatable and get there with lockfiles and pinned images.)

The build runs on a clean, ephemeral runner every time, never on someone's laptop. That makes “works on my machine” a problem the pipeline catches, not one your users find. Caches for dependencies and build layers keep a clean build fast without making it dirty.

In real pipelinesnpm ci && npm run build, mvn package, go build ./..., cargo build --release; dependency caching with actions/setup-node (cache: npm) or Docker layer cache.

Pin everything the build pulls in: dependencies with a lockfile (npm ci fails if it doesn't match package.json), base images by digest, and CI actions by version or commit SHA. Then next month's rebuild of the same commit gives the same output.

Automated tests are the gauge. The probe drops onto the part and the dial reads it against a tolerance, here 13.00 ± 0.20 mm. Real suites check thousands of these tolerances: unit tests, integration tests, type checks, linters and security scans.

Suites are usually shaped like a pyramid: many fast unit tests at the base, fewer integration tests that exercise real dependencies (a database in a container, for example), and a thin layer of slow end-to-end tests on top. Static checks such as linting, type checking and security scanning run alongside them, often in parallel jobs.

When a reading is out of tolerance, the kicker throws the part into the reject bin and the line stops for that commit. Nothing downstream runs, no artifact is made, and the version already in production stays live. A red build blocks the change, never the users.

In real pipelinesnpm test, pytest, go test ./...; plus eslint, tsc --noEmit, dependency audits (npm audit, Dependabot), SAST (CodeQL, Semgrep) and secret scanning.

Fail fast: run the quickest checks first so a broken commit is rejected in seconds. A flaky test (one that sometimes fails for no reason) is a gauge you can't trust. Fix it or quarantine it, and never “retry until green”.

A part that passes is packed into a crate and labelled with a version. That crate is the artifact: a container image, a .jar, a zip or a binary. It must be treated as immutable. Once it is labelled, nobody opens it again.

Immutability has to be enforced, though. A container tag like app:1.4.2 is only a movable pointer, and anyone with push rights can point it at a different image. The image's digest (app@sha256:…) is a hash of its content and can never change. Pipelines deploy by digest, or turn on tag immutability in the registry.

This is the most important habit in delivery: build once, deploy many. The same crate goes to staging and then to production. Nothing is rebuilt between environments, so what you tested is exactly what you ship. What changes between environments is configuration (URLs, credentials, feature flags), which is injected at deploy time and never baked into the crate.

In real pipelinesdocker build -t registry/app:1.4.2 . then docker push, which prints the sha256 digest; or upload to an artifact store (npm, Maven, S3). Registries such as ECR, GHCR and Artifactory can reject overwrites of existing tags.

Tag artifacts with something you can trace: a semantic version, the commit hash, or both, and record the digest. Version numbers are never reused, even for a release that was rolled back.

The crane is the deployment tool: it moves crates and never changes what's inside. Its first stop is staging, an environment built like production but with no real users on it.

There the artifact gets smoke tests and end-to-end checks against real databases, real networking and real configuration. Bugs that only appear when the pieces run together show up here, including database migrations, which should run in staging before they ever touch production.

Staging uses its own configuration and secrets, injected by the deployment. The crate is the same one production will get; only the settings around it differ.

In real pipelinesdeploy.sh staging app@sha256:… then smoke-test.sh https://staging.example.com; environments with their own secrets in GitHub Actions, GitLab or Argo CD; short-lived preview environments per pull request.

Keep staging as close to production as you can afford: same infrastructure code, same migrations, a realistic data shape. Every difference between them is a place where a bug can hide.

Production has two bays, often called blue and green. One is live and the lever (a load balancer, router or service selector) sends all traffic to it. The crane retires whatever was sitting in the idle bay, sets the new crate there, waits for its health checks to pass, and only then does the lever move: new requests switch in one step, and requests already in flight on the old bay are allowed to finish (connection draining), so users see no downtime.

The old version stays in its bay, warm and ready. That is what makes the next step cheap. It also means both versions may briefly run side by side against the same database, so every change has to be backward compatible with the version before it.

Blue/green is one strategy among several. Choosing a deployment strategy, after the stations, compares them.

In real pipelinesBlue/green: AWS CodeDeploy, Azure App Service slots, Argo Rollouts. Rolling updates: Kubernetes Deployment (pods are replaced a few at a time). Canary: Argo Rollouts, Flagger, weighted load-balancer rules (1 %, 10 %, 50 %, 100 % of traffic).

Continuous delivery means every green run produces a crate that is ready to ship, and a person presses the button. Continuous deployment means the line ships it with no one pressing anything.

Shipping isn't the end of the pipeline. The dial watches the live bay's error rate under real traffic, with a red mark at the threshold the team agreed on. Here that threshold is 1 % of requests failing.

Some bugs only appear under real load, real data or real users. The tests passed and staging looked fine, but the dial climbs anyway. Watching for a few minutes after every deploy is how you find out.

Teams usually watch the four golden signals: latency, traffic, errors and saturation. They compare them with a service level objective (SLO), such as “99.9 % of requests succeed over 30 days”. In mature pipelines a machine reads the dial: automated analysis compares the new version with the old one and triggers the rollback itself.

In real pipelinesError rate, latency (p95/p99), traffic, saturation; dashboards and alerts in Prometheus + Grafana, Datadog or CloudWatch; OpenTelemetry traces; automated canary analysis in Argo Rollouts or Flagger.

Decide the threshold before you deploy, not while the dial is climbing. Tag every metric and log with the version, so a spike points straight at the release that caused it.

When the dial crosses the red mark, the lever flips back to the previous bay. New requests go to the last good version as soon as the router switches (seconds with a load balancer, longer if the switch depends on DNS caches expiring), and nothing had to be rebuilt or redeployed. The faulty crate sits idle until the next deploy retires it.

The lever only moves code, not data. If the release changed the database schema or wrote data in a new format, flipping back is safe only when the old version can still read it. That is why migrations are made backward compatible (see below).

Then the fix goes through the whole line again as a new commit, with a new version number. A rollback undoes a deployment, not history. When a rollback isn't possible, teams roll forward instead: they ship a small fix quickly through the same line.

In real pipelinesswitch-traffic.sh previous, kubectl rollout undo deployment/app, argo rollouts abort, a feature flag turned off, or redeploying the last good digest.

A rollback you have never practised is not a plan. Automate it, and run it on purpose now and then. Feature flags give you a second, finer lever: turn off one feature without moving any traffic.