
A slow pipeline is a tax that every developer pays on every change. People context-switch while they wait, batch up larger changes to avoid waiting again, and eventually stop trusting a build that fails for no clear reason. None of that shows up as a single outage, which is why it's so easy to live with.
The good news is that most pipeline slowness comes from a short list of repeat offenders. Here are eight of them, with fixes you can apply this week. The examples use GitHub Actions, but the ideas carry over to GitLab CI, Jenkins, CircleCI, and anything else.
1. Skipping caching
If every run downloads the same dependencies and rebuilds the same layers from scratch, you're paying for work that has already been done.
Most CI platforms can cache package managers with one extra line:
- uses: actions/setup-node@v7
with:
node-version: 24
cache: npm
- run: npm ci(Action versions move on, so check the action's releases page for the current major version before copying.)
For Docker builds, order your Dockerfile so the slow, rarely changing steps come first and your source code is copied last. That way the dependency layer stays cached until the lockfile changes.
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run buildFresh CI runners start with an empty local cache, so you can also push the build cache to a registry and reuse it:
# One-time setup: a builder that supports registry cache export
docker buildx create --name ci-builder --driver docker-container --use
docker buildx build --push \
--cache-from type=registry,ref=registry.example.com/app:buildcache \
--cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max \
-t registry.example.com/app:latest .The extra builder step matters: Docker's default build driver only supports registry caching when the containerd image store is enabled. In GitHub Actions, the docker/setup-buildx-action step handles this for you.
2. Running everything in sequence
Linting, unit tests, type checks, and security scans rarely depend on each other, yet many pipelines run them one after another. Total time becomes the sum of every step instead of the length of the slowest one.
Split independent work into parallel jobs and only chain the jobs that truly depend on earlier results:
jobs:
lint:
runs-on: ubuntu-latest
steps:
# checkout, setup, and cache steps omitted
- run: npm run lint
unit-tests:
runs-on: ubuntu-latest
steps:
- run: npm test
build:
needs: [lint, unit-tests]
runs-on: ubuntu-latest
steps:
- run: npm run buildIf one test suite is the bottleneck, shard it across several runners (Jest supports this natively from version 28 onward, and other runners such as Playwright offer a similar flag):
unit-tests:
runs-on: ubuntu-latest
strategy:
matrix:
shard: [1, 2, 3, 4]
steps:
- run: npx jest --shard=${{ matrix.shard }}/43. Running every check on every change
A one-line README fix doesn't need your full end-to-end suite. Treating all changes the same wastes minutes on each push and trains developers to ignore the pipeline.
Build a tiered pipeline:
On every pull request: fast checks such as lint, unit tests, and a build.
After merge or on a schedule: slower checks such as full end-to-end suites, performance tests, and deep security scans.
In monorepos, use path filters so a change to one service doesn't rebuild the others:
on:
pull_request:
paths:
- 'services/api/**'
- '.github/workflows/api.yml'One gotcha: if a workflow is marked as a required check but never runs because of a path filter, the pull request can get stuck waiting for it. Use a small always-running gate job as your required check, or configure the filtering inside the pipeline instead.
4. Living with flaky tests
A flaky test passes and fails on the same code. Each one costs reruns, wasted minutes, and a little more of the team's trust. Once developers believe "just rerun it" is the answer, real failures start slipping through.
Automatic retries can hide the problem rather than fix it. A healthier approach:
Track flakiness. Record which tests fail and then pass on retry.
Quarantine fast. Move a known flaky test out of the blocking path so it stops hurting everyone, but keep it visible and assigned to an owner.
Fix the cause. Common culprits are shared state between tests, hard-coded sleeps, real network calls, and time or ordering assumptions.
Set a policy. For example, a flaky test gets fixed or deleted within an agreed number of days.
A green build should mean something. If it doesn't, the pipeline isn't doing its job.
5. Rebuilding artifacts for every environment
If staging and production each build the code separately, then what you tested isn't exactly what you shipped. Different dependency resolutions, build-time variables, or even timing can produce subtly different results. It's also slower, since you repeat the build for every environment.
Build once, promote everywhere. Produce a single immutable artifact, tag it with the commit SHA, and move that same artifact through each environment. Only configuration should change.
docker build -t registry.example.com/app:${GIT_SHA} .
docker push registry.example.com/app:${GIT_SHA}
# Staging and production deploy the SAME image.
# Only environment-specific config differs.This also makes rollbacks simple: redeploy the previous artifact instead of rebuilding old code.
6. Leaving manual steps in the critical path
Every manual handoff adds waiting time: a ticket for a deployment window, a person who has to click "approve," a checklist run by hand. These steps also vary from person to person, which is where mistakes creep in.
Audit your release process and ask of each manual step: what is this protecting us from, and can a machine check it instead?
Replace manual QA checks with automated smoke tests that run right after deploy.
Use automated gates, such as passing tests, security scans, and health checks, instead of sign-offs.
Keep human approval only where compliance or risk genuinely requires it, and make it a single, fast step.
Consider progressive delivery (canary releases, feature flags) so that a bad release affects few users and can be rolled back automatically.
7. Starving the pipeline of compute (and wasting what you have)
Sometimes the pipeline is fast once it starts, but jobs wait in a queue for a runner. Other times the opposite happens: nobody has cancelled a stale run, and half your capacity is busy testing commits that were already superseded.
Cancel superseded runs. If a developer pushes again, the previous run is rarely useful.
Right-size runners. A heavy build on an underpowered machine can cost more in waiting than a larger runner costs in money.
Watch queue time separately from run time. They have different fixes.
Prefer ephemeral runners. Clean, disposable environments avoid mysterious "works on that one machine" failures.
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: trueUse this on pull request checks, not on deployment workflows. Cancelling a deploy halfway through can leave an environment in a half-updated state, so deployments should queue instead.
8. Never measuring or maintaining the pipeline
Pipelines tend to be built once and then only ever grow. Steps get added and never removed, YAML gets copy-pasted across repositories, and nobody can say how long a typical run takes or how often it fails.
You can't improve what you don't measure. Track a few numbers over time:
Median and 90th percentile pipeline duration
Queue time
Failure rate, and the share of failures caused by flaky tests or infrastructure rather than real bugs
Time from commit to production, which is the DORA metric lead time for changes
Set a team budget that feels right, for example "pull request checks finish in about ten minutes," and treat regressions as bugs. Give the pipeline an owner, review it periodically, and delete steps that no longer earn their keep.
To reduce copy-paste drift, move shared logic into reusable workflows or templates:
jobs:
ci:
uses: my-org/.github/.github/workflows/node-ci.yml@v1The shared workflow file needs an on: workflow_call trigger to be callable this way.
Quick wins checklist
Dependency and Docker layer caching enabled
Independent jobs run in parallel
Pull requests run fast checks; slow suites run after merge or nightly
Flaky tests are tracked, quarantined, and owned
One artifact is built once and promoted across environments
Manual steps reviewed, with automation replacing what it can
Superseded runs are cancelled automatically
Pipeline duration and failure rate are measured
Final thoughts
You don't need to fix all eight at once. Pull up your last 20 pipeline runs, find where the time actually goes, and start with the biggest single delay. Cutting a pipeline from thirty minutes to ten isn't just a nicer experience; it changes how often people merge, how small their changes are, and how safe every release feels.
Which of these slows your team down the most? Share your experience in the comments.
Comments(0)
Join the discussion by logging into your account.
No comments yet. Be the first to comment!