A release pipeline failed after every test had passed.

The application code was fine. The packaging step was fine. The failure happened when the release job installed its tooling at runtime and pulled newer plugin versions than the container’s Node.js runtime could execute.

That looked like a one-line dependency pin. It was actually evidence of a much larger problem.

The same release block had been copied across a sizeable GitLab group, the project template contained the same defect, and only one repository had moved to the shared CI/CD components that were intended to replace those copied pipelines.

Over the next two days I turned that single failure into a repeatable workflow for auditing, testing and safely backporting CI/CD changes across 37 repositories.

This article describes a real internal standardisation exercise. Repository names, hostnames, package registries and operational details have been generalised.

The failure that exposed the estate

The failed job used an older release container with Node.js 19. During the job it ran an unpinned install similar to this:

before_script:
  - npm install @semantic-release/git @semantic-release/changelog -D

The latest plugin majors required a newer Node.js runtime. npm emitted an EBADENGINE warning but continued installing the packages. The job then failed while loading the plugins:

SyntaxError: Unexpected token 'with'

Pinning compatible plugin versions stopped the immediate failure:

before_script:
  - >-
    npm install
    @semantic-release/git@10.0.1
    @semantic-release/exec@6.0.3
    @semantic-release/changelog@6.0.3
    -D

That fixed one repository, but it did not answer the more important questions:

  • Which other repositories had copied the same release job?
  • Which apparently dormant repositories were carrying the same deferred failure?
  • Was the project template still creating new repositories with the defect?
  • Why had an earlier fix in another repository never propagated?
  • How could I test changes without triggering production schedules, deployments or package publishing?

The problem was no longer a broken job. It was configuration drift across an estate.

What the audit found

I queried the GitLab group and collected only the files needed for a CI/CD audit: .gitlab-ci.yml, semantic-release configuration and the main Python packaging files.

The resulting inventory covered 37 projects. Eighteen had a root CI configuration. The initial scan found:

  • nine repositories with the same unpinned semantic-release failure mode;
  • the same defect in the Python project template;
  • two repositories that had already pinned the plugins but were still tied to the old runtime;
  • one repository using the shared Python CI/CD component;
  • several hand-built pipeline variants with different Python versions, release mechanisms and trigger syntax;
  • active packaged projects with no visible root pipeline;
  • scheduled jobs mixed into normal CI files, including jobs with production side effects.

The most important finding had nothing to do with the number of broken repositories: the group already had a reusable component model, but most repositories still owned private copies of shared pipeline logic.

A copied pipeline is effectively a dependency that cannot be centrally patched.

Why dormant repositories still matter

Several affected repositories had not changed for months. It would have been easy to deprioritise them.

That would have been a mistake.

A dormant repository with a broken release path is a deferred incident. The next developer to merge a small fix may discover the problem months later, when the original failure and its context have been forgotten.

Dormant repositories therefore remained in the risk inventory, even when they were not first in the migration order.

I split the work into three horizons

Trying to replace every pipeline immediately would have mixed urgent remediation with structural migration. I separated them instead.

Horizon 1: stop the immediate failures

The first horizon was deliberately mechanical:

  1. pin the known compatible semantic-release plugin versions;
  2. replace the obsolete singular branch setting with the supported branches configuration where required;
  3. fix the project template early, because templates multiply defects;
  4. verify the release job actually loaded semantic-release and reached a valid outcome.

A valid outcome did not have to create a release. Either of these was acceptable:

The next release version is ...
There are no relevant changes, so no new version is released

The important test was that semantic-release ran rather than crashing during plugin loading.

Horizon 2: migrate to shared components

The second horizon addressed the underlying drift.

Pure Python package repositories were a good fit for a shared component that supplied linting, tests, builds, publishing and release automation. Service repositories with unusual deployment models could adopt only the shared release component and retain their local build or deployment jobs.

The target was a small set of composable components with typed inputs and a narrow local configuration.

A consuming pipeline should look more like this:

include:
  - component: >-
      $CI_SERVER_FQDN/platform/ci-components/python@2.1.0
    inputs:
      python-build-version: "3.12"

A repository-specific scheduled job can remain in the same .gitlab-ci.yml. Included configuration and local jobs can coexist, so standardisation does not require deleting legitimate local behaviour.

Horizon 3: enforce and monitor

The final horizon turns a one-off audit into a control:

  • run the drift scan on a schedule;
  • flag unpinned release tooling and moving component references;
  • publish and document the approved components;
  • use semantic versions for components rather than @main;
  • let dependency automation propose component upgrades through merge requests;
  • enforce protected branches and repository settings consistently;
  • provide a short new-repository path that starts from the standard component.

The objective reaches past repairing the current estate: the same class of drift should become visible before it causes another release failure.

The sandbox was the control plane

Changing CI/CD across many repositories is risky because a pipeline is executable infrastructure. A test commit can publish a package, rotate a password, deploy an application or call a production API.

I therefore created a separate GitLab group as a disposable sandbox.

The production repositories stayed read-only throughout the experiment. Repositories were mirrored into the sandbox, changed there, and tested with real pipelines. Only verified file differences were later proposed back to the original repositories through merge requests.

The workflow was driven by a JSON manifest containing project metadata, scope, protected branches, variable metadata, schedule inventory, sandbox location and migration status.

{
  "source_group": "platform/automation",
  "sandbox_group": "platform/ci-sandbox",
  "projects": [
    {
      "rel_path": "example-service",
      "default_branch": "main",
      "in_scope": true,
      "status": "verified-green",
      "sandbox": {
        "id": 12345,
        "path_with_namespace": "platform/ci-sandbox/example-service"
      }
    }
  ]
}

Every later script keyed off this manifest. That made the process resumable and prevented a collection of loosely related scripts from developing separate ideas about scope or state.

Mirroring Git history mattered

A normal shallow clone was not enough for the sandbox creation step.

Release automation derives previous versions from Git tags. If the sandbox contained only the default branch without the historical tags, semantic-release could calculate an incorrect next version and the test would not represent production.

The sandbox copy therefore used a mirror clone and mirror push against a new, empty target project:

git clone --mirror $SourceUrl $MirrorPath
git -C $MirrorPath push --mirror $TargetUrl

This preserved branches, tags and other refs needed to make the release test meaningful.

git push --mirror is intentionally powerful. I only used it against newly created sandbox projects, never against an existing production repository.

Safety rules were executable, not advisory

The sandbox had six important guardrails.

1. Do not copy pipeline schedules

Some source pipelines contained scheduled jobs with production effects. The sandbox inventory recorded those schedules so a later test could assert that the sandbox had none.

2. Start with CI disabled

Mirrored projects were inert until their variables, branch protections and safety policy had been checked. Enabling CI did not trigger a pipeline; it only made a later manual or push-triggered test possible.

3. Copy variables with a default-deny policy

Pipelines needed some read-only build and dependency credentials, but the sandbox did not need deployment, token-rotation or production publishing credentials.

The copy script therefore used an allowlist and explicit deny rules. It preserved metadata such as variable type, protection, masking, raw expansion and environment scope, but it did not write secret values into the manifest or logs.

4. Recreate protected branches

Protected variables are only available to pipelines on protected branches or tags. A sandbox copy with an unprotected main branch can fail in a confusing way because required variables silently disappear.

The branch protection state therefore formed part of the sandbox reconstruction.

5. Refuse production paths

Scripts that triggered pipelines or pushed commits checked that the target namespace was beneath the sandbox group and was not beneath the source group.

if ($ProjectPath -notlike "$SandboxGroup/*") {
    throw "Refusing to run outside the sandbox group."
}

if ($ProjectPath -like "$SourceGroup/*") {
    throw "Refusing to run against a source repository."
}

6. Check the live sandbox before every first run

A read-only safety script queried GitLab and failed if it found:

  • any pipeline schedule;
  • any denied variable;
  • an excluded high-risk repository;
  • a remote mirror pointing back to production;
  • an unexpected project with CI already enabled.

Warnings are easy to overlook. A non-zero exit code is much harder to ignore.

Verification needed a semantic verdict

A green pipeline alone was not enough.

The script that watched a sandbox pipeline downloaded every failed trace and the release trace. It then classified the release job using both job status and anchored log messages.

This distinction mattered because release logs and generated changelogs can contain old commit messages describing earlier failures. A naive search for Unexpected token 'with' could find that phrase in a successful changelog and incorrectly label the job as broken.

The verifier therefore evaluated status first:

if ($ReleaseJob.status -eq 'success') {
    if ($Trace -match 'Published release|The next release version is') {
        $Verdict = 'fixed'
    }
    elseif ($Trace -match 'no release necessary|no relevant changes') {
        $Verdict = 'fixed'
    }
}
elseif (
    $Trace -match "An error occurred while running semantic-release: SyntaxError: Unexpected token 'with'"
) {
    $Verdict = 'baseline-bug'
}

Only a valid release verdict advanced the manifest to verified-green.

Backports were deliberately narrow

Once a sandbox repository was verified, the production change still had to be proposed safely.

The backport script compared the verified sandbox files with the live production default branch, not with the original audit snapshot. That avoided proposing changes that another developer had already merged after the audit was collected.

For the immediate fix, only these files were eligible:

.gitlab-ci.yml
.releaserc.json

The script refused to proceed if a diff contained sandbox-only markers or introduced allow_failure. It defaulted to a review-only mode and required an explicit switch before pushing a branch and opening a merge request.

It also refused to push directly to a default branch.

This produced small merge requests with an evidence trail: the exact live diff, the sandbox pipeline URL, and a description based on the change the repository actually needed.

A red merge request did not automatically mean the change was wrong

Several backport merge requests encountered failing security jobs even though the same repositories had passed in the sandbox.

The first comparison checked whether the same job already failed on the target branch. That was useful, but not conclusive. Vulnerability scanners consume live advisory databases, so a target-branch pipeline from several months earlier can be a stale baseline.

The workflow therefore downloaded the failing traces and classified the content:

  • references to semantic-release or the pinned plugins required investigation;
  • CVE or vulnerability findings were likely unrelated dependency drift;
  • ambiguous logs stayed unclassified until read.

The lesson was simple: a changed status is evidence, not causation.

What I would keep from this approach

Several parts of the workflow proved more valuable than the original fix.

Treat the template as infrastructure

A defect in one repository affects one repository. A defect in a template affects every future repository created from it.

Templates should be audited and upgraded before the long tail of consumers.

Make the manifest the source of truth

The manifest connected discovery, sandbox creation, verification and backporting. It also made interrupted work safe to resume.

Build refusal paths into automation

The most useful scripts did not only automate the happy path. They refused production namespaces, default branches, denied variables, schedules and sandbox-only changes.

Version the shared component

Moving copied YAML into a component removes per-repository drift, but consuming the component from @main creates another moving dependency. Shared components should be released, pinned and upgraded through reviewed merge requests.

Separate remediation from migration

The dependency pin was small enough to backport quickly. Component adoption was a larger design change and belonged in a separate wave. Keeping those changes separate made review and rollback easier.

The resulting content series

I have documented the reusable parts of the work separately:

References