A copied pipeline looks convenient when the first few repositories need the same build and release jobs.

The cost appears later. Each repository gradually acquires its own runtime versions, package commands, release plugins, branch rules and local fixes. A defect patched in one copy can remain dormant in every other copy until those pipelines run again.

That was the pattern I found while auditing CI/CD drift across a GitLab group. Nine repositories carried the same release-tooling failure, two had a local fix that still depended on an old runtime, and only one project had adopted the shared component model already available to the group.

The sustainable fix was to stop maintaining copies altogether: move common pipeline behaviour into versioned GitLab CI/CD components and leave each consuming repository with a small, explicit contract.

The examples use fictional project paths and package registries. The design is based on a real private-repository standardisation exercise.

What should be centralised?

A component should own behaviour that is common across a repository class, such as:

  • selecting the supported build image;
  • installing dependencies consistently;
  • running linting, tests and package builds;
  • publishing test reports and build artifacts;
  • applying standard release rules;
  • generating releases and package publications;
  • exposing controlled inputs for genuine repository differences.

The consuming repository should still own application-specific facts:

  • its package metadata and dependencies;
  • test configuration;
  • optional feature flags;
  • deployment destinations that are genuinely unique;
  • any extra jobs that are not part of the common lifecycle.

The boundary matters. A component with dozens of repository-specific conditionals becomes another monolith. A component with no inputs encourages consumers to override its internals. The useful middle ground is a stable workflow with a small, documented input surface.

The copy-paste model

A typical repository-owned pipeline can contain hundreds of lines:

stages:
  - validate
  - test
  - build
  - release

variables:
  PYTHON_VERSION: "3.10"

lint:
  image: python:${PYTHON_VERSION}
  stage: validate
  script:
    - pip install ruff
    - ruff check .

unit-test:
  image: python:${PYTHON_VERSION}
  stage: test
  script:
    - pip install -r requirements-dev.txt
    - pytest --junitxml=reports/junit.xml
  artifacts:
    reports:
      junit: reports/junit.xml

release:
  image: node:19
  stage: release
  before_script:
    - npm install @semantic-release/git @semantic-release/changelog -D
  script:
    - npx semantic-release
  only:
    - main

Even when this file began as a standard template, it is now a private fork. Updating the template does not update the repository.

Across an estate, the same block tends to drift in several dimensions:

Dimension Typical drift
Build runtime Python 3.8, 3.9, 3.10, 3.11 and 3.12
Release runtime Multiple Node.js majors
Dependency installation Pinned, unpinned and globally installed tools
Rules only, except, rules and custom branch checks
Publishing Twine, Poetry, npm or bespoke scripts
Release configuration Different branch keys and plugin sets
Artifacts Different paths, expiry periods and report formats

This is why line-by-line standardisation does not scale. Every repository remains responsible for future maintenance.

The component contract

A consuming repository can instead include a versioned component and pass only the values that differ:

include:
  - component: $CI_SERVER_FQDN/platform/ci-components/python-package@1.4.0
    inputs:
      python-version: "3.12"
      package-directory: "."
      run-lint: true

This file answers three useful questions immediately:

  1. Which shared workflow does the repository use?
  2. Which released version does it depend on?
  3. Which supported variations has it selected?

The implementation lives in the component project. A simplified component might start like this:

spec:
  inputs:
    python-version:
      description: "Python runtime used by build and test jobs"
      default: "3.12"
    package-directory:
      description: "Directory containing the package metadata"
      default: "."
    run-lint:
      type: boolean
      default: true
---

python-lint:
  image: python:$[[ inputs.python-version ]]
  stage: test
  rules:
    - if: '"$[[ inputs.run-lint ]]" == "true"'
  script:
    - cd "$[[ inputs.package-directory ]]"
    - python -m pip install --upgrade pip
    - python -m pip install -e ".[dev]"
    - ruff check .

python-test:
  image: python:$[[ inputs.python-version ]]
  stage: test
  script:
    - cd "$[[ inputs.package-directory ]]"
    - python -m pip install --upgrade pip
    - python -m pip install -e ".[dev]"
    - pytest --junitxml=reports/junit.xml
  artifacts:
    when: always
    reports:
      junit: "$[[ inputs.package-directory ]]/reports/junit.xml"

The exact job names and implementation will vary, but the design principle is consistent: inputs describe supported variation; consumers do not edit shared job internals.

Pin the component version

Moving from copied YAML to a component does not automatically remove supply-chain risk. A moving reference such as @main means the behaviour can change without a commit in the consuming repository:

include:
  - component: $CI_SERVER_FQDN/platform/ci-components/python-package@main

That is convenient during component development, but it is not a stable production dependency.

Use a released version:

include:
  - component: $CI_SERVER_FQDN/platform/ci-components/python-package@1.4.0

For the strongest immutability, use a full commit SHA:

include:
  - component: $CI_SERVER_FQDN/platform/ci-components/python-package@3e1b9d46994c85a6f7e543e679fa1272b847de11

The important outcome is that a pipeline change becomes visible and reviewable in the consumer. Updating from 1.4.0 to 1.5.0 is an ordinary merge request rather than a silent estate-wide change.

Treat components like products

A useful component needs the same lifecycle discipline as any other shared dependency.

Define compatibility

Document the supported runtime and repository shapes. For example:

  • Python 3.11 and 3.12;
  • packages using pyproject.toml;
  • pytest for test reports;
  • semantic-release for versioning;
  • protected release branches only;
  • package publication through an approved registry variable.

A repository outside that contract should not be forced into the component until the component is extended deliberately.

Test the component in realistic consumers

The component project should test more than its own YAML syntax. Maintain small fixture repositories or downstream test projects representing the supported patterns:

  • a library with no publish step;
  • a publishable package;
  • a project in a subdirectory;
  • a project with linting disabled;
  • a project with a prerelease branch;
  • a repository that should never release from a merge request.

The same isolated sandbox approach used for migration can also validate new component versions against representative repositories.

Publish release notes

A component release should explain:

  • changed images and runtime versions;
  • changed defaults;
  • new or removed inputs;
  • job-name changes that can affect needs or overrides;
  • required variable changes;
  • migration steps;
  • whether the release is backwards-compatible.

Consumers can then upgrade intentionally instead of reading a large YAML diff in the component repository.

Set a deprecation policy

Shared components can accumulate old inputs and compatibility branches just as copied pipelines do. Define how long deprecated inputs remain supported and how consumers will be identified before removal.

An estate inventory makes this possible. When every include records a component version, it is straightforward to find repositories still using an old release.

A staged migration strategy

I grouped the work into three horizons rather than attempting an immediate big-bang conversion.

Horizon 1: stop current failures

Apply the smallest compatible fix to repositories that are already broken or about to break. In this case that meant pinning release plugins while retaining the existing runtime.

This reduced operational risk without pretending the old design was now healthy.

Horizon 2: migrate repository classes

Choose one coherent class of repositories, such as standard Python packages, and migrate them to the component contract.

For each repository:

  1. record its existing runtime, build and release behaviour;
  2. identify any real deviation from the component defaults;
  3. test the proposed include in the sandbox;
  4. confirm test, build and release semantics from job traces;
  5. backport only the verified CI and release configuration;
  6. open a normal merge request in the source repository.

The verified backport workflow kept the change narrow and auditable.

Horizon 3: enforce the standard

Once the component supports the target repository class:

  • update the project template to use it;
  • detect new copied pipeline blocks in audits;
  • reject moving component references in protected repositories;
  • report consumers running unsupported versions;
  • define ownership for component releases;
  • schedule controlled upgrade campaigns.

This prevents the next generation of drift rather than merely cleaning up the current one.

Avoid migration by visual similarity

Two pipeline files can look almost identical while having different operational effects.

Before replacing a local pipeline, compare behaviour rather than just job names:

  • Which branches and tags run each job?
  • Does a merge request pipeline publish anything?
  • Which variables are required and at what scope?
  • Are jobs manual, automatic or scheduled?
  • Which artifacts and reports are consumed elsewhere?
  • Does semantic-release need the complete tag history?
  • Does the repository use a non-standard package path?
  • Are downstream projects dependent on a particular job name?

This is one reason I mirrored complete refs and inspected release traces during testing. A green build alone did not prove that release behaviour was equivalent.

Keep local escape hatches explicit

There will always be exceptional repositories. The objective is to stop shared lifecycle behaviour from being privately copied, and eliminating every local job was never the aim.

A consumer can include the component and add a genuinely local job:

include:
  - component: $CI_SERVER_FQDN/platform/ci-components/python-package@1.4.0
    inputs:
      python-version: "3.12"

validate-generated-schema:
  image: python:3.12
  stage: test
  script:
    - python tools/validate_generated_schema.py

That exception is visible and does not fork the common test and release implementation.

Overrides of component jobs should be rarer. They couple the consumer to internal job names and can make future upgrades difficult. When several repositories need the same override, it is evidence that the component contract needs a new supported input.

The operational result

A component-based estate changes the maintenance model:

Copied pipelines Versioned components
Fix every repository Fix and release one component
Drift is hidden in large YAML files Version adoption is inventoryable
Template updates affect only new repos Existing repos can receive explicit upgrades
Shared behaviour is privately editable Supported variation is expressed through inputs
Changes can arrive accidentally through copy-paste Upgrades are normal reviewed dependency changes

The central gain is a controlled dependency relationship between the platform workflow and the repositories that consume it. Fewer lines of YAML is a side effect.

What I would standardise first

Based on the audit, my initial component contract would cover:

  1. a supported Python runtime matrix;
  2. dependency installation from pyproject.toml;
  3. linting and pytest reports;
  4. package build and artifact retention;
  5. semantic-release on approved branches;
  6. package publication using protected, scoped variables;
  7. explicit inputs for package path and optional publication;
  8. rules that separate merge request validation from release activity.

I would leave repository-specific deployments and high-risk scheduled maintenance jobs outside the first version. Those workflows need a separate threat model and more restrictive component contracts.

This article is part of a wider GitLab CI/CD standardisation series:

References