Levandor

GitLab CI Gotcha — Publish Docker Dev on feature/* needs optional Test dependency

2 min read

Enabling Publish Docker Dev on feature/* branches in the BESS Python consumer repos requires making the Test job dependency optional, or the pipeline refuses to start. Discovered while deploying the BE-1595 Arrow Flight feature to dev.

Context

The 3 Python consumer repos — bess-optimization, bess-forecast-day-ahead, bess-trader-dashboard — needed feature/arrow branches that build a dev ECR image consuming the new flight-enabled py-mando wheel. To enable dev image builds on those branches we added to the Publish Docker Dev job rules:

- if: '$CI_COMMIT_BRANCH =~ /^feature\//'

Symptoms

The pipeline failed to even start (invalid pipeline, no jobs run):

'Publish Docker Dev' job needs 'Test' job, but 'Test' does not exist in the pipeline.
This might be because of the only, except, or rules keywords.
To need a job that sometimes does not exist in the pipeline, use needs:optional.

Root Cause

Publish Docker Dev extends a hidden .Publish template whose needs: includes - job: Test (artifacts: true). But the Test job’s rules only fire on merge_request_event, develop, rc/*, main, and release/* — not on feature/*.

So enabling Publish on feature branches creates a hard dependency on a job that does not exist in that pipeline → GitLab rejects the whole pipeline.

Fix

Mark the Test need as optional in .Publish:

.Publish:
  needs:
    - job: Dynamic Env Variable Setup
      artifacts: true
    - job: Test
      artifacts: true
      optional: true   # so Publish can run on feature/* where Test doesn't exist

Repo-Specific Nuance

Not all three repos are the same

  • bess-optimization and bess-forecast-day-ahead — .Publish has needs: - job: Test. Both hit the bug and need the optional: true patch.
  • bess-trader-dashboard — .Publish only needs Dynamic Env Variable Setup (no Test dependency). It does not hit this and needs no change.

For Agents

Any automation patching these CI files must guard the optional-injection on the presence of a - job: Test need under .Publish. Blindly injecting optional: true where there is no Test need (trader-dashboard) is wrong.

Tradeoff

Making Test optional means the dev image ships without running the test suite on feature branches.

  • Alternative considered: add /^feature\// to the Test job rules → would run the full suite, but then unrelated test failures block the dev image.
  • For disposable feature/arrow deploy branches, optional was chosen — fast disposable dev images over gating on the full suite.

Graph View

Backlinks