Mando CI/CD

GitLab CI/CD pipeline and deployment configuration for the Mando workspace.

This pipeline is hand-rolled, unlike the Python repos

mando’s jobs are written out in its own .gitlab-ci.yml. The Python-side BESS repos (bess-optimization, forecast, algo services) instead consume the shared component library bess-os-ci-components and keep ~40-line pipelines.

Pipeline Stages

graph LR
    SEC["Security<br/>(Snyk)"] --> CONT["Pipeline<br/>Container"]
    CONT --> SETUP["Setup"]
    SETUP --> BUILD["Build &<br/>Test"]
    BUILD --> PUB["Publish"]
    PUB --> REL["Release"]
    REL --> PAGES["Pages"]

1. Security — Snyk vulnerability scanning
2. Pipeline Container — Build Docker base/build images
3. Setup — Environment preparation
4. Build & Test — Compilation and test suite
5. Publish — Artifact/package publication
6. Release — Release tagging
7. Pages — Documentation deployment

Branch Strategy

Branch Pattern	Target	ECR
feature/*, bugfix/*, rc/*, poc/*, develop	Dev	Dev ECR
release/*	Production	Prod ECR

poc/* added 2026-08-05

A one-line addition to .branch_rules:dev so poc/e2e-tests gets the full dev pipeline (including the new E2E job below). A throwaway feature/e2e-tests mirror branch created before this existed was deleted — and deleting a branch mid-pipeline kills its not-yet-started jobs at git fetch (missing ref). See mando-e2e-ci-green-2026-08-05.

E2E Data Suite Linux Dev (added 2026-08-05)

First green run: pipeline 2735150017, 72s

Downloads the released mando-cli binary cross-project via CI_JOB_TOKEN + job-token allowlist, brings up postgres:17 + wiremock/wiremock:3x as services: (shared netns → WireMock on localhost:8081), starts mando_bess as a background host process from the build artifact, runs refinery migrations on boot, then mando e2e run --external-stack. Junit uploaded (currently one case for the whole test_set_1). Full record incl. the required boot env set: mando-e2e-ci-green-2026-08-05.

Container Images

Production Image (Dockerfile)

FROM debian:13.1-slim
# Installs: ca-certificates, libc6, libssl3
# Copies: certs, assets, libduckdb.so, pre-built target/release/mando_bess
EXPOSE 8080
CMD ["mando_bess"]

Updated 2026-07-10

samba-libs/libsmbclient removed from the runtime image by the remotefs-smb to smb migration (pure-Rust smb crate). The build images still install them (harmless leftover). Toolchain is now 1.89.0 via rust-toolchain.toml even though the build image base tag remains rust:1.88.0-bookworm.

Build Image (container.linux.Dockerfile)

Base: rust:1.88.0-bookworm (toolchain 1.89.0 installed via rust-toolchain.toml)

Tools installed:

• Docker, jq, AWS CLI
• CMake, Go
• Poetry, pipx
• pkg-config, samba dev libs
• Maturin 1.9.2

Windows Variants

• container.win.base.Dockerfile — Windows base image
• container.win.chef.build.Dockerfile — Chef-based dependency caching

Chef dependency cache cooked the wrong feature set (fixed 2026-08-05)

container.linux.chef.build.Dockerfile cooked deps with a bare --release, while build.sh uses --features flight and test.sh uses --all-features. Cargo keys its cache per feature set, so every job missed the cooked layer and recompiled arrow-flight/tonic. Now cooks both variants.

Benefit only lands after merge to develop

The container rebake trigger is develop-only, so branch pipelines keep the old image. Escalations if still slow: scheduled weekly rebake (against Cargo.lock drift) or sccache+S3. GitLab cache: cannot help — it cannot hold /init/chef/cook/target, which is outside CI_PROJECT_DIR. The real wall-clock whale is PyMando Win Dev at ~2656s, untouched by any of this.

Pipeline Triggers

Container images rebuild when any of these change:

• rust-toolchain.toml
• container.*.Dockerfile
• Cargo.toml / Cargo.lock
• py-mando/pyproject.toml / poetry.lock

System Dependencies (Linux Runtime)

Package	Purpose
ca-certificates	TLS certificate store
libc6	C standard library
libssl3	OpenSSL / TLS

Quality Gates

• Lint gate: cargo clippy --release --all-features must be clean
• Test gate: cargo test --all-features --release --lib -- --test-threads=1 (.gitlab/scripts/test.sh; single-threaded mandatory: shared in-memory DB pools)
• No cargo fmt gate — never run bare cargo fmt locally (reformats ~64 legacy files); scope to touched files
• See Mando AGENTS.md Master Guide for the full pre-push checklist

The test gate is --lib-only (correction 2026-07-22)

The --lib in .gitlab/scripts/test.sh means CI compiles and runs ONLY lib unit tests. */tests/ integration targets are never built by the pipeline and rot silently; the separate “Integration Test Linux Dev” job covers only the mando-lib archiver Postgres suite. Concrete rot: mando-bess/tests/flight_end_to_end.rs is compile-broken on develop (E0603). Local gates must use --lib for CI parity. Full record: mando-ci-lib-only-test-gate-2026-07-22.

Known Flaky Tests

Intermittent CI failures with no underlying code defect are tracked in mando-known-flaky-tests-2026-07-15. Prefer retrying the CI job; real fixes land in dedicated follow-up tickets.

Build Requirements

Tool	Version	Purpose
Rust	1.89.0 toolchain (MSRV 1.88.0)	Compiler
Python	3.12	py-mando build
Maturin	1.9.2	PyO3 wheel builder
Poetry	2.1.3	Python dependency management
CMake	—	Native dependency builds
Go	—	Tool compilation
AWS CLI	—	ECR push, deployment