Levandor

Mando AGENTS Guide (mirror)

36 min read

Mirror

Canonical file: /Volumes/bandi/coding/poc/mando/AGENTS.md (kept untracked in the mando repo by design, wired into sessions via untracked mando/CLAUDE.md). Re-sync this mirror when the canonical changes. Last sync: 2026-07-22 (CI —lib test gate propagation: sections 2.4, 10, 13.5 + new landmine row). See Mando AGENTS.md Master Guide for provenance and change history.

AGENTS.md - Mando Engineering Guide

Master conventions document for AI agents (and humans) writing code in this repository. Compiled 2026-07-11 against origin/develop @ 92bfe1c8 (v1.16.0), CI configuration, and accumulated review feedback; extended 2026-07-13 with mined MR review themes, the working method (§13), and deployment ceremonies (§14); refreshed 2026-07-15 against tip 53912a70 (fingerprint activity + Arrow Flight merged). When this document and the code on develop disagree, the code wins - then update this file. Branches created before 2026-07-08 (v1.15.0 and earlier) predate a flow-engine restructure - see §11 “Branch drift” before working on one.

1. Orientation

What: Battery Energy Storage System (BESS) optimization platform. Alpiq Sales & Origination / Flexible Assets. Stack: Rust 2021 workspace + Python 3.12 bindings. Axum REST API, Polars/Arrow data plane, PostgreSQL/DuckDB/SQLite storage. Host: GitLab - gitlab.com/alpiq_cicd/sales-and-origination/flexible-assets/bess/poc/mando. Default branch: develop. Use glab (never gh).

Crate map & dependency direction (never invert)

mando-core (no internal deps: error! macro, DataPointId, model primitives, DF validation)
  └── mando-lib (data layer: repositories, MandoService, adapters, workflow engine)
        ├── mando-flow-step       step-service trait + concrete step services
        │                         (mdr, metis, position_manager, volue_*) [lib]
        │     └── mando-bess-lib  per-flow business services
        │                         (as_auction_update, …) extracted from mando-bess [lib]
        ├── mando-bess            REST API server + flow registry (+ mando-codegen) [binary]
        ├── py-mando              PyO3 extension (+ mando-codegen)   [cdylib]
        ├── py-mando-simulation   simulation-runner PyO3 package (own maturin build) [cdylib]
        ├── mandarrow-client      Arrow Flight gRPC client for streaming datapoints
        │                         (mando-core only; consumed by mando-bess + py-mando) [lib]
        ├── position-manager      market position bookkeeping        [lib]
        └── mando-simulator       flow-simulation server (+ position-manager) [binary]
mando-codegen           build-time generator (YAML + Askama templates → Rust), not a proc macro
mando-flow-step-derive  proc macros: #[derive(ParamEnum)] and #[derive(ParamMeta, attributes(param))]
  • New shared primitives with no I/O → mando-core. Anything touching DB/network/external systems → mando-lib (adapters) / mando-flow-step (step services). Flow business logic → mando-bess-lib. HTTP endpoints & flow registry → mando-bess. Python surface → py-mando / py-mando-simulation.
  • mando-core must stay lightweight (it is compiled into the Python extension).

Feature flags

CrateFeatureGatesConsumers
mando-libappAxum server plumbing, OTel, utoipamando-bess, mando-simulator
mando-libpythonPyO3 bindings (pulls mando-core/python)py-mando
mando-libworkflowreserved (empty), always enabled in runtime crates-
mando-corepythonPyO3 bindingspy-mando
mando-coretest-utiltracing test helpers + DataFrame fixturesdev-dependencies
mando-codegenpythonPython schema generationpy-mando

CI tests with --all-features - new code must compile under every feature combination.

2. Non-negotiables

  1. Never hand-edit generated code. mando-bess/build/generated/**.rs (domain.rs, flows/{market}/*.rs, flows/flow_handlers.rs, flows/manifest.rs) is generated by build.rs from mando-bess/config/**.yaml + mando-codegen/templates/*.askama - but it is tracked in git. Change the YAML/template, run cargo build, commit the regenerated output together with the input change. Same for *.out golden files (mando-codegen/test/domain/): regenerate, never edit (rustfmt ignores them).
  2. Never call tracing::error! or log::error! - banned via clippy.toml. Every error must be a thiserror enum variant, logged with mando_core::error!(err, message = "...") (see §6).
  3. Never run bare cargo fmt - it reformats dozens of legacy files and poisons your diff. CI has no fmt gate and the tip tree is not fmt-clean. cargo fmt -- path/to/file.rs does NOT scope either (field-verified 2026-07-13: it still touched unrelated files) - format touched files with rustfmt --edition 2021 path/to/file.rs directly. Note: the import/comment options in rustfmt.toml (group_imports, imports_granularity, wrap_comments, format_code_in_doc_comments) are nightly-only and silently ignored on the stable toolchain - the std/external/crate import grouping is maintained by hand; match it yourself.
  4. Tests run single-threaded: cargo test --all-features --release --lib -- --test-threads=1 (CI command, .gitlab/scripts/test.sh - the --lib is real: CI gates ONLY lib unit tests; tests/ integration targets are never compiled by the pipeline, see §11). Shared in-memory DB pools corrupt under parallel tests. Never “fix” a flaky test by removing this.
  5. Clippy is the lint gate: cargo clippy --release --all-features must be clean. Workspace policy (root Cargo.toml [workspace.lints]): clippy::all allowed, correctness/perf/suspicious denied + cherry-picked denies (panic_in_result_fn, exit, print_stderr, undocumented_unsafe_blocks, …).
  6. No unsafe (unsafe_code = "deny" workspace-wide, zero blocks in production code). No unwrap()/expect() in production paths - if a value can be parsed or constructed at build time, do it there; otherwise return a Result and propagate with ? into typed errors (reviewer-stated policy, MR !345). Tests may unwrap freely.
  7. Commits: Conventional Commits, lowercase, imperative, title-only. No descriptions, ever: no bodies, no bullets, no trailers, nothing after the title line. No scopes in the title (fix: normalize ebs host, never fix(ebs): …) - reviewer-mandated in MR !322. Types: feat:/fix:/chore:/refactor:/ci:/test:/style:/release:. Keep titles short and fluff-free: what changed, nothing more.
  8. Commit cadence: commit after roughly every 3 completed tasks (or one coherent unit of work), not one mega-commit at the end and not per-keystroke. Each commit must build. Keep the branch linear: no merge commits inside a feature branch and no unrelated commits riding along (both were review rejections - MR !322, !474).
  9. Branches: feature/BE-xxxx / bugfix/BE-xxxx off develop (Jira key mandatory); release/* → prod; rc/* release candidates. MRs target develop. Never post MR/issue comments unless explicitly asked; never override the squash/merge commit message when merging.
  10. Dependencies are pinned (=x.y.z) in the root Cargo.toml under #region sections - add new deps there (pinned, in the right region), reference from member crates with { workspace = true }. Prefer an already-installed crate over a new one.
  11. Comment-sparse codebase. Do not add comments or doc comments to new code; the code must be self-explanatory. Only accepted markers: //TODO BE-XXXX: reason.
  12. No em dashes. Never use the em dash character in anything written into this repo or its surroundings: code, commit titles, docs, MR titles/descriptions, YAML. Use a plain hyphen, a colon, or restructure the sentence.
  13. Subagent-driven development, always. Delegate implementation work to subagents by default and without asking whether to do so - it is the standing mode of operation here, not a choice to surface. The orchestrator scopes, reviews, and integrates; subagents write the code. (Background subagents cannot write files; dispatch write-performing subagents in the foreground.)

3. Toolchain & build

  • Toolchain pinned: 1.89.0 (rust-toolchain.toml); MSRV rust-version = "1.88.0"; edition 2021. Workspace version single-sourced at [workspace.package] version (1.16.0 on develop) - only release: commits change it.
  • Git LFS required (git lfs install) - lib/libduckdb/** binaries are LFS-tracked. A pre-push hook checks for git-lfs.
  • Registries: Nexus alpiq / alpiq-internal tokens in ~/.cargo/credentials.toml; local .cargo/config.toml created from .cargo/config.toml.example (sets platform DYLD_LIBRARY_PATH).
  • Build/run: 
    cargo build                        # whole workspace
cargo run -p mando_bess            # REST API on :8080 (needs MANDO_MODE + adapter env vars)
  • Runtime config via env vars (envconfig): core prefix MANDO_ (MANDO_MODE={DuckDb|Sqlite|Postgres|DuckDbPostgres}, MANDO_PORT, MANDO_ENVIRONMENT, MANDO_TEXT_LOG, MANDO_CACHE_CLEAR_OBSOLETE_SCHEDULE); per-adapter prefixes (EBS_SMB_*, VOLUE_EMS_*, FINGRID_*, METIS_*, ENTRA_ID_*). No secrets in code or committed files. bess-service.yaml at root is a local, gitignored run config.
  • Config-crate direction: reviewers are steering NEW configuration toward the config crate (config::Config) instead of Envconfig derives (flagged twice by two reviewers in MR !556). Existing adapters keep envconfig; do not introduce new #[derive(Envconfig)] structs without checking whether config::Config fits.

4. Architecture - where does new code go?

Layering inside mando-lib (bottom → top)

LayerLocationRoleKey exemplars
Repositorymando-lib/src/repo.rs + repo_{sqlite,postgres,duckdb,passthrough}.rs#[async_trait] traits (Repository, DataPointRepository, CacheRepository) + one impl per backendrepo.rs:28-104 (RepoError), trait defs repo.rs:184+
Service basemando-lib/src/service_base.rsMandoServiceBase<…> generic over the 4 data-type repos + dp registry; calculation/virtual-DP resolution
Service dispatchmando-lib/src/service.rs + service_{sqlite,duckdb,postgres,mock}.rsMandoService facade + MandoServiceInner enum (Sqlite/DuckDb/Postgres/DuckDbPostgres/Mock) selected at runtime via MANDO_MODE; dispatch!/dispatch_async! macros
Adaptermando-lib/src/adapter/{alpiq,volue,fingrid,mando}/One module per external system: XyzConfig (Envconfig) + XyzError (thiserror) + XyzClientadapter/alpiq/ebs.rs, adapter/volue/ems.rs
Service (integration)mando-lib/src/service/…Business-logic wrappers over adaptersservice/alpiq/position_manager.rs
Step servicesmando-flow-step/src/service/…One service per flow step against an external system; each implements FlowStepService (mando-flow-step/src/lib.rs): type Params: ParamMeta; type Response; + async from_config(config: &str, providers: &StepProviders)service/mdr.rs, service/volue_ems_spot.rs
Flow business logicmando-bess-lib/src/service/{flow}/…Per-flow orchestration/business servicesservice/as_auction_update/…
Workflow enginemando-lib/src/workflow/WorkflowStep tree, FlowService, FlowRequest<T>, StepStatus (Skip/Success/Warning/Error/Fatal), ErrorWithStepStatus, param_meta, flow persistence reposworkflow/descriptor.rs, workflow/flow.rs

Invariants: flows call services, not repositories directly (exceptions: audit, settings). All data access is async (tokio). Backend selection is runtime (enum dispatch), never compile-time. Adapters own their config, error type, and auth provider.

Flow engine & versioning

  • Flow definitions live in mando-bess/config/flows/{market}/{type}_{version}.yaml (e.g. fi/manual_schedule_v3.yaml). config/flows/manifest.yaml is the deployment catalog: under setup: {setup_version}: flows: each flow name maps to path, schedule_env (e.g. MANDO_FLOW_SCHEDULE_AUCTION) and semaphore_group - scheduling and concurrency grouping are configured HERE, not in code.
  • build.rs (mando-codegen) generates - all tracked in git - build/generated/flows/{market}/{type}_{version}.rs (step structs, params, workflow tree, the flow-handler impl), flows/flow_handlers.rs, flows/manifest.rs, flows/mod.rs. The old hand-written per-version wiring dirs (src/flow/{type}/v{n}/) are gone.
  • Runtime: mando-bess/src/flow/flow_registry.rs - the FlowHandler trait each generated flow implements (flow_type, flow_version, descriptor, is_visible, schedule_env, semaphore_group, manual_only, params_meta, run_manual, run_scheduled) plus FlowContext (env config, error-notification service, flow/setting repos, per-group semaphores of capacity 1, Arc<MandoService>). HTTP surface stays in route/flow_routes.rs / route/flow_type_provider.rs.
  • Step implementations live in mando-flow-step (external-system step services) and mando-bess-lib (per-flow business services); every step type must be registered with type_entry!(...) in mando-bess/build.rs (the type registry codegen resolves YAML type: names against). Steps receive shared state via StepProviders (mando-flow-step/src/providers.rs: flow repository, datapoint registry, all auth providers, simulation_enabled); parameters use #[derive(ParamMeta)] / #[derive(ParamEnum)] (mando-flow-step-derive).
  • Scheduling: each handler’s schedule_env() (sourced from the manifest) names the env var holding the cron string; cron strings may hold multiple schedules separated by ;; jobs run in Europe/Berlin and audit-log with flow type + execution id (src/flow/flow_scheduler.rs). Semaphores have capacity 1 - one running flow per semaphore_group.
  • Canonical exemplar: fi/manual_schedule v3 (config/flows/fi/manual_schedule_v3.yaml → build/generated/flows/fi/manual_schedule_v3.rs).
  • Released flow versions are immutable: never change a released vN’s YAML or generated code - breaking changes go into v(N+1).

Data model

  • DataPointId (mando-core/src/model/datapoint.rs): hierarchical /-separated ASCII-alphanumeric paths (Market/DA/Price). Immutable once created (used as FK). Types: Normal, Virtual(alias), Calculated{expression, inputs}. Runtime catalog: DataPointRegistry (mando-lib/src/datapoint/registry.rs), handed to steps via StepProviders.
  • New datapoints: add to mando-bess/config/defaults-bess-datapoint.yaml (or config/parts/*.yaml), cargo build regenerates build/generated/domain.rs, commit both. (ResolvedDataPoint is a codegen-internal model in mando-codegen - not a runtime/Python type.)
  • DataFrames (Polars): standard columns time/value_time, value, override, fetch_time, generation_time; DateTime<Utc> timestamps, chrono-tz for zone math. Prefer lazy (.lazy() … .collect()) in query paths.
  • Migrations: refinery, mando-bess/database/{YYYY_QN}/U{version}__{description}.sql (integer or timestamp version, e.g. U202601081116__add_data_point_update_table.sql). Embedded via embed_migrations! and applied idempotently on startup.

5. Rust style

Formatting is rustfmt-owned (rustfmt.toml): max_width = 200, group_imports = "StdExternalCrate", imports_granularity = "Module", wrap_comments, format_code_in_doc_comments. On top of that, house rules enforced in review:

  • Imports at top of file only, module-granular, grouped std → external → crate (maintained by hand, see §2.3; never add use inside function bodies - test modules may use super::*;). Prefer use crate::… over super:: (~14:1 in codebase). Once something is imported, never call it by its full path at the use site (FlowRepository::new(…), not mando_lib::workflow::repository::FlowRepository::new(…)) - the single most repeated review nit (MR !345, !389, !437, !556).
  • Blank line before return/final Ok(…)/Err(…), around standalone blocks (multi-line chains, match blocks), between every function definition, and before comments (MR !322, !345, !474). See adapter/alpiq/ebs.rs for the rhythm. Do not add blank lines beyond that - stray empty lines are review rejections too (MR !437, !495).
  • Inline format! args in new code: format!("{x}"), not format!("{}", x). This is a review rule, not the codebase majority (positional still outnumbers inline ~3:1 on tip) - don’t churn legacy call sites, but don’t add more positional ones.
  • Module organization: directories get a mod.rs declaring submodules; a module that owns submodules internally can be a named file (ems.rs declaring pub mod spot_data;). Shared module-level types live at the top of mod.rs (e.g. adapter/mod.rs defines StepStatus, HasStepStatus). No types.rs dumping grounds.
  • Visibility: plain pub dominates (pub(crate) is rare, ~19 uses total) - match the module you’re editing.
  • Naming: types PascalCase with role suffixes - …Error, …Repository, …Service, …Provider, …Config, …Client, …Filter/…Request; files/fns snake_case; consts/statics SCREAMING_SNAKE_CASE; generic params single capitals.
  • Derives: #[derive(Error, Debug)] for errors; #[derive(Clone, Debug)] (+ Serialize, Deserialize, PartialEq as needed) for data. Serde attrs per context: rename_all = "camelCase" (external APIs), "kebab-case" (mando-bess DTOs/enums, paired with #[strum(serialize_all = "kebab-case")]), "PascalCase" (some configs), tag = "type" for tagged enums.
  • Async: #[async_trait] on all async traits; bounds Send + Sync + 'static for shared services. tokio::sync::{RwLock, OnceCell} / std::sync::OnceLock - the once_cell crate is not used. No spawn_blocking in mando-lib (py-mando is the exception, §9).
  • Tracing: #[tracing::instrument("area.component.action", skip_all, fields(...))] on significant async fns; structured fields, message as a field: info!(message = "Generating file", hub, customer, reference_date = d.format("%Y-%m-%d").to_string()). Custom flow_span! macro inside flow code. Levels: info!/debug!/warn!; error only via mando_core::error!.
  • Enums with string forms: strum_macros::{Display, EnumString} instead of hand-rolled FromStr.
  • #[allow(…)] only item-level and narrow (#[allow(dead_code)] on test/mock fields is the common case). Never module-wide blanket allows.
  • No custom builder types, no single-implementation traits, no speculative abstractions - reuse the existing shape of the module you’re in.

6. The error system (this repo’s signature pattern)

  1. Every component defines a thiserror enum named …Error, one variant per failure mode: 
    #[derive(Error, Debug)]
pub enum RepoError {
    #[error("cannot retrieve connection from pool: '{0}'")]
    CannotGetConnectionFromPool(#[from] r2d2::Error),
    #[error(transparent)]
    ModelError(#[from] ModelError),
}
    #[from] for conversions, #[error(transparent)] for pure wrapping, named-field variants where context helps (mando-simulator/src/error.rs). anyhow appears only in init/setup paths and flow-step glue (ErrorWithStepStatus), never in adapter/repository signatures.
  2. Log errors only with mando_core::error! (mando-core/src/error.rs). No derive needed - the macro resolves everything at runtime: it walks the source() chain and emits a tracing::error! with error.kind/error.code/error.type (qualified Enum::Variant extracted from the Debug repr), error.message (the root cause message), and error.stack (the full source chain joined with ": "). It is compile-time restricted to std::error::Error values. Forms: 
    error!(err, message = "Failed to fetch container metadata");
error!(err, http_context = ctx, message = "...");   // adds http.method/url/request.body/status fields
error!(@resolved code = ..., msg = ..., stack = ..., ...); // low-level arm used by the workflow logger - rarely needed by hand
  3. Flow steps wrap failures in ErrorWithStepStatus (mando-lib/src/workflow/mod.rs) carrying a StepStatus (Warning/Error/Fatal) plus a logged_at_site flag. Constructors: new(status, err) when the step boundary should emit the ERROR line, logged(status, err) when you already error!-logged at the detection site (the boundary then logs only WARN - this is the double-logging guard; pick the right one). StepResult::log emits error.fingerprint = "{error_code}|{step_path}|{activity}" (activity optional, set via the consuming with_activity() builder - e.g. the Volue EMS data group name) for cross-execution aggregation - don’t invent your own fingerprint fields. (In-flight: BE-3541 removes logged_at_site/logged() in favor of single boundary emission with HttpContext carried on the error - once merged, always construct with new/message and never log at the detection site.)
  4. HTTP mapping (mando-bess): map service errors deterministically in a map_*_error_to_response fn to ErrorResponse { status, error, message } (mando-bess/src/model/error_response.rs) → JSON body {"error": "FlowDisabled", "message": "…"}. Same variant ⇒ same status code, always. Examples: SystemDisabled/FlowDisabled → 423, ConcurrentFlowError → 429, UnknownFlowError → 400, invalid JSON → 400 "InvalidJson".
  5. Redesign status: the observability redesign has partially landed on develop (commit ae6d1098): root-cause error.message, error.stack chain, error.fingerprint, logged_at_site. Not landed: error.trace field, .step_context() helper. The old plan doc (docs/error-handling-redesign-plan.md) and doc/errors.json are not tracked on develop - they are local artifacts in older checkouts; don’t resurrect or commit them.

7. mando-bess REST API conventions

Startup: main.rs → propagator → init_log → metrics → init(APP_NAME) (pools, repos, services) → init_router → run. Failures wrap in AppInitError + error!. The shared state is AppServices (lib.rs): mando, flow_env_config, flow_registry, flow_context, flow_repository, audit_repository, setting_repository, component_versions. In DuckDb mode an optional DuckDB UI server runs on :4213; in the hybrid Postgres mode the cache-cleanup cron job is spawned.

  • Routing: OpenApiRouter::with_openapi(ApiDoc::openapi()) (mando-bess/src/lib.rs:496+); every handler registered with .routes(routes!(handler)) (utoipa-axum) or merged sub-routers. Swagger at /swagger, Scalar at /docs, spec at /api-docs/openapi.json. Static files: .nest_service("/assets", ServeDir::new("assets")).
  • URLs & enums: kebab-case paths (/flow-config, /audit/logs); path enums serialize kebab-case via serde + strum (model/flow_type.rs). Versioning is per-flow (FlowVersion::V2/V3/V4), not in the URL.
  • Handler shape (extractor order matters): 
    pub async fn handler(
    State(app_services): State<AppServices>,
    Extension(user): Extension<Option<User>>,   // when audit/user context needed
    Path(p): Path<T>, Query(q): Query<Q>,
    ValidatedJson(body): ValidatedJson<B>,      // writes; GETs skip body
) -> impl IntoResponse
    Return Json(dto).into_response(), a mapped ErrorResponse, or a bare StatusCode. 204 for empty-success PATCH. Exception: flow-run endpoints take the body as a raw String - the codegen-generated flow handler deserializes it into the typed param struct (run_flow in route/flow_routes.rs).
  • Every handler carries #[utoipa::path(...)]: method, path, camelCase operation_id, description, all success + error responses (error body = ErrorResponseBody), and a tag from api_docs.rs (System, Query, Flow, Settings - add new tags there).
  • Validation: request bodies use the custom ValidatedJson<T> extractor (route/validated_json.rs) - T implements the local JsonSchema trait returning a JSON Schema (jsonschema crate validates before deserialize). No garde/validator derives.
  • Auth: token_layer (mando-lib/src/app/route/token_layer.rs) parses the Bearer JWT into Extension<Option<User>> (username only, for audit logs). Signature validation is currently disabled (insecure_disable_signature_validation) - known dev posture; do not build authz on top of it and do not “fix” it without a ticket.
  • Middleware order (do not reorder): OtelInResponseLayer → OtelAxumLayer → HttpMetricsLayer → DefaultBodyLimit::disable() → RequestBodyLimitLayer (10 MB) → token_layer → routes. No CORS layer exists.
  • State-changing endpoints write an audit event via audit_repository.log_event(AuditEvent::…, json!(...), user.map(|u| u.username)) (see setting_routes.rs).
  • Background jobs: tokio-cron-scheduler through scheduler.rs (create_job, Europe/Berlin timezone, cron from env). Job failures log via error! and return.

8. py-mando (PyO3) conventions

  • Layout: Rust in py-mando/src/ with #[pymodule] pub mod py_mando re-exporting submodules via #[pymodule_export] (lib.rs); Python package in py-mando/python/py_mando/ (wrappers: pandas.py, logger.py, tracing.py, codegen-produced bess/ clients; py.typed marker). A #[pyfunction] init() runs at module import: tracing subscriber with the custom JsonFormatter + rustls ring provider, idempotent. abi3-py38 stable ABI - wheel is cp38-abi3 (package itself requires Python ≥3.10).
  • py-mando-simulation (split out of py-mando in 1.16.0, commit 3559042d): sibling maturin package exporting SimulationRunner, ServerConfig and the simulator request/response types; requires Python ≥3.11, depends on ddtrace (no polars/pandas). Breaking: py_mando.SimulationRunner no longer exists - import py_mando_simulation. Same init()-at-import, logger, and libduckdb-preload patterns as py-mando.
  • Classes: #[pyclass] + #[pymethods]; fields exposed with #[pyo3(get)]/#[pyo3(get, set)]; constructors #[new] with #[pyo3(signature = (...))] for defaults; #[staticmethod] for factories; enums #[pyclass(eq, eq_int)]; enum-with-fields constructors via #[pyo3(constructor = (...))] (model/backend_service.rs).
  • Async bridging - two sanctioned patterns: 
    // blocking API (sync Python callers):
py.allow_threads(|| {
    pyo3_async_runtimes::tokio::get_runtime().block_on(async { /* .await work */ })
})
// awaitable API (the *_async method variants return a Python coroutine):
pyo3_async_runtimes::tokio::future_into_py(py, async move { /* clone services in */ })
    Never hold the GIL across .await; never Python::with_gil inside an async block; clone Py<T> with clone_ref(py), never Clone.
  • Errors: single exception type PyMandoError (create_exception!, mando-lib/src/python/error.rs, re-used by both packages); ~21 map_*_error fns there (repo, polars, duckdb, uuid, io, envconfig, …) plus py-mando-local map_rustls_error/method_not_supported. Every boundary maps: .map_err(map_repo_error)?. Add a new map_x_error fn for new error enums - all format as "{ErrorType}: {msg}".
  • Data: DataFrames cross via pyo3-polars::PyDataFrame, stored as Py<PyAny>; the pandas backend is a thin Python-side wrapper. Required frame columns: value_time, generation_time, fetch_time, value.
  • Python style: snake_case functions, PascalCase classes, light type hints (Optional[str], list[T]), pytest with @pytest.mark.parametrize (no fixture farms), tests in py-mando/test/test_*.py; notebooks validated with pytest --nbval (testpaths: test, example/notebooks).
  • Local build (macOS, per package): python3.12 venv → pip install poetry==2.1.3 → poetry install → maturin develop [--release] → poetry run pytest (py-mando adds --nbval for the notebooks). build.rs copies lib/libduckdb/1.4.2/{platform}/ into data/platlib (shipped in the wheel via data = "data"); each package’s __init__.py preloads it - ctypes RTLD_GLOBAL on Linux/macOS, os.add_dll_directory on Windows. If symbols still fail to resolve: export DYLD_LIBRARY_PATH=$PWD/data/platlib.

9. Testing ceremonies

  • Placement: inline #[cfg(test)] mod tests at the bottom of the file under test - 87 such modules on tip. One exception since 2026-07: mando-lib/tests/ holds Postgres-backed integration suites (archiver) that CI runs via .gitlab/scripts/integration-test.sh with a postgres service and TEST_PG_HOST; locally they fail on the missing env var - do not count them against your local gate, and do not add new tests/ dirs without the same CI wiring. Test fns test_… (dominant, ~255) or behavioral should_… (~11). The restructure crates (mando-flow-step, mando-flow-step-derive, mando-bess-lib) currently ship without tests - when you add logic there, still add inline tests per workspace convention; don’t copy their bareness.
  • Frameworks (already in workspace - use these, add nothing new):
    • #[tokio::test] for async; #[test_log::test] / #[test(tokio::test)] when tracing output helps.
    • #[test_case(400 ; "bad request")] (test-case) and #[parameterized(case = {…})] for input matrices (adapter/volue/ems.rs, util/instrumented_http_client.rs).
    • httpmock::MockServer for HTTP adapters - every adapter test mocks the remote, no live calls.
    • Assertions: plain assert_eq!/assert!.
  • Fixtures: reuse the existing helpers - mando-core/src/util/test.rs DataFrame factories (generate_double_df, generate_pvc_df, generate_static_df, get_df_value_*, via the test-util feature), mando-core/src/test_util.rs (capture for asserting tracing output), and mando-lib/src/test.rs (init_log, get_result_* accessors). Do not hand-roll frame builders.
  • DB tests: in-process, in-memory (DuckdbConnectionManager::memory() behind a setup_test_pool() helper). No testcontainers. This is why --test-threads=1 is mandatory.
  • Credential-dependent tests: mark #[ignore = "used for local connectivity testing"] so CI skips them.
  • New logic ships with a test: unit tests beside the code; parameterize edge cases; mock externals. Coverage tooling isn’t enforced - reviewer judgment is the gate.

10. Git & CI ceremonies

Pre-push checklist (mirror of CI)

cargo build --release
cargo clippy --release --all-features                       # gate
cargo test --all-features --release --lib -- --test-threads=1   # gate; --lib = CI parity (field-verified 2026-07-22); see RTK warning below; dropping --lib also builds tests/ targets CI never gates: flight_end_to_end is compile-broken on develop, and mando-lib/tests/ archiver needs TEST_PG_HOST
rustfmt --edition 2021 $(git diff --name-only origin/develop -- '*.rs')   # rustfmt directly; cargo fmt does not scope
# if py-mando touched:
cd py-mando && maturin develop --release && poetry run pytest --nbval && cd ..
# if py-mando-simulation touched:
cd py-mando-simulation && maturin develop --release && poetry run pytest && cd ..

RTK false-green warning (field-verified 2026-07-13): on machines with the RTK shell hook, plain cargo test ... -- --test-threads=1 gets rewritten and the --test-threads=1 arg becomes a test-name FILTER: every test is filtered out and the command exits 0 having run NOTHING (output looks like 0 passed, N filtered out ... 0.00s). Run the gate as rtk proxy cargo test --all-features --release --lib -- --test-threads=1 and always confirm the summed N passed totals are nonzero before claiming green. Exit code 0 alone is not evidence a test ran.

rustfmt false-green warning (field-verified 2026-07-13): rustfmt --check --skip-children <file> errors out (Unrecognized option) and prints NOTHING, which reads as fmt-clean. Use rustfmt --edition 2021 --check <file> without --skip-children, and note rustfmt FOLLOWS mod declarations into child files: attribute hunks by the filename in each Diff in <path>: header, not by the file you passed. To separate pre-existing churn from yours, check the base revision’s copy AT ITS REAL PATH (stash/checkout), NEVER by copying it to a scratch dir: rustfmt cannot resolve mod children from a new location and silently reports 0 hunks (false green, field-verified 2026-07-13). Also: running rustfmt on a module-declaring file WRITES into its child files - format leaf files, or revert unintended children afterwards. When a touched legacy file has pre-existing fmt hunks, hand-fix only your own lines rather than reformatting the whole file (theme 1).

Pipeline (.gitlab-ci.yml + .gitlab/*.yml)

Stages: security (Snyk) → pipeline container → setup (version calc) → build & test (Linux always; Windows on develop/rc; separate jobs for the Rust workspace, py-mando pytest --nbval, and py-mando-simulation pytest) → publish (service + simulator docker images, py-mando and py-mando-simulation wheels to Nexus; dev targets for feature/*|bugfix/*|rc/*|develop, prod for release/*) → release (annotated tag v{version}, changelog from commit log + BE-\d+ Jira keys) → pages (disabled).

  • Version suffixes appended by CI: +dev.{pipeline}.{sha} (develop/rc), +feat.… (feature/bugfix). release: commit sets the plain version; bump Cargo.toml version only on release/*.
  • No MR/issue templates and no CODEOWNERS - review ceremony is informal; conventional commit titles are checked at review time.
  • Merging release/* flows back to main and develop.
  • Publishing an image is NOT deploying it - deployment is a separate ceremony in the optimization-universe-iac repo, see §14.

Etiquette

  • MR flow via glab; formal actions (push, labels, re-request review) are fine; comments/notes on MRs only when explicitly requested; never set the squash/merge commit message on merge - reviewers own it.
  • MR descriptions: GitLab default only. Never write a custom MR body; the auto-generated default (e.g. Closes BE-XXXX) is the whole description. The MR title is a valid no-scope conventional commit title.
  • Working documents (docs/ plans/handovers, .agents/, benchmarks, screenshots) stay untracked - never commit them.

Recurring review feedback (mined from MRs !322-!558)

Every theme below was flagged by reviewers at least twice across this repo’s MR history. Treat them as pre-review gates: check the diff against this list before pushing.

  1. Zero unrelated changes in the diff. The ticket’s changes and nothing else: no drive-by renames, no incidental level/config tweaks, no touched-but-unchanged files. Revert accidental churn (e.g. a YAML file picked up by tooling) before pushing. Asked in MR !437, !474, !495, !548, !556.
  2. Delete unused code before review, all of it. Leftover structs, unused router registrations, dead params, “for later” scaffolding: reviewers reject them every time, with escalating patience loss (“How many more times do I have to ask you to remove the unused code?” - MR !345). If it is not called, it does not ship.
  3. Failures are errors, never warnings. If an operation actually failed, log it through mando_core::error! with an error-code enum variant and, in flows, fail the step (Fatal if it must kill the execution). Fifteen-plus “should be error” comments in MR !481 alone. Do not downgrade real failures to warn! and do not invent custom error message strings: the error’s to_string() is the message, human context goes in the message = field.
  4. Put code where it lives. Route helpers next to their routes, mod declarations grouped together, generic helpers in util, and name modules for what they are (a save endpoint is not a “formatter”). MR !322, !345, !556.
  5. Finish the pass before requesting review. “you forgot this”, “half done”, “don’t skip this” are recurring. Sweep every TODO you created, every call site of a thing you changed, and every sibling config (a datapoint added to one data group usually belongs in its mirror group too - MR !362). Address every review comment before re-requesting.
  6. Green pipeline and no conflicts before review. Reviewers open with “pipeline is broken” / “resolve conflicts” (MR !345, !474, !558). Rebase on origin/develop and confirm CI before assigning.
  7. rustfmt every NEW file (MR !556) - new files have no legacy-diff excuse; use rustfmt <file> directly (any form of cargo fmt, even with a path, leaks into unrelated files).
  8. Keep Jira in sync: if the implementation drifted from the ticket text, update the ticket so it reflects what was actually built (MR !437).
  9. Reuse the enforced idioms: import errors from mando_core rather than redefining, drop redundant thiserror: path prefixes, no unwrap() (parse at build time or return Result), collapse per-type match duplication by binding the service once (MR !345, !481, !556).

11. Landmines (verified)

LandmineReality
cargo fmt in any formBare run reformats ~64 unrelated files, and even cargo fmt -- <file> leaks beyond the named file (field-verified). Use rustfmt <file> directly. CI won’t catch it, reviewers will.
RTK hook vs cargo testThe RTK shell hook rewrites cargo test ... -- --test-threads=1 so that every test is filtered out: exit 0, 0 passed, nothing verified. Use rtk proxy cargo test ... and check the passed totals (see section 10).
rustfmt --skip-childrenUnrecognized on the pinned stable rustfmt: errors out printing nothing, which looks fmt-clean. Check without it and attribute hunks by the Diff in <path>: headers (see section 10).
Stale Cargo.lock on developThe committed lock can lag workspace versions (seen: mandarrow-client 1.16.0 vs 1.16.1); any plain cargo build regenerates it. If a lock hunk you did not cause appears in your diff, verify tip’s lock is stale before reverting it - keeping the regenerated line can be correct.
Parallel testsIn-memory DB pools shared across tests - must run --test-threads=1.
CI test gate is --lib-only.gitlab/scripts/test.sh runs cargo test ... --lib: */tests/ integration targets are NEVER compiled by the pipeline and can rot silently (archiver runs via the separate integration-test job). mando-bess/tests/flight_end_to_end.rs fails E0603 on develop (DataPointUpdateInfo imported privately via mando_lib::service, broken since the mando-repository split, field-verified 2026-07-22) - a full local cargo test without --lib hits it; keep --lib for CI parity and fix the flight test as its own change.
Local develop refOften stale - branch from / diff against origin/develop.
Generated code is committedmando-bess/build/generated/** is tracked; regenerate via cargo build and commit alongside the YAML/template change.
doc/errors.json, bench/, docs/, .ai/Local/untracked artifacts - leave them out of commits.
bess-service.yamlLocal run config, gitignored - never reference it as a committed file.
JWT signature validation disabledtoken_layer parses but doesn’t verify; User is audit metadata, not authz.
Git LFSlib/libduckdb/** binaries; pushes need git-lfs installed.
macOS py-mandolibduckdb preload quirk; DYLD_LIBRARY_PATH=$PWD/data/platlib fallback.
py_mando.SimulationRunnerGone since 1.16.0 - the simulation runner lives in the separate py_mando_simulation package. Old snippets/notebooks importing it from py_mando break.
rustfmt configThe import-grouping/comment options are nightly-only and ignored on stable - running fmt does NOT produce the repo’s import style; write imports in std→external→crate groups yourself.
Stale parent docspoc/CLAUDE.md references mando/CLAUDE.md, a mando-lib-macro crate and an ErrorCode derive - none exist on develop. The only proc-macro crate is mando-flow-step-derive (ParamEnum/ParamMeta - not ErrorCode; error codes are runtime-extracted by mando_core::error!). Trust this file and the code.
.worktrees/Contains in-flight experiment branches (incl. an ErrorCode-deriving mando-lib-macro) - never grep-and-copy from there as if it were develop.
Branch driftBranches based before 2026-07-08 / v1.16.0 (e.g. bugfix/BE-3333) predate the flow restructure: 7 crates instead of 11, hand-written src/flow/{type}/v{n}/ wiring, flow YAMLs not yet under config/flows/{market}/, simulation runner still inside py-mando. Match the structure of the branch you are on; expect merge friction in flow code.

12. Recipes (exemplar-driven)

A. New external-system adapter - copy the shape of mando-lib/src/adapter/alpiq/ebs.rs or adapter/volue/ems.rs: adapter/<vendor>/<system>.rs with XyzConfig (#[derive(Envconfig)], XYZ_* env vars) + XyzError (thiserror) + XyzClient (reqwest/graphql/smb) → declare in adapter/<vendor>/mod.rs → business wrapper in service/<vendor>/ → if flow steps need it, add its auth/client to StepProviders (mando-flow-step/src/providers.rs) and wire construction in mando-bess/src/lib.rs. Tests: httpmock, parameterized error cases.

B. New repository capability - trait method in mando-lib/src/repo.rs → implement in repo_sqlite.rs, repo_postgres.rs, repo_duckdb.rs (+ passthrough if cached) → surface through MandoServiceBase → delegate in MandoService via dispatch_async!. All backends or none.

C. New REST endpoint - handler file under mando-bess/src/route/ + pub mod in route/mod.rs → DTOs with Serialize/Deserialize + ToSchema (kebab-case) → #[utoipa::path] with every response documented → register with .routes(routes!(handler)) in lib.rs → map errors via a map_*_error_to_response → audit-log state changes → ValidatedJson + JsonSchema impl for write bodies.

D. New error variant - add variant with #[error("…")] to the component’s enum → convert with #[from]/impl From where it crosses layers → error!(err, message = "…") at the detection site → if it reaches HTTP, add the deterministic arm in the mapping fn.

E. New flow version - new mando-bess/config/flows/{market}/{type}_v{n+1}.yaml (copy v{n} as the template) + entry in config/flows/manifest.yaml → cargo build regenerates build/generated/flows/{market}/{type}_v{n+1}.rs + flow_handlers.rs + manifest.rs → new step services (if any) per Recipe F → commit YAML + regenerated files + new services together. Never touch released versions. A new market = new config/flows/{market}/ dir + manifest entries; codegen handles the rest.

F. New flow step service - shared/external-system step → mando-flow-step/src/service/<step>.rs; market/flow business step → mando-bess-lib/src/service/<flow>/<step>.rs. Implement FlowStepService (type Params/type Response + from_config(config, providers)), param struct derives #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ParamMeta)] (+ ParamEnum on enum params), shared state comes from StepProviders → register the type with type_entry!(...) in mando-bess/build.rs (36 registrations there today) → reference it by type name from flow YAML. Copy an existing sibling (service/mdr.rs, service/as_auction_update/…) rather than inventing a shape.

G. Expose Rust to Python - impl in a py-mando/src/ module (#[pyclass]/#[pymethods], async via the §8 bridge patterns, errors via map_*_error) → #[pymodule_export] in lib.rs → maturin develop → pytest in py-mando/test/test_*.py. Simulation-related surface goes in py-mando-simulation instead.

H. New datapoint - entry in mando-bess/config/defaults-bess-datapoint.yaml / config/parts/*.yaml (id path, type, timezone, unit, versioned_metadata) → cargo build → commit YAML + regenerated domain.rs.

I. New migration - mando-bess/database/{YYYY_QN}/U{timestamp}__{snake_description}.sql; additive and idempotent-safe; applied automatically at startup.

13. Working method - how to think before you type

The rules above say WHAT the code must look like. This section is HOW to work so that safe, simple, non-redundant code is the default output, not a review outcome. It encodes the failure modes actually observed in this repo’s MR history (§10) and the conventions agents keep re-breaking.

13.1 Comprehension before code

  • Read the exemplar first. Every recipe in §12 names one. New code in this repo is pattern replication, not invention: find the nearest sibling (adapter, step service, route, repo method), read it fully, and match its shape. If your design differs from the sibling’s, the sibling wins unless you can say why in one sentence.
  • Trace the whole flow end to end before editing: HTTP route → handler → service → repo/adapter, or YAML → generated handler → step service. A change made at the wrong layer is a second bug even when it works.
  • Grep for prior art before writing anything reusable. This workspace already has: DataFrame test factories (mando-core/src/util/test.rs), tracing capture (test_util.rs), 21 Python error mappers, ValidatedJson, dispatch macros, retry/rate-limit utils, resolution converters. Re-implementing a helper that exists a few files over is the most common form of redundancy.
  • Find every caller before changing shared code. Fix at the shared function, never per-caller; sibling configs mirror each other (a datapoint added to one data group usually belongs in its twin - MR !362).

13.2 The simplicity ladder (repo-tuned)

Stop at the first rung that holds:

  1. Does it need to exist? No scaffolding “for later”, no speculative abstractions, no single-implementation traits. Reviewers delete unused code on sight (§10 theme 2).
  2. Workspace already does it? Reuse the helper/type/pattern that exists (see grep list above).
  3. Std or tokio does it? std::sync::OnceLock / tokio::sync::OnceCell, not new crates (once_cell is deliberately absent).
  4. An already-pinned dependency does it? itertools, strum, serde_with are in. Never add a new crate for what a few lines can do - the mime crate was rejected in review exactly this way (MR !345).
  5. Only then: the minimum code that works, written in the module’s existing style.

One escape hatch, used sparingly: a deliberate simplification with a known ceiling gets a //TODO BE-XXXX: reason naming the ceiling. That is the only sanctioned comment form here.

13.3 Safety defaults

  • Typed errors from the first line: define/extend the component’s thiserror enum before writing the happy path, propagate with ?, and decide the logging site once (detection site with error! + ErrorWithStepStatus::logged, or step boundary with ::new - never both).
  • Never weaken a trust boundary to simplify: ValidatedJson/JsonSchema on write bodies, DataPointId validation, and status-code determinism stay, even when the shortcut is tempting.
  • Change symmetrically or not at all: repo trait methods land in all four backends; new code compiles under --all-features; generated files move only together with their YAML/template inputs; released flow versions are never edited.
  • Prefer additive, reversible changes: extend enums/traits rather than reshaping them; do not churn legacy call sites (fmt, positional format!, envconfig) while adding features - unrelated churn is both risk and the top review rejection.
  • Prove it before claiming it: run the §10 gate commands and paste-real results; a green pipeline and a conflict-free rebase are preconditions for review, not afterthoughts.

13.4 Non-redundancy defaults

  • One source of truth per fact: an error message lives in the enum variant (to_string() IS the message - no parallel custom strings); a schedule lives in the manifest; a version lives in [workspace.package]; config lives in one struct.
  • Bind once, dispatch once: when the same match arms repeat per type, bind the service/value once and reuse it (asked explicitly in MR !556); shared logic between two steps/adapters moves to util or the shared crate at the second occurrence, not the fifth.
  • Copy the sibling’s shape, share its parts: duplicate STRUCTURE across siblings is convention (every adapter looks alike); duplicate LOGIC is debt. Extract the logic, keep the shape.
  • Delete as you go: replacing a code path removes the old one in the same MR; nothing ships commented-out, unregistered, or “kept for reference”.

13.5 The pre-done gate

Before declaring any task done, in order:

  1. git diff origin/develop and read your own diff line by line: only ticket-related lines, no stray blank lines, no unused imports/code, no full paths for imported items, no warn! hiding a real failure.
  2. Walk the §10 recurring-review list as a checklist - it is literally what the reviewers will run.
  3. Run the gates: cargo clippy --release --all-features, cargo test --all-features --release --lib -- --test-threads=1, scoped rustfmt on touched files (always on new files; never any form of cargo fmt), plus the relevant pytest suite if Python surface changed.
  4. New non-trivial logic has an inline test next to it; externals are mocked with httpmock; edge cases are parameterized.
  5. Commit per the cadence (§2.7-8): short no-scope title, nothing after the title line.

When two designs both pass every rule here, choose the one that already exists in this codebase.

14. Deployment - getting a build onto an environment

Deployment is a two-repo ceremony: this repo builds and publishes images; the sibling repo optimization-universe-iac (terraform, ECS on AWS) decides which image version runs where. You never deploy from the mando repo alone.

14.1 The environment matrix

EnvironmentAWS accountComponent groupIaC branch (plan / apply)ECR (mando image)Apply
dev794038257734 (bit-bessos-dev)dev_testplan: develop, feature/*, bugfix/*; apply: develop only843164609896.dkr.ecr.eu-central-1.amazonaws.com/poc/mando/deploymanual
test071128452852 (bit-bessos-test)dev_testdevelop, rc/*same dev ECRmanual
int621553445748 (bit-bessos-int)int_prodrelease/*748634852998.dkr.ecr.eu-central-1.amazonaws.com/poc/mandomanual
prod282467977019 (bit-bessos-prod, SIMULATION_MODE: false)int_prodmainsame prod ECRmanual
  • terraform/main.tf maps env → group (dev/test → dev_test, int/prod → int_prod) and builds the image ref as ${ecr_repos[group]["mando"]}:${components[group]["mando"].version}.
  • Naming trap: int_prod is a shared version GROUP, not “production” - in practice applies consuming int_prod values have only run for INT, and the real prod deploy mechanism is still TBD. Verify with the team before treating any int_prod change as customer-facing.
  • Every terraform_apply:{env} job is when: manual - a push never applies by itself; someone clicks Apply in the IaC pipeline after reviewing the plan artifact. Feature/bugfix branches of the IaC repo only get a dev PLAN - the pin bump must reach the IaC repo’s develop before terraform_apply:dev exists to click.

14.2 Deploy to develop (the standard flow)

  1. Migration superset check: the deployed branch’s refinery migrations must be a SUPERSET of what the target DB has already applied - a branch missing a migration the DB has crash-loops mando on boot (V...__... is missing from the filesystem). Merge develop into your branch before building anything you intend to deploy.
  2. Produce the image: merge your MR into mando develop. The Publish Service Docker Dev job (runs only on develop and rc/*) pushes the image to the dev ECR tagged {cargo_version}-dev.{pipeline_id}.{short_sha} (the + from APP_VERSION becomes - in the docker tag, e.g. 1.16.0-dev.2555942905.50575825). Grab the exact tag from the publish job’s Docker image pushed to ... line or from ECR.
  3. The optimization-universe change: in optimization-universe-iac, on its develop branch, edit terraform/terraform.auto.tfvars.json and set components.dev_test.mando.version to the new tag. Commit convention observed in that repo: feat: bump component versions (title-only, same commit rules as here).
  4. Plan: push; terraform_plan:dev runs automatically and attaches the plan as an artifact/report. Read it - the only expected change for a version bump is the mando container image in the bess-os ECS task definition.
  5. Apply: trigger the manual terraform_apply:dev job. Known false-red: terraform_apply:dev always shows red because of a pre-existing customer-portal S3 403 HeadObject that is not ours - open the job log and confirm Apply complete!, then ignore the red. ECS rolls the service with the new image. (dev_test covers test too: terraform_plan:test/apply:test run from develop/rc/* with the test env-var block.)
  6. Wire-type coupling: if the change touched shared wire types (Arrow schemas, py-mando interfaces), everything that speaks them must come from the SAME build - mando image, py-mando wheel, and the optimization/forecast consumer images - and consumers must be locked to the matching wheel. Bump them together in the tfvars, never mando alone.

14.3 Deploying a feature-branch build (the CI branch-name change)

Mando publishes images only from develop and rc/*. To test an unmerged branch on the dev environment:

  1. In mando’s .gitlab-ci.yml, temporarily add your branch name to the only: list of Publish Service Docker Dev (and its Setup:DevTest / build dependencies if they carry the same restriction) - e.g. add - /^bugfix/BE-3333$/. Precedent: the BE-1595 deploys did exactly this, plus optional: true on the publish job’s needs: entries that referenced jobs absent on feature branches.
  2. Push, let the pipeline publish the image, note the tag.
  3. Bump components.dev_test.mando.version in the IaC repo as in §14.2 and apply (the bump itself still has to land on the IaC repo’s develop - its feature branches only plan).
  4. Drop the CI commit before the MR merges (rebase it away or revert it) - a branch-name edit in .gitlab-ci.yml is exactly the “unrelated change” reviewers reject (§10 theme 1), and leaving it merged would publish every future push of that branch.
  5. Mind §14.2 step 0: a feature branch behind develop on migrations will crash-loop on the dev DB.

14.4 Runtime configuration changes (no rebuild)

Mando’s per-environment runtime config (flow cron schedules MANDO_FLOW_SCHEDULE_*, MANDO_FLOW_ACTIVE_VERSION, adapter hosts like EBS_SMB_HOST, SIMULATION_MODE, CPU/memory sizing, notification channels) lives in the IaC repo’s .gitlab-ci.yml under the .environment-vars:{dev|test|int|prod} blocks, flowing into terraform as TF_VAR_*. Changing a schedule or host = edit that block in the right env, then the same plan → manual apply ceremony. Constraint: MANDO_FLOW_ACTIVE_VERSION (e.g. V1_4) must name a setup version that exists in mando’s config/flows/manifest.yaml - bump them together when a new flow setup ships.

14.5 Other environments

  • int: cut/update a release/* branch in the IaC repo; it plans/applies the int_prod group, whose images come from mando release/* builds (prod ECR, tag {version}-{pipeline_id}.{short_sha}). Set components.int_prod.mando.version accordingly.
  • prod: same int_prod group, deployed from the IaC repo’s main branch. Same manual-apply gate. Remember the §14.1 naming trap before assuming blast radius.
  • Never edit components.int_prod.* as part of a dev deploy MR - version bumps are per-group, and mixing groups in one MR invites the unrelated-changes rejection.

14.6 Post-deploy verification (the testing ceremony)

  1. Version check first: GET https://mando.{env-domain}/version returns the running mando version plus its component versions - confirm it equals the tag you bumped. GET /health must be 200.
  2. Smoke the surface: /swagger (or /docs) loads; a read-only data query endpoint returns data.
  3. Watch the logs: Datadog EU site, filter env:dev, services bess-os-service-mando (plus bess-os-algo-optimization / bess-os-algo-forecast when consumers moved); look for error.kind/error.code events and the startup migration lines (refinery applies on boot - the §14.2 step-0 crash-loop is the most common bad-deploy signature).
  4. Flows: dev runs SIMULATION_MODE=true; confirm scheduled flows fire per their cron (MANDO_FLOW_SCHEDULE_*) and land Success/Warning step statuses in the flow execution endpoints rather than Fatal.
  5. Local pre-deploy E2E exists too: test/pi1/end-to-end.py posts mock data against a locally running service - use it before the IaC bump when the change touches data ingestion.

14.7 When editing the IaC repo itself

  • Gate dev-only resources on var.environment == "dev", NEVER on local.environment_group - the group collapses dev+test and int+prod, so a group-gated “dev” resource silently appears in test (and int+prod respectively).
  • The bess_os_ecs security group uses inline ingress blocks: a standalone aws_vpc_security_group_ingress_rule resource gets silently reaped on the next apply. New ingress goes inside the resource (use a dynamic "ingress" block for conditional rules).
  • Topology to keep in mind: mando’s ALB is internal-only (443 → one target group on 8080); the algo containers co-locate in the same bess-os ECS task and reach mando over localhost; the bastion is SSM-only with zero ingress; the VPC is a data lookup owned by another team - no NAT/VPC-endpoint changes from this repo.

Graph View

Backlinks