Alpiq BESS Topics

For Agents

Living index of themes for the Alpiq BESS / Mando project. Each H2 is a topic; bullets are wikilinks to related notes. Updated by obsidian-documenter when documenting work. Read by historian at bootstrap. Topics kept alphabetical.

Arrow Flight / Deployment

be-1595-arrow-flight-dev-deploy-runbook-2026-06-24 — Dev deploy runbook (recurring facts). Three artifacts move together (mando image + py-mando wheel + optimization/forecast consumer images; same build on a wire change). Migration drift → mando crash-loop (V…__… is missing from the filesystem), fix by merging develop so branch migrations ⊇ DB’s. IaC = optimization-universe-iac .worktrees/mando-arrow terraform terraform.auto.tfvars.json (mando/optimization-algo/forecast-algo image pins). terraform_apply:dev always red on a pre-existing customer-portal S3 403 HeadObject despite Apply complete! (harmless, not ours). Runtime switch MANDO_FETCH_STRATEGY=flight + MANDO_FLIGHT_HOST/PORT on algo task defs. Verify in Datadog EU env:dev: bess-os-service-mando/bess-os-algo-optimization/bess-os-algo-forecast.
be-1595-flight-rest-graceful-degradation-2026-06-24 — F4 capability-gap fallback. Flight/DuckDB passthrough only streams within its cache window; a range predating the cache (Permanent storage) returns RepoError::MethodNotSupported (repo_passthrough.rs:335). Was mapped to gRPC FailedPrecondition + client only fell back on is_retryable() → propagated Fatal, crashed forecast. Fix (commit 7926278c): map MethodNotSupported → gRPC Unimplemented (mando-bess/src/flight.rs repo_error_to_status) + ClientError::is_unimplemented() (mandarrow-client/src/error.rs); client falls back on is_retryable() || is_unimplemented() (py-mando/src/polars.rs). KEEP CyclicDependency as FailedPrecondition (real error). Net: optimization fully on Flight, forecast logs falling back to REST then completes. Resolves the “Known limit” in be-1595-flight-execution-id-parity-2026-06-17.
be-1595-flight-wire-type-version-skew-2026-06-24 — #1 deploy hazard: wheel ↔ server skew. Wheel (mandarrow-client Flight wire types + bess client send(update_id=...)) and server deploy independently; a develop merge bumped shared DataPointUpdateInfo (fetch_time→update_time, added update_id) → old-wheel↔new-server broke. Fix (commit 73cbdcf3): tolerant wire types — #[serde(default)] on every optional/added field, #[serde(alias="old")] on renames, NEVER #[serde(deny_unknown_fields)] on boundary structs, + compat tests. Can’t retroactively fix released versions (deploy mando + wheel from the same build on wire changes); send-side signature mismatch only resolves by rebuilding consumers on the matching wheel.
be-1595-arrow-consumer-lock-script-2026-06-24 — Consumer wheel-lock script (deploy-arrow-consumers.ps1 / base64 iex one-liner). Windows on corp net, Nexus must resolve. Clone via token URL (https://oauth2:$Token@…) + GIT_TERMINAL_PROMPT=0/GCM_INTERACTIVE=Never to stop the GitLab credential popup hanging. The base64 one-liner is a frozen snapshot → a stale clipboard silently re-locks the WRONG wheel; re-copy fresh + verify the echoed wheel. Dev images gated behind the python-docker-publish component (Publish Docker Dev only on develop/rc) → script appends a local rules-override for feature/*.
be-1595-flight-execution-id-parity-2026-06-17 — Brings the Flight streaming path to parity with REST for execution-bound manual overrides; fixes the dev regression where MANDO_FETCH_STRATEGY=flight hard-broke the optimization/forecast algo runners (they always fetch with execution_id). Shared mando_lib::app::execution_override::apply_execution_overrides helper (DRY across REST get_data + Flight do_get); execution_id: Option<Uuid> added to both QueryTicket + FlightTicket (#[serde(default)]); overrides injected into DataPointFilters at the HANDLER layer before retrieve_stream (no trait change); MandoFlightService gained an Arc<FlowRepository>; client guard relaxed to error only on access_token (headers + access_token-fallback deferred). Deploy needs mando image + wheel + consumer images.
be-1595-enabling-arrow-flight-consumers-2026-06-16 — How to turn ON the Flight client path in BESS consumers. Runtime switch: py-mando env MANDO_FETCH_STRATEGY (default rest; Flight only when == flight; flight.rs:16, read per-fetch polars.rs:224-236) + MANDO_FLIGHT_HOST/PORT/PROTOCOL. Dev wheel ships -F flight so it’s a pure env flip, no rebuild. Gotcha: Flight-fetch error → SILENT REST fallback (polars.rs:234); confirm via logs/metrics. Per-service topology: algo-optimization + algo-forecast share the bess-os ECS task (bess_os_ecs.tf, localhost) → add 3 env vars (done on feature/mando-arrow, d3db63e); trader-dashboard is a separate task (trader_dashboard_ecs.tf, over local.mando_domain) with no network path to the gRPC port → needs NLB/ALB-HTTP2 plumbing, left on REST. Server side: var.mando_flight_port (50051) exposed via port mapping + SG rule.
Arrow Flight Streaming Design — Flight streaming server/client design.
Arrow Flight Gap Analysis 2026-04-30 — gap analysis.
be-1595-publish-docker-dev-feature-branch-test-need-2026-06-16 — CI gotcha enabling dev-image builds on feature/* for the Flight consumer repos.

BESS AM / Real-time Telemetry

BESS AM (BE-2262) - mando-bess-am — BESS Asset Management (ticket BE-2262, in flight, authored by Gergely Vászon). Headless Kinesis stream processor (new crate mando-bess-am) for per-second WAGO battery telemetry (real-time SoC/SoH) on the FI/Valkeakoski/Beskar asset. Pipeline WAGO Box → AWS IoT Core → Kinesis (bess-am-events) → mando-bess-am → Kinesis (bess-os-events) → mando-bess: consumes raw per-second events, persists to Postgres schema bess_am, computes 1-min aggregates (Mean/Max/Min/Last/StdDev/Count/Sum) via windowing + grace-period closure, forwards to bess-os-events. Introduces a NEW Event-typed data-point class distinct from the time-series flows (NOT one of Trading/Manual Schedule/Auction/Intraday/Data Update/AFRR). Kinesis adapter at mando-lib/src/adapter/kinesis/; Event model mando-core/src/model/{event,wago}.rs; Event datapoints in battery_online.yaml. Builds on BE-2132 (interval/Event groundwork), depends on BE-2341 (generic Kinesis producer, folded into the MR). Core MR !512 OPEN/requested_changes; nothing merged to develop yet.
Dev Deploy 2026-07-14 (error telemetry) - the May feat deploy of the BE-2262 POC left U202605131400__create_event_table + U202605141900__create_kinesis_checkpoint_table in dev’s refinery_schema_history; the files exist only on the unmerged branches, so EVERY develop-based image crash-loops on dev at boot (MigrationError::PatchApplicationFailed). Worked around by restoring both files byte-exact (single canonical blob f45b6dd4 across all source commits). OPEN ACTION: land the two files on develop (small MR, recommended) or clean dev’s history rows.

Bugs / Investigations

fatal-step-double-logging-ancestor-boundary-2026-09-02 — Fatal steps emit the SAME error at the leaf boundary and again at every ancestor wrapper node (run_step logs unconditionally at mod.rs:499; status_or_error() returns Err only for Fatal, ? re-surfaces it at the wrapper). Fingerprints diverge by step path so copies don’t aggregate. Not the BE-3541 axis; still live on develop b5770766. Trigger was a DuckDB duplicate PK already fixed by MR !610 (dab6cba6) but absent from prod 7c1ef44f.
pymando-rust-log-trace-correlation-gap-2026-09-02 — Rust tracing events inside the Python algo containers carry no dd.trace_id/dd.span_id/logger.name/thread_name: py-mando/src/log_formatter.rs is a second, thinner formatter, and dd_formatter’s TraceInfo/lookup_trace_info are private AND behind the app feature py-mando never enables. TraceFilter contextvars (tracing.py) also never reach Rust events. Reusing dd_formatter’s helper is RULED OUT (it needs OtelData and py-mando installs no OTel layer); fix is B1 metadata parity + B2 bridging the ddtrace span context Python→Rust per-call.
be-1595-flow-repository-sqlite-bugs-2026-06-17 — Two latent FlowRepositorySqlite bugs (local Sqlite dev/test path only, NOT Postgres prod; worth a ticket), found writing the test seed for the Flight parity work. (1) create_flow (flow_repository_sqlite.rs:111) uses lazy stmt.query() for an INSERT + drops Rows without iterating → SQL may never run, row never persists (use .execute()/.insert()). (2) started_at schema default current_timestamp stores TEXT but get_execution reads column 2 as i64 → row created without explicit integer started_at fails to read back (InvalidColumnType TEXT vs i64).
calculated-virtual-dp-range-cutoff-bug-2026-05-18 — Calculated & Virtual DPs leak rows past to (all 4 retrieve methods); root cause in service_base.rs:94-159, mechanisms in convert_resolution.rs upsampling + evaluation.rs Full-join/concat. Diagnosed, fix pending.
flow-step-log-message-dropped-2026-05-26 — ErrorWithStepStatus::log(status, message) drops message at parent log site; StepResult::log() (mod.rs:155-159) destructures with .., Display is "status: {status}" only (mod.rs:443), status_or_error collapses Log to Ok (mod.rs:231-239), top-level catch in mando-bess/.../flow.rs:289 sees "status: Error" only. Error(anyhow) arm OK; only ::Log broken. Tests at mod.rs:548-605 only assert log level + wrapper string. Affects all envs. Cross-check follow-up commits on feature/BE-2272.
mando-cli-v0.4.0-compose-bugs-triage-2026-05-26 — Two confirmed compose-runtime bugs in v0.4.0: context: . resolves to runconfig/ not project root (build.yml:11, build_dev.yml:18); bare service slug passed for mocked runconfig activates malformed override stub via render_override (templates.rs:298-325) — profiles: ["{run_tag}"] line at templates.rs:317 is a load-bearing invariant keeping mocked stubs dormant. Fixes identified, not yet committed.
mando-cli-v0.4.0-mando-bess-binary-missing-2026-06-25 — The NEXT failure after the above triage’s fixes (commit 2233959). On a fresh clone, default profile mando-mocked-algos aborts docker compose build at COPY target/release/mando_bess → failed to compute cache key … not found. Design gap: mando/Dockerfile is a thin runtime image copying a PRE-built binary (no Rust stage), bess-service.yaml:23-32 context_includes it (CI assumption), and the default profile maps mando→build runconfig (runprofile.rs:225) while nothing in up compiles it (--cargo is artifact-only, cli.rs:55). Workaround: mando up --mando=pull (pulls the registry image). Durable fix pending decision: flip default → pull (open risk: confirm default MANDO_IMAGE → registry not mando:dev) + add an up preflight guard for missing binary. Diagnosed, not implemented. Reported by gabi on WSL.
mando-cli-v0.4.0-piped-output-invisible-failures-2026-06-25 — Observability bug found while validating the above’s --mando=pull workaround. When up output is piped/redirected (2>&1 | tee log), failures are INVISIBLE: captured .mando/compose-up.log empty, only a bare docker compose up … failed (exit 1) shown. Two stacking root causes (both required): (A) docker compose writes progress+errors to /dev/tty, bypassing mando-cli’s pipe (compose.rs run_streaming) → fix: top-level --progress plain (compose.rs ~L231); (B) CliBuffer::add_log_line (opts.rs:57) uses indicatif::MultiProgress::println, a silent no-op on non-TTY, dropping all streamed output + the failure-tail replay (compose.rs:263-272) → fix: eprintln! fallback when !stderr().is_terminal(). Validated macOS: redirected run 35→138 lines, real pull access denied + tail + log pointer now visible. Reusable: pipe-captured docker compose needs --progress plain/--ansi never; indicatif println needs eprintln! fallback on non-TTY. Tester’s now-visible failure = registry pull access denied (needs docker login, distinct from mando-cli’s GitLab-API auth). Both fixes built+validated, NOT committed; linux/amd64 build pending.
BESS AM (BE-2262) - mando-bess-am (Update 2026-06-24) — FIXED 2026-06-25. mando-codegen casing bug: expand_variants mis-cased python_field for multi-segment Named variants — Min1/Mean → market.da.price.min1.Mean instead of all-lowercase …min1.mean (lowercased first path segment but not the value segment). Affected BESS AM 1-minute aggregate datapoint names (Min1/Mean, Min1/Max, Min1/StdDev…). Originated in the branch’s own feat: separate out bess-am commit 5a739baa, NOT the develop merge (mando-codegen byte-identical before/after). Fix: lowercase the value segment at mando-codegen/src/util.rs:41 → format!("{pf}.{name_lower}.{}", v.to_lowercase()); commit cc4720b2 on feature/BE-2262-bess-am-poc, pushed to MR !512. Verified cargo test -p mando_codegen now 51/51 (was 50/1). Residual (unrelated): two pre-existing rustfmt drifts remain in that same file (~L50, ~L390, author’s code), left untouched. (Same update also records origin/develop merged into the !512 branch as 80a17e78 — compiles clean, 428 lib tests pass.)
Dev Deploy 2026-07-14 (error telemetry) - dev crash-loop at boot: MigrationError::PatchApplicationFailed: migration V202605131400__create_event_table is missing from the filesystem. Root cause: dev’s refinery_schema_history carries two migrations from the UNMERGED BE-2262 bess-am branches (May feat deploy), so refinery validation fails for any image built without those files; the INVERSE of the BE-1595 drift case, merging develop cannot fix it. Fix: byte-exact restore of both files (blob f45b6dd4, checksums pass) as 6bc418cd. STANDING WARNING until develop carries the files or the rows are cleaned.
fr-region-missing-datadog-logs-2026-07-21 - FR region services silently absent from Datadog logs: modules/bess_os/cloudwatch.tf creates the -fr log groups but the dd-logs-forwarder subscription-filter modules were dropped at module extraction (they live only in top-level cloudwatch.tf). Diagnosed with live AWS evidence, fix not applied.
mando-ci-lib-only-test-gate-2026-07-22 - mando-bess/tests/flight_end_to_end.rs is compile-broken on develop itself (E0603: DataPointUpdateInfo imported via mando_lib::service where it is only a private use since the b3ce27a8 repository crate split), undetected because the CI test gate is --lib-only; fix is a pending one-liner (import from mando_core::model::datapoint), its own change on Andras’s yes.
mando-cli-full-audit-2026-07-22 — full ~18.7k-line mando-cli audit: 37 verified bugs (2 HIGH — unguarded kill -TERM -{pid} mass-kill in host_process.rs:108-133 with no pid<=1 guard; orphaned untracked host procs on record failure in up.rs:262-270 — 18 MEDIUM, 17 LOW). Themes: exit-0-on-failure (migrate/get/pull), no reqwest timeouts anywhere, wiremock.enable() silent no-op. Also a −280-line/−5-dep over-engineering cut list, 9 duplication clusters (compose-context prologue copy-pasted across 8 command files), and the simulator runtime spec unimplemented on main. No fixes applied.
mando-e2e-rebase-2026-08-05 - rebasing mando poc/e2e-tests onto develop after ~135 commits of drift surfaced an upstream rename (crate::service::MandoServiceConfig → mando_repository::model::DataPointRepositoryConfig) that a default cargo test CANNOT see, because the app module is feature-gated and gets silently skipped — the feature-flag twin of the --lib-only CI gate lesson. Also two live env-var races (MANDO_TEST_ENDPOINTS in the new CSV route tests, fixed; MANDO_DEBUG_MOCK_ERROR upstream in debug_error, reported).
pymando-logger-init-race-flaky-2026-08-10 - test_dd_conformance.py::test_init_populates_identity_and_dd_env intermittently sees identity["service"] == None. Root cause spans the Rust/Python boundary: constructing AlgoRunner (algo_runner.rs:~113) std::thread::spawns a permanent Axum server thread that test_algo_runner.py LEAKS; each thread re-runs logger.init() with no service arg, and logger.py init() sets identity then _initialized several slow lines apart (gap includes import ddtrace.auto), so a background init(service=None) overwrites identity mid-window. A threading.Lock mitigation shipped (MR !601, feature/BE-4067) but a residual microsecond window remains — the real fix (stop leaking the server thread / isolate the test) needs its own ticket. Tell: interleaved Starting Algo runner service, listening on 127.0.0.1:3003/3002 log lines before the failing test.

CI/CD

bess-os-ci-components — the org’s shared GitLab CI component library (alpiq_cicd/sales-and-origination/flexible-assets/bess/poc/bess-os-ci-components, tags v1.0.0..v1.4.0, semantic-release; templates/<name>/template.yml: mr-jira-check, python-setup, python-test, python-docker-publish, release-notes, simulator-pipeline). Consumer .gitlab-ci.yml = ~40 lines of include: component: + spec: inputs:, pinned @v1.x.0. Two patterns worth stealing: python-setup’s dotenv broadcast (one job derives PROJECT_VERSION/ECR/release metadata from branch + pyproject.toml, exports via artifacts: reports: dotenv: build.env, downstream inherits through needs:) and python-test’s service-less + prebuilt-wheel shape (no services:, DB_DISABLED=true, poetry pulls a prebuilt py-mando wheel from Nexus so there is ZERO Rust compilation — that is why the 54/68-case suites cost 7.4/13.8 min; pre_test_script hook, --junitxml + junitparser merge + artifacts: when: always). Branch rules: MR events, develop, rc/*, main, release/*. The Rust mando pipeline does NOT consume it.
BESS AM (BE-2262) - mando-bess-am — adds a dedicated CI job “Publish BESS-AM Docker Dev” building the mando-bess-am image from container.bess-am.Dockerfile (ECR tag prefix bess-am-<version>). Crate-split branch feature/BE-2262-bess-am-poc-build-test (commit 5a739baa) separates it into its own image.
be-1595-publish-docker-dev-feature-branch-test-need-2026-06-16 — GitLab gotcha: enabling Publish Docker Dev on feature/* in the BESS Python consumer repos needs optional: true on the .Publish → Test dependency, or the pipeline won’t start.
be-1595-arrow-consumer-lock-script-2026-06-24 — consumer dev images gated behind the python-docker-publish component (Publish Docker Dev only on develop/rc); the wheel-lock script appends a local rules-override so feature/* builds. Also the GitLab credential-popup hang fix (GIT_TERMINAL_PROMPT=0/GCM_INTERACTIVE=Never + token-URL clone).
be-1595-arrow-flight-dev-deploy-runbook-2026-06-24 — terraform_apply:dev always red on a pre-existing customer-portal S3 403 HeadObject despite Apply complete! (harmless); IaC image pins in optimization-universe-iac .worktrees/mando-arrow terraform.auto.tfvars.json.
clippy-disallowed-macros-allow-placement-2026-06-22 — keeping clippy green across versions: disallowed_macros allow must be a module-root #![allow] at the macro’s definition (mando-core/src/error.rs), not at the call site (ineffective on clippy 1.95). Workspace pins 1.88.0; verify clippy-clean under the pinned toolchain since the call-site/block form is version-fragile.
mando-cli-github-build-mirror — GitHub-Actions build pipeline, GitLab Package Registry publishing, parallel-release flow (init → matrix-publish → finalize).
mando-cli-gitlab-release-flow-2026-08-05 — mando-cli’s FIRST-EVER GitLab pipelines, unblocked by clearing the project’s stale ci_config_path (canonical diagnosis for “committed .gitlab-ci.yml, no pipeline created, no error shown”). Windows builds removed entirely. Single long-lived release branch flow: release mints the tag + GitLab release + moves latest; main only builds/publishes versioned packages. First release blocked solely on MR !5 approval — a group-level approval rule the author cannot self-serve from the CLI. UPDATE 2026-08-05 evening: v0.4.0 SHIPPED (tag v0.4.0+2734905610, full pipeline green Version → musl builds → Publish+latest → Release). Two reusable GitLab gotchas from the one failed Release run: dependencies: overrides needs: for artifact download (both were declared, so the Version job’s reports: dotenv PACKAGE_VERSION never arrived and tag= was empty — an empty dotenv var with no error is the signature), and markdown backticks in release descriptions get shell-evaluated by release-cli’s busybox wrapper (keep release notes plain text).
mando-cli-wsl-linux-build — local macOS (Apple Silicon) cross-compile of a Linux x86_64 / WSL mando binary; --platform linux/amd64 + poc/-parent-mount gotchas. The manual counterpart to the CI mirror.
Mando AGENTS.md Master Guide — the verified mando gate set: NO cargo fmt gate in CI (never bare cargo fmt — ~64 legacy files reformat; scope to touched files); lint gate cargo clippy --release --all-features; canonical test cargo test --all-features --release -- --test-threads=1 (single-threaded MANDATORY — shared in-memory DB pools). Runtime image now ca-certificates/libc6/libssl3 only (samba dropped); build image base still rust:1.88.0-bookworm + samba (toolchain 1.89.0 via rust-toolchain.toml). Pipeline etiquette: title-only Conventional Commits, feature/BE-xxxx branches, py-mando publish is manual, release/* flows back to main+develop. Round 2 (2026-07-13): rustfmt.toml’s import-grouping/comment options are nightly-only + silently ignored on stable 1.89 (tip tree NOT fmt-clean — import grouping maintained by hand); CI gained a “PyMando Simulation Linux Dev” job (pytest without --nbval) + a simulator docker publish child pipeline; no MR/issue templates, no CODEOWNERS.
Mando Deployment Ceremony - the two-repo deploy ceremony (AGENTS.md section 14): mando CI only publishes images (Publish Service Docker Dev, only: develop + rc/, tag {version}-dev.{pipeline_id}.{short_sha}, APP_VERSION + becomes -); optimization-universe-iac pins versions with ALL terraform_apply jobs when: manual. CORRECTED 2026-08-18: pins are NO LONGER in terraform/terraform.auto.tfvars.json under a dev_test group - since IaC 3f50c77 (2026-07-14) that file is gone and each env has its own terraform/environments/{dev,test,int,prod}.tfvars with a flat components.mando.version, selected by terraform plan -var-file=environments/${CI_ENVIRONMENT_NAME}.tfvars; local.environment_group no longer exists anywhere in terraform/, and ecr_repos moved into the per-env files too. dev and test are independently pinnable and have diverged in practice. Verified branch rules on origin/develop @ 9600b34c: dev plan develop|feature/*|bugfix/*, dev apply develop, test plan+apply develop|rc/* (rc/ DOES still drive test), int release/*, prod main. Feature-branch deploy = temporary only: edit that MUST be reverted before the MR merges. Never deploy from mando alone.
Test Env Deployment State 2026-08-18 - worked example of reading the new per-env pin end to end, plus the apply-failure pattern: three terraform_apply:test runs on 2026-08-11, the first two (pipelines 2750514125 @ 12:37 UTC, 2750720634 @ 13:24 UTC) failed and the third (2750845368 @ 14:12 UTC, IaC 9600b34c) went green - the failures track the two intervening “application asset tag” fix commits (cb07b0b, 9600b34c), not the mando bump. Recipe for “is MR X live on env Y”: git merge-base --is-ancestor <merge-sha> <deployed-sha>.
Dev Deploy 2026-07-14 (error telemetry) - worked example of the feature-branch deploy on BOTH repos: throwaway mando branch feature/BE-3482-dev-deploy (develop tip 51d2b516 + !569 cherry-picks + only: edit + the PyMando Win Dev needs/dependencies entry commented out; that job is absent on feature branches and a dangling need kills pipeline creation) + IaC branch feature/error-hdl (version bump + “ci: enable dev apply from error-hdl branch” adding the branch to terraform_apply:dev only:, which normally exists ONLY on IaC develop). All CI edits temporary; strip on merge.
mando-ci-lib-only-test-gate-2026-07-22 - the CI test gate (.gitlab/scripts/test.sh) runs cargo test --all-features --release --lib -- --test-threads=1: ONLY lib unit tests are gated; */tests/ integration targets are never compiled by the pipeline and rot silently (the separate “Integration Test Linux Dev” job covers only the mando-lib archiver Postgres suite). Local gates need --lib for CI parity. --lib arrived with the archiver split commit 7278f36a. Companion lesson 2026-07-23 (BE-3657 phase 3): per-crate task gates (-p mando_lib) likewise miss CONSUMER crates - the final whole-branch review caught 2 tuple-syntax construction sites in mando-simulator that no task gate ever compiled (a Critical); workspace-wide cargo clippy --release --all-features (no -p) is the only gate that catches cross-crate fallout.
mando-e2e-ci-green-2026-08-05 — “E2E Data Suite Linux Dev” GREEN in real CI (mando pipeline 2735150017, 72s): cross-project download of the released mando-cli binary via CI_JOB_TOKEN + job-token allowlist, postgres:17 + wiremock/wiremock:3x as services: (shared netns → WireMock on localhost:8081), mando_bess as a background host process from the build artifact, refinery migrations on boot, junit uploaded. mando’s .branch_rules:dev gained poc/* (one line). THREE REUSABLE CI LESSONS: deleting a branch mid-pipeline kills its not-yet-started jobs at git fetch (missing ref); cargo-chef must cook the SAME feature sets the jobs build (container.linux.chef.build.Dockerfile cooked bare --release while build.sh uses --features flight and test.sh --all-features, so every job recompiled arrow-flight/tonic — now cooks both, but the rebake trigger is develop-only so the win lands only after merge); and GitLab cache: cannot hold /init/chef/cook/target because it is outside CI_PROJECT_DIR (escalations: weekly scheduled rebake, sccache+S3). PyMando Win Dev at ~2656s is the real pipeline whale.
mando-cli-junit-per-assertion-2026-08-06 — junit granularity in the e2e job raised from one case per test set to one case per assertion, so the MR widget names the failing assertion. Carries the reusable GitLab reporting rule: testcase names must be stable across pass and fail because GitLab keys a test’s history on (classname, name) — derive the name from the expectation, never from the result. Plus the false-green-by-omission guard for tests that never executed.
mando-cli-flow-coverage-2026-08-06 — the E2E job’s GitLab coverage: value is flows exercised / flows the service declares, parsed from the runner’s own last line via '/flow coverage: \d+\/\d+ flows $(\d+\.\d+)%$/' (mando 96b5b89c); the regex needs a decimal digit, so summary format and regex are one cross-repo contract and a mismatch shows as no coverage, no error. An interim cargo-llvm-cov job (green at 40.36%) was fully reverted — do not reintroduce line coverage here. First live value: pipeline 2736559170, 20.0% = 1/5 flows. Badge /badges/poc%2Fe2e-tests/coverage.svg?job=E2E+Data+Suite+Linux+Dev.

Clippy / Lints

clippy-disallowed-macros-allow-placement-2026-06-22 — clippy disallowed_macros (mando bans tracing::error!/log::error! in clippy.toml to force logging through the mando_core::error! wrapper): a per-invocation/per-block/per-fn #[allow(clippy::disallowed_macros)] at the call site is silently ineffective on a newer clippy (1.95) and is flagged “unused, applied to a macro invocation”. The lint resolves at the lexical site where the banned tokens live (the macro definition), so the only robust fix is a module-root #![allow(clippy::disallowed_macros)] in the defining file (mando-core/src/error.rs) — one allow covers all cross-crate callers. Caveat: workspace pins 1.88.0; the block form on develop presumably passes CI on 1.88 but is fragile across clippy versions; module-root #![allow] can’t be worse on 1.88 and is confirmed on newer clippy. General rule: for lints that fire on macro expansions, #[allow] at the call site no-ops — put #![allow] at the macro’s definition.

Conventions / Agent Guides

Mando AGENTS.md Master Guide — master conventions doc for AI agents at /Volumes/bandi/coding/poc/mando/AGENTS.md (untracked, deliberately never committed; synthesized 2026-07-10 from 5 parallel researchers on develop). 12 sections + 8 recipes: crate map/dependency direction, 10 non-negotiables, Rust style, mando_core::error! system, mando-bess API + py-mando PyO3 conventions, testing/git/CI ceremonies, verified-landmines table. Headline corrections: mando-bess/build/generated/** is generated BUT tracked in git (edit YAML/template → build → commit both, never hand-edit); NO ErrorCode derive / NO mando-lib-macro on develop (worktree experiments only; parent poc/CLAUDE.md stale); JWT token_layer signature validation DISABLED (audit metadata, not authz); flow versions immutable once released. Rebuilt 2026-07-13 (round 2) against the true develop tip 92bfe1c8 (Jul-9, v1.16.0) via 5 re-run researchers on a .worktrees/develop worktree — the stale-merge-base caveat is obsolete; local develop fast-forwarded + now tracks origin/develop. Round-2 facts: FlowStepService = construction trait (type Params: ParamMeta; type Response; + async from_config(config, &StepProviders)), manifest.yaml deployment catalog (schedule_env/semaphore_group, semaphores capacity 1), 36 type_entry! registrations, error redesign partially landed (ae6d1098), py-mando-simulation split (BREAKING), second async bridge (future_into_py), rustfmt opts nightly-only/ignored, zero tests in the new crates, no MR templates/CODEOWNERS. Maintenance rule: code wins, then update the file. Rebuild process gotcha: agentops:researcher subagents can’t SendMessage — collect reports via scratchpad files or transcript extraction (~/.claude/projects/<project>/<session>.jsonl, longest assistant text).
Mando MR Review Patterns - evidence base behind the guide’s “Recurring review feedback” section: all 21 of Andras’s mando MRs (!322 through !558; 17 merged / 2 closed / 2 open) mined 2026-07-13 via read-only glab API; 108 reviewer notes (krisztian.fekete1 87 / gabor.nagy6 13 / balazs.mracsko.alpiq 7). Nine themes with source MRs: zero unrelated diff changes; delete ALL unused code before review (most repeated complaint); failures are errors never warnings (mando_core::error! + enum variant, no custom message strings); placement (route helpers next to routes, util in util); completeness before review (mirror sibling configs); green pipeline + no conflicts before assigning review; rustfmt every new file; Jira ticket in sync; idioms (errors imported from mando_core, no unwrap, bind service once). Plus config direction (new config: config::Config over Envconfig) and the dictated rules: commits title-only NO scopes/descriptions (reviewer-mandated !322), commit every ~3 tasks, never em dashes anywhere, always subagent-driven development.
mando-repos-history-rewrite-2026-08-05 - the standing attribution-free history policy applied across both repos (mando-cli main/release/4 re-pointed tags, mando poc/e2e-tests), and the verification discipline it forced: rtk-wrapped grep/log pipelines produced fabricated zeros, so an rtk-mediated negative is never evidence — re-prove absences with rtk proxy git redirected to a file. Third member of the rtk hazard family alongside the --test-threads=1 false-green and the piped-exit-code trap (all three now in Agent Context).
BE-3482 Datadog Logs and APM Conformance - source of the 2026-07-13 fmt correction: cargo fmt -- <file> does NOT scope (field-verified, it touched api_docs.rs which was never named); format single files with rustfmt --edition 2021 <file>, every form of cargo fmt banned. Propagated to AGENTS.md sections 2.3, 10, 13.5 + the landmines table, Mando AGENTS Guide (mirror) resynced, fmt warnings in Mando AGENTS.md Master Guide / Agent Context amended.
BE-3482 Datadog Logs and APM Conformance - second 2026-07-13 tooling landmine: the RTK shell hook rewrites cargo test --all-features --release -- --test-threads=1 so the threads flag becomes a test-name filter (every test filtered out, exit 0 in 0.00s, signature 0 passed, N filtered out; two gate runs false-green before detection). On RTK machines run rtk proxy cargo test --all-features --release -- --test-threads=1 and confirm the summed passed totals are nonzero; exit 0 alone proves nothing. Adjacent pitfall: cargo test | grep | tail; echo $? reports tail’s exit, not cargo’s. Recorded in AGENTS.md section 10 + the landmines table; mirror resynced.
BE-3482 pymando Branch Review - third 2026-07-13 tooling landmine: rustfmt --check --skip-children is UNRECOGNIZED on the pinned stable rustfmt and prints nothing (false green); check without the flag, attribute reported hunks by the Diff in <path>: headers (rustfmt follows mod declarations into child files), and baseline against git show <base>:<path> to separate pre-existing churn from branch-introduced churn. Recorded in AGENTS.md section 10 + the landmines table; Mando AGENTS Guide (mirror) resynced.
BE-3482 pymando Branch Review - fourth and fifth 2026-07-13 tooling landmines (from the rework session): (a) rustfmt base-file checks that COPY the file to a scratch dir are FALSE GREEN (rustfmt cannot resolve the mod children from the new location and reports 0 hunks), and running rustfmt on a module-declaring file WRITES INTO its child files (6 unrelated py-mando files were touched and reverted); the only sound method is checking out the base revision at its real path; (b) develop’s committed Cargo.lock can be STALE (mandarrow-client 1.16.0 vs workspace 1.16.1); plain cargo build regenerates it, so a lock hunk in a branch diff may be legitimate, not churn. Recorded in AGENTS.md section 10 + the landmines table; Mando AGENTS Guide (mirror) resynced.

Data Point Retrieval

BESS AM (BE-2262) - mando-bess-am — introduces a NEW Event-typed data-point class (real-time telemetry) alongside the existing TimeSeriesDouble/TimeSeriesDoubleMatrix/StaticData. Event datapoints (BATTERY_SOC_ONLINE/BATTERY_SOH_ONLINE) defined in mando-bess/config/parts/battery_online.yaml under Asset/FI/Valkeakoski/Beskar/Battery/.../Online, keyed to Kinesis external IDs [bess-am-events, OnlineSOC/OnlineSOH]. Model at mando-core/src/model/event.rs; built on BE-2132 interval/Event groundwork.
calculated-virtual-dp-range-cutoff-bug-2026-05-18 — [from, to) range not enforced for Virtual/Calculated DPs in handle_data_point_types; EvaluationMetaData.range plumbed but unused by final output.
mando-data-insert-wire-format-2026-08-06 — the insert side of the contract, pinned: POST /data/insert (mando-lib/src/app/route/save_data_route.rs) takes {"data_points": {"<dp id>": [<bytes>]}} where the bytes are an Arrow IPC STREAM encoded as a JSON array of decimal u8 (not base64 — polars 0.49.1’s deserialize_map_bytes only reaches bytes via visit_seq under serde_json); reader is polars-arrow StreamReader, update_info optional, 204 on success, Content-Type never enforced. Required schema: value_time/generation_time/fetch_time Timestamp(Microsecond, None) naive-UTC, value/value_x/value_y Float64, id Utf8 for matrix/trade; a flag column and any null cell are rejected; get_column_date_times ignores tz and normalises ns/us/ms. Fixture fingerprint: 936 IPC bytes → ~9KB JSON.
mando-cli-e2e-live-green-2026-08-05 — two breaking DataPointId changes landed upstream (found live 2026-08-05): (1) its Deserialize/Serialize were rewritten from a {id, id_fragments} struct to a bare JSON string, so any hand-mirrored query body sending the object shape now gets a 400 from POST /data/query/v2; (2) DataPointId::new now rejects path fragments that are not purely ASCII alphanumeric — ids with an underscore inside a segment (bess/fi_north/soc_state, the in_/out_ corpus naming) 400 on query. The second is UNCONFIRMED as intentional and needs raising with the mando team; mando-cli’s docs/e2e-guide.md still documents underscore ids as valid.

Docker Compose / Runtime

mando-cli-simulator-runtime-2026-05-30 — adds second compose project mando-sim as a peer to dev mando; two-stack ergonomics via shared service_stack::select_profile resolver (single source of truth for logs/exec/volume); env merged via fill_build_args (NOT --env-file, which hard-fails on missing .env); Python healthcheck; Gurobi license as file mount, not multiline env. (§A env contract SUPERSEDED 2026-06-02 — see mando-cli-simulator-env-contract-2026-06-02)
mando-cli-simulator-env-contract-2026-06-02 — realigned simulator env to real images: orchestrator SIMULATOR_DATABASE_* (replaces POSTGRES_*), per-runner SIMULATOR_<NAME>_HOST (replaces SIMULATOR_RUNNERS CSV), repo URL/branch/commit_hash moved orchestrator → runners. Runners read unprefixed SIMULATOR_REPO_URL/BRANCH/COMMIT_HASH (assumption — verify when image lands). Healthcheck Python → curl; Gurobi mount → GUROBI_LIC env. Schema bess_simulation → simulator.
mando-cli-v0.4.0-compose-bugs-triage-2026-05-26 — v0.4.0 baseline compose-runtime triage; context: . relative-path bug; load-bearing profiles: ["{run_tag}"] invariant in render_override.
mando-cli-v0.4.0-mando-bess-binary-missing-2026-06-25 — next compose-build failure once the above fixes (commit 2233959) landed: default mando-mocked-algos resolves mando to the build runconfig (runprofile.rs:225), but mando/Dockerfile is a thin runtime image that COPYs a pre-built target/release/mando_bess and the up flow never compiles it → docker compose build aborts on a fresh clone (failed to compute cache key: "/target/release/mando_bess": not found). Workaround mando up --mando=pull; durable fix (flip default → pull + preflight guard) pending decision.
mando-cli-v0.4.0-piped-output-invisible-failures-2026-06-25 — the streaming compose runner (compose.rs run_streaming) loses ALL output under pipe/redirect: docker compose writes progress+errors to /dev/tty (bypassing the pipe) → add top-level --progress plain (compose.rs ~L231); and CliBuffer::add_log_line (opts.rs:57) routes lines through indicatif::MultiProgress::println, a non-TTY silent no-op → fall back to eprintln! on !stderr().is_terminal(). Together they restore the captured .mando/compose-up.log, the streamed lines, and the --- last 40 log lines --- failure tail (compose.rs:263-272) for piped/CI/tee runs. Both fixes required.
mando-cli-full-audit-2026-07-22 — 2026-07-22 audit of the compose runtime and command layer: 2 HIGH bugs in host-process signalling (host_process.rs), MEDIUM compose/volume/down flag-drop + name-mapping bugs (down.rs, volume.rs, exec.rs/logs.rs), a 9-cluster duplication list headed by the compose-context prologue duplicated across 8 command files (→ compose::prepare), and the simulator runtime spec still unimplemented on main.
mando-cli-datadog-local-logs-2026-07-22 — mando up --datadog renders an optional infra template (src/runtime/templates/infra/datadog.yml → .infra/datadog.builtin.yaml) with a local Datadog agent (container_name local-dd-agent, name:^mando-.* include filter), included existence-driven by compose::assemble (NOT in INFRA_BUILTIN_FILES) so a flag-off up can delete it. Compose interpolates env on EVERY verb → the template MUST use ${DD_API_KEY:-} not :? (a :? hard-fails down/logs when the var is absent while the file exists). Extracted build_args::atomic_write_with_dir unifying 3 write sites. UPDATE 2026-07-23: docs/datadog-guide.md (tip c2becf1) verified end to end, every claim holds: 4 fail-fast paths, mando-only tailing, EU shipping (103 logs), teardown at both levels (flag-off up --remove-orphans + down). UPDATE 2026-07-23 (ddtags override): the embedded ddtags env WINS when mando emits one - Datadog JSON parsing promotes the body ddtags (MANDO_ENVIRONMENT=local → env:local, unindexed by the org → 202-accepted lines VANISH); the “no env in ddtags” claim holds only while MANDO_ENVIRONMENT puts none there; set MANDO_ENVIRONMENT=dev-local-<user> for ANY local mando (containerized included); the per-developer env tag is the only wall from the shared org’s REAL env:prod telemetry under the same service name.
mando-local-host-run-recipe-2026-07-23 - running a locally built mando_bess as a HOST process against the mando-cli infra stack (mando up --datadog -p infra: postgres + wiremock + local-dd-agent). Env from .cargo/config.toml.example + overrides (file is STALE, see Env / Configuration); infra pg creds bess_transactional_db/postgres/postgres @ localhost:5432; MANDO_PORT=8081 (wiremock owns 8080), adapter hosts at localhost:8080 for hermetic 404s. DANGER: mando’s refinery races the mando-cli migrations container in DuckDbPostgres mode (idle-in-transaction boot wedge, move_schema_objects blocked; recovery = pg_terminate_backend + drop public/data/bess_os cascade + single-writer relaunch; DuckDb mode skips pg migrations but the stale container set lacks the setting table). Host processes are invisible to the local dd agent (mando-* filter): ship to Datadog via direct intake. Teardown: mando down + pkill mando_bess. UPDATE 2026-07-23 (epilogue): the first direct-intake forward VANISHED - Datadog promotes the embedded ddtags from the JSON line, so mando’s env:local (from MANDO_ENVIRONMENT=local) overrode the envelope’s env:dev-local-levander and the org indexes no env:local; fix = rewrite embedded ddtags before forwarding; durable = set MANDO_ENVIRONMENT=dev-local-<user> in the run env.

DuckDB

BE-1597 DuckDB UI Exposure Research — BE-1597: exposing the DuckDB UI of running mando-bess + remote client access to its in-memory DuckDB. Root cause both prior attempts failed: the UI JS bundle hardcodes "localhost:4213" === window.location.host ? "duckdb_ui" : "web", so any non-localhost:4213 origin (path prefix, custom hostname, ALB) silently degrades — killing Attempt A (socat 4214 + ALB TG + /duckdb* listener rule; died on the 8080 /health check) and Attempt B (reqwest reverse proxy mando-bess/src/route/duckdb_proxy.rs + <base href>/JS string-patch). Fix = SSM AWS-StartPortForwardingSessionToRemoteHost forwarding the LOCAL port 4213 → browse http://localhost:4213 (literal localhost, not 127.0.0.1); no ALB (already internal=true); bastion hop required since ECS Exec is interactive-only. UI ext: only ui_local_port(4213)/ui_remote_url/ui_polling_interval(284ms) settings, NO bind-address (binds ::1 → socat), no auth, proxies assets from ui.duckdb.org per session (internet required), not statically linked (verified nm -D on vendored 1.4.2 .so → only Icu/Json). Remote client → in-mem DB: only via Quack (DuckDB 1.5.3+, beta until 2.0). Includes a full 1.4.2→1.5.4 upgrade assessment (low-risk, in-memory-only sidesteps storage churn; =1.4.2 pin Cargo.toml:60 vs the new 1.MAJOR_MINOR_PATCH.x crate scheme, 1.5.4→1.10504.0; C API additive-only; date_trunc/lambda SQL breaks clear).
BE-1597 DuckDB UI — Design — the approved decision/spec built on the research above (status: design approved, NOT implemented; owner Geri Vászon; repos mando + optimization-universe-iac). Phased: Approach 1 dev-only NOW via SSM AWS-StartPortForwardingSessionToRemoteHost forwarding the LOCAL port 4213 → http://localhost:4213 (literal localhost, not 127.0.0.1; no ALB / target group / health check / /duckdb* prefix / reverse proxy / bundle patching — the bastion hop is only needed because ECS Exec is interactive-only); Approach 2 = Quack for a remote client → in-memory DB, deferred to DuckDB 2.0 (Sept 2026) since Quack is Beta in 1.5.x. Container change: add socat + a docker-entrypoint.sh bridging TCP4:4214 → TCP6:[::1]:4213 when DUCK_DB_UI_SERVER=true (NO Rust change; env already read at lib.rs:181/:301). Never in int/prod (proprietary MotherDuck frontend, no auth). Blocking open Q: Fargate egress to duckdb.org UNVERIFIED.

Env / Configuration

BESS AM (BE-2262) - mando-bess-am — BESS AM service config uses env prefix BESS_AM_* and config/bess-am.yaml; reads the bess-am-events Kinesis stream and writes bess-os-events; Postgres schema bess_am. Local-dev Kinesis via doc/kinesis-local-setup.md + MiniStack (ministackorg/ministack).
be-1595-enabling-arrow-flight-consumers-2026-06-16 — Flight client toggled by MANDO_FETCH_STRATEGY (default rest, Flight only when == flight, case-insensitive) + MANDO_FLIGHT_HOST/PORT/PROTOCOL. Pure runtime env flip on the -F flight dev wheel; co-located consumers reach mando over localhost, separate tasks can’t (no network path to the gRPC port). Silent REST fallback on error.
be-1595-arrow-flight-dev-deploy-runbook-2026-06-24 — deploy-side view of the same switch: MANDO_FETCH_STRATEGY=flight + MANDO_FLIGHT_HOST/PORT set on the algo task defs; verify in Datadog EU env:dev (mando/optimization/forecast) — expect forecast falling back to REST, optimization fully on Flight.
mando-cli-simulator-env-contract-2026-06-02 — three reusable patterns: (1) arbitrary extras passthrough — filter workspace .env keys against a managed-set const (MANAGED_SIM_ENV_KEYS), auto-inject survivors as KEY: "${KEY}", sorted alphabetically, with tracing::info! per injection; (2) sim-postgres aliasing — when two services share a default credential, route the dependent service’s native env (POSTGRES_USER) through the orchestrator’s contract (SIMULATOR_DATABASE_USERNAME) so one .env override controls both; (3) derive UI summary strings from the actual invocation-args function — never hardcode the verb+flags in summary text or it goes stale (pre-fix bug: literal "up -d --build --remove-orphans" in success row while up_compose_args had dropped --build for sim).
BE-1597 DuckDB UI — Design — the DuckDB-UI dev gate is DUCK_DB_UI_SERVER (already read in mando-bess at lib.rs:181/:301; NO Rust change), set in terraform as DUCK_DB_UI_SERVER = tostring(var.environment == "dev") on the mando container and mirrored by the docker-entrypoint.sh if [ "$DUCK_DB_UI_SERVER" = "true" ] socat guard. Use var.environment, NOT local.environment_group — the group collapses int+prod and dev+test, so it’s the wrong knob for a dev-only gate.
Mando MR Review Patterns - config direction from review (two reviewers, MR !556): NEW config uses config::Config, not Envconfig; existing envconfig adapters stay as-is (no mass migration).
Mando Deployment Ceremony - per-env mando runtime config WITHOUT rebuild. CORRECTED 2026-08-18: this is NO LONGER in the .environment-vars:{env} blocks of the IaC .gitlab-ci.yml - since IaC 3f50c77 those anchors carry ONLY ASSUME_ROLE_ARN/ACCOUNT_ID/ACCOUNT_NAME/AWS_REGION/environment:name, and SIMULATION_MODE, MANDO_FLOW_SCHEDULE_*, EBS_SMB_HOST, MANDO_FLOW_ACTIVE_VERSION grep to NOTHING there. Runtime config now lives in terraform/environments/<env>.tfvars as lower-case terraform vars: simulation_mode, ebs_smb_host/ebs_smb_share, mando_flow_schedule_{trading,idc_trading,intraday,auction,data_update}; sizing and active version moved per region under bess_os_regions.{fi,fr} (mando_flow_active_version, ecs_resources.mando_cpu/mando_memory) - dev has fi+fr, test has fi only. Constraint unchanged: mando_flow_active_version (dev fi V1_4, dev fr V1_4_SIMULATION) must name a setup version existing in mando config/flows/manifest.yaml; bump the two together. dev and test both run simulation_mode = "true".
fr-region-missing-datadog-logs-2026-07-21 - mando reads the DD_SERVICE env since commit ab8cfd11 on feature/BE-3657 (resolve_dd_service() + OnceLock-cached dd_service() in mando-lib/src/app/mod.rs, hardcoded bess-os-service-mando as fallback also on empty string, all 6 call sites routed incl. 4 raw global::meter literals); deployment-neutral on the main stack because IaC bess_os_ecs.tf:218 already sets the var on the mando container, and it unblocks the FR -fr suffix.
mando-cli-json-output-agent-skill-2026-07-23 - MANDO_OUTPUT=json (exact-match "json" only; the --json flag beats it) is the env knob that flips mando-cli output to terse NDJSON on stdout. Any other value / unset = human mode (byte-identical to pre-feature).
mando-local-host-run-recipe-2026-07-23 - .cargo/config.toml.example is STALE (field-verified 2026-07-23): MANDO_FLOW_ACTIVE_VERSION is DEAD, the config now wants MANDO_SETUP_ACTIVE_VERSION (V1_4) + MANDO_SETUP_SCHEDULER_DISABLED (prefix MANDO_SETUP, config-crate style); FINGRID_API_KEY must be NON-EMPTY (init panics via unwrap: “must contain at least one key”); the postgres cert loader reads <cwd>/certs/eu-central-1-bundle.pem, so running from a workspace root needs a certs symlink to mando-bess/certs.
mando-e2e-rebase-2026-08-05 - .cargo/config.toml is gitignored local config (it holds secrets), so a fresh mando checkout has none and py_mando/py_mando_simulation fail to link on Apple Silicon with unresolved _Py_* symbols. Fix: cp .cargo/config.toml.example .cargo/config.toml — the example carries the [target.aarch64-apple-darwin] rustflags -undefined dynamic_lookup that PyO3 cdylib targets need. Nothing bootstraps this for you.
mando-e2e-ci-green-2026-08-05 - /Volumes/bandi/coding/poc/compose.override.yml is the canonical known-good env set for mando_bess — diff against it rather than deriving requirements from code (deriving from code turned CI boot into one 15-minute pipeline per missing var). Two boot-order warts to raise upstream: DataPlatformConfig::init_from_env().unwrap() runs BEFORE its own DATA_PLATFORM_DISABLED flag is read, so FINGRID_DATABASE/OUTPUT_LOCATION must exist even when disabled; and FINGRID_API_KEY hard-panics at fingrid.rs:29. Final missing set for a CI boot: FINGRID_API_KEY, AFRR_AUCTION_RESULT_DEADLINE, FCR_AUCTION_RESULT_DEADLINE, BATTERY_STATIC_DATA_MDR_PATH.
mando-cli-e2e-live-green-2026-08-05 - the MANDO_FLOW_* → MANDO_SETUP_* migration reaching the e2e suite, found only by a live run: MANDO_SETUP_ACTIVE_VERSION=V1_4 is now REQUIRED (the service crashloops at boot without it), and MANDO_FLOW_SCHEDULER_DISABLED was renamed MANDO_SETUP_SCHEDULER_DISABLED — nothing rejects the obsolete name, the scheduler just stays on, and the 0 5 * * * * data-update cron then fires mid-suite and breaks unmatched_max: 0 at a nondeterministic point. General rule: a renamed env var that is merely ignored fails as an unrelated-looking downstream symptom.

Error Handling

fatal-step-double-logging-ancestor-boundary-2026-09-02 — Fatal steps emit the SAME error at the leaf boundary and again at every ancestor wrapper node (run_step logs unconditionally at mod.rs:499; status_or_error() returns Err only for Fatal, ? re-surfaces it at the wrapper). Fingerprints diverge by step path so copies don’t aggregate. Not the BE-3541 axis; still live on develop b5770766. Trigger was a DuckDB duplicate PK already fixed by MR !610 (dab6cba6) but absent from prod 7c1ef44f.
BE-3117 Per-Flow Error Context Store — “OEF v2”: one aggregate business ERROR per flow run with full fidelity. ErrorRecord + FLOW_ERROR_CONTEXT task-local Vec + record_error! macro (TT-muncher, in mando-lib not mando-core) + flush_flow_errors/FlowCompletionGuard. Invariant: record exactly once per flow, gated by logged_at_site. Hand-written redacting Serialize (drops bodies when http_context.sensitive). Removes OEF v1’s dd_formatter ERROR→WARN demotion. Fixed pre-existing MarketNotFound double-log via mark_logged().
BE-2272 — DD log JSON flattening to root (flow.*, step.*, error.* siblings).
flow-step-log-message-dropped-2026-05-26 — ErrorWithStepStatus::Log { message } payload silently dropped by StepResult::log() destructure + Display impl + status_or_error collapse. Only ::Log arm broken; ::Error(anyhow) correctly logs.
clippy-disallowed-macros-allow-placement-2026-06-22 — tooling gotcha behind the error-logging path: mando bans tracing::error!/log::error! (clippy.toml) to force logging through mando_core::error!; that wrapper’s own #![allow(clippy::disallowed_macros)] must be a module-root attribute in mando-core/src/error.rs, not at the call site (call-site/block allow is ineffective on clippy 1.95). Found during the BE-3117 error-handling redesign.
Mando AGENTS.md Master Guide — correction: NO ErrorCode derive and NO mando-lib-macro crate on develop (verified git grep ErrorCode origin/develop empty) — worktree experiments only (poc-error-extractor, BE-2023, BE-1595/BE-3482); mando-lib-macro marked outdated. error.kind (fully-qualified path) / error.code (Enum::Variant) / error.source are extracted at runtime by mando_core::error! from the Debug repr, no derive. Canonical pattern: every component defines a thiserror enum …Error; anyhow only in init/flow-step glue; HTTP mapping via deterministic map_*_error_to_response (same variant ⇒ same status); py-mando maps everything to the single PyMandoError via map_*_error fns (mando-lib/src/python/error.rs). doc/errors.json = untracked generated catalog (generator not in CI — likely poc-error-extractor worktree). Direction of travel: log-once-at-step-boundary (docs/error-handling-redesign-plan.md). Round 2 (2026-07-13): the redesign PARTIALLY landed in commit ae6d1098 — error.message = root cause, error.stack = source chain, error.fingerprint = {code}|{step_path}, ErrorWithStepStatus.logged_at_site double-log guard with new()/logged() constructors; NOT landed: error.trace, .step_context(); the plan doc + doc/errors.json remain untracked on develop.
Mando MR Review Patterns - review-mandated error policy (15+ “should be error” comments + policy statement from balazs, MR !481): failures are ERRORS, never warnings; log via mando_core::error! with an error-code enum variant; Fatal fails the execution; NO custom error message strings (the error’s to_string() IS the message, human context goes in the message field).
BE-3657 error_stack Adoption - LOCKED decision (2026-07-16, planned/awaiting go): full error_stack =0.8.0 adoption over the ErrorCode-derive direction. Principle “leave what we can to error_stack”: mando adds only the ErrorCode full-path attachment (captured statically in CodedExt at change_context/new sites) + extraction fns + an error!(report=...) arm. Field contract: error.code==kind==type==outermost full-path code; error.errors = codes array; error.details = [{code,file,line,message}]; fingerprint full-path. Crate’s own tree rendering with codes as attachment sub-lines. Boundary message template "{outermost}: {static}: {chain}". Phase-2 direction: per-variant #[error_meta(event_type, resolution step, business_message, description)] → details enrichment + resolution-doc codegen (phase 1 must not preclude; attachments are open-ended). Branch feature/BE-3657 not yet created, off BE-3656 head a0735ebb; spec/plan under docs/superpowers/{specs,plans}/2026-07-15-*error-stack*. UPDATE 2026-07-22: phases 1+2 BUILT + PUSHED @ 2baea68a (13 commits atop the rebased BE-3656 head 8f50ef2a); boundary template FLIPPED static-first "{static}: {outermost}[: {chain}]" (2026-07-16); MR only after !578 merges + explicit yes, then rebase --onto develop; standalone cargo build -p mando_core broken by a pre-existing BE-3643 uuid/serde issue (workspace graph fine). UPDATE 2026-07-22 afternoon: !578 MERGED (merge 4a0d297f = develop tip); rebased onto it zero-conflict (2baea68a → 22ccae93, patch-id byte-identical, pipeline 2696324182 green) + DD_SERVICE env fix appended (ab8cfd11, branch now 14 commits); MR pending ONLY on Andras’s yes. UPDATE 2026-07-22 evening: phases 1+2 ride MR !580; phase 3 PLANNED + approved: scope “plumbing + adapters” (rejected: plumbing-only, full workspace breadth), mechanics = seam replication per the Volue pilot (rejected: native Report signatures, boundary-only wrapping), same ticket (branch feature/BE-3657-phase3 off develop AFTER !580 merges, one phase = one MR); conformance-tightened against Balazs’s logs-and-apm.md (line 157 absolute: ALL tuple/transparent variants restructured; error.details.attributes via new mando-core ErrorAttr landing before any adapter; line 159 http roll-up = outermost-wins HttpContext pick, with_report fills http_context when unset; auth providers: no bodies/tokens, method/url/status only); adapter order auth providers → volue/ems completion → volue/atp → alpiq/ebs (SMB, share/path ErrorAttr) → mdr/opl/data_platform/et_3000 → fingrid → metis → mando group → position_manager; vetoable defaults: archiver deferred to the repos phase, startup mock ported to mainline behind MANDO_DEBUG_MOCK_ERROR default-off; spec/plan docs/superpowers/{specs,plans}/2026-07-22-*phase3* (15 tasks); execution NOT started, gated on !580. UPDATE 2026-07-23: !580 MERGED (squash de7cd4c6, merge 4c3442df, zero review comments); phase 3 EXECUTED subagent-driven on feature/BE-3657-phase3 (14 tasks + final whole-branch review; 20 commits, HEAD e0184bc3, pushed + ls-remote verified, CI running, NO MR yet): mando-core ErrorAttr + per-level error.details attributes + error_attr_value + report_first_attachment + with_report http roll-up; ALL in-scope adapter groups migrated (*_reported seams, exhaustive no-wildcard *_error_from_report, zero error(transparent) left in mando-lib/src/adapter incl. archiver); auth providers first (no bodies/tokens, sentinel leak tests); PyO3 map_report_error forward plumbing (grep-proven unreachable this phase); shared error_json_field_value across all three log formatters; MANDO_DEBUG_MOCK_ERROR env-gated report-backed mock. KEY FINDING (formal spec exception): only volue/ems carries Reports to the step boundary in production (Error = Report<VolueEmsError>); the other seven groups are WALLED (flat-typed service towers + pub(crate) seams make cross-crate wiring structurally impossible), chains flatten at legacy wrappers until a services-phase tower restructure - every wall independently verified genuine. Gates at e0184bc3: workspace clippy 0, 574 lib tests / 0 failed, no lock churn, 61 files +4390/-1482 vs develop; follow-up bundle triaged (auth-provider pre-existing body logs, ErrorAttr ids on non-success terminals only, py delegation tests CI-only, data_platform test panic-safety, opl clear_strategies portfolio attr, map_err closure-form drift). develop since moved (6fae64a5); pre-MR rebase possible on go. UPDATE 2026-07-23 (local e2e): phase 3 telemetry VERIFIED LOCALLY end to end from the real runtime (branch binary b394e0d4, rebase-descendant of e0184bc3, run as a host process against the mando-cli infra stack): boot mock 4-level report (error.errors 4 full-path codes, error.details real file:line + attributes incl. data_group, error.stack box tree with the ErrorAttr sub-line + the opaque HttpContext attachment, codekindtype, root-cause message, trace correlation); Volue EMS spot single-level fingerprint code|step_path (spot_data.rs:79); Volue EMS retrieve TWO-LEVEL auth chain live (ems.rs:217 + ems_auth_provider.rs:155, the final-review auth level confirmed); Metis full-path code with NO error.errors (walled-tower flattening as designed); dd_formatter real nested JSON; 12 error lines to Datadog EU via direct intake (env:dev-local-levander). Run recipe + friction: mando-local-host-run-recipe-2026-07-23. UPDATE 2026-07-23 evening (MR + short-code decision): phase 3 IN REVIEW as MR !585 “feat: adopt error stack across all adapters” (feature/BE-3657-phase3 → develop, head a58df341, 22 commits, no conflicts, branch pipeline green on the same sha; reviewers krisztian.fekete1 / gabor.nagy6 / balint.budavoelgyi / jozsef.nagy1). Same head implements the NEW DECISION (Andras, 2026-07-23) amending the field contract: error.code goes SHORT (Enum::Variant, bare struct name when variantless, per the team spec’s line-100 mobile-readability intent) while error.kind/error.type/error.errors/error.details codes/error.fingerprint ALL stay full module path - mando-core short_error_code resolver (PascalCase-segment heuristic, valid due to house naming), applied at the 4 macro emission sites only (span/boundary layers passthrough), pinned by test kind_type_fingerprint_stay_full_while_code_goes_short, 577/0 workspace lib tests, verified LIVE locally + forwarded to Datadog (env:dev-local-levander). Consequences: the “mobile readability rationale is dead” pending team sentence WITHDRAWN (conformance restored); DD cutover impact for @error.code-keyed assets shrinks (short form resembles legacy codes); @error.source RETIRED (renamed to error.message in the redesign, pinned by a mando-core regression test) - delete or rekey any facet/monitor still on it to @error.message/@error.errors (an empty error.source consumer was hit and fixed by Andras 2026-07-23). UPDATE 2026-07-24 (Caused-by stack renderer): NEW DECISION (Andras) REVISING the locked “crate’s own tree, codes as sub-lines” trade-off: error.stack rendering switches from the error-stack crate’s Debug tree to a custom prefix-style “Caused by” chain renderer, because Datadog collapses newlines in several views and the crate’s ASCII connectors (|-, lone |, |->) turned into unreadable pipe soup in the real DD UI. Per-level format: {short code}: {message} head (first level bare, subsequent prefixed Caused by: ) + at file:line:column + with key = value per ErrorAttr; opaque attachments no longer mentioned. Commit 1f9cd611 (new MR !585 head, 23 commits): render_stack_tree walks frames like report_details; crate debug-hook plumbing (install_debug_hook, Charset/ColorMode, HOOK_INIT) deleted as dead; 579/0 workspace lib tests incl. an exact multiline golden pin; verified live + forwarded to Datadog (2026-07-24 env:dev-local-levander events carry the new stacks AND the short error.code together). Datadog research recorded: Error Tracking for logs needs error status + service + (error.kind OR a valid error.stack) and mando always emits error.kind (stack format never gates tracking/grouping); DD’s valid-stack bar wants a frame with a FUNCTION NAME + filename, which error_stack Locations never carry (mando stacks never frame-parseable regardless of format); the source tag auto-parses only conventional language stacks; remappers relocate but do not parse - so error.stack is purely presentational, optimized for human readability and graceful newline-collapse (which the Caused by: separators survive). UPDATE 2026-07-24 (error-authoring DX overhaul): decision series (Andras), north star: error reporting must not break Rust’s natural flow, ? is the API; 3 commits (branch now head f771805e, ~27 commits, pushed, riding MR !585): f5beabd4 report vocabulary (new_coded/CodedExt/change_context_coded DELETED workspace-wide → ToReport::into_report (crate already exports IntoReport, hence the name) / Result::reported() / .wrap() / .attr(k,v) / flatten_report for Clone enums collapsing 6 conversion fns to one-liners; 307 sites, 37 files; bonus find: 20 map_err(new_coded) fn-pointer sites - the exact #[track_caller]-losing form - had degraded locations until swept); 4e519269 short error.details[].code (the details panel is the human panel, file:line carries precision; errors/kind/type/fingerprint stay full path; split pinned by a test in ONE emission); f771805e native report signatures REVISING the phase-3 seam-replication decision (dual *_reported seams “not fit for a codebase this big”: ~35 flat twins deleted, bare MandoReport<C>(Report<C>) newtype in mando-core with Deref / as_report / into_inner / delegating Display+Debug / std::error::Error source-None / plain From<Report<C>> ? auto-lift + MandoResult<T,E> alias; adapters ONE fn per op returning MandoResult; ~50 consumer edits flattening at tower edges; ems towers keep Report<VolueEmsError> internally with into_inner at 5 sites; 5 dead conversion fns deleted; the walls stand, the seam mechanics are gone). Gates 580/0 workspace lib tests + clippy clean. NEXT MR spec’d (docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md): #[derive(MandoReport)] From-impl generation (#[report(from(Src))] display-mapping default, #[report(wraps(Inner, message = "..."))] chain levels, MandoError marker bounds at boundaries + clippy ban on Report::new); spike (scratchpad/spike-trackcaller, rustc 1.89.0, error-stack 0.8.0) proved direct From on Report<LocalEnum> orphan-blocked (E0117, naive design dead) while #[track_caller] through the local newtype propagates ?-site locations EXACTLY (no ergonomics-for-locations sacrifice needed, despite Andras offering it); grounding pass caught 2 design regressions pre-implementation (from(Src) must display-map or source fields kill Clone/flatten; wraps() needs an explicit static message for struct variants). UPDATE 2026-07-27: error.stack REVERTED to error-stack’s default Debug tree (commit e813a84c, MR !585 head; RE-REVISES the 2026-07-24 Caused-by renderer back toward the original locked decision 3 - ASCII, no color, full-path codes per frame via the ErrorCode debug hook, the DD newline-collapse quirk accepted as display-only) while error.details[].code stays SHORT (deliberate asymmetry); revert conflicts hand-resolved, 578 lib tests + clippy clean. MandoReport FINALIZED as the name for both the newtype and the derive (over Trace/ErrChain, zero churn). derive(MandoReport) MR STARTED on stacked branch feature/BE-3657-derive off the phase-3 tip e813a84c (transplant at merge: git rebase --onto origin/develop e813a84c feature/BE-3657-derive); MR !585 still open, zero review comments, not_approved the only gate. UPDATE 2026-07-27 (short codes REVERSED): Andras reversed his own 2026-07-23/24 short-code direction - commit 4df2562a “refactor: restore full path error codes” (the NEW MR !585 head) reverts a58df341 (short error.code) + 4e519269 (short error.details[].code), so error.code AND error.details[].code are FULL module path again, uniform with kind/type/errors/fingerprint; the short/full split is gone. short_error_code helper DELETED with its tests (zero refs workspace-wide); clean revert + one import fix; clippy clean + 575 lib tests / 0 filtered. DD-cutover consequence: full-path break is back to FULL impact (nothing resembles the legacy short codes), @error.details.code facet is full-path, and the “full-path-kills-mobile-readability” team sentence (logs-and-apm.md line 100) is UN-WITHDRAWN (needs re-raising). Derive branch feature/BE-3657-derive (zero own commits) fast-forwarded e813a84c → 4df2562a; transplant now git rebase --onto origin/develop 4df2562a feature/BE-3657-derive; plan docs/superpowers/plans/2026-07-27-be3657-report-derive.md. Error-postfix question answered: *_reported twins GONE (~35 deleted by f771805e), 13 *_error_from_report converters remain BY DESIGN (the walls; removal deferred to the services phase). UPDATE 2026-07-28 (DX overhaul v2, tip b7b04b71, 4 commits/21 total): SECOND Andras-driven DX pass on feature/BE-3657-derive reaching bare ? for EVERY source + a single #[mando_error] enum annotation. Native MandoReport birth - into_report()/reported() return MandoReport/MandoResult at CONSTRUCTION and the newtype’s std Error impl is REMOVED (the anyhow trick), making the blanket From<S: MandoFrom> coherent (5acd801b); #[cause(Src1, Src2)] bare source lists replace the 07-27 from/mapped bridge grammar with .mapped()/MappedExt DELETED (8ebd6ad9; wraps/.wrapped()? kept); the #[mando_error] attribute macro (Error + Debug + marker + cause routes in one annotation, shared generate_routes engine) replaces #[derive(MandoReport)], whose export is DELETED (4221f00b + 28-enum sweep b7b04b71, thiserror imports dropped in 20 files). LESSONS: a SECOND scratch-crate false-green (local FakeReport masked open-world coherence, orphan on 07-27 - scratch crates lie); the .lift() measurement found 227 broken seams not ~11 (implicit ? seams dwarf visible .into() sites), ~160 vanishing under native birth (towers pay .into_inner() at ~40-50 sites, .lift()/LiftExt only at tower edges). Reviewer disproved the attribute-ordering hazard (attribute macros expand before derives). Byte-identical (A/B + 3 goldens) + wire e2e PASS vs 86f3439f; intended delta: mando-simulator init moved to the report arm. UPDATE 2026-07-28 (later): the derive branch was RENAMED to feature/BE-4000 (its own Jira key; worktree dir stays .worktrees/BE-3657-derive; transplant now git rebase --onto origin/develop feature/BE-3657-phase3 feature/BE-4000). A SECOND assurance pass (3 Opus lenses over 86f3439f..cd5d53a7, run because the 07-27 five-lens predated DX v2) returned CLEAN on all six dimensions (reflexive-From disjointness + attribute-ordering both proven by scratch compile; native-birth tower into_inner/lift level-preserving; 585 lib + 15 trybuild + A/B 5 + 3 goldens; clippy 0; single shared route-gen engine), with three latent no-instance edges documented (last-segment Error/Debug dedup vs a custom derive_more::Debug; generic error enums unsupported by route-gen, pre-existing; hardcoded ::thiserror path) and an OPEN judgment call for Andras (an into_bare_report() helper collapsing ~52 into_report().into_inner() tower sites, worth it only if unwalling is NOT the next MR). A 2-commit standards fix-wave (479cc312 scoped rustfmt of the MandoFrom on_unimplemented note + cd5d53a7 renaming the two “duplicate report mapping” strings to “duplicate cause mapping” with 3 regenerated trybuild .stderr) moved the BE-4000 tip to cd5d53a7 (clippy 0, report_derive 8, lib 585/0). On phase-3 !585, Andras RESTORED the opl create_orders dead code review round 1 questioned (commit 0427c68d, adapted to the current idiom, still dead; phase-3 tip ee84ea50 → 0427c68d, all 3 !585 comments resolved; the BE-4000 base stays ee84ea50, transplant excludes it). BE-4000 = the FINAL derive branch, DX v2 DOUBLE-ASSURED; both await !585 approval → merge → transplant → Andras’s MR yes. UPDATE 2026-07-28 (rebase + pipeline green): MR !585 was REBASED onto develop 29ed8e34 (22 commits ahead, opening a merge conflict) to clear it - phase-3 tip 0427c68d → c3519e73, force-with-lease pushed, all 30 commits preserved (git range-diff 27=/3!), gates 605 lib/0 + clippy clean, pipeline SUCCESS on the rebased head. LANDMINE: develop had merged 2 NEW consumers of the simulator-client API phase-3 migrated (mando-lib/src/service/mando/asset_simulator.rs + mando-bess/src/live_simulator_init.rs) that BROKE THE BUILD after the rebase WITHOUT being a textual conflict (brand-new files, no overlapping hunk; only a full workspace build catches it) - both migrated to simulator_client_error_from_report(e.as_report()) and FOLDED into the signature-owning commit so every commit builds (footprint 81 → 83 files). krisztian.fekete1 had APPROVED !585 after round 1; the rebase’s new commits RESET the approval (GitLab drops approvals on ANY new commit), so !585 is not_approved again (conflict gone, pipeline green) and needs re-approval. Repo handover docs/superpowers/HANDOVER-error-handling-2026-07-29.md (untracked) supersedes the 07-24 one. Current tips: BE-4000 cd5d53a7, !585 phase 3 c3519e73, develop 29ed8e34.
BE-3482 Datadog Logs and APM Conformance - BE-3482 status note (2026-07-13): core conformance MERGED 2026-07-08 (MR !558 squash ae6d1098); the unpushed continuation adds the fingerprint ACTIVITY component (error.fingerprint = code|path|activity via ErrorWithStepStatus.activity: Option<String> + consuming with_activity(), all constructors default None so behavior is bit-identical when absent; reference wiring in mando-lib/src/service/volue/ems/ts_data_send.rs, every per-data-group error carries the data group name). Orphan commit 478cd993 (adds dd_formatter::app_version(), removes 3 panicking unwraps in mando-bess, fixes the known local mando-bess panic) was salvaged same day: cherry-picked onto the fingerprint branch as bc3727ba (Andras’s call, instead of a separate MR); the branch has 2 commits, gates green on both (clippy exit 0, 387 passed / 0 failed across 22 suites), pushed 2026-07-13 and open as MR !569 awaiting review; the old worktree + local feature/BE-3482 are purely historical and archivable. Known limit: at-site-logged variants (VolueEmsError::ApiError) reach the boundary as WARN with no fingerprint until the single-emission migration (own future ticket). UPDATE 2026-07-22: !569 MERGED (merge 53912a70); the single-emission migration MERGED as !571 (BE-3541 Single Error Emission), closing the limit; !570 (pymando) + wheel also landed.
BE-3541 Single Error Emission - MERGED via MR !571 (squash d1cf9c75 “refactor: emit step errors once by removing the logged-at-site path”, merge 1c4c33a5): one ERROR emission per step failure at the step boundary with HttpContext carried on the error; ErrorWithStepStatus.logged_at_site + logged() REMOVED - construct with new/message, never log at the detection site; unlocks fingerprint + http.* for previously at-site-logged variants.
mando-cli-datadog-local-logs-2026-07-22 - mando up --datadog verifies BE-3482’s dd_formatter error/log JSON (error.code/error.fingerprint) in the real Datadog EU UI from a local run. Confirms the body ddtags service tag overrides infra enrichment while dd_formatter ddtags carries NO env (agent DD_ENV authoritative). Filter env:dev-local-<user> → service:bess-os-service-mando.
BE-4067 whole-mando error unwalling - FINISHED the error-stack unwalling (BE-4047 began it): removed ALL production raw error_stack::Report<C> from mando-lib and adopted the full BE-4000 #[mando_error]/#[cause(...)]/bare-? idiom. GOAL ACHIEVED, verified: git grep into_mando_result -- mando-lib/src = 0, the only production into_report().into_inner() is the intentional wrap_http_call helper body, zero raw-Report-returning fns. Branch feature/BE-4067 @ 0dff4f2d stacked on feature/BE-4047 (!599), 9 commits (one per subsystem: wrap_http_call helper / fingrid / data_platform / metis / ebs / opl report_polars born-lift / ems service tower / atp VolueAtpAuthenticationProviderError deletion / fingrid cleanup). NEW PATTERN: wrap_http_call<E,F>(source: HttpCallError, make: F) -> MandoReport<E> (adapter/http_cause.rs) - a SHARED helper (NOT a #[cause] route) replacing the triplicated HttpCallError->ApiError wrap at metis/fingrid/ems, because ApiError is multi-field + reattaches an opaque HttpContext a message-only #[cause] route would drop; MUST keep #[track_caller]. Recipe step (d): message-only foreign-error maps → #[cause(ForeignErr)] + bare ?; multi-field variants (NonSuccessStatus/InvalidUrl/CannotParseJson) and context-prefixed/multi-source maps (ebs SambaError, data_platform SdkError) KEEP manual construction. EMS SERVICE tower gotcha: type Error = Report<VolueEmsError> → type Error = MandoReport<VolueEmsError> (the assoc Error IS E, NOT MandoResult); lives under service/volue/ems/*, not adapter/. Emission pinned via per-subsystem BE-4014 Datadog Error-Rendering Test Harness goldens (location-trimmed, 11 dd_emission tests pass). Final review READY-TO-MERGE no findings; lib 376/0, clippy —all-features clean; atp new() error change absorbed by anyhow ? at mando-bess/lib.rs:198 (no cross-crate ripple). North star: the mando-ERRORS axis of conformant traces+errors across all services; still open: mando trace conformance, downstream bess-optimization/bess-forecast-day-ahead/dashboards, phase 4 #[error_meta], the unlanded error.trace field + .step_context().
BE-4067 error DX clarity - the grammar-clarity + mando-bess-lib adoption follow-on to the unwalling (branch feature/BE-4067-grammar @ 0a3c2382, 10 commits, LOCAL/NOT pushed, stacked on unwalling tip 9038b0c6). Phase 1 (emission-NEUTRAL) NEW GRAMMAR replacing generic #[cause] on #[mando_error] enums with self-naming verbs: #[reframe(Src)] (flatten FOREIGN error → your {message}, bare ?), #[chain(Inner)]/#[chain(Inner,"msg")] (wrap a MANDO error + frame, .wrapped()?), #[transparent(Inner)] (re-export; macro injects thiserror error(transparent)+#[from]+a MandoFrom; #[error(transparent)] inside #[mando_error] now a hard macro error). MENTAL MODEL: field shape = behavior ({message}=reframe, unit=chain, (Err)=transparent); reframe REPLACES with your message, chain ADDS on top, transparent SHOWS THROUGH (reframe vs transparent = source becomes TEXT vs stays ERROR). Plus reflexive MandoFrom<Self> → bare Result<T,E> ?s into MandoResult (.reported() now rarely needed, ~35 sites dropped); MandoReport OPAQUE (Deref dropped, as_report()/into_inner() only) + flatten_report clippy-banned (workspace-wide via disallowed_methods=deny overriding all=allow) = the no-silent-stack-loss guard. Phase 2 (SEMANTIC / telemetry change): 25 mando-bess-lib service enums adopted (#[mando_error], 72 newtypes → #[transparent]) + ~21 boundaries switched to the rich report path via ErrorWithStepStatus::reported/from_report = a deliberate DD telemetry change on financial flows (idc_order, auction): error.errors/error.details ADDED, error.stack colon-chain → tree, error.message source shifts; code/kind/type UNCHANGED; captured as inline dd_emission asserts (mando-lib Phase-1 goldens stay byte-identical). GOTCHAS: workspace all=allow disables disallowed_methods except mando-core (fixed P2.3); py_mando “could not compile (lib)” = macOS cdylib LINK failure not a compile error (cargo check clean); cargo check skips #[cfg(test)]; the #[chain] reshape drops the transparent #[from] (breaks bare ? at other construction sites); whole-file rustfmt reflows legacy lines. Gates @ 0a3c2382: clippy --release --all-features clean, -p mando_lib --lib 379/0, -p mando_bess_lib 10/0. PENDING: telemetry sign-off, final review, push + MR (each a separate yes).
BE-4014 Datadog Error-Rendering Test Harness - develop-based pub test helper render_dd(|| error!(...)) -> Vec<serde_json::Value> (+ render_dd_one) in mando_lib::app::dd_formatter::test_support driving the REAL error! macro through the REAL DatadogFormatter (promotes the previously-private capture harness from dd_formatter.rs’s test module); --features app,test-util (NEW mando-lib test-util = ["mando_core/test-util"]), 9 worked examples, run cargo test -p mando_lib --features app,test-util. Bundles a develop DD error-emission contract survey (now test-pinned; the error-stack !585/BE-4000 branches change/extend it): report arm = errors[] full-path outermost-first + details[] {code,file,line,message} per level, NO fingerprint, codekindtypeoutermost, messagedeepest; plain + http arms = BARE fingerprint==code (the {code}|{step_path}|{activity} shape is StepResult::log’s, NOT the macro’s) + NO errors/details; NO render-layer redaction (http.*.body verbatim, no-leak = CALLER passes None); error.details has NO attributes on develop (ErrorAttr branch-only). Branch feature/BE-4014 @ 9a8a7538 PUSHED, not yet an MR. UPDATE 2026-07-29 (golden matcher): the harness gained a golden/snapshot MATCHER (Andras’s A/B style: assert a STATIC expected error.* JSON against the generated) - assert_dd_matches(generated, expected) + assert_error_renders(body, expected) in the same test_support; objects match as a recursive SUBSET (extra generated keys ignored so you omit volatile file/line), arrays same-length element-wise, strings exact OR a leading * = ends_with (grep affordance for full-path codes), scalars exact, PANICS with the JSON path on mismatch; the field-by-field ABSENCE tests kept as complementary (a positive golden cannot assert a field is absent); self-tested 8 tests incl. should_panic negatives + 3 golden json!({...}) example tests as the new whole-shape template; 39/0 under --features app,test-util; tip eac5fb85 → 9a8a7538.
Test Env Deployment State 2026-08-18 - where the error chain actually IS deployed. On test (image 1.17.0-dev.2749963346.336ce49f, mando develop 336ce49f): !571 BE-3541, !585 BE-3657-phase3, !592 BE-4000, !597 BE-4014 and !599 BE-4047 are ALL live (ancestry-verified); !601 BE-4067 is NOT (merge c2c9c9a2 landed 19:26 UTC, ~5h after the test apply, and test has not been bumped since). So any Datadog read of error.errors/error.details/tree-shaped error.stack on test measures the PRE-!601 world. Green-but-unpinned images that DO carry !601: 1.17.0-dev.2751762903.c2c9c9a2 and 1.17.0-dev.2756529188.b5770766 (develop tip).

External Integrations

Agent Context - the External Integrations table was rebuilt 2026-08-13 against poc/e2e-tests: ~15 external systems + 6 auth providers (was 7 systems), each with its planned Maintenance/{country}/{Service} datapoint, protocol, auth model and env prefix. Authoritative in-code registry = the system = "..." attribute on #[step(...)]. Likron = OPL (system = "OPL (Likron)", mando-bess-lib/src/service/intraday/idc_order.rs:122). GOTCHA: system strings are NOT normalized (volue_atp_order_book.rs says "Likron" but talks to Volue ATP; as_auction_update/energy_bids.rs says bare "Volue"), so aggregation by flow.step.system is broken. Config split: OPL/Volue ATP/Fingrid/Entra ID use parse_config_with_prefix; EBS/MDR/Metis/ET-3000/Volue EMS/Position Manager/OnePassport still #[derive(Envconfig)].
External Service Outage Gate - bitmask design - the 13 Maintenance/{country}/{Service} interval-double datapoints that back the planned outage gate, and how flow steps declare external: deps against them.

Infrastructure / AWS

AWS CLI Access via Identity Center - how AWS CLI auth works since 2026-07-21: [sso-session alpiq-sso] in ~/.aws/config (IdC start URL ssoins-69878836cb6e09c7, eu-central-1) + four Developer-role profiles bessos-{dev,test,int,prod} mapping to accounts 794038257734 / 071128452852 / 621553445748 / 282467977019 (same set as the section 14.1 deploy matrix); one aws sso login --sso-session alpiq-sso covers all four; saml2aws retired but left on disk. Landmine: the RTK hook mangles aws output, always rtk proxy aws .... Used for ECR/ECS access during mando deployments.
BE-1597 DuckDB UI Exposure Research — reusable mando AWS-infra facts gathered for BE-1597: mando ALB aws_lb.bess_os_service is internal=true, 443-only, single target group on port 8080 (bess_os_load_balancer.tf); bastion (bastion.tf) t3.medium in a private subnet, no public IP, SSM Session Manager only, egress-all + ZERO ingress; mando runs on ECS Fargate desired_count=1 with enable_execute_command = environment=="dev" (bess_os_ecs.tf:499), and ECS Exec is interactive-only — no port forwarding (SSM AWS-StartPortForwardingSessionToRemoteHost via the bastion is the actual port-forward path); the bess_os_ecs SG (security_group.tf:4-23) uses INLINE ingress blocks — a standalone aws_vpc_security_group_ingress_rule is silently deleted on the next apply; must use a dynamic "ingress" block inside the existing resource; NO NAT gateway / VPC endpoints defined (the VPC is a base-vpc tag data lookup owned by another team) → Fargate egress to duckdb.org UNVERIFIED; int + prod both map to account group int_prod (main.tf:5-6).
BE-1597 DuckDB UI — Design — the approved terraform/infra shape for BE-1597: open port 4214 to the DuckDB UI via a dynamic "ingress" block INSIDE the bess_os_ecs SG referencing the bastion SG (not the VPC CIDR), gated var.environment == "dev" — a standalone aws_vpc_security_group_ingress_rule would be silently reaped because that SG uses inline ingress blocks. No ALB listener / target group / health check (mando ALB already internal=true); the bastion SSM hop (AWS-StartPortForwardingSessionToRemoteHost) is required because ECS Exec is interactive-only. Blocking pre-check: Fargate egress to extensions/ui.duckdb.org is UNVERIFIED (no NAT / VPC endpoints; base-vpc owned by another team) — if blocked, Phase 1 can’t ship and no terraform fixes it. desired_count=1 + Fargate re-IPs each deploy, so the duckdb-ui.sh helper re-resolves the task IP every run.
Mando Deployment Ceremony - the environment matrix. REWRITTEN 2026-08-18: there are no component groups any more. 4 envs = 4 independent terraform/environments/<env>.tfvars files; terraform/main.tf reads var.components["mando"].version and var.ecr_repos["mando"] FLAT and local.environment_group is gone from the whole terraform/ tree. ECR strings moved into the per-env files (dev+test identical 843164609896…/poc/mando/deploy, int+prod identical 748634852998…/poc/mando, but as separate copies that can drift). Accounts: dev 794038257734, test 071128452852, int 621553445748, prod 282467977019. int_prod naming trap survives only in older notes/AGENTS.md. Pins observed 2026-08-18 on origin/develop @ 9600b34c: dev+test 1.17.0-dev.2749963346.336ce49f, int+prod 1.17.0-2718305160.c3f6530b. ⚠️ The local clone is on rc/1.11.0 from 2026-05-27 - always git -C <iac> show origin/develop:terraform/environments/<env>.tfvars. Expected plan diff on a version bump: the mando container image in the bess-os ECS task def ONLY.
Test Env Deployment State 2026-08-18 - the test environment as of 2026-08-18: mando image 1.17.0-dev.2749963346.336ce49f from mando develop 336ce49f (MR !610, built 09:07 UTC 2026-08-11), pinned by IaC 16e8241 and applied green at 14:12 UTC; nothing applied to test since. Includes the re-derivation recipe and the ancestry table of which error-handling MRs are live.
Dev Deploy 2026-07-14 (error telemetry) - terraform plan-reading lesson: a version bump producing 17 add / 19 change / 5 destroy was IaC develop drift catch-up (12 genuinely new resources: the bess_os_fr incoming_sync lambda stack + 2 RDS analytics SG rules) plus immutable task-definition replacement mechanics (all destroys were “must be replaced” task defs), NOT danger; the second apply after the migration fix was down to 4 added / 19 changed / 4 destroyed.
fr-region-missing-datadog-logs-2026-07-21 - the FR deployment shape: dev-only module "bess_os_fr" (terraform/bess_os_fr.tf, count = var.environment == "dev" ? 1 : 0) instantiating shared terraform/modules/bess_os with region_code = "fr" in account 794038257734/eu-central-1; log shipping is awslogs → CloudWatch → subscription filter → datadog-forwarder Lambda (dd-logs-forwarder module from bit-tf-modules v3.21.4), the DD agent sidecar does metrics/APM only. Landmine: modules/bess_os lacks the forwarder wiring, so every regional instantiation ships log groups with no Datadog forwarding.
mando-cli-datadog-local-logs-2026-07-22 - DD_API_KEY acquisition route (2026-07-23): the deployed Datadog API key lives in AWS Secrets Manager, dev account, secret DdApiKeySecret-pkeeEykkaqu3 (referenced from optimization-universe-iac terraform/secret.tf via CI var DATADOG_API_KEY_SECRET_NAME); fetch with aws sso login --profile bessos-dev then aws secretsmanager get-secret-value --secret-id DdApiKeySecret-pkeeEykkaqu3 --query SecretString --output text (eu-central-1). Needed because the Alpiq Standard Operations Datadog role lacks “API Keys Read” (org key page blocked) and Personal Settings ddpat_/ddapp_ tokens do NOT work as DD_API_KEY (real API keys are plain 32-hex; app keys are 40-hex/ddapp_ and pair with an API key for queries only). Follow-up: request the permission or a personal org key from the Datadog admin.

Licensing

BE-1597 DuckDB UI Exposure Research — DuckDB UI frontend = proprietary MotherDuck code, blocks production use. DuckDB core, the prebuilt libduckdb binaries, duckdb-rs, libduckdb-sys, the ui extension SOURCE (github.com/duckdb/duckdb-ui), quack, ICU (Unicode-3.0), yyjson are all MIT/permissive (IP: Stichting DuckDB Foundation) — safe to vendor closed-source with attribution/NOTICE. BUT the DuckDB UI frontend assets served from ui.duckdb.org are proprietary, source unpublished (DuckDB’s own 2025-03-12 launch post says so); MotherDuck said Mar 2025 they were “reviewing licensing options” and as of Jul 2026 there is still no published license / terms of use. Served with no auth and (per DuckDB docs) “can access the data you load into DuckDB” → fine on a dev laptop / dev container, NOT acceptable for the int_prod trading service. (The Quack remote-protocol extension, by contrast, is a core DuckDB-signed MIT extension — no such issue.) Cf. the GPL-avoidance decision in remotefs-smb to smb migration.
BE-1597 DuckDB UI — Design — the licensing finding turned into an approved decision: because the DuckDB UI frontend is proprietary MotherDuck JS served live from ui.duckdb.org with no auth (source unpublished, still no license as of Jul 2026), the spec forbids running the UI in int/prod and ships it dev-only (Phase 1) / on the developer’s own laptop via Quack (Phase 2, where MotherDuck’s JS is already trusted). Everything else in the supply chain (DuckDB core / libduckdb / duckdb-rs / libduckdb-sys / the ui-ext SOURCE / quack / ICU / yyjson) is MIT/permissive and safe to vendor closed-source with a NOTICE bundle. Evidence: License finding (the important one).

mando-cli

mando-cli-v2 — current architecture (AppContext, CommandDelegate, 13 commands).
mando-cli-docker-lifecycle — DockerClient + ContainerSpec, bollard 0.19 wiring.
mando-cli-mock-command-2026-04-28 — mando mock subcommands and WireMock infra.
mando-cli-mock-down-idempotent-2026-05-06 — mando mock down + WireMock healthcheck endpoint fix; canonical 7-step lifecycle pattern; idempotent-teardown rule (404 = success).
mando-cli-status-readonly-2026-05-06 — mando status made read-only and bounded (connect_readonly, table_exists, statement_timeout = '2s'); status-must-be-pure-reads rule.
mando-cli-build-context-filter-2026-05-06 — yaml-driven context_includes and build_filtered_tar to stop COPY-everything build-context hangs.
mando-cli-build-variants-shelved-2026-05-06 — SHELVED design for profile-driven build variants (dev vs release Dockerfile, pre_command, 3-field profile syntax).
mando-cli-wsl-linux-build — how to build a Linux x86_64 / WSL release binary locally from an Apple Silicon Mac (musl static target, Docker --platform linux/amd64, optional-query path-dep gotcha).
mando-cli-v0.4.0-compose-bugs-triage-2026-05-26 — v0.4.0 compose-runtime triage: two confirmed bugs in the post-bollard rewrite; context: . relative-path resolution + bare-slug positional arg activating a malformed render_override stub. Documents the load-bearing profiles: ["{run_tag}"] invariant.
mando-cli-v0.4.0-mando-bess-binary-missing-2026-06-25 — design gap surfaced after the triage fixes (commit 2233959): the default profile builds mando from mando/Dockerfile, a thin runtime image that COPYs a pre-compiled target/release/mando_bess, but no builtin profile maps mando to pull/artifact and nothing in up runs cargo build --release (--cargo is artifact-only). Fresh mando up aborts the compose build. Workaround mando up --mando=pull; durable fix (default → pull + missing-binary preflight in up) pending user decision. Reported by gabi (WSL).
mando-cli-v0.4.0-piped-output-invisible-failures-2026-06-25 — mando up failures are invisible when output is piped/redirected (tester does 2>&1 | tee log): the user gets only a bare failed (exit 1) and an empty .mando/compose-up.log. Two fixes in the compose runner + buffer: (A) --progress plain on the docker compose call (compose.rs ~L231) so compose stops writing progress/errors to /dev/tty; (B) eprintln! fallback in CliBuffer::add_log_line (opts.rs:57) when stderr isn’t a TTY, because indicatif::MultiProgress::println silently no-ops there. Validated macOS (35→138 lines). Also clarifies mando-cli’s “registry auth” (GitLab API creds) ≠ docker login (container registry) — the now-visible tester failure is a registry pull access denied. Built+validated, not committed.
mando-cli-simulator-runtime-2026-05-30 — second first-class Docker Compose stack mando-sim coexisting with the dev mando project; six runners generated from a data-driven SIMULATOR_RUNNERS list; own Postgres on host port 5433; RunProfile.compose_project/layers() routes commands; new src/runtime/service_stack.rs::select_profile shared resolver for logs/exec/volume; defines the §A orchestrator↔runner env contract that the (WIP) mando_simulator crate, simulator-runner image, and six sim repos must implement. Supersedes GitLab MR !2 (balint). Landed b244f2e on feature/simulator-runtime. Process lesson: use full cargo test, not --bin mando (skips tests/).
mando-cli-simulator-env-contract-2026-06-02 — follow-up to the simulator runtime: §A env contract realigned to real images (simulator:1.11.0-feat.2569583284.7d69a070 + runner shape f7aa42f); adds arbitrary-extras passthrough via filter_extras / MANAGED_SIM_ENV_KEYS; sim-postgres aliased to SIMULATOR_DATABASE_* so one .env override controls both. Landed c3f0af7 on feature/simulator-runtime (8 files, +440/−67, 813 tests).
mando-cli-datadog-local-logs-2026-07-22 - mando up --datadog local Datadog log shipping; 2026-07-23 end-to-end test of docs/datadog-guide.md (tip c2becf1): guide accurate, every claim verified. HAZARD: a stale pre-feature binary reporting the SAME 0.4.0 version silently swallows --datadog as a service arg and starts a real up; the guide’s flag-position warning cannot protect against it (suggested guide follow-up: a version-check note). Nit: the new error strings use em dash characters, against Andras’s conventions elsewhere.
mando-cli-as-forecast-removal-2026-07-24 - removed obsolete bess-os-algo-forecast-as (Project::AsForecast) via compiler-driven deletion (enum variant/slug/aliases/config default; builtin profiles now 4 services; README five repos/two algos). LESSON: an enum variant used as a persisted HashMap<Project,_> map key in .bessstate.json — deleting it made a stale "AsForecast" key fail deser and the unwrap_or_default fallback silently WIPE the whole projects map (all overrides/toggles). Fix: per-key tolerant filter_known_projects mirroring save()’s PascalCase keys via Project::deserialize (NOT from_alias), warn! per dropped key, fixture restored as regression guard. Rule: deleting an enum variant = audit every persisted format keyed by it before shipping.
mando-cli-flow-run-2026-07-28 - mando flow list/run/status: reproducible TEST-MATCHABLE flow execution (golden-file-diffable, one command + one params file). Linchpin: trigger id DB exec id every log line’s root flow.exec_id (one Uuid); local builds ALWAYS emit dd_formatter JSON; server pre-creates step rows as Queued; trigger returns immediately (tokio::spawn). Follow = verbatim byte-preserved local streaming (single-writer, filtered by root flow.exec_id) + universal step digest (exit 1 on Error/Fatal). Terminal conformance kv pointers (dd_query/dd_url/started_at/stopped_at) into Datadog. Flows auction/intraday/data-update/as-auction-update/manual-schedule (kebab URL, snake_case flow_key body, server pins versions). Working tree, UNCOMMITTED 2026-07-28.
mando-cli-otel-capture-and-tail-2026-07-28 - always-on OTel trace capture (mando-otel-collector contrib 0.109.0 in every mando up, raw OTLP JSON NDJSON to <workspace>/.otel/traces/spans.ndjson, 50MB/3 backups; OTEL_EXPORTER_OTLP_ENDPOINT injected only-when-absent; up --datadog adds APM fan-out via agent OTLP receiver + collector otlphttp) + new local-only mando tail [logs|traces|all] [FILTER...] zero-query-language substring filter over gron-flattened telemetry (verbatim default, --flat shows matching assignments). flow run gained --span-file/--bundle. Bug fixed: ensure_infra hardcoded the collector config every command, so mando status after up --datadog reverted it and killed APM on next restart — now derived from datadog overlay file presence. Consolidation: ONE piped-child seam (system/logstream.rs), ONE OTLP walk/reader (flow/spans.rs). Working tree, UNCOMMITTED 2026-07-28.
mando-cli-json-output-agent-skill-2026-07-23 - mando --json <cmd> / MANDO_OUTPUT=json (flag wins, exact-match "json") renders all output as terse NDJSON on stdout (kv/step/err/data + ALWAYS-LAST {"t":"result","ok","code"} from main.rs’s single exit path); human mode byte-identical, spinners suppressed, logs/exec/compose chatter stay raw on stderr. One pure mapper src/ui/json.rs::element_to_event covers all 21 commands via the Element seam (format!-built for t-first order; payload keys alphabetical, preserve_order off). Agent skill .claude/skills/mando/SKILL.md (64 lines): grammar + minimal invocations + error→remedy + frugality (status before logs, trust exit codes, never parse human tables); mock -p goes BEFORE the subcommand. NO top-level warn event (warnings are step s:"warn"). Working tree, UNCOMMITTED 2026-07-23.
mando-cli-e2e-harness-2026-08-04 - two-repo data-level E2E harness (poc/e2e-tests in mando-cli AND mando, UNCOMMITTED, never run live). mando e2e run <suite> phases wipe/up/ready/migrate/seed/per-set; mando verify <expect.yaml> standalone (flow/mock/logs/spans/datapoints/outbound). New dev-gated POST /data/insert/csv/{*datapoint_id} server-side (Arrow-IPC insert body makes CSV impossible client-side). mando query REMOVED, dropping mandarrow-client/mando-core/arrow-array worktree deps so --all-features builds clean. Two criticals: self-confirming query-body tests, and zero WireMock mappings in the runner’s own stack. Superseded 2026-08-05: committed, rebased and GREEN 8/8 live.
mando-cli-e2e-live-green-2026-08-05 - the harness’s FIRST live run: GREEN 8/8 twice from wiped volumes against the rebased mando (2af0c6c9). The 08-04 “only a live run catches schema drift” prediction fired verbatim — upstream rewrote DataPointId serde from {id, id_fragments} to a bare JSON string, unit tests stayed green, only the live run saw the 400 (fix: src/e2e/datapoints.rs data_point_id() → json!(dp_id), 4 tests updated; MR !6 bugfix/e2e-datapoint-id-string). Plus two env drifts in mando’s e2e/suite.yaml and a NEW upstream restriction (DataPointId::new rejects non-ASCII-alphanumeric path fragments) that makes underscore-bearing ids 400 — docs/e2e-guide.md doc drift to fix.
mando-cli-junit-per-assertion-2026-08-06 - --junit now emits one testcase per assertion instead of one per test set (feature/junit-per-assertion, 9c39a88); mando verify gained --junit too. The fix was plumbing, not logic — verify::run was discarding an already-per-assertion Vec<Assertion> and returning a bool. Introduces the dual-label pattern: Label { name, case } where name is the terminal’s machine label (mock.requests[GET /path]) and case is JUnit’s human sentence (GET /path at least 1x), kept separate because the two consumers have different stability requirements. THE RULE: case names must derive from the EXPECTATION, never the observed result — GitLab keys test history on (classname, name), so a moving label reads as a test vanishing plus a new one appearing rather than a pass→fail transition (pinned by a_case_label_does_not_move_when_its_assertion_flips_to_failing). False-green trap: a set that never reached the verify engine has zero assertions, so set_cases emits a fallback case or the report would be green by omission. Known gap: a mock.requests entry with no count/min/max asserts nothing and now shows up as a named green testcase (“any number of times”) that mimics real coverage.
mando-data-insert-wire-format-2026-08-06 - why the harness KEEPS mando’s dev-gated CSV seed endpoint instead of inserting through the normal POST /data/insert. Client-side Arrow is feasible (~250 LOC + ~17-20 crates; mando-repository/src/arrow.rs:59-88 proves arrow-rs 56.2 StreamWriter output reads through the polars reader) but was rejected on drift risk — the body encoding is an undocumented internal of polars’ Deserialize for DataFrame that already changed between =0.45.1 and =0.49.1, and nothing would test the producing side. Also notes the seed request’s Content-Type: text/csv is decorative (the server takes body: String).
mando-repos-history-rewrite-2026-08-05 - attribution-free history rewrite of mando-cli main/release/4 re-pointed tags + mando poc/e2e-tests (2af0c6c9); merged source branch deleted, stale clones must --tags --force refetch. Carries the NEW rtk landmine: rtk-wrapped grep/log pipelines returned fabricated zeros; only rtk proxy git written to a file is trustworthy, and an rtk-mediated negative result is never evidence of absence.
mando-cli-flow-coverage-2026-08-06 - mando e2e run (main 5a135b8) closes with flow not covered: <name> lines + flow coverage: N/M flows (X.X%), denominator fetched from the live service flow inventory (the mando flow list endpoint — no list maintained in the suite); a failed fetch warns and skips, never fails the suite; +36 tests → 1686. Same day: test set test_set_1 renamed data-update-beskar-soc with no JUnit history impact — since fabc589 classnames come from the case’s flow: field (dir name only disambiguates two cases sharing a flow), so case dirs rename freely and the junit note’s <test set>.<section> table is superseded, docs/ is now untracked + gitignored as AI-internal (so repo-doc pointers in older notes are dead), agent skill gained an e2e case-anatomy section for authoring case.yaml.

py-mando / Python Bindings

pymando-rust-log-trace-correlation-gap-2026-09-02 — Rust tracing events inside the Python algo containers carry no dd.trace_id/dd.span_id/logger.name/thread_name: py-mando/src/log_formatter.rs is a second, thinner formatter, and dd_formatter’s TraceInfo/lookup_trace_info are private AND behind the app feature py-mando never enables. TraceFilter contextvars (tracing.py) also never reach Rust events. Reusing dd_formatter’s helper is RULED OUT (it needs OtelData and py-mando installs no OTel layer); fix is B1 metadata parity + B2 bridging the ddtrace span context Python→Rust per-call.
Mando AGENTS.md Master Guide — round-2 (2026-07-13) verified py-mando facts: py-mando-simulation split out of py-mando in commit 3559042d — py_mando.SimulationRunner is GONE (breaking change); the new package requires Python ≥3.11, ships ddtrace, no polars/pandas. Both packages run init() at import (JsonFormatter + rustls ring) and preload libduckdb (ctypes RTLD_GLOBAL / add_dll_directory); build.rs copies lib/libduckdb/1.4.2 into data/platlib. Two sanctioned async bridges: blocking py.allow_threads(|| pyo3_async_runtimes::tokio::get_runtime().block_on(…)) + pyo3_async_runtimes::tokio::future_into_py for the *_async awaitable variants. Errors: single PyMandoError via map_*_error fns (mando-lib/src/python/error.rs). Test posture: py-mando-simulation Rust side zero tests, 1 Python integration test; CI job “PyMando Simulation Linux Dev” (pytest without --nbval).
py-mando — crate overview (abi3-py38 stable ABI, algo params, DataFrame ops, algo runner HTTP server).
BE-3482 pymando Branch Review - 2026-07-13 read-only review of feature/BE-3482-pymando (13 commits, py-mando DD log/APM conformance): NEEDS REWORK BEFORE MR despite genuinely green gates (clippy clean; 366 passed / 0 failed / 35 ignored via rtk proxy; fmt_util proven feature-flag-clean by building mando_lib with and without --features python). Blockers: 13/13 scoped commit titles (banned per !322), 7 orphaned ContextVars in tracing.py, byte-identical duplicates of the fingerprint branch’s version-fallback fix. Landing order: fingerprint MR first, then rebase dropping the duplicates; sharpest option also drops the fmt_util extraction for a zero-conflict 9-commit MR. Rework EXECUTED same day: rebuilt from scratch as feature/BE-3482-pymando-v2 (worktree .worktrees/BE-3482-pymando-v2), 15 title-only no-scope commits off develop tip 51d2b516, zero cherry-pick conflicts, 10 files +374/-55; 9b0c4e70 kept, 86d7ffd8/34e749d5/d0986230 dropped (dropping the extraction cost zero extra code, level_to_status was still inline); all blockers/should-fixes done (build_ddtags public, bare-code fingerprint, caplog + genuine global restore); gates green (clippy 0, 434/0/35 cargo tests, rustfmt 0 branch hunks, pytest 120 passed / 1 skipped vs baseline 114); NOT pushed, no MR; v1 untouched as fallback. UPDATE 2026-07-22: pushed + MERGED as MR !570 (squash fde2425e “feat: pymando logging conformance”, merge 14dfb3ad); conformant wheel py-mando==1.16.1+dev.2691658359.14dfb3ad published to Nexus dev.
BE-3613 Algo Services py-mando Conformance - moving the algo services (bess-os-algo-optimization + bess-os-algo-forecast; forecast-as OUT of scope) onto the conformant wheel py-mando==1.16.1+dev.2691658359.14dfb3ad: pin bump + fork replacement + log_error adoption per repo; code COMPLETE 2026-07-22 on both feature/BE-3613 branches (bess-optimization bb0555e/d47212d/71fb146; bess-forecast-day-ahead 50231b6/927e1e5/e1d5ade + fork-test file deletions), UNPUSHED, blocked only on VPN/Nexus for poetry lock --no-update && poetry install && poetry run pytest.
BE-3657 error_stack Adoption - phase 3 (planned 2026-07-22 evening, execution gated on !580) plans a shared error_json_field_value fn in mando-core unifying the error.errors/error.details JSON parsing across mando-lib’s dd_formatter and BOTH python package log formatters (py-mando/src/log_formatter.rs, py-mando-simulation/src/log_formatter.rs); today only dd_formatter has that parse. UPDATE 2026-07-23: SHIPPED on feature/BE-3657-phase3 @ e0184bc3 (unmerged) - the shared hook unifies all three formatters; PyO3 also gained map_report_error forward plumbing, its wiring grep-proven unreachable this phase; py-mando/py-mando-simulation pytest deferred to CI (no local venv, Nexus VPN-gated).
BE-4014 Datadog Error-Rendering Test Harness - carries a DIAGNOSED-not-fixed Windows py-mando CI red: 3 test_dd_conformance.py tests (test_log_error_uses_deepest_cause / _without_step_context_omits_path / _extra_does_not_override_error_fields) fail with IndexError on empty caplog.records. Root cause = test-isolation leak, NOT a logic bug: log_error is a pure logger.error, dictConfig disable_existing_loggers=True + the logger_state fixture not restoring per-logger disabled/propagate makes caplog stop capturing after the first caplog test (first passes, next 3 fail = classic leak signature). Feature fde2425e (pymando logging conformance). Fix direction: disable_existing_loggers=False and/or reset the dd-conformance-error logger in the error_log fixture.
pymando-logger-init-race-flaky-2026-08-10 - another test_dd_conformance.py isolation leak, this one from the Rust side: AlgoRunner::new (algo_runner.rs:~113) std::thread::spawns a permanent Axum server thread on CONSTRUCTION (no shutdown inside pytest), so test_algo_runner.py leaks two threads whose start_server re-runs py_mando.logger.init() with service=None, racing the non-thread-safe init() and overwriting the module-global identity. test_init_populates_identity_and_dd_env then asserts identity["service"] and sees None. Shipped mitigation: a threading.Lock around init() (MR !601). Two globals (identity, _initialized) mutated by leaked background threads = the shared-state collision class. Same category as the caplog leak above and the env-var races in mando-known-flaky-tests-2026-07-15.

Testing

mando-cli-e2e-harness-2026-08-04 - the data-level E2E harness (mando e2e run / mando verify), CSV suites in the team’s test-data taxonomy. Also the canonical write-up of the self-confirming-test trap: unit tests that assert a hand-invented wire shape against the same invented struct prove only self-consistency; wire-format tests need ground truth from outside the code under test.
bess-os-ci-components — how the Python-side suites are actually run in CI: python-test is a plain runner with NO services: (bess-optimization sets DB_DISABLED=true) installing a prebuilt py-mando wheel from Nexus, so no Rust compiles and the 54/68-case suites cost only 7.4/13.8 min. Reporting convention: pytest --junitxml per group → junitparser merge → artifacts: when: always + reports: junit: (always-upload is what makes failures readable in the MR widget). mando-cli’s e2e job takes this reporting shape but needs wiremock/postgres services: from mando’s .integration-test-linux instead.
mando-e2e-rebase-2026-08-05 - two lessons about gates that lie. Feature-gated code is invisible to a bare cargo test: the app module is feature-gated, so an upstream type rename left the new CSV route uncompilable while the suite stayed green — only --features app caught it. And any test mutating a process-global env var is a latent cross-test race: MANDO_TEST_ENDPOINTS raced across parallel tests (fixed by passing the flag into test_router + a Mutex around set/read/clear), and the identical bug exists upstream on MANDO_DEBUG_MOCK_ERROR where CI’s --test-threads=1 hides it entirely.
mando-cli-e2e-live-green-2026-08-05 - the self-confirming-test trap’s SECOND ACT, and the strongest available evidence for the rule. The 08-04 note predicted in writing that only a live run could catch drift in the hand-mirrored query schema; the mirror test query_body_deserializes_as_the_service_would_read_it then stayed green through a full upstream serde rewrite (DataPointId struct → bare string) while the live query 400’d. A hand-mirrored wire type has no mechanism to notice upstream moving — only a live run gates that seam.
mando-e2e-ci-green-2026-08-05 - the e2e suite moved from “green on my machine” to green in CI, proving mando e2e run --external-stack against a CI-provided stack rather than a locally-orchestrated compose project. Shape worth reusing: prebuilt binary pulled cross-project + GitLab services: sharing the job’s network namespace (so sidecars are on localhost:<port>) + the service-under-test as a background host process. Its one open improvement — junit reporting the whole test_set_1 as ONE case — is CLOSED: per-assertion cases shipped 9c39a88 and render as 8 named cases in pipeline 2735668937 (see below).
mando-cli-junit-per-assertion-2026-08-06 - how to name a CI testcase so its history survives: GitLab attaches a test’s past runs to (classname, name), so a case label built from the observed result reads as one test disappearing and another appearing instead of a pass→fail transition. Labels must come from the expectation (step X is Success from the expected status; GET /path at least 1x from the expectation’s min), which is why the terminal label and the JUnit label are two separate fields on the same Label. Also the false-green-by-omission rule: a test that never ran must still emit a case, while an expectation merely absent from config must not.
mando-data-insert-wire-format-2026-08-06 - the build-vs-borrow rule for test seams: do not re-implement a wire format whose only specification is another crate’s private serde impl (polars’ Deserialize for DataFrame), because a dependency bump splits producer from reader with no test in either repo spanning both. Keeps the E2E harness on mando’s server-side CSV seed endpoint (14 tests, same repo as the polars pin) over a client-side Arrow insert. Open ~3-line improvement: replace the runtime MANDO_TEST_ENDPOINTS gate with a cargo feature test-endpoints so test routes are absent from prod binaries, not merely unreachable.
mando-known-flaky-tests-2026-07-15 - running catalogue of mando flakes; now three entries, all “no code defect, retry the job”: a wall-clock-boundary scheduler flake, the debug_error env-var race between gate_enabled_when_var_is_true and gate_defaults_off_when_var_unset (~2/3 parallel failure rate, masked in CI by --test-threads=1), and the py-mando logger-init race in pymando-logger-init-race-flaky-2026-08-10.
pymando-logger-init-race-flaky-2026-08-10 - the deepest of the flakes: a leaked background thread (spawned by merely CONSTRUCTING AlgoRunner) re-runs a non-thread-safe logger.init() and overwrites a module global that test_dd_conformance asserts on. Reusable lesson: a test that asserts on process-global state is a latent race whenever any live background thread can mutate that state — and a Lock narrowing the window is a mitigation, not a fix; the fix removes the shared-state seam (stop leaking the thread, or isolate the test in a fresh process / local monkeypatch).
mando-cli-flow-coverage-2026-08-06 - what “coverage” means for this suite: flows exercised over flows the service declares, computed against the live flow inventory rather than a hand-kept list, and deliberately chosen over cargo-llvm-cov line coverage (the interim job hit 40.36% green and was reverted). The metric immediately reframed the earlier 8/8 assertion green as 1 of 5 flows (20.0%) — assertion counts say nothing about behavioural breadth. Design detail worth copying: a coverage report must degrade to a warning, never fail a functional suite.

Tracing / Observability

pymando-rust-log-trace-correlation-gap-2026-09-02 — Rust tracing events inside the Python algo containers carry no dd.trace_id/dd.span_id/logger.name/thread_name: py-mando/src/log_formatter.rs is a second, thinner formatter, and dd_formatter’s TraceInfo/lookup_trace_info are private AND behind the app feature py-mando never enables. TraceFilter contextvars (tracing.py) also never reach Rust events. Reusing dd_formatter’s helper is RULED OUT (it needs OtelData and py-mando installs no OTel layer); fix is B1 metadata parity + B2 bridging the ddtrace span context Python→Rust per-call.
clippy-disallowed-macros-allow-placement-2026-06-22 — mando bans tracing::error! directly (clippy.toml disallowed-macros) so all error logging routes through the mando_core::error! wrapper; the wrapper suppresses the lint for its own expansion via a module-root #![allow(clippy::disallowed_macros)] in mando-core/src/error.rs (call-site allow is ineffective on clippy 1.95).
BE-3117 Per-Flow Error Context Store — FLOW_ERROR_CONTEXT task-local store created inside the spawned flow task (task-locals don’t cross tokio::spawn); drained after scope returns into ONE tracing::error! aggregate with flow.error.* flat facets + full flow.errors JSON. Hand-written Serialize redacts HTTP bodies when http_context.sensitive.
BE-2272 — DD log JSON flattening to root.
BE-1842 Datadog Observability — span-scope walking, tokio::spawn context-inheritance gotcha, dd_formatter field plumbing.
BE-3482 Datadog Logs and APM Conformance - Datadog logs/APM conformance state + ranked open deferrals: single-emission migration (biggest win, unlocks fingerprint + http.* on at-site errors) > error.*/http.* on trace spans > verify dd.trace_id correlation in DD > flow.step.timeout_ms. The fingerprint ACTIVITY component (pushed 2026-07-13, open as MR !569; MERGED by 2026-07-15 as 53912a70) makes DA vs AS schedule failures in the auction scheduling step group SEPARATELY in Datadog (previously ONE error group). Authoritative fingerprint standard: architecture-design repo, logs-and-apm.md (error.code + flow.step.path + specific activity).
BE-3482 pymando Branch Review - review outcome for the py-mando DD conformance branch: solid work (flat flow.step.path, conformant logging.conf, log_error helper with root-cause unwinding, set_log_tags ddtags bridge into the Rust formatter) but needs rework before MR. Sharpest conformance finding: error.fingerprint serializes code|None when no step context is set (tracing.py:109), polluting DD fingerprint aggregation for log_error calls outside a trace()/run_with_trace scope. The same-day rework FIXED it (bare code, no "None", with a new test) and added a NEW flow.step.system span tag: the non-Mando counterparty taken straight from StepMetadata.system in mando-lib/src/workflow/mod.rs (the structured source flow.step.connection is formatted from), one line, no string parsing; runtime JSON conformance proven from the Rust stream during pytest (120 passed / 1 skipped).
Dev Deploy 2026-07-14 (error telemetry) - first LIVE validation of the BE-3482 telemetry on dev, observed from the deploy’s own crash-loop logs: error.code == error.kind == error.type (spec-verified: architecture-design logs-and-apm.md lines 87-104 explicitly mandate three fields with the same value for APM/logs/mobile), error.message = root cause, error.stack = source chain (wrapper-embeds-source Display pattern causes cosmetic text duplication), error.fingerprint absent as expected for non-step errors. Follow-up idea: emit a bare-code fingerprint from the Rust error! macro outside flow steps (parity with py-mando log_error), needs one sentence added to the standard.
fr-region-missing-datadog-logs-2026-07-21 - why FR services show APM/metrics but zero logs in Datadog: container logs travel CloudWatch → datadog-forwarder Lambda via subscription filters (NOT the agent sidecar), and the -fr log groups have no subscription filters because modules/bess_os/cloudwatch.tf never got the dd-logs-forwarder blocks.
BE-3657 error_stack Adoption - Datadog verification of the dev throwaway build 1.16.1-feat.2682437891.b7d15e2c (2026-07-16..22): nested error.details (4-level chain, real file:line), error.errors array queries (element/wildcard/negation), nested http.{request{method,url,version,body,content_length}, response{status_code,body,content_length}}, trace correlation confirmed (sample trace 0dc8d42739a08ac0805808b2d14b6644); startup mock deep error fires once per task boot (still the typed error! arm, not the report arm). PENDING DD cutover AT BE-3657 MERGE: create facets @error.fingerprint/@error.errors/@error.details.code; inventory monitors keyed on short codes, flat http.*, exact message “Failed running step” - all code/fingerprint values change format at that deploy. UPDATE 2026-07-23: !580 (phases 1+2) MERGED (merge 4c3442df), so the cutover checklist becomes actionable with the first develop-based dev deploy carrying it; the mock is superseded on the phase 3 branch by an env-gated report-backed port (MANDO_DEBUG_MOCK_ERROR, default off, unmerged). UPDATE 2026-07-23 evening: the short-code decision (commit a58df341, in review with MR !585) narrows that format break: error.code becomes the SHORT Enum::Variant form (close to the legacy shape, so @error.code-keyed monitor impact shrinks), while error.kind/error.type/error.fingerprint/error.errors/error.details codes stay full module path and DO still change format at that deploy. NEW cutover item: @error.source is RETIRED (renamed to error.message in the redesign, pinned by a mando-core regression test) - delete or rekey any facet/monitor still on it to @error.message/@error.errors (an empty error.source consumer already hit and fixed by Andras 2026-07-23). UPDATE 2026-07-24 (DX batch + live stream): error.details[].code also goes SHORT (commit 4e519269; errors/kind/type/fingerprint stay full path), so the @error.details.code facet keeps a readable short form at cutover. Local LIVE stream now operational: server on 8081 + a CONTINUOUS forwarder shipping error/warning lines to Datadog EU within ~5s under env:dev-local-levander (MANDO_ENVIRONMENT=dev-local-levander set at source, no forward-time ddtags rewrite); complete field contract verified live (code short, errors full, details codes short per level, fingerprint full, Caused-by stack, nested http blocks on 5xx paths). GOTCHAS: facet queries with :: in the value MUST be quoted (@error.code:"VolueEmsError::ApiError", unquoted silently matches nothing); a survived stale local server held port 8081 so a new launch died AddrInUse while old-format emissions kept flowing (diagnose with lsof on the port + the holder’s binary path). UPDATE 2026-07-27 (short codes REVERSED): the short-code decisions are undone (commit 4df2562a, the new MR !585 head) - error.code AND error.details.code are FULL module path again. This REVERSES the shrink/readable-short-form notes above: the @error.details.code facet is full-path (no longer short), and the DD-cutover format break is back to FULL impact across @error.code/@error.kind/@error.type/@error.fingerprint/@error.errors/@error.details.code (nothing resembles the legacy short codes now); ::-quoting rule still applies. The short_error_code helper is deleted.
mando-cli-flow-run-2026-07-28 - mando flow run emits machine-consumable conformance pointers at terminal for a downstream test suite: dd_query (@flow.exec_id:<id>, env-scoped from the rendered .infra/datadog.builtin.yaml DD_ENV when the local datadog overlay is active), dd_url (DD_SITE-aware deep link with epoch-ms bounds), started_at/stopped_at. Local builds ALWAYS emit dd_formatter JSON (init_log hardcodes dd_enabled=true). Trace-backend hint: local+datadog→Datadog, local-without→generic OTel (no Jaeger UI in repo), remote→Datadog. Local follow filters container logs by the root flow.exec_id JSON key.
mando-cli-otel-capture-and-tail-2026-07-28 - every mando up stack now captures OTel traces to disk via a bundled mando-otel-collector (raw OTLP JSON NDJSON, rotated) and up --datadog fans out to Datadog APM (agent OTLP receiver + collector otlphttp); mando tail reads those capture files + local logs with a gron-flatten + substring-AND filter (flow.exec_id = "<id>), verbatim/golden-friendly, local-only by design (remote = Datadog dd_query pointers). flow run --span-file/--bundle extract this run’s spans by flow.exec_id attribute. Callout bug: ensure_infra collector-config revert killed APM after any mando status — fixed via overlay-file-derived variant.
mando-cli-datadog-local-logs-2026-07-22 - Datadog host-tag join latency (2026-07-23 guide test): env:dev-local-<user> rides DD_ENV HOST metadata, and the host-tag join takes up to ~10 min for a brand-new agent host (~8 min observed), so freshly shipped logs are initially queryable by service:/container_name: only and get the env tag retroactively; an env filter that looks broken right after agent start is just the join lag. Logs Search API api.datadoghq.eu/api/v2/logs/events/search works with the org API key + a Personal Settings ddapp_ application key.
BE-4014 Datadog Error-Rendering Test Harness - unit-test harness for DD error rendering: render_dd(|| error!(...)) / render_dd_one in mando_lib::app::dd_formatter::test_support (real error! macro through real DatadogFormatter, --features app,test-util). Test-pins develop’s DD emission contract: report arm errors[]/details[] + NO fingerprint (codekindtypeoutermost, messagedeepest); plain + http arms bare fingerprint==code (the {code}|{step_path}|{activity} shape is StepResult::log’s); NO render-layer redaction of http.*.body (caller-owned no-leak); no attributes on error.details on develop (branch-only). Develop-based, independent of the error-stack train.

Workflow / Flow Engine

fatal-step-double-logging-ancestor-boundary-2026-09-02 — Fatal steps emit the SAME error at the leaf boundary and again at every ancestor wrapper node (run_step logs unconditionally at mod.rs:499; status_or_error() returns Err only for Fatal, ? re-surfaces it at the wrapper). Fingerprints diverge by step path so copies don’t aggregate. Not the BE-3541 axis; still live on develop b5770766. Trigger was a DuckDB duplicate PK already fixed by MR !610 (dab6cba6) but absent from prod 7c1ef44f.
External Service Outage Gate - bitmask design - gating flow/step execution on declared external-service outages collapses to ONE bitmask AND: each service is one bit (severity 1 = Warning never blocks, so only 0/2 matter), required = fold s: acc | (R(s) << i(s)) is a codegen-emitted per-flow const, blockers = required & outages, can_run = blockers == 0. R(s) = OR over declarations of (propagates_failure(step) AND NOT transient); the expected min over thresholds degenerates to OR. GOTCHA: propagates_failure must fold up the ANCESTOR chain (a hard step under a failure_status: Warning parent contributes nothing) or the guard is stricter than the runtime. transient: true and failure_status: Warning are identical AT THE GATE - one code path only. OR being idempotent+associative means dedup is a structural WorkflowStep tree fold, no HashSet. Blocked run maps to HTTP 423 beside FlowDisabled/SystemDisabled.
be-1595-flow-repository-sqlite-bugs-2026-06-17 — Two latent FlowRepositorySqlite bugs (Sqlite dev/test only): create_flow lazy-query() INSERT that never persists, and started_at current_timestamp TEXT default unreadable as i64 in get_execution.
be-1595-flight-execution-id-parity-2026-06-17 — Execution-bound manual overrides resolved via FlowRepository.execution_id lookup, now shared between REST and Flight paths (apply_execution_overrides); MandoFlightService carries an Arc<FlowRepository> so the Flight handler can resolve execution overrides like REST.
BE-3117 Per-Flow Error Context Store — flow boundary in workflow/flow.rs wraps execution in FLOW_ERROR_CONTEXT.scope(...) + FlowCompletionGuard (Drop-flush for panic/cancel). Step-level recording in StepResult::log via ErrorRecord::from(&ErrorWithStepStatus), gated on !logged_at_site. The logged_at_site flag is the successor to logged_at_failure_site.
flow-step-log-message-dropped-2026-05-26 — Step-failure logging contract: ErrorWithStepStatus::log(status, message) is the “I haven’t logged yet, parent please log me” path (logged_at_failure_site: false), but the parent’s StepResult::log() drops message. Documents the constructor/Display/status_or_error/top-level-catch chain and the c945514e/4c543cb1 cross-check.
Mando AGENTS.md Master Guide — released flow versions are immutable: never change a released vN’s YAML or generated code; breaking behavior → v(N+1); exemplar manual_schedule v3. Definition YAML (mando-bess/config/flows/fi/*.yaml) → build.rs → generated Rust that is TRACKED in git (commit YAML + output together). Tip flows @ 2026-07-08: as_auction_update_v2, auction_v4, data_update_v2, intraday_v2, manual_schedule_v3. Jul-08 develop restructured src-side wiring from per-type src/flow/{type}/v{n}/ dirs to flow_registry.rs, backed by new crates mando-flow-step (FlowStepService trait: uniform async step construction from YAML) + mando-flow-step-derive (ParamEnum/ParamMeta derives). Round 2 (2026-07-13): FlowStepService is a construction trait, NOT execute()-style — type Params: ParamMeta; type Response; + async from_config(config: &str, providers: &StepProviders); StepProviders carries flow_repository/data_point_registry/6 auth providers/simulation_enabled; 36 type_entry! registrations in mando-bess/build.rs; param structs derive #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ParamMeta)]; config/flows/manifest.yaml is the deployment catalog (setup: {version}: flows: {name}: {path, schedule_env, semaphore_group} — scheduling env + concurrency groups in the manifest, not code; semaphores capacity 1/group); the flow-step crates ship with zero tests.
BE-3656 APM Span Enrichment - SpanEnrichmentLayer copies error.*/http.* off ERROR-level tracing events onto the active OTel span as Datadog APM tags; reads opentelemetry::Context::current().span() NOT tracing::Span::current() (empty inside Layer::on_event under scoped subscribers). MERGED via MR !578 2026-07-22 (squash 46a9be1b, merge 4a0d297f). Verified live on dev (3 commits, a0735ebb review-hardening: bodies off spans, first-error-wins guard, no MR yet). Dev-deploy prototype d17227ac validates the BE-3657 field shapes: full-path error.code, startup mock deep error, error.errors/error.details as real JSON arrays (targeted dd_formatter parse), flow.step.system from !570; DD array queries verified (element/wildcard/object-field/numeric/negation). Note: DD flattens array SPAN attrs to key.0/key.1 (breaks faceting) so arrays are log-side only, spans keep scalar error.code. Three tooling landmines added: Bash cwd resets to main checkout between calls, RTK garbles grep/sed (Read to ground-truth), grep "test result" | tail -1 cargo false green.
mando-cli-flow-run-2026-07-28 - the mando flow CLI (list/run/status) triggers + follows a flow run. Wire model: kebab URL segments (auction/intraday/data-update/as-auction-update/manual-schedule), snake_case flow_key body wrapper, server pins the version. Terminal detection is sound because the server pre-creates ALL step rows as Queued and the returned trigger id the DB execution id every log line’s root flow.exec_id; step digest maps Success/Warning/Skip/Error/Fatal/Canceled, exit 1 on Error/Fatal.
BE-3482 pymando Branch Review - the step info_span! in mando-lib/src/workflow/mod.rs gained a flow.step.system tag (2026-07-13 rework, commit 52b4d270 on feature/BE-3482-pymando-v2, unpushed): the non-Mando counterparty taken directly from StepMetadata.system (the same structured source connection() formats), no string parsing; verified all 14 DataFlow::Wrapper steps declare system: "Mando" and externals declare real names (Volue EMS x19, Metis x14, Position Manager x8); 4-case #[test_case] matrix covers Receive/Send/Wrapper semantics. Scope tension: one mando-lib commit on an otherwise py-mando-only branch, trivially splittable.

Levandor

Explorer

Alpiq BESS Topics

Arrow Flight / Deployment

BESS AM / Real-time Telemetry

Bugs / Investigations

CI/CD

Clippy / Lints

Conventions / Agent Guides

Data Point Retrieval

Docker Compose / Runtime

DuckDB

Env / Configuration

Error Handling

External Integrations

Infrastructure / AWS

Licensing

mando-cli

py-mando / Python Bindings

Testing

Tracing / Observability

Workflow / Flow Engine

Graph View

Table of Contents