Levandor

Alpiq BESS Topics

51 min read

For Agents

Living index of themes for the Alpiq BESS / Mando project. Each H2 is a topic; bullets are wikilinks to related notes. Updated by obsidian-documenter when documenting work. Read by historian at bootstrap. Topics kept alphabetical.

Arrow Flight / Deployment

  • be-1595-arrow-flight-dev-deploy-runbook-2026-06-24 — Dev deploy runbook (recurring facts). Three artifacts move together (mando image + py-mando wheel + optimization/forecast consumer images; same build on a wire change). Migration drift → mando crash-loop (V…__… is missing from the filesystem), fix by merging develop so branch migrations ⊇ DB’s. IaC = optimization-universe-iac .worktrees/mando-arrow terraform terraform.auto.tfvars.json (mando/optimization-algo/forecast-algo image pins). terraform_apply:dev always red on a pre-existing customer-portal S3 403 HeadObject despite Apply complete! (harmless, not ours). Runtime switch MANDO_FETCH_STRATEGY=flight + MANDO_FLIGHT_HOST/PORT on algo task defs. Verify in Datadog EU env:dev: bess-os-service-mando/bess-os-algo-optimization/bess-os-algo-forecast.
  • be-1595-flight-rest-graceful-degradation-2026-06-24 — F4 capability-gap fallback. Flight/DuckDB passthrough only streams within its cache window; a range predating the cache (Permanent storage) returns RepoError::MethodNotSupported (repo_passthrough.rs:335). Was mapped to gRPC FailedPrecondition + client only fell back on is_retryable() → propagated Fatal, crashed forecast. Fix (commit 7926278c): map MethodNotSupported → gRPC Unimplemented (mando-bess/src/flight.rs repo_error_to_status) + ClientError::is_unimplemented() (mandarrow-client/src/error.rs); client falls back on is_retryable() || is_unimplemented() (py-mando/src/polars.rs). KEEP CyclicDependency as FailedPrecondition (real error). Net: optimization fully on Flight, forecast logs falling back to REST then completes. Resolves the “Known limit” in be-1595-flight-execution-id-parity-2026-06-17.
  • be-1595-flight-wire-type-version-skew-2026-06-24 — #1 deploy hazard: wheel ↔ server skew. Wheel (mandarrow-client Flight wire types + bess client send(update_id=...)) and server deploy independently; a develop merge bumped shared DataPointUpdateInfo (fetch_time→update_time, added update_id) → old-wheel↔new-server broke. Fix (commit 73cbdcf3): tolerant wire types — #[serde(default)] on every optional/added field, #[serde(alias="old")] on renames, NEVER #[serde(deny_unknown_fields)] on boundary structs, + compat tests. Can’t retroactively fix released versions (deploy mando + wheel from the same build on wire changes); send-side signature mismatch only resolves by rebuilding consumers on the matching wheel.
  • be-1595-arrow-consumer-lock-script-2026-06-24 — Consumer wheel-lock script (deploy-arrow-consumers.ps1 / base64 iex one-liner). Windows on corp net, Nexus must resolve. Clone via token URL (https://oauth2:$Token@…) + GIT_TERMINAL_PROMPT=0/GCM_INTERACTIVE=Never to stop the GitLab credential popup hanging. The base64 one-liner is a frozen snapshot → a stale clipboard silently re-locks the WRONG wheel; re-copy fresh + verify the echoed wheel. Dev images gated behind the python-docker-publish component (Publish Docker Dev only on develop/rc) → script appends a local rules-override for feature/*.
  • be-1595-flight-execution-id-parity-2026-06-17 — Brings the Flight streaming path to parity with REST for execution-bound manual overrides; fixes the dev regression where MANDO_FETCH_STRATEGY=flight hard-broke the optimization/forecast algo runners (they always fetch with execution_id). Shared mando_lib::app::execution_override::apply_execution_overrides helper (DRY across REST get_data + Flight do_get); execution_id: Option<Uuid> added to both QueryTicket + FlightTicket (#[serde(default)]); overrides injected into DataPointFilters at the HANDLER layer before retrieve_stream (no trait change); MandoFlightService gained an Arc<FlowRepository>; client guard relaxed to error only on access_token (headers + access_token-fallback deferred). Deploy needs mando image + wheel + consumer images.
  • be-1595-enabling-arrow-flight-consumers-2026-06-16 — How to turn ON the Flight client path in BESS consumers. Runtime switch: py-mando env MANDO_FETCH_STRATEGY (default rest; Flight only when == flight; flight.rs:16, read per-fetch polars.rs:224-236) + MANDO_FLIGHT_HOST/PORT/PROTOCOL. Dev wheel ships -F flight so it’s a pure env flip, no rebuild. Gotcha: Flight-fetch error → SILENT REST fallback (polars.rs:234); confirm via logs/metrics. Per-service topology: algo-optimization + algo-forecast share the bess-os ECS task (bess_os_ecs.tf, localhost) → add 3 env vars (done on feature/mando-arrow, d3db63e); trader-dashboard is a separate task (trader_dashboard_ecs.tf, over local.mando_domain) with no network path to the gRPC port → needs NLB/ALB-HTTP2 plumbing, left on REST. Server side: var.mando_flight_port (50051) exposed via port mapping + SG rule.
  • Arrow Flight Streaming Design — Flight streaming server/client design.
  • Arrow Flight Gap Analysis 2026-04-30 — gap analysis.
  • be-1595-publish-docker-dev-feature-branch-test-need-2026-06-16 — CI gotcha enabling dev-image builds on feature/* for the Flight consumer repos.

BESS AM / Real-time Telemetry

  • BESS AM (BE-2262) - mando-bess-am — BESS Asset Management (ticket BE-2262, in flight, authored by Gergely Vászon). Headless Kinesis stream processor (new crate mando-bess-am) for per-second WAGO battery telemetry (real-time SoC/SoH) on the FI/Valkeakoski/Beskar asset. Pipeline WAGO Box → AWS IoT Core → Kinesis (bess-am-events) → mando-bess-am → Kinesis (bess-os-events) → mando-bess: consumes raw per-second events, persists to Postgres schema bess_am, computes 1-min aggregates (Mean/Max/Min/Last/StdDev/Count/Sum) via windowing + grace-period closure, forwards to bess-os-events. Introduces a NEW Event-typed data-point class distinct from the time-series flows (NOT one of Trading/Manual Schedule/Auction/Intraday/Data Update/AFRR). Kinesis adapter at mando-lib/src/adapter/kinesis/; Event model mando-core/src/model/{event,wago}.rs; Event datapoints in battery_online.yaml. Builds on BE-2132 (interval/Event groundwork), depends on BE-2341 (generic Kinesis producer, folded into the MR). Core MR !512 OPEN/requested_changes; nothing merged to develop yet.
  • Dev Deploy 2026-07-14 (error telemetry) - the May feat deploy of the BE-2262 POC left U202605131400__create_event_table + U202605141900__create_kinesis_checkpoint_table in dev’s refinery_schema_history; the files exist only on the unmerged branches, so EVERY develop-based image crash-loops on dev at boot (MigrationError::PatchApplicationFailed). Worked around by restoring both files byte-exact (single canonical blob f45b6dd4 across all source commits). OPEN ACTION: land the two files on develop (small MR, recommended) or clean dev’s history rows.

Bugs / Investigations

  • be-1595-flow-repository-sqlite-bugs-2026-06-17 — Two latent FlowRepositorySqlite bugs (local Sqlite dev/test path only, NOT Postgres prod; worth a ticket), found writing the test seed for the Flight parity work. (1) create_flow (flow_repository_sqlite.rs:111) uses lazy stmt.query() for an INSERT + drops Rows without iterating → SQL may never run, row never persists (use .execute()/.insert()). (2) started_at schema default current_timestamp stores TEXT but get_execution reads column 2 as i64 → row created without explicit integer started_at fails to read back (InvalidColumnType TEXT vs i64).
  • calculated-virtual-dp-range-cutoff-bug-2026-05-18 — Calculated & Virtual DPs leak rows past to (all 4 retrieve methods); root cause in service_base.rs:94-159, mechanisms in convert_resolution.rs upsampling + evaluation.rs Full-join/concat. Diagnosed, fix pending.
  • flow-step-log-message-dropped-2026-05-26 — ErrorWithStepStatus::log(status, message) drops message at parent log site; StepResult::log() (mod.rs:155-159) destructures with .., Display is "status: {status}" only (mod.rs:443), status_or_error collapses Log to Ok (mod.rs:231-239), top-level catch in mando-bess/.../flow.rs:289 sees "status: Error" only. Error(anyhow) arm OK; only ::Log broken. Tests at mod.rs:548-605 only assert log level + wrapper string. Affects all envs. Cross-check follow-up commits on feature/BE-2272.
  • mando-cli-v0.4.0-compose-bugs-triage-2026-05-26 — Two confirmed compose-runtime bugs in v0.4.0: context: . resolves to runconfig/ not project root (build.yml:11, build_dev.yml:18); bare service slug passed for mocked runconfig activates malformed override stub via render_override (templates.rs:298-325) — profiles: ["{run_tag}"] line at templates.rs:317 is a load-bearing invariant keeping mocked stubs dormant. Fixes identified, not yet committed.
  • mando-cli-v0.4.0-mando-bess-binary-missing-2026-06-25 — The NEXT failure after the above triage’s fixes (commit 2233959). On a fresh clone, default profile mando-mocked-algos aborts docker compose build at COPY target/release/mando_bess → failed to compute cache key … not found. Design gap: mando/Dockerfile is a thin runtime image copying a PRE-built binary (no Rust stage), bess-service.yaml:23-32 context_includes it (CI assumption), and the default profile maps mando→build runconfig (runprofile.rs:225) while nothing in up compiles it (--cargo is artifact-only, cli.rs:55). Workaround: mando up --mando=pull (pulls the registry image). Durable fix pending decision: flip default → pull (open risk: confirm default MANDO_IMAGE → registry not mando:dev) + add an up preflight guard for missing binary. Diagnosed, not implemented. Reported by gabi on WSL.
  • mando-cli-v0.4.0-piped-output-invisible-failures-2026-06-25 — Observability bug found while validating the above’s --mando=pull workaround. When up output is piped/redirected (2>&1 | tee log), failures are INVISIBLE: captured .mando/compose-up.log empty, only a bare docker compose up … failed (exit 1) shown. Two stacking root causes (both required): (A) docker compose writes progress+errors to /dev/tty, bypassing mando-cli’s pipe (compose.rs run_streaming) → fix: top-level --progress plain (compose.rs ~L231); (B) CliBuffer::add_log_line (opts.rs:57) uses indicatif::MultiProgress::println, a silent no-op on non-TTY, dropping all streamed output + the failure-tail replay (compose.rs:263-272) → fix: eprintln! fallback when !stderr().is_terminal(). Validated macOS: redirected run 35→138 lines, real pull access denied + tail + log pointer now visible. Reusable: pipe-captured docker compose needs --progress plain/--ansi never; indicatif println needs eprintln! fallback on non-TTY. Tester’s now-visible failure = registry pull access denied (needs docker login, distinct from mando-cli’s GitLab-API auth). Both fixes built+validated, NOT committed; linux/amd64 build pending.
  • BESS AM (BE-2262) - mando-bess-am (Update 2026-06-24) — FIXED 2026-06-25. mando-codegen casing bug: expand_variants mis-cased python_field for multi-segment Named variants — Min1/Mean → market.da.price.min1.Mean instead of all-lowercase …min1.mean (lowercased first path segment but not the value segment). Affected BESS AM 1-minute aggregate datapoint names (Min1/Mean, Min1/Max, Min1/StdDev…). Originated in the branch’s own feat: separate out bess-am commit 5a739baa, NOT the develop merge (mando-codegen byte-identical before/after). Fix: lowercase the value segment at mando-codegen/src/util.rs:41 → format!("{pf}.{name_lower}.{}", v.to_lowercase()); commit cc4720b2 on feature/BE-2262-bess-am-poc, pushed to MR !512. Verified cargo test -p mando_codegen now 51/51 (was 50/1). Residual (unrelated): two pre-existing rustfmt drifts remain in that same file (~L50, ~L390, author’s code), left untouched. (Same update also records origin/develop merged into the !512 branch as 80a17e78 — compiles clean, 428 lib tests pass.)
  • Dev Deploy 2026-07-14 (error telemetry) - dev crash-loop at boot: MigrationError::PatchApplicationFailed: migration V202605131400__create_event_table is missing from the filesystem. Root cause: dev’s refinery_schema_history carries two migrations from the UNMERGED BE-2262 bess-am branches (May feat deploy), so refinery validation fails for any image built without those files; the INVERSE of the BE-1595 drift case, merging develop cannot fix it. Fix: byte-exact restore of both files (blob f45b6dd4, checksums pass) as 6bc418cd. STANDING WARNING until develop carries the files or the rows are cleaned.
  • fr-region-missing-datadog-logs-2026-07-21 - FR region services silently absent from Datadog logs: modules/bess_os/cloudwatch.tf creates the -fr log groups but the dd-logs-forwarder subscription-filter modules were dropped at module extraction (they live only in top-level cloudwatch.tf). Diagnosed with live AWS evidence, fix not applied.
  • mando-ci-lib-only-test-gate-2026-07-22 - mando-bess/tests/flight_end_to_end.rs is compile-broken on develop itself (E0603: DataPointUpdateInfo imported via mando_lib::service where it is only a private use since the b3ce27a8 repository crate split), undetected because the CI test gate is --lib-only; fix is a pending one-liner (import from mando_core::model::datapoint), its own change on Andras’s yes.
  • mando-cli-full-audit-2026-07-22 — full ~18.7k-line mando-cli audit: 37 verified bugs (2 HIGH — unguarded kill -TERM -{pid} mass-kill in host_process.rs:108-133 with no pid<=1 guard; orphaned untracked host procs on record failure in up.rs:262-270 — 18 MEDIUM, 17 LOW). Themes: exit-0-on-failure (migrate/get/pull), no reqwest timeouts anywhere, wiremock.enable() silent no-op. Also a −280-line/−5-dep over-engineering cut list, 9 duplication clusters (compose-context prologue copy-pasted across 8 command files), and the simulator runtime spec unimplemented on main. No fixes applied.

CI/CD

  • BESS AM (BE-2262) - mando-bess-am — adds a dedicated CI job “Publish BESS-AM Docker Dev” building the mando-bess-am image from container.bess-am.Dockerfile (ECR tag prefix bess-am-<version>). Crate-split branch feature/BE-2262-bess-am-poc-build-test (commit 5a739baa) separates it into its own image.
  • be-1595-publish-docker-dev-feature-branch-test-need-2026-06-16 — GitLab gotcha: enabling Publish Docker Dev on feature/* in the BESS Python consumer repos needs optional: true on the .Publish → Test dependency, or the pipeline won’t start.
  • be-1595-arrow-consumer-lock-script-2026-06-24 — consumer dev images gated behind the python-docker-publish component (Publish Docker Dev only on develop/rc); the wheel-lock script appends a local rules-override so feature/* builds. Also the GitLab credential-popup hang fix (GIT_TERMINAL_PROMPT=0/GCM_INTERACTIVE=Never + token-URL clone).
  • be-1595-arrow-flight-dev-deploy-runbook-2026-06-24 — terraform_apply:dev always red on a pre-existing customer-portal S3 403 HeadObject despite Apply complete! (harmless); IaC image pins in optimization-universe-iac .worktrees/mando-arrow terraform.auto.tfvars.json.
  • clippy-disallowed-macros-allow-placement-2026-06-22 — keeping clippy green across versions: disallowed_macros allow must be a module-root #![allow] at the macro’s definition (mando-core/src/error.rs), not at the call site (ineffective on clippy 1.95). Workspace pins 1.88.0; verify clippy-clean under the pinned toolchain since the call-site/block form is version-fragile.
  • mando-cli-github-build-mirror — GitHub-Actions build pipeline, GitLab Package Registry publishing, parallel-release flow (init → matrix-publish → finalize).
  • mando-cli-wsl-linux-build — local macOS (Apple Silicon) cross-compile of a Linux x86_64 / WSL mando binary; --platform linux/amd64 + poc/-parent-mount gotchas. The manual counterpart to the CI mirror.
  • Mando AGENTS.md Master Guide — the verified mando gate set: NO cargo fmt gate in CI (never bare cargo fmt — ~64 legacy files reformat; scope to touched files); lint gate cargo clippy --release --all-features; canonical test cargo test --all-features --release -- --test-threads=1 (single-threaded MANDATORY — shared in-memory DB pools). Runtime image now ca-certificates/libc6/libssl3 only (samba dropped); build image base still rust:1.88.0-bookworm + samba (toolchain 1.89.0 via rust-toolchain.toml). Pipeline etiquette: title-only Conventional Commits, feature/BE-xxxx branches, py-mando publish is manual, release/* flows back to main+develop. Round 2 (2026-07-13): rustfmt.toml’s import-grouping/comment options are nightly-only + silently ignored on stable 1.89 (tip tree NOT fmt-clean — import grouping maintained by hand); CI gained a “PyMando Simulation Linux Dev” job (pytest without --nbval) + a simulator docker publish child pipeline; no MR/issue templates, no CODEOWNERS.
  • Mando Deployment Ceremony - the two-repo deploy ceremony (AGENTS.md section 14): mando CI only publishes images (Publish Service Docker Dev, only: develop + rc/, tag {version}-dev.{pipeline_id}.{short_sha}, APP_VERSION + becomes -); optimization-universe-iac pins versions in terraform/terraform.auto.tfvars.json with ALL terraform_apply jobs when: manual (dev/test from develop, int from release/, prod from main). Feature-branch deploy = temporary only: edit that MUST be reverted before the MR merges. Never deploy from mando alone.
  • Dev Deploy 2026-07-14 (error telemetry) - worked example of the feature-branch deploy on BOTH repos: throwaway mando branch feature/BE-3482-dev-deploy (develop tip 51d2b516 + !569 cherry-picks + only: edit + the PyMando Win Dev needs/dependencies entry commented out; that job is absent on feature branches and a dangling need kills pipeline creation) + IaC branch feature/error-hdl (version bump + “ci: enable dev apply from error-hdl branch” adding the branch to terraform_apply:dev only:, which normally exists ONLY on IaC develop). All CI edits temporary; strip on merge.
  • mando-ci-lib-only-test-gate-2026-07-22 - the CI test gate (.gitlab/scripts/test.sh) runs cargo test --all-features --release --lib -- --test-threads=1: ONLY lib unit tests are gated; */tests/ integration targets are never compiled by the pipeline and rot silently (the separate “Integration Test Linux Dev” job covers only the mando-lib archiver Postgres suite). Local gates need --lib for CI parity. --lib arrived with the archiver split commit 7278f36a. Companion lesson 2026-07-23 (BE-3657 phase 3): per-crate task gates (-p mando_lib) likewise miss CONSUMER crates - the final whole-branch review caught 2 tuple-syntax construction sites in mando-simulator that no task gate ever compiled (a Critical); workspace-wide cargo clippy --release --all-features (no -p) is the only gate that catches cross-crate fallout.

Clippy / Lints

  • clippy-disallowed-macros-allow-placement-2026-06-22 — clippy disallowed_macros (mando bans tracing::error!/log::error! in clippy.toml to force logging through the mando_core::error! wrapper): a per-invocation/per-block/per-fn #[allow(clippy::disallowed_macros)] at the call site is silently ineffective on a newer clippy (1.95) and is flagged “unused, applied to a macro invocation”. The lint resolves at the lexical site where the banned tokens live (the macro definition), so the only robust fix is a module-root #![allow(clippy::disallowed_macros)] in the defining file (mando-core/src/error.rs) — one allow covers all cross-crate callers. Caveat: workspace pins 1.88.0; the block form on develop presumably passes CI on 1.88 but is fragile across clippy versions; module-root #![allow] can’t be worse on 1.88 and is confirmed on newer clippy. General rule: for lints that fire on macro expansions, #[allow] at the call site no-ops — put #![allow] at the macro’s definition.

Conventions / Agent Guides

  • Mando AGENTS.md Master Guide — master conventions doc for AI agents at /Volumes/bandi/coding/poc/mando/AGENTS.md (untracked, deliberately never committed; synthesized 2026-07-10 from 5 parallel researchers on develop). 12 sections + 8 recipes: crate map/dependency direction, 10 non-negotiables, Rust style, mando_core::error! system, mando-bess API + py-mando PyO3 conventions, testing/git/CI ceremonies, verified-landmines table. Headline corrections: mando-bess/build/generated/** is generated BUT tracked in git (edit YAML/template → build → commit both, never hand-edit); NO ErrorCode derive / NO mando-lib-macro on develop (worktree experiments only; parent poc/CLAUDE.md stale); JWT token_layer signature validation DISABLED (audit metadata, not authz); flow versions immutable once released. Rebuilt 2026-07-13 (round 2) against the true develop tip 92bfe1c8 (Jul-9, v1.16.0) via 5 re-run researchers on a .worktrees/develop worktree — the stale-merge-base caveat is obsolete; local develop fast-forwarded + now tracks origin/develop. Round-2 facts: FlowStepService = construction trait (type Params: ParamMeta; type Response; + async from_config(config, &StepProviders)), manifest.yaml deployment catalog (schedule_env/semaphore_group, semaphores capacity 1), 36 type_entry! registrations, error redesign partially landed (ae6d1098), py-mando-simulation split (BREAKING), second async bridge (future_into_py), rustfmt opts nightly-only/ignored, zero tests in the new crates, no MR templates/CODEOWNERS. Maintenance rule: code wins, then update the file. Rebuild process gotcha: agentops:researcher subagents can’t SendMessage — collect reports via scratchpad files or transcript extraction (~/.claude/projects/<project>/<session>.jsonl, longest assistant text).
  • Mando MR Review Patterns - evidence base behind the guide’s “Recurring review feedback” section: all 21 of Andras’s mando MRs (!322 through !558; 17 merged / 2 closed / 2 open) mined 2026-07-13 via read-only glab API; 108 reviewer notes (krisztian.fekete1 87 / gabor.nagy6 13 / balazs.mracsko.alpiq 7). Nine themes with source MRs: zero unrelated diff changes; delete ALL unused code before review (most repeated complaint); failures are errors never warnings (mando_core::error! + enum variant, no custom message strings); placement (route helpers next to routes, util in util); completeness before review (mirror sibling configs); green pipeline + no conflicts before assigning review; rustfmt every new file; Jira ticket in sync; idioms (errors imported from mando_core, no unwrap, bind service once). Plus config direction (new config: config::Config over Envconfig) and the dictated rules: commits title-only NO scopes/descriptions (reviewer-mandated !322), commit every ~3 tasks, never em dashes anywhere, always subagent-driven development.
  • BE-3482 Datadog Logs and APM Conformance - source of the 2026-07-13 fmt correction: cargo fmt -- <file> does NOT scope (field-verified, it touched api_docs.rs which was never named); format single files with rustfmt --edition 2021 <file>, every form of cargo fmt banned. Propagated to AGENTS.md sections 2.3, 10, 13.5 + the landmines table, Mando AGENTS Guide (mirror) resynced, fmt warnings in Mando AGENTS.md Master Guide / Agent Context amended.
  • BE-3482 Datadog Logs and APM Conformance - second 2026-07-13 tooling landmine: the RTK shell hook rewrites cargo test --all-features --release -- --test-threads=1 so the threads flag becomes a test-name filter (every test filtered out, exit 0 in 0.00s, signature 0 passed, N filtered out; two gate runs false-green before detection). On RTK machines run rtk proxy cargo test --all-features --release -- --test-threads=1 and confirm the summed passed totals are nonzero; exit 0 alone proves nothing. Adjacent pitfall: cargo test | grep | tail; echo $? reports tail’s exit, not cargo’s. Recorded in AGENTS.md section 10 + the landmines table; mirror resynced.
  • BE-3482 pymando Branch Review - third 2026-07-13 tooling landmine: rustfmt --check --skip-children is UNRECOGNIZED on the pinned stable rustfmt and prints nothing (false green); check without the flag, attribute reported hunks by the Diff in <path>: headers (rustfmt follows mod declarations into child files), and baseline against git show <base>:<path> to separate pre-existing churn from branch-introduced churn. Recorded in AGENTS.md section 10 + the landmines table; Mando AGENTS Guide (mirror) resynced.
  • BE-3482 pymando Branch Review - fourth and fifth 2026-07-13 tooling landmines (from the rework session): (a) rustfmt base-file checks that COPY the file to a scratch dir are FALSE GREEN (rustfmt cannot resolve the mod children from the new location and reports 0 hunks), and running rustfmt on a module-declaring file WRITES INTO its child files (6 unrelated py-mando files were touched and reverted); the only sound method is checking out the base revision at its real path; (b) develop’s committed Cargo.lock can be STALE (mandarrow-client 1.16.0 vs workspace 1.16.1); plain cargo build regenerates it, so a lock hunk in a branch diff may be legitimate, not churn. Recorded in AGENTS.md section 10 + the landmines table; Mando AGENTS Guide (mirror) resynced.

Data Point Retrieval

  • BESS AM (BE-2262) - mando-bess-am — introduces a NEW Event-typed data-point class (real-time telemetry) alongside the existing TimeSeriesDouble/TimeSeriesDoubleMatrix/StaticData. Event datapoints (BATTERY_SOC_ONLINE/BATTERY_SOH_ONLINE) defined in mando-bess/config/parts/battery_online.yaml under Asset/FI/Valkeakoski/Beskar/Battery/.../Online, keyed to Kinesis external IDs [bess-am-events, OnlineSOC/OnlineSOH]. Model at mando-core/src/model/event.rs; built on BE-2132 interval/Event groundwork.
  • calculated-virtual-dp-range-cutoff-bug-2026-05-18 — [from, to) range not enforced for Virtual/Calculated DPs in handle_data_point_types; EvaluationMetaData.range plumbed but unused by final output.

Docker Compose / Runtime

  • mando-cli-simulator-runtime-2026-05-30 — adds second compose project mando-sim as a peer to dev mando; two-stack ergonomics via shared service_stack::select_profile resolver (single source of truth for logs/exec/volume); env merged via fill_build_args (NOT --env-file, which hard-fails on missing .env); Python healthcheck; Gurobi license as file mount, not multiline env. (§A env contract SUPERSEDED 2026-06-02 — see mando-cli-simulator-env-contract-2026-06-02)

  • mando-cli-simulator-env-contract-2026-06-02 — realigned simulator env to real images: orchestrator SIMULATOR_DATABASE_* (replaces POSTGRES_*), per-runner SIMULATOR_<NAME>_HOST (replaces SIMULATOR_RUNNERS CSV), repo URL/branch/commit_hash moved orchestrator → runners. Runners read unprefixed SIMULATOR_REPO_URL/BRANCH/COMMIT_HASH (assumption — verify when image lands). Healthcheck Python → curl; Gurobi mount → GUROBI_LIC env. Schema bess_simulation → simulator.

  • mando-cli-v0.4.0-compose-bugs-triage-2026-05-26 — v0.4.0 baseline compose-runtime triage; context: . relative-path bug; load-bearing profiles: ["{run_tag}"] invariant in render_override.

  • mando-cli-v0.4.0-mando-bess-binary-missing-2026-06-25 — next compose-build failure once the above fixes (commit 2233959) landed: default mando-mocked-algos resolves mando to the build runconfig (runprofile.rs:225), but mando/Dockerfile is a thin runtime image that COPYs a pre-built target/release/mando_bess and the up flow never compiles it → docker compose build aborts on a fresh clone (failed to compute cache key: "/target/release/mando_bess": not found). Workaround mando up --mando=pull; durable fix (flip default → pull + preflight guard) pending decision.

  • mando-cli-v0.4.0-piped-output-invisible-failures-2026-06-25 — the streaming compose runner (compose.rs run_streaming) loses ALL output under pipe/redirect: docker compose writes progress+errors to /dev/tty (bypassing the pipe) → add top-level --progress plain (compose.rs ~L231); and CliBuffer::add_log_line (opts.rs:57) routes lines through indicatif::MultiProgress::println, a non-TTY silent no-op → fall back to eprintln! on !stderr().is_terminal(). Together they restore the captured .mando/compose-up.log, the streamed lines, and the --- last 40 log lines --- failure tail (compose.rs:263-272) for piped/CI/tee runs. Both fixes required.

  • mando-cli-full-audit-2026-07-22 — 2026-07-22 audit of the compose runtime and command layer: 2 HIGH bugs in host-process signalling (host_process.rs), MEDIUM compose/volume/down flag-drop + name-mapping bugs (down.rs, volume.rs, exec.rs/logs.rs), a 9-cluster duplication list headed by the compose-context prologue duplicated across 8 command files (→ compose::prepare), and the simulator runtime spec still unimplemented on main.

  • mando-cli-datadog-local-logs-2026-07-22 — mando up --datadog renders an optional infra template (src/runtime/templates/infra/datadog.yml → .infra/datadog.builtin.yaml) with a local Datadog agent (container_name local-dd-agent, name:^mando-.* include filter), included existence-driven by compose::assemble (NOT in INFRA_BUILTIN_FILES) so a flag-off up can delete it. Compose interpolates env on EVERY verb → the template MUST use ${DD_API_KEY:-} not :? (a :? hard-fails down/logs when the var is absent while the file exists). Extracted build_args::atomic_write_with_dir unifying 3 write sites. UPDATE 2026-07-23: docs/datadog-guide.md (tip c2becf1) verified end to end, every claim holds: 4 fail-fast paths, mando-only tailing, EU shipping (103 logs), teardown at both levels (flag-off up --remove-orphans + down). UPDATE 2026-07-23 (ddtags override): the embedded ddtags env WINS when mando emits one - Datadog JSON parsing promotes the body ddtags (MANDO_ENVIRONMENT=local → env:local, unindexed by the org → 202-accepted lines VANISH); the “no env in ddtags” claim holds only while MANDO_ENVIRONMENT puts none there; set MANDO_ENVIRONMENT=dev-local-<user> for ANY local mando (containerized included); the per-developer env tag is the only wall from the shared org’s REAL env:prod telemetry under the same service name.

  • mando-local-host-run-recipe-2026-07-23 - running a locally built mando_bess as a HOST process against the mando-cli infra stack (mando up --datadog -p infra: postgres + wiremock + local-dd-agent). Env from .cargo/config.toml.example + overrides (file is STALE, see Env / Configuration); infra pg creds bess_transactional_db/postgres/postgres @ localhost:5432; MANDO_PORT=8081 (wiremock owns 8080), adapter hosts at localhost:8080 for hermetic 404s. DANGER: mando’s refinery races the mando-cli migrations container in DuckDbPostgres mode (idle-in-transaction boot wedge, move_schema_objects blocked; recovery = pg_terminate_backend + drop public/data/bess_os cascade + single-writer relaunch; DuckDb mode skips pg migrations but the stale container set lacks the setting table). Host processes are invisible to the local dd agent (mando-* filter): ship to Datadog via direct intake. Teardown: mando down + pkill mando_bess. UPDATE 2026-07-23 (epilogue): the first direct-intake forward VANISHED - Datadog promotes the embedded ddtags from the JSON line, so mando’s env:local (from MANDO_ENVIRONMENT=local) overrode the envelope’s env:dev-local-levander and the org indexes no env:local; fix = rewrite embedded ddtags before forwarding; durable = set MANDO_ENVIRONMENT=dev-local-<user> in the run env.

DuckDB

  • BE-1597 DuckDB UI Exposure Research — BE-1597: exposing the DuckDB UI of running mando-bess + remote client access to its in-memory DuckDB. Root cause both prior attempts failed: the UI JS bundle hardcodes "localhost:4213" === window.location.host ? "duckdb_ui" : "web", so any non-localhost:4213 origin (path prefix, custom hostname, ALB) silently degrades — killing Attempt A (socat 4214 + ALB TG + /duckdb* listener rule; died on the 8080 /health check) and Attempt B (reqwest reverse proxy mando-bess/src/route/duckdb_proxy.rs + <base href>/JS string-patch). Fix = SSM AWS-StartPortForwardingSessionToRemoteHost forwarding the LOCAL port 4213 → browse http://localhost:4213 (literal localhost, not 127.0.0.1); no ALB (already internal=true); bastion hop required since ECS Exec is interactive-only. UI ext: only ui_local_port(4213)/ui_remote_url/ui_polling_interval(284ms) settings, NO bind-address (binds ::1 → socat), no auth, proxies assets from ui.duckdb.org per session (internet required), not statically linked (verified nm -D on vendored 1.4.2 .so → only Icu/Json). Remote client → in-mem DB: only via Quack (DuckDB 1.5.3+, beta until 2.0). Includes a full 1.4.2→1.5.4 upgrade assessment (low-risk, in-memory-only sidesteps storage churn; =1.4.2 pin Cargo.toml:60 vs the new 1.MAJOR_MINOR_PATCH.x crate scheme, 1.5.4→1.10504.0; C API additive-only; date_trunc/lambda SQL breaks clear).
  • BE-1597 DuckDB UI — Design — the approved decision/spec built on the research above (status: design approved, NOT implemented; owner Geri Vászon; repos mando + optimization-universe-iac). Phased: Approach 1 dev-only NOW via SSM AWS-StartPortForwardingSessionToRemoteHost forwarding the LOCAL port 4213 → http://localhost:4213 (literal localhost, not 127.0.0.1; no ALB / target group / health check / /duckdb* prefix / reverse proxy / bundle patching — the bastion hop is only needed because ECS Exec is interactive-only); Approach 2 = Quack for a remote client → in-memory DB, deferred to DuckDB 2.0 (Sept 2026) since Quack is Beta in 1.5.x. Container change: add socat + a docker-entrypoint.sh bridging TCP4:4214 → TCP6:[::1]:4213 when DUCK_DB_UI_SERVER=true (NO Rust change; env already read at lib.rs:181/:301). Never in int/prod (proprietary MotherDuck frontend, no auth). Blocking open Q: Fargate egress to duckdb.org UNVERIFIED.

Env / Configuration

  • BESS AM (BE-2262) - mando-bess-am — BESS AM service config uses env prefix BESS_AM_* and config/bess-am.yaml; reads the bess-am-events Kinesis stream and writes bess-os-events; Postgres schema bess_am. Local-dev Kinesis via doc/kinesis-local-setup.md + MiniStack (ministackorg/ministack).
  • be-1595-enabling-arrow-flight-consumers-2026-06-16 — Flight client toggled by MANDO_FETCH_STRATEGY (default rest, Flight only when == flight, case-insensitive) + MANDO_FLIGHT_HOST/PORT/PROTOCOL. Pure runtime env flip on the -F flight dev wheel; co-located consumers reach mando over localhost, separate tasks can’t (no network path to the gRPC port). Silent REST fallback on error.
  • be-1595-arrow-flight-dev-deploy-runbook-2026-06-24 — deploy-side view of the same switch: MANDO_FETCH_STRATEGY=flight + MANDO_FLIGHT_HOST/PORT set on the algo task defs; verify in Datadog EU env:dev (mando/optimization/forecast) — expect forecast falling back to REST, optimization fully on Flight.
  • mando-cli-simulator-env-contract-2026-06-02 — three reusable patterns: (1) arbitrary extras passthrough — filter workspace .env keys against a managed-set const (MANAGED_SIM_ENV_KEYS), auto-inject survivors as KEY: "${KEY}", sorted alphabetically, with tracing::info! per injection; (2) sim-postgres aliasing — when two services share a default credential, route the dependent service’s native env (POSTGRES_USER) through the orchestrator’s contract (SIMULATOR_DATABASE_USERNAME) so one .env override controls both; (3) derive UI summary strings from the actual invocation-args function — never hardcode the verb+flags in summary text or it goes stale (pre-fix bug: literal "up -d --build --remove-orphans" in success row while up_compose_args had dropped --build for sim).
  • BE-1597 DuckDB UI — Design — the DuckDB-UI dev gate is DUCK_DB_UI_SERVER (already read in mando-bess at lib.rs:181/:301; NO Rust change), set in terraform as DUCK_DB_UI_SERVER = tostring(var.environment == "dev") on the mando container and mirrored by the docker-entrypoint.sh if [ "$DUCK_DB_UI_SERVER" = "true" ] socat guard. Use var.environment, NOT local.environment_group — the group collapses int+prod and dev+test, so it’s the wrong knob for a dev-only gate.
  • Mando MR Review Patterns - config direction from review (two reviewers, MR !556): NEW config uses config::Config, not Envconfig; existing envconfig adapters stay as-is (no mass migration).
  • Mando Deployment Ceremony - per-env mando runtime config WITHOUT rebuild: .environment-vars:{env} blocks in optimization-universe-iac .gitlab-ci.yml flow to TF_VAR_* (MANDO_FLOW_SCHEDULE_* crons, MANDO_FLOW_ACTIVE_VERSION, EBS_SMB_HOST, SIMULATION_MODE, CPU/memory, notification channels). Constraint: MANDO_FLOW_ACTIVE_VERSION (V1_4 in all envs) must name a setup version existing in mando config/flows/manifest.yaml; bump the two together. dev runs SIMULATION_MODE=true.
  • fr-region-missing-datadog-logs-2026-07-21 - mando reads the DD_SERVICE env since commit ab8cfd11 on feature/BE-3657 (resolve_dd_service() + OnceLock-cached dd_service() in mando-lib/src/app/mod.rs, hardcoded bess-os-service-mando as fallback also on empty string, all 6 call sites routed incl. 4 raw global::meter literals); deployment-neutral on the main stack because IaC bess_os_ecs.tf:218 already sets the var on the mando container, and it unblocks the FR -fr suffix.
  • mando-cli-json-output-agent-skill-2026-07-23 - MANDO_OUTPUT=json (exact-match "json" only; the --json flag beats it) is the env knob that flips mando-cli output to terse NDJSON on stdout. Any other value / unset = human mode (byte-identical to pre-feature).
  • mando-local-host-run-recipe-2026-07-23 - .cargo/config.toml.example is STALE (field-verified 2026-07-23): MANDO_FLOW_ACTIVE_VERSION is DEAD, the config now wants MANDO_SETUP_ACTIVE_VERSION (V1_4) + MANDO_SETUP_SCHEDULER_DISABLED (prefix MANDO_SETUP, config-crate style); FINGRID_API_KEY must be NON-EMPTY (init panics via unwrap: “must contain at least one key”); the postgres cert loader reads <cwd>/certs/eu-central-1-bundle.pem, so running from a workspace root needs a certs symlink to mando-bess/certs.

Error Handling

  • BE-3117 Per-Flow Error Context Store — “OEF v2”: one aggregate business ERROR per flow run with full fidelity. ErrorRecord + FLOW_ERROR_CONTEXT task-local Vec + record_error! macro (TT-muncher, in mando-lib not mando-core) + flush_flow_errors/FlowCompletionGuard. Invariant: record exactly once per flow, gated by logged_at_site. Hand-written redacting Serialize (drops bodies when http_context.sensitive). Removes OEF v1’s dd_formatter ERROR→WARN demotion. Fixed pre-existing MarketNotFound double-log via mark_logged().
  • BE-2272 — DD log JSON flattening to root (flow.*, step.*, error.* siblings).
  • flow-step-log-message-dropped-2026-05-26 — ErrorWithStepStatus::Log { message } payload silently dropped by StepResult::log() destructure + Display impl + status_or_error collapse. Only ::Log arm broken; ::Error(anyhow) correctly logs.
  • clippy-disallowed-macros-allow-placement-2026-06-22 — tooling gotcha behind the error-logging path: mando bans tracing::error!/log::error! (clippy.toml) to force logging through mando_core::error!; that wrapper’s own #![allow(clippy::disallowed_macros)] must be a module-root attribute in mando-core/src/error.rs, not at the call site (call-site/block allow is ineffective on clippy 1.95). Found during the BE-3117 error-handling redesign.
  • Mando AGENTS.md Master Guide — correction: NO ErrorCode derive and NO mando-lib-macro crate on develop (verified git grep ErrorCode origin/develop empty) — worktree experiments only (poc-error-extractor, BE-2023, BE-1595/BE-3482); mando-lib-macro marked outdated. error.kind (fully-qualified path) / error.code (Enum::Variant) / error.source are extracted at runtime by mando_core::error! from the Debug repr, no derive. Canonical pattern: every component defines a thiserror enum …Error; anyhow only in init/flow-step glue; HTTP mapping via deterministic map_*_error_to_response (same variant ⇒ same status); py-mando maps everything to the single PyMandoError via map_*_error fns (mando-lib/src/python/error.rs). doc/errors.json = untracked generated catalog (generator not in CI — likely poc-error-extractor worktree). Direction of travel: log-once-at-step-boundary (docs/error-handling-redesign-plan.md). Round 2 (2026-07-13): the redesign PARTIALLY landed in commit ae6d1098 — error.message = root cause, error.stack = source chain, error.fingerprint = {code}|{step_path}, ErrorWithStepStatus.logged_at_site double-log guard with new()/logged() constructors; NOT landed: error.trace, .step_context(); the plan doc + doc/errors.json remain untracked on develop.
  • Mando MR Review Patterns - review-mandated error policy (15+ “should be error” comments + policy statement from balazs, MR !481): failures are ERRORS, never warnings; log via mando_core::error! with an error-code enum variant; Fatal fails the execution; NO custom error message strings (the error’s to_string() IS the message, human context goes in the message field).
  • BE-3657 error_stack Adoption - LOCKED decision (2026-07-16, planned/awaiting go): full error_stack =0.8.0 adoption over the ErrorCode-derive direction. Principle “leave what we can to error_stack”: mando adds only the ErrorCode full-path attachment (captured statically in CodedExt at change_context/new sites) + extraction fns + an error!(report=...) arm. Field contract: error.code==kind==type==outermost full-path code; error.errors = codes array; error.details = [{code,file,line,message}]; fingerprint full-path. Crate’s own tree rendering with codes as attachment sub-lines. Boundary message template "{outermost}: {static}: {chain}". Phase-2 direction: per-variant #[error_meta(event_type, resolution step, business_message, description)] → details enrichment + resolution-doc codegen (phase 1 must not preclude; attachments are open-ended). Branch feature/BE-3657 not yet created, off BE-3656 head a0735ebb; spec/plan under docs/superpowers/{specs,plans}/2026-07-15-*error-stack*. UPDATE 2026-07-22: phases 1+2 BUILT + PUSHED @ 2baea68a (13 commits atop the rebased BE-3656 head 8f50ef2a); boundary template FLIPPED static-first "{static}: {outermost}[: {chain}]" (2026-07-16); MR only after !578 merges + explicit yes, then rebase --onto develop; standalone cargo build -p mando_core broken by a pre-existing BE-3643 uuid/serde issue (workspace graph fine). UPDATE 2026-07-22 afternoon: !578 MERGED (merge 4a0d297f = develop tip); rebased onto it zero-conflict (2baea68a → 22ccae93, patch-id byte-identical, pipeline 2696324182 green) + DD_SERVICE env fix appended (ab8cfd11, branch now 14 commits); MR pending ONLY on Andras’s yes. UPDATE 2026-07-22 evening: phases 1+2 ride MR !580; phase 3 PLANNED + approved: scope “plumbing + adapters” (rejected: plumbing-only, full workspace breadth), mechanics = seam replication per the Volue pilot (rejected: native Report signatures, boundary-only wrapping), same ticket (branch feature/BE-3657-phase3 off develop AFTER !580 merges, one phase = one MR); conformance-tightened against Balazs’s logs-and-apm.md (line 157 absolute: ALL tuple/transparent variants restructured; error.details.attributes via new mando-core ErrorAttr landing before any adapter; line 159 http roll-up = outermost-wins HttpContext pick, with_report fills http_context when unset; auth providers: no bodies/tokens, method/url/status only); adapter order auth providers → volue/ems completion → volue/atp → alpiq/ebs (SMB, share/path ErrorAttr) → mdr/opl/data_platform/et_3000 → fingrid → metis → mando group → position_manager; vetoable defaults: archiver deferred to the repos phase, startup mock ported to mainline behind MANDO_DEBUG_MOCK_ERROR default-off; spec/plan docs/superpowers/{specs,plans}/2026-07-22-*phase3* (15 tasks); execution NOT started, gated on !580. UPDATE 2026-07-23: !580 MERGED (squash de7cd4c6, merge 4c3442df, zero review comments); phase 3 EXECUTED subagent-driven on feature/BE-3657-phase3 (14 tasks + final whole-branch review; 20 commits, HEAD e0184bc3, pushed + ls-remote verified, CI running, NO MR yet): mando-core ErrorAttr + per-level error.details attributes + error_attr_value + report_first_attachment + with_report http roll-up; ALL in-scope adapter groups migrated (*_reported seams, exhaustive no-wildcard *_error_from_report, zero error(transparent) left in mando-lib/src/adapter incl. archiver); auth providers first (no bodies/tokens, sentinel leak tests); PyO3 map_report_error forward plumbing (grep-proven unreachable this phase); shared error_json_field_value across all three log formatters; MANDO_DEBUG_MOCK_ERROR env-gated report-backed mock. KEY FINDING (formal spec exception): only volue/ems carries Reports to the step boundary in production (Error = Report<VolueEmsError>); the other seven groups are WALLED (flat-typed service towers + pub(crate) seams make cross-crate wiring structurally impossible), chains flatten at legacy wrappers until a services-phase tower restructure - every wall independently verified genuine. Gates at e0184bc3: workspace clippy 0, 574 lib tests / 0 failed, no lock churn, 61 files +4390/-1482 vs develop; follow-up bundle triaged (auth-provider pre-existing body logs, ErrorAttr ids on non-success terminals only, py delegation tests CI-only, data_platform test panic-safety, opl clear_strategies portfolio attr, map_err closure-form drift). develop since moved (6fae64a5); pre-MR rebase possible on go. UPDATE 2026-07-23 (local e2e): phase 3 telemetry VERIFIED LOCALLY end to end from the real runtime (branch binary b394e0d4, rebase-descendant of e0184bc3, run as a host process against the mando-cli infra stack): boot mock 4-level report (error.errors 4 full-path codes, error.details real file:line + attributes incl. data_group, error.stack box tree with the ErrorAttr sub-line + the opaque HttpContext attachment, codekindtype, root-cause message, trace correlation); Volue EMS spot single-level fingerprint code|step_path (spot_data.rs:79); Volue EMS retrieve TWO-LEVEL auth chain live (ems.rs:217 + ems_auth_provider.rs:155, the final-review auth level confirmed); Metis full-path code with NO error.errors (walled-tower flattening as designed); dd_formatter real nested JSON; 12 error lines to Datadog EU via direct intake (env:dev-local-levander). Run recipe + friction: mando-local-host-run-recipe-2026-07-23. UPDATE 2026-07-23 evening (MR + short-code decision): phase 3 IN REVIEW as MR !585 “feat: adopt error stack across all adapters” (feature/BE-3657-phase3 → develop, head a58df341, 22 commits, no conflicts, branch pipeline green on the same sha; reviewers krisztian.fekete1 / gabor.nagy6 / balint.budavoelgyi / jozsef.nagy1). Same head implements the NEW DECISION (Andras, 2026-07-23) amending the field contract: error.code goes SHORT (Enum::Variant, bare struct name when variantless, per the team spec’s line-100 mobile-readability intent) while error.kind/error.type/error.errors/error.details codes/error.fingerprint ALL stay full module path - mando-core short_error_code resolver (PascalCase-segment heuristic, valid due to house naming), applied at the 4 macro emission sites only (span/boundary layers passthrough), pinned by test kind_type_fingerprint_stay_full_while_code_goes_short, 577/0 workspace lib tests, verified LIVE locally + forwarded to Datadog (env:dev-local-levander). Consequences: the “mobile readability rationale is dead” pending team sentence WITHDRAWN (conformance restored); DD cutover impact for @error.code-keyed assets shrinks (short form resembles legacy codes); @error.source RETIRED (renamed to error.message in the redesign, pinned by a mando-core regression test) - delete or rekey any facet/monitor still on it to @error.message/@error.errors (an empty error.source consumer was hit and fixed by Andras 2026-07-23). UPDATE 2026-07-24 (Caused-by stack renderer): NEW DECISION (Andras) REVISING the locked “crate’s own tree, codes as sub-lines” trade-off: error.stack rendering switches from the error-stack crate’s Debug tree to a custom prefix-style “Caused by” chain renderer, because Datadog collapses newlines in several views and the crate’s ASCII connectors (|-, lone |, |->) turned into unreadable pipe soup in the real DD UI. Per-level format: {short code}: {message} head (first level bare, subsequent prefixed Caused by: ) + at file:line:column + with key = value per ErrorAttr; opaque attachments no longer mentioned. Commit 1f9cd611 (new MR !585 head, 23 commits): render_stack_tree walks frames like report_details; crate debug-hook plumbing (install_debug_hook, Charset/ColorMode, HOOK_INIT) deleted as dead; 579/0 workspace lib tests incl. an exact multiline golden pin; verified live + forwarded to Datadog (2026-07-24 env:dev-local-levander events carry the new stacks AND the short error.code together). Datadog research recorded: Error Tracking for logs needs error status + service + (error.kind OR a valid error.stack) and mando always emits error.kind (stack format never gates tracking/grouping); DD’s valid-stack bar wants a frame with a FUNCTION NAME + filename, which error_stack Locations never carry (mando stacks never frame-parseable regardless of format); the source tag auto-parses only conventional language stacks; remappers relocate but do not parse - so error.stack is purely presentational, optimized for human readability and graceful newline-collapse (which the Caused by: separators survive). UPDATE 2026-07-24 (error-authoring DX overhaul): decision series (Andras), north star: error reporting must not break Rust’s natural flow, ? is the API; 3 commits (branch now head f771805e, ~27 commits, pushed, riding MR !585): f5beabd4 report vocabulary (new_coded/CodedExt/change_context_coded DELETED workspace-wide → ToReport::into_report (crate already exports IntoReport, hence the name) / Result::reported() / .wrap() / .attr(k,v) / flatten_report for Clone enums collapsing 6 conversion fns to one-liners; 307 sites, 37 files; bonus find: 20 map_err(new_coded) fn-pointer sites - the exact #[track_caller]-losing form - had degraded locations until swept); 4e519269 short error.details[].code (the details panel is the human panel, file:line carries precision; errors/kind/type/fingerprint stay full path; split pinned by a test in ONE emission); f771805e native report signatures REVISING the phase-3 seam-replication decision (dual *_reported seams “not fit for a codebase this big”: ~35 flat twins deleted, bare MandoReport<C>(Report<C>) newtype in mando-core with Deref / as_report / into_inner / delegating Display+Debug / std::error::Error source-None / plain From<Report<C>> ? auto-lift + MandoResult<T,E> alias; adapters ONE fn per op returning MandoResult; ~50 consumer edits flattening at tower edges; ems towers keep Report<VolueEmsError> internally with into_inner at 5 sites; 5 dead conversion fns deleted; the walls stand, the seam mechanics are gone). Gates 580/0 workspace lib tests + clippy clean. NEXT MR spec’d (docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md): #[derive(MandoReport)] From-impl generation (#[report(from(Src))] display-mapping default, #[report(wraps(Inner, message = "..."))] chain levels, MandoError marker bounds at boundaries + clippy ban on Report::new); spike (scratchpad/spike-trackcaller, rustc 1.89.0, error-stack 0.8.0) proved direct From on Report<LocalEnum> orphan-blocked (E0117, naive design dead) while #[track_caller] through the local newtype propagates ?-site locations EXACTLY (no ergonomics-for-locations sacrifice needed, despite Andras offering it); grounding pass caught 2 design regressions pre-implementation (from(Src) must display-map or source fields kill Clone/flatten; wraps() needs an explicit static message for struct variants).
  • BE-3482 Datadog Logs and APM Conformance - BE-3482 status note (2026-07-13): core conformance MERGED 2026-07-08 (MR !558 squash ae6d1098); the unpushed continuation adds the fingerprint ACTIVITY component (error.fingerprint = code|path|activity via ErrorWithStepStatus.activity: Option<String> + consuming with_activity(), all constructors default None so behavior is bit-identical when absent; reference wiring in mando-lib/src/service/volue/ems/ts_data_send.rs, every per-data-group error carries the data group name). Orphan commit 478cd993 (adds dd_formatter::app_version(), removes 3 panicking unwraps in mando-bess, fixes the known local mando-bess panic) was salvaged same day: cherry-picked onto the fingerprint branch as bc3727ba (Andras’s call, instead of a separate MR); the branch has 2 commits, gates green on both (clippy exit 0, 387 passed / 0 failed across 22 suites), pushed 2026-07-13 and open as MR !569 awaiting review; the old worktree + local feature/BE-3482 are purely historical and archivable. Known limit: at-site-logged variants (VolueEmsError::ApiError) reach the boundary as WARN with no fingerprint until the single-emission migration (own future ticket). UPDATE 2026-07-22: !569 MERGED (merge 53912a70); the single-emission migration MERGED as !571 (BE-3541 Single Error Emission), closing the limit; !570 (pymando) + wheel also landed.
  • BE-3541 Single Error Emission - MERGED via MR !571 (squash d1cf9c75 “refactor: emit step errors once by removing the logged-at-site path”, merge 1c4c33a5): one ERROR emission per step failure at the step boundary with HttpContext carried on the error; ErrorWithStepStatus.logged_at_site + logged() REMOVED - construct with new/message, never log at the detection site; unlocks fingerprint + http.* for previously at-site-logged variants.
  • mando-cli-datadog-local-logs-2026-07-22 - mando up --datadog verifies BE-3482’s dd_formatter error/log JSON (error.code/error.fingerprint) in the real Datadog EU UI from a local run. Confirms the body ddtags service tag overrides infra enrichment while dd_formatter ddtags carries NO env (agent DD_ENV authoritative). Filter env:dev-local-<user> → service:bess-os-service-mando.

Infrastructure / AWS

  • AWS CLI Access via Identity Center - how AWS CLI auth works since 2026-07-21: [sso-session alpiq-sso] in ~/.aws/config (IdC start URL ssoins-69878836cb6e09c7, eu-central-1) + four Developer-role profiles bessos-{dev,test,int,prod} mapping to accounts 794038257734 / 071128452852 / 621553445748 / 282467977019 (same set as the section 14.1 deploy matrix); one aws sso login --sso-session alpiq-sso covers all four; saml2aws retired but left on disk. Landmine: the RTK hook mangles aws output, always rtk proxy aws .... Used for ECR/ECS access during mando deployments.
  • BE-1597 DuckDB UI Exposure Research — reusable mando AWS-infra facts gathered for BE-1597: mando ALB aws_lb.bess_os_service is internal=true, 443-only, single target group on port 8080 (bess_os_load_balancer.tf); bastion (bastion.tf) t3.medium in a private subnet, no public IP, SSM Session Manager only, egress-all + ZERO ingress; mando runs on ECS Fargate desired_count=1 with enable_execute_command = environment=="dev" (bess_os_ecs.tf:499), and ECS Exec is interactive-only — no port forwarding (SSM AWS-StartPortForwardingSessionToRemoteHost via the bastion is the actual port-forward path); the bess_os_ecs SG (security_group.tf:4-23) uses INLINE ingress blocks — a standalone aws_vpc_security_group_ingress_rule is silently deleted on the next apply; must use a dynamic "ingress" block inside the existing resource; NO NAT gateway / VPC endpoints defined (the VPC is a base-vpc tag data lookup owned by another team) → Fargate egress to duckdb.org UNVERIFIED; int + prod both map to account group int_prod (main.tf:5-6).
  • BE-1597 DuckDB UI — Design — the approved terraform/infra shape for BE-1597: open port 4214 to the DuckDB UI via a dynamic "ingress" block INSIDE the bess_os_ecs SG referencing the bastion SG (not the VPC CIDR), gated var.environment == "dev" — a standalone aws_vpc_security_group_ingress_rule would be silently reaped because that SG uses inline ingress blocks. No ALB listener / target group / health check (mando ALB already internal=true); the bastion SSM hop (AWS-StartPortForwardingSessionToRemoteHost) is required because ECS Exec is interactive-only. Blocking pre-check: Fargate egress to extensions/ui.duckdb.org is UNVERIFIED (no NAT / VPC endpoints; base-vpc owned by another team) — if blocked, Phase 1 can’t ship and no terraform fixes it. desired_count=1 + Fargate re-IPs each deploy, so the duckdb-ui.sh helper re-resolves the task IP every run.
  • Mando Deployment Ceremony - the environment matrix: 4 envs map to 2 terraform component groups in main.tf (dev+test → dev_test, int+prod → int_prod); per-group ECR registries (843164609896…/poc/mando/deploy for dev_test, 748634852998…/poc/mando for int_prod); int_prod naming trap reconfirmed (historically INT, not real prod). Observed pins on rc/1.11.0: dev_test 1.11.0-dev.2555942905.50575825, int_prod 1.10.0-2533146896.b06b6b59. Expected plan diff on a version bump: the mando container image in the bess-os ECS task def ONLY.
  • Dev Deploy 2026-07-14 (error telemetry) - terraform plan-reading lesson: a version bump producing 17 add / 19 change / 5 destroy was IaC develop drift catch-up (12 genuinely new resources: the bess_os_fr incoming_sync lambda stack + 2 RDS analytics SG rules) plus immutable task-definition replacement mechanics (all destroys were “must be replaced” task defs), NOT danger; the second apply after the migration fix was down to 4 added / 19 changed / 4 destroyed.
  • fr-region-missing-datadog-logs-2026-07-21 - the FR deployment shape: dev-only module "bess_os_fr" (terraform/bess_os_fr.tf, count = var.environment == "dev" ? 1 : 0) instantiating shared terraform/modules/bess_os with region_code = "fr" in account 794038257734/eu-central-1; log shipping is awslogs → CloudWatch → subscription filter → datadog-forwarder Lambda (dd-logs-forwarder module from bit-tf-modules v3.21.4), the DD agent sidecar does metrics/APM only. Landmine: modules/bess_os lacks the forwarder wiring, so every regional instantiation ships log groups with no Datadog forwarding.
  • mando-cli-datadog-local-logs-2026-07-22 - DD_API_KEY acquisition route (2026-07-23): the deployed Datadog API key lives in AWS Secrets Manager, dev account, secret DdApiKeySecret-pkeeEykkaqu3 (referenced from optimization-universe-iac terraform/secret.tf via CI var DATADOG_API_KEY_SECRET_NAME); fetch with aws sso login --profile bessos-dev then aws secretsmanager get-secret-value --secret-id DdApiKeySecret-pkeeEykkaqu3 --query SecretString --output text (eu-central-1). Needed because the Alpiq Standard Operations Datadog role lacks “API Keys Read” (org key page blocked) and Personal Settings ddpat_/ddapp_ tokens do NOT work as DD_API_KEY (real API keys are plain 32-hex; app keys are 40-hex/ddapp_ and pair with an API key for queries only). Follow-up: request the permission or a personal org key from the Datadog admin.

Licensing

  • BE-1597 DuckDB UI Exposure Research — DuckDB UI frontend = proprietary MotherDuck code, blocks production use. DuckDB core, the prebuilt libduckdb binaries, duckdb-rs, libduckdb-sys, the ui extension SOURCE (github.com/duckdb/duckdb-ui), quack, ICU (Unicode-3.0), yyjson are all MIT/permissive (IP: Stichting DuckDB Foundation) — safe to vendor closed-source with attribution/NOTICE. BUT the DuckDB UI frontend assets served from ui.duckdb.org are proprietary, source unpublished (DuckDB’s own 2025-03-12 launch post says so); MotherDuck said Mar 2025 they were “reviewing licensing options” and as of Jul 2026 there is still no published license / terms of use. Served with no auth and (per DuckDB docs) “can access the data you load into DuckDB” → fine on a dev laptop / dev container, NOT acceptable for the int_prod trading service. (The Quack remote-protocol extension, by contrast, is a core DuckDB-signed MIT extension — no such issue.) Cf. the GPL-avoidance decision in remotefs-smb to smb migration.
  • BE-1597 DuckDB UI — Design — the licensing finding turned into an approved decision: because the DuckDB UI frontend is proprietary MotherDuck JS served live from ui.duckdb.org with no auth (source unpublished, still no license as of Jul 2026), the spec forbids running the UI in int/prod and ships it dev-only (Phase 1) / on the developer’s own laptop via Quack (Phase 2, where MotherDuck’s JS is already trusted). Everything else in the supply chain (DuckDB core / libduckdb / duckdb-rs / libduckdb-sys / the ui-ext SOURCE / quack / ICU / yyjson) is MIT/permissive and safe to vendor closed-source with a NOTICE bundle. Evidence: License finding (the important one).

mando-cli

  • mando-cli-v2 — current architecture (AppContext, CommandDelegate, 13 commands).
  • mando-cli-docker-lifecycle — DockerClient + ContainerSpec, bollard 0.19 wiring.
  • mando-cli-mock-command-2026-04-28 — mando mock subcommands and WireMock infra.
  • mando-cli-mock-down-idempotent-2026-05-06 — mando mock down + WireMock healthcheck endpoint fix; canonical 7-step lifecycle pattern; idempotent-teardown rule (404 = success).
  • mando-cli-status-readonly-2026-05-06 — mando status made read-only and bounded (connect_readonly, table_exists, statement_timeout = '2s'); status-must-be-pure-reads rule.
  • mando-cli-build-context-filter-2026-05-06 — yaml-driven context_includes and build_filtered_tar to stop COPY-everything build-context hangs.
  • mando-cli-build-variants-shelved-2026-05-06 — SHELVED design for profile-driven build variants (dev vs release Dockerfile, pre_command, 3-field profile syntax).
  • mando-cli-wsl-linux-build — how to build a Linux x86_64 / WSL release binary locally from an Apple Silicon Mac (musl static target, Docker --platform linux/amd64, optional-query path-dep gotcha).
  • mando-cli-v0.4.0-compose-bugs-triage-2026-05-26 — v0.4.0 compose-runtime triage: two confirmed bugs in the post-bollard rewrite; context: . relative-path resolution + bare-slug positional arg activating a malformed render_override stub. Documents the load-bearing profiles: ["{run_tag}"] invariant.
  • mando-cli-v0.4.0-mando-bess-binary-missing-2026-06-25 — design gap surfaced after the triage fixes (commit 2233959): the default profile builds mando from mando/Dockerfile, a thin runtime image that COPYs a pre-compiled target/release/mando_bess, but no builtin profile maps mando to pull/artifact and nothing in up runs cargo build --release (--cargo is artifact-only). Fresh mando up aborts the compose build. Workaround mando up --mando=pull; durable fix (default → pull + missing-binary preflight in up) pending user decision. Reported by gabi (WSL).
  • mando-cli-v0.4.0-piped-output-invisible-failures-2026-06-25 — mando up failures are invisible when output is piped/redirected (tester does 2>&1 | tee log): the user gets only a bare failed (exit 1) and an empty .mando/compose-up.log. Two fixes in the compose runner + buffer: (A) --progress plain on the docker compose call (compose.rs ~L231) so compose stops writing progress/errors to /dev/tty; (B) eprintln! fallback in CliBuffer::add_log_line (opts.rs:57) when stderr isn’t a TTY, because indicatif::MultiProgress::println silently no-ops there. Validated macOS (35→138 lines). Also clarifies mando-cli’s “registry auth” (GitLab API creds) ≠ docker login (container registry) — the now-visible tester failure is a registry pull access denied. Built+validated, not committed.
  • mando-cli-simulator-runtime-2026-05-30 — second first-class Docker Compose stack mando-sim coexisting with the dev mando project; six runners generated from a data-driven SIMULATOR_RUNNERS list; own Postgres on host port 5433; RunProfile.compose_project/layers() routes commands; new src/runtime/service_stack.rs::select_profile shared resolver for logs/exec/volume; defines the §A orchestrator↔runner env contract that the (WIP) mando_simulator crate, simulator-runner image, and six sim repos must implement. Supersedes GitLab MR !2 (balint). Landed b244f2e on feature/simulator-runtime. Process lesson: use full cargo test, not --bin mando (skips tests/).
  • mando-cli-simulator-env-contract-2026-06-02 — follow-up to the simulator runtime: §A env contract realigned to real images (simulator:1.11.0-feat.2569583284.7d69a070 + runner shape f7aa42f); adds arbitrary-extras passthrough via filter_extras / MANAGED_SIM_ENV_KEYS; sim-postgres aliased to SIMULATOR_DATABASE_* so one .env override controls both. Landed c3f0af7 on feature/simulator-runtime (8 files, +440/−67, 813 tests).
  • mando-cli-datadog-local-logs-2026-07-22 - mando up --datadog local Datadog log shipping; 2026-07-23 end-to-end test of docs/datadog-guide.md (tip c2becf1): guide accurate, every claim verified. HAZARD: a stale pre-feature binary reporting the SAME 0.4.0 version silently swallows --datadog as a service arg and starts a real up; the guide’s flag-position warning cannot protect against it (suggested guide follow-up: a version-check note). Nit: the new error strings use em dash characters, against Andras’s conventions elsewhere.
  • mando-cli-as-forecast-removal-2026-07-24 - removed obsolete bess-os-algo-forecast-as (Project::AsForecast) via compiler-driven deletion (enum variant/slug/aliases/config default; builtin profiles now 4 services; README five repos/two algos). LESSON: an enum variant used as a persisted HashMap<Project,_> map key in .bessstate.json — deleting it made a stale "AsForecast" key fail deser and the unwrap_or_default fallback silently WIPE the whole projects map (all overrides/toggles). Fix: per-key tolerant filter_known_projects mirroring save()’s PascalCase keys via Project::deserialize (NOT from_alias), warn! per dropped key, fixture restored as regression guard. Rule: deleting an enum variant = audit every persisted format keyed by it before shipping.
  • mando-cli-json-output-agent-skill-2026-07-23 - mando --json <cmd> / MANDO_OUTPUT=json (flag wins, exact-match "json") renders all output as terse NDJSON on stdout (kv/step/err/data + ALWAYS-LAST {"t":"result","ok","code"} from main.rs’s single exit path); human mode byte-identical, spinners suppressed, logs/exec/compose chatter stay raw on stderr. One pure mapper src/ui/json.rs::element_to_event covers all 21 commands via the Element seam (format!-built for t-first order; payload keys alphabetical, preserve_order off). Agent skill .claude/skills/mando/SKILL.md (64 lines): grammar + minimal invocations + error→remedy + frugality (status before logs, trust exit codes, never parse human tables); mock -p goes BEFORE the subcommand. NO top-level warn event (warnings are step s:"warn"). Working tree, UNCOMMITTED 2026-07-23.

py-mando / Python Bindings

  • Mando AGENTS.md Master Guide — round-2 (2026-07-13) verified py-mando facts: py-mando-simulation split out of py-mando in commit 3559042d — py_mando.SimulationRunner is GONE (breaking change); the new package requires Python ≥3.11, ships ddtrace, no polars/pandas. Both packages run init() at import (JsonFormatter + rustls ring) and preload libduckdb (ctypes RTLD_GLOBAL / add_dll_directory); build.rs copies lib/libduckdb/1.4.2 into data/platlib. Two sanctioned async bridges: blocking py.allow_threads(|| pyo3_async_runtimes::tokio::get_runtime().block_on(…)) + pyo3_async_runtimes::tokio::future_into_py for the *_async awaitable variants. Errors: single PyMandoError via map_*_error fns (mando-lib/src/python/error.rs). Test posture: py-mando-simulation Rust side zero tests, 1 Python integration test; CI job “PyMando Simulation Linux Dev” (pytest without --nbval).
  • py-mando — crate overview (abi3-py38 stable ABI, algo params, DataFrame ops, algo runner HTTP server).
  • BE-3482 pymando Branch Review - 2026-07-13 read-only review of feature/BE-3482-pymando (13 commits, py-mando DD log/APM conformance): NEEDS REWORK BEFORE MR despite genuinely green gates (clippy clean; 366 passed / 0 failed / 35 ignored via rtk proxy; fmt_util proven feature-flag-clean by building mando_lib with and without --features python). Blockers: 13/13 scoped commit titles (banned per !322), 7 orphaned ContextVars in tracing.py, byte-identical duplicates of the fingerprint branch’s version-fallback fix. Landing order: fingerprint MR first, then rebase dropping the duplicates; sharpest option also drops the fmt_util extraction for a zero-conflict 9-commit MR. Rework EXECUTED same day: rebuilt from scratch as feature/BE-3482-pymando-v2 (worktree .worktrees/BE-3482-pymando-v2), 15 title-only no-scope commits off develop tip 51d2b516, zero cherry-pick conflicts, 10 files +374/-55; 9b0c4e70 kept, 86d7ffd8/34e749d5/d0986230 dropped (dropping the extraction cost zero extra code, level_to_status was still inline); all blockers/should-fixes done (build_ddtags public, bare-code fingerprint, caplog + genuine global restore); gates green (clippy 0, 434/0/35 cargo tests, rustfmt 0 branch hunks, pytest 120 passed / 1 skipped vs baseline 114); NOT pushed, no MR; v1 untouched as fallback. UPDATE 2026-07-22: pushed + MERGED as MR !570 (squash fde2425e “feat: pymando logging conformance”, merge 14dfb3ad); conformant wheel py-mando==1.16.1+dev.2691658359.14dfb3ad published to Nexus dev.
  • BE-3613 Algo Services py-mando Conformance - moving the algo services (bess-os-algo-optimization + bess-os-algo-forecast; forecast-as OUT of scope) onto the conformant wheel py-mando==1.16.1+dev.2691658359.14dfb3ad: pin bump + fork replacement + log_error adoption per repo; code COMPLETE 2026-07-22 on both feature/BE-3613 branches (bess-optimization bb0555e/d47212d/71fb146; bess-forecast-day-ahead 50231b6/927e1e5/e1d5ade + fork-test file deletions), UNPUSHED, blocked only on VPN/Nexus for poetry lock --no-update && poetry install && poetry run pytest.
  • BE-3657 error_stack Adoption - phase 3 (planned 2026-07-22 evening, execution gated on !580) plans a shared error_json_field_value fn in mando-core unifying the error.errors/error.details JSON parsing across mando-lib’s dd_formatter and BOTH python package log formatters (py-mando/src/log_formatter.rs, py-mando-simulation/src/log_formatter.rs); today only dd_formatter has that parse. UPDATE 2026-07-23: SHIPPED on feature/BE-3657-phase3 @ e0184bc3 (unmerged) - the shared hook unifies all three formatters; PyO3 also gained map_report_error forward plumbing, its wiring grep-proven unreachable this phase; py-mando/py-mando-simulation pytest deferred to CI (no local venv, Nexus VPN-gated).

Tracing / Observability

  • clippy-disallowed-macros-allow-placement-2026-06-22 — mando bans tracing::error! directly (clippy.toml disallowed-macros) so all error logging routes through the mando_core::error! wrapper; the wrapper suppresses the lint for its own expansion via a module-root #![allow(clippy::disallowed_macros)] in mando-core/src/error.rs (call-site allow is ineffective on clippy 1.95).
  • BE-3117 Per-Flow Error Context Store — FLOW_ERROR_CONTEXT task-local store created inside the spawned flow task (task-locals don’t cross tokio::spawn); drained after scope returns into ONE tracing::error! aggregate with flow.error.* flat facets + full flow.errors JSON. Hand-written Serialize redacts HTTP bodies when http_context.sensitive.
  • BE-2272 — DD log JSON flattening to root.
  • BE-1842 Datadog Observability — span-scope walking, tokio::spawn context-inheritance gotcha, dd_formatter field plumbing.
  • BE-3482 Datadog Logs and APM Conformance - Datadog logs/APM conformance state + ranked open deferrals: single-emission migration (biggest win, unlocks fingerprint + http.* on at-site errors) > error.*/http.* on trace spans > verify dd.trace_id correlation in DD > flow.step.timeout_ms. The fingerprint ACTIVITY component (pushed 2026-07-13, open as MR !569; MERGED by 2026-07-15 as 53912a70) makes DA vs AS schedule failures in the auction scheduling step group SEPARATELY in Datadog (previously ONE error group). Authoritative fingerprint standard: architecture-design repo, logs-and-apm.md (error.code + flow.step.path + specific activity).
  • BE-3482 pymando Branch Review - review outcome for the py-mando DD conformance branch: solid work (flat flow.step.path, conformant logging.conf, log_error helper with root-cause unwinding, set_log_tags ddtags bridge into the Rust formatter) but needs rework before MR. Sharpest conformance finding: error.fingerprint serializes code|None when no step context is set (tracing.py:109), polluting DD fingerprint aggregation for log_error calls outside a trace()/run_with_trace scope. The same-day rework FIXED it (bare code, no "None", with a new test) and added a NEW flow.step.system span tag: the non-Mando counterparty taken straight from StepMetadata.system in mando-lib/src/workflow/mod.rs (the structured source flow.step.connection is formatted from), one line, no string parsing; runtime JSON conformance proven from the Rust stream during pytest (120 passed / 1 skipped).
  • Dev Deploy 2026-07-14 (error telemetry) - first LIVE validation of the BE-3482 telemetry on dev, observed from the deploy’s own crash-loop logs: error.code == error.kind == error.type (spec-verified: architecture-design logs-and-apm.md lines 87-104 explicitly mandate three fields with the same value for APM/logs/mobile), error.message = root cause, error.stack = source chain (wrapper-embeds-source Display pattern causes cosmetic text duplication), error.fingerprint absent as expected for non-step errors. Follow-up idea: emit a bare-code fingerprint from the Rust error! macro outside flow steps (parity with py-mando log_error), needs one sentence added to the standard.
  • fr-region-missing-datadog-logs-2026-07-21 - why FR services show APM/metrics but zero logs in Datadog: container logs travel CloudWatch → datadog-forwarder Lambda via subscription filters (NOT the agent sidecar), and the -fr log groups have no subscription filters because modules/bess_os/cloudwatch.tf never got the dd-logs-forwarder blocks.
  • BE-3657 error_stack Adoption - Datadog verification of the dev throwaway build 1.16.1-feat.2682437891.b7d15e2c (2026-07-16..22): nested error.details (4-level chain, real file:line), error.errors array queries (element/wildcard/negation), nested http.{request{method,url,version,body,content_length}, response{status_code,body,content_length}}, trace correlation confirmed (sample trace 0dc8d42739a08ac0805808b2d14b6644); startup mock deep error fires once per task boot (still the typed error! arm, not the report arm). PENDING DD cutover AT BE-3657 MERGE: create facets @error.fingerprint/@error.errors/@error.details.code; inventory monitors keyed on short codes, flat http.*, exact message “Failed running step” - all code/fingerprint values change format at that deploy. UPDATE 2026-07-23: !580 (phases 1+2) MERGED (merge 4c3442df), so the cutover checklist becomes actionable with the first develop-based dev deploy carrying it; the mock is superseded on the phase 3 branch by an env-gated report-backed port (MANDO_DEBUG_MOCK_ERROR, default off, unmerged). UPDATE 2026-07-23 evening: the short-code decision (commit a58df341, in review with MR !585) narrows that format break: error.code becomes the SHORT Enum::Variant form (close to the legacy shape, so @error.code-keyed monitor impact shrinks), while error.kind/error.type/error.fingerprint/error.errors/error.details codes stay full module path and DO still change format at that deploy. NEW cutover item: @error.source is RETIRED (renamed to error.message in the redesign, pinned by a mando-core regression test) - delete or rekey any facet/monitor still on it to @error.message/@error.errors (an empty error.source consumer already hit and fixed by Andras 2026-07-23). UPDATE 2026-07-24 (DX batch + live stream): error.details[].code also goes SHORT (commit 4e519269; errors/kind/type/fingerprint stay full path), so the @error.details.code facet keeps a readable short form at cutover. Local LIVE stream now operational: server on 8081 + a CONTINUOUS forwarder shipping error/warning lines to Datadog EU within ~5s under env:dev-local-levander (MANDO_ENVIRONMENT=dev-local-levander set at source, no forward-time ddtags rewrite); complete field contract verified live (code short, errors full, details codes short per level, fingerprint full, Caused-by stack, nested http blocks on 5xx paths). GOTCHAS: facet queries with :: in the value MUST be quoted (@error.code:"VolueEmsError::ApiError", unquoted silently matches nothing); a survived stale local server held port 8081 so a new launch died AddrInUse while old-format emissions kept flowing (diagnose with lsof on the port + the holder’s binary path).
  • mando-cli-datadog-local-logs-2026-07-22 - Datadog host-tag join latency (2026-07-23 guide test): env:dev-local-<user> rides DD_ENV HOST metadata, and the host-tag join takes up to ~10 min for a brand-new agent host (~8 min observed), so freshly shipped logs are initially queryable by service:/container_name: only and get the env tag retroactively; an env filter that looks broken right after agent start is just the join lag. Logs Search API api.datadoghq.eu/api/v2/logs/events/search works with the org API key + a Personal Settings ddapp_ application key.

Workflow / Flow Engine

  • be-1595-flow-repository-sqlite-bugs-2026-06-17 — Two latent FlowRepositorySqlite bugs (Sqlite dev/test only): create_flow lazy-query() INSERT that never persists, and started_at current_timestamp TEXT default unreadable as i64 in get_execution.
  • be-1595-flight-execution-id-parity-2026-06-17 — Execution-bound manual overrides resolved via FlowRepository.execution_id lookup, now shared between REST and Flight paths (apply_execution_overrides); MandoFlightService carries an Arc<FlowRepository> so the Flight handler can resolve execution overrides like REST.
  • BE-3117 Per-Flow Error Context Store — flow boundary in workflow/flow.rs wraps execution in FLOW_ERROR_CONTEXT.scope(...) + FlowCompletionGuard (Drop-flush for panic/cancel). Step-level recording in StepResult::log via ErrorRecord::from(&ErrorWithStepStatus), gated on !logged_at_site. The logged_at_site flag is the successor to logged_at_failure_site.
  • flow-step-log-message-dropped-2026-05-26 — Step-failure logging contract: ErrorWithStepStatus::log(status, message) is the “I haven’t logged yet, parent please log me” path (logged_at_failure_site: false), but the parent’s StepResult::log() drops message. Documents the constructor/Display/status_or_error/top-level-catch chain and the c945514e/4c543cb1 cross-check.
  • Mando AGENTS.md Master Guide — released flow versions are immutable: never change a released vN’s YAML or generated code; breaking behavior → v(N+1); exemplar manual_schedule v3. Definition YAML (mando-bess/config/flows/fi/*.yaml) → build.rs → generated Rust that is TRACKED in git (commit YAML + output together). Tip flows @ 2026-07-08: as_auction_update_v2, auction_v4, data_update_v2, intraday_v2, manual_schedule_v3. Jul-08 develop restructured src-side wiring from per-type src/flow/{type}/v{n}/ dirs to flow_registry.rs, backed by new crates mando-flow-step (FlowStepService trait: uniform async step construction from YAML) + mando-flow-step-derive (ParamEnum/ParamMeta derives). Round 2 (2026-07-13): FlowStepService is a construction trait, NOT execute()-style — type Params: ParamMeta; type Response; + async from_config(config: &str, providers: &StepProviders); StepProviders carries flow_repository/data_point_registry/6 auth providers/simulation_enabled; 36 type_entry! registrations in mando-bess/build.rs; param structs derive #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ParamMeta)]; config/flows/manifest.yaml is the deployment catalog (setup: {version}: flows: {name}: {path, schedule_env, semaphore_group} — scheduling env + concurrency groups in the manifest, not code; semaphores capacity 1/group); the flow-step crates ship with zero tests.
  • BE-3656 APM Span Enrichment - SpanEnrichmentLayer copies error.*/http.* off ERROR-level tracing events onto the active OTel span as Datadog APM tags; reads opentelemetry::Context::current().span() NOT tracing::Span::current() (empty inside Layer::on_event under scoped subscribers). MERGED via MR !578 2026-07-22 (squash 46a9be1b, merge 4a0d297f). Verified live on dev (3 commits, a0735ebb review-hardening: bodies off spans, first-error-wins guard, no MR yet). Dev-deploy prototype d17227ac validates the BE-3657 field shapes: full-path error.code, startup mock deep error, error.errors/error.details as real JSON arrays (targeted dd_formatter parse), flow.step.system from !570; DD array queries verified (element/wildcard/object-field/numeric/negation). Note: DD flattens array SPAN attrs to key.0/key.1 (breaks faceting) so arrays are log-side only, spans keep scalar error.code. Three tooling landmines added: Bash cwd resets to main checkout between calls, RTK garbles grep/sed (Read to ground-truth), grep "test result" | tail -1 cargo false green.
  • BE-3482 pymando Branch Review - the step info_span! in mando-lib/src/workflow/mod.rs gained a flow.step.system tag (2026-07-13 rework, commit 52b4d270 on feature/BE-3482-pymando-v2, unpushed): the non-Mando counterparty taken directly from StepMetadata.system (the same structured source connection() formats), no string parsing; verified all 14 DataFlow::Wrapper steps declare system: "Mando" and externals declare real names (Volue EMS x19, Metis x14, Position Manager x8); 4-case #[test_case] matrix covers Receive/Send/Wrapper semantics. Scope tension: one mando-lib commit on an otherwise py-mando-only branch, trivially splittable.

Graph View