Levandor

Alpiq BESS Activity Log

78 min read

For Agents

Reverse-chronological session log. Newest entries at top, grouped by date (## YYYY-MM-DD). Each bullet: one piece of work, short summary, wikilinks to docs touched. Updated by obsidian-documenter on every project doc write. Read by historian at bootstrap (top ~15 entries).

2026-07-24

  • BE-3657 error-authoring DX overhaul: 3-commit decision series (Andras), north star = error reporting must not break Rust’s natural flow, ? is the API (feature/BE-3657-phase3 now head f771805e, ~27 commits, pushed, riding MR !585). (1) f5beabd4 “refactor: adopt report vocabulary for error authoring”: new_coded/CodedExt/change_context_coded DELETED workspace-wide, replaced by ToReport::into_report (error-stack 0.8 already exports an IntoReport trait, hence ToReport; the method name stays unambiguous), Result::reported(), .wrap(), .attr(k,v) sugar, flatten_report for Clone enums (6 conversion fns became one-liners); 307 sites swept across 37 files; BONUS FIND: 20 map_err(new_coded) fn-pointer sites - the exact #[track_caller]-losing form - had degraded locations until this commit. (2) 4e519269 “feat: shorten error detail codes”: error.details[].code goes SHORT (details = the human panel, file:line carries precision); errors/kind/type/fingerprint stay full path; split pinned by a test asserting both forms in ONE emission. (3) f771805e “refactor: collapse adapter seams into native report signatures”: the dual *_reported seam pattern (Andras: “not fit for a codebase this big”) ELIMINATED, ~35 flat twins deleted; bare newtype MandoReport<C>(Report<C>) in mando-core (Deref, as_report, into_inner, delegating Display/Debug, std::error::Error with source None, plain From<Report<C>> enabling ? auto-lift, MandoResult<T,E> alias); adapters ONE fn per op returning MandoResult; ~50 consumer edits (flatten at tower edges); ems towers keep Error = Report<VolueEmsError> internally (into_inner at 5 sites); 5 dead conversion fns deleted per the branch’s own theme-2 precedent. REVISES phase-3 locked decision 2 (seam replication): the once-rejected native-signatures alternative adopted; the walled-towers exception stands but the seam mechanics are gone. Gates: 580/0 workspace lib tests, clippy clean. → Error-authoring DX overhaul (2026-07-24).
  • NEXT BE-3657 MR designed + spec’d: #[derive(MandoReport)] (docs/superpowers/specs/2026-07-24-be3657-report-derive-design.md, untracked): derive-generated From impls so error handling becomes plain ? - #[report(from(SrcType))] display-mapping default (preserves the Clone/message architecture), #[report(wraps(Inner, message = "..."))] chain levels, MandoError marker trait bounds at boundaries + clippy ban on Report::new. Spike (scratchpad/spike-trackcaller, rustc 1.89.0, error-stack 0.8.0) PROVED: direct From impls on Report<LocalEnum> are orphan-blocked (E0117, the naive design is dead), but through the local newtype #[track_caller] propagates ?-site locations EXACTLY (without the attribute they degrade to the from body) - Andras had offered to sacrifice site tracking for ergonomics, no sacrifice needed. Spec grounding pass caught two design regressions pre-implementation: from(Src) must display-map (source fields would kill Clone/flatten), wraps() needs an explicit static message for struct variants. → Error-authoring DX overhaul (2026-07-24).
  • BE-3657 local LIVE error stream operational + 2 Datadog operational gotchas: server on 8081 (build carrying the DX batch) with a CONTINUOUS forwarder shipping error/warning lines to Datadog EU within ~5s under env:dev-local-levander (MANDO_ENVIRONMENT=dev-local-levander now set AT SOURCE - no forward-time ddtags rewrite); complete field contract verified live: code short, errors full, details codes short per level, fingerprint full, Caused-by stack, nested http blocks on 5xx paths. GOTCHAS: Datadog facet queries with :: in the value MUST be quoted (@error.code:"VolueEmsError::ApiError"; unquoted silently matches nothing); the 2026-07-23 session’s survived server instance held port 8081 so the new launch died AddrInUse while the old build’s old-format emissions kept flowing - diagnose with lsof on the port + the holder’s binary path. → Error-authoring DX overhaul (2026-07-24), mando-local-host-run-recipe-2026-07-23.
  • mando-cli: bess-os-algo-forecast-as (Project::AsForecast) REMOVED as obsolete (Andras’s call; working tree, uncommitted). Compiler-driven: enum variant + slug/aliases + config default YAML deleted; builtin profiles (mocked-algos/full/fast-dev) now 4 services; e2e + docs truth pass (README five repos/two algos, skill table, datadog guide). LESSON: removing an enum variant used as a persisted-state map key broke .bessstate.json backcompat — a stale "AsForecast" key fails the derived HashMap<Project,_> deser and the unwrap_or_default fallback silently WIPES the whole projects map; caught because the first implementer removed the e2e fixture “so it would deserialize” (the dodge was the tell). Fix: per-key tolerant filter_known_projects mirroring save()’s PascalCase keys via Project::deserialize (NOT from_alias), warn! per dropped key, red-green tested, fixture restored as regression guard. Verified: dead-slug user profiles fail cleanly, leftover dirs ignored. Gate: 403 lib + 8 integration, clippy 0. → mando-cli-as-forecast-removal-2026-07-24.
  • mando-cli --datadog local log shipping ACCEPTED — Andras confirmed acceptance after a live run; logs land in Datadog EU as designed. Note’s Acceptance section moved pending→confirmed. → Acceptance.
  • NEW DECISION (Andras): error.stack switches to a custom “Caused by” chain renderer (REVISES the 2026-07-15 locked decision 3 “crate’s own tree, codes as sub-lines, accepted trade-off”; the once-rejected prefix style is now adopted after seeing the tree in the real DD UI): Datadog collapses newlines in several views and the error-stack crate’s ASCII connectors (|-, lone |, |->) turned into unreadable pipe soup. New per-level format: {short code}: {message} head (first level bare, subsequent prefixed Caused by: ), at file:line:column location line, with key = value per ErrorAttr; opaque attachments no longer mentioned. Implemented as commit 1f9cd611 on feature/BE-3657-phase3 (new MR !585 head, 23 commits, pushed + ls-remote verified): render_stack_tree rewritten to walk frames like report_details; crate debug-hook plumbing (install_debug_hook, Charset/ColorMode, HOOK_INIT) deleted as dead; 579/0 workspace lib tests incl. an exact multiline golden pin; verified LIVE locally and forwarded to Datadog (env:dev-local-levander 2026-07-24 events carry the new stacks AND the short error.code together). Datadog research backing it (worth keeping): Error Tracking for logs needs error status + service + (error.kind OR a valid error.stack) and mando always emits error.kind, so stack format never gates tracking/grouping; DD’s “valid stack trace” bar wants at least two lines with one meaningful frame carrying a FUNCTION NAME + filename, and error_stack Locations have no function names, so mando stacks can never be frame-parsed regardless of format; the source tag auto-parses only conventional language stacks; log remappers relocate attributes but do not parse. Conclusion recorded: mando’s error.stack is purely presentational - optimize for human readability and graceful newline-collapse (which the Caused by: separators survive). → error.stack Caused-by renderer decision (2026-07-24).
  • Standing “bess-am migrations MR” dev-DB crash-loop hazard RESOLVED with no action needed (tracked since the 2026-07-22 error-handling handover; verified 2026-07-24). The throwaway dev-deploy commit 6bc418cd had added exactly two migrations relative to its develop merge-base (2026_Q2/U202605131400__create_event_table.sql + 2026_Q2/U202605141900__create_kinesis_checkpoint_table.sql); both are now on origin/develop, brought in by 3ebcc897 “feat: add bess-am POC” (feature/BE-2262-bess-am-poc, merge 6fae64a5, 2026-07-23). mando-bess/database/ on 6bc418cd and origin/develop are byte-identical (git ls-tree diff), so develop-built images now carry a superset (identical set) of what dev’s refinery_schema_history has applied and the refinery “missing from the filesystem” crash-loop for develop-based dev deploys is gone. The three standing-warning callouts (Dev Deploy, BESS AM, Mando Deployment Ceremony notes) marked resolved; repo handover docs/superpowers/HANDOVER-error-handling-2026-07-24.md item 4 already updated. Same session also verified MR !585 (BE-3657 phase 3, head f771805e) still open, zero comments, has_conflicts false against develop even after the BE-3685 merge daf6b6df, head pipeline success, only gate = not_approved. → Dev Deploy 2026-07-14 (error telemetry), BESS AM (BE-2262) - mando-bess-am, Mando Deployment Ceremony, BE-3657 error_stack Adoption.

2026-07-23

  • mando-cli JSON output + agent skill VALIDATED (2-judge pass → PASS): live-behavior judge ran the debug binary with side-effect-free commands and confirmed 6/6 observable contracts (per-line NDJSON, result always-last on success AND error paths, exit code == result.code, MANDO_OUTPUT exact-match — "JSON" falls through to human mode, human byte-behavior unchanged, stdout purity under stderr redirect, --json position-independent/global=true); fix-closure judge confirmed all 4 inline fixes CLOSED with line evidence + clean standards census (0 comments, 0 non-test unwraps, SKILL.md 64 lines), gate green incl. release-profile 0 warnings and query unchanged at 2 pre-existing errors. Report .agents/council/2026-07-23-vibe-json-output.md; awaiting user commit. → Validation.
  • (evening) BE-3657 phase 3 IN REVIEW: MR !585 OPENED - “feat: adopt error stack across all adapters”, description Closes BE-3657 (GitLab default), feature/BE-3657-phase3 → develop at head a58df341 (22 commits), no conflicts, branch pipeline green on the same sha; reviewers krisztian.fekete1 / gabor.nagy6 / balint.budavoelgyi / jozsef.nagy1. Head commit = the short error.code implementation, so it rides the review. → MR !585 opened (2026-07-23 evening).
  • mando-cli JSON output mode + agent skill (working tree, UNCOMMITTED) - mando --json <cmd> / MANDO_OUTPUT=json (flag wins, exact-match "json") switches all Element-rendered output to terse NDJSON on stdout (kv/step/err/data events + an ALWAYS-LAST {"t":"result","ok","code"} from main.rs’s single exit path); human mode byte-identical, spinners suppressed, child/compose chatter + logs/exec stay on stderr. One pure mapper (src/ui/json.rs element_to_event) covers all 21 commands via the Element seam, format!-built for t-first key order (payload keys alphabetical, preserve_order off). Agent skill at .claude/skills/mando/SKILL.md (64 lines, token-minimal: grammar + minimal invocations + error→remedy map + frugality rules). Review caught a fabricated port-conflict remedy sourced from a TEST fixture (replaced with real workspace-lock contention row), 3 wrong subcommand tables (migrate/profile/volume), an emitterless {"t":"warn"} grammar (removed — warnings are step s:"warn"), and mock -p position (before subcommand). Gate: 401 lib + 8 integration tests, clippy 0, release warnings 0, query feature unchanged. → mando-cli-json-output-agent-skill-2026-07-23.
  • (evening) NEW DECISION (Andras): error.code goes SHORT, everything else stays full path - error.code now carries Enum::Variant (e.g. VolueEmsError::AuthenticationError; bare struct name when variantless) per the team spec’s line-100 mobile-readability intent; error.kind/error.type/error.errors/error.details codes/error.fingerprint ALL stay full module path. Implemented as commit a58df341 (the MR !585 head): short_error_code resolver in mando-core (PascalCase-segment heuristic, valid due to house naming), applied at the 4 macro emission sites only (span/boundary layers inherit as passthrough), split pinned by test kind_type_fingerprint_stay_full_while_code_goes_short, 577/0 workspace lib tests; verified LIVE in the local run and forwarded to Datadog (env:dev-local-levander). Consequences: the “mobile readability rationale is dead” pending team sentence WITHDRAWN (conformance restored); DD cutover impact for @error.code-keyed assets shrinks (short form resembles legacy codes); @error.source confirmed RETIRED (renamed to error.message in the redesign, pinned by a mando-core regression test) - any facet/monitor still keyed on it must be deleted or rekeyed to @error.message/@error.errors (Andras hit exactly this today: an empty error.source consumer, since fixed by him). → Short error.code decision (2026-07-23).
  • MR !580 (BE-3657 phases 1+2) MERGED (squash de7cd4c6, merge 4c3442df, ZERO review comments). The pending DD cutover checklist (facets @error.fingerprint/@error.errors/@error.details.code + monitor inventory for short codes / flat http.* / exact message “Failed running step”) becomes actionable with the first develop-based dev deploy carrying it. → BE-3657 error_stack Adoption.
  • BE-3657 phase 3 EXECUTED subagent-driven (2026-07-22 evening through 2026-07-23) on feature/BE-3657-phase3: 14 tasks + a final whole-branch review, every task under an adversarial spec + quality review; initially stacked on the in-review !580 head ab8cfd11, later rebase --onto develop (zero conflicts, patch-id byte-identical); 20 commits, final HEAD e0184bc3, PUSHED (ls-remote verified), CI running, NO MR yet (awaits Andras’s explicit yes). Delivered: mando-core ErrorAttr + per-level error.details attributes + error_attr_value one-level nesting + report_first_attachment outermost-wins pick + with_report http roll-up; ALL in-scope adapter groups migrated with *_reported seams and exhaustive no-wildcard *_error_from_report conversions; full tuple/transparent elimination (final sweep: zero error(transparent) in mando-lib/src/adapter, archiver included); auth providers first under the hard no-bodies/no-tokens rule with sentinel leak tests everywhere; PyO3 map_report_error forward plumbing (wiring grep-proven unreachable this phase); shared error_json_field_value hook unifying dd_formatter + both python package log formatters; env-gated report-backed mock (MANDO_DEBUG_MOCK_ERROR, default off) replacing the dev-deploy branch’s hardcoded-JSON mock. KEY FINDING (formal spec exception): only volue/ems carries Reports to the step boundary in production (its towers now use Error = Report<VolueEmsError>); the other SEVEN adapter groups are WALLED (service towers fix flat error types + pub(crate) seam visibility makes cross-crate wiring structurally impossible), so their rich chains flatten at legacy wrappers until a services-phase tower restructure; every wall independently verified genuine during review. Fix waves on 3 tasks (dead-code deletions per review theme 2: opl dead order code, et3000 + position_manager dead variants); final fix wave de-transparented 4 remaining variants after the reviewer ruled line 157 absolute (transparent named-source singles duplicate the inner level’s message in error.details). Gates at e0184bc3: workspace clippy exit 0, 574 lib tests passed / 0 failed (--lib CI parity), transparent sweep zero, no Cargo.lock churn, 61 files +4390/-1482 vs develop; py-mando/py-mando-simulation pytest deferred to CI (no local venv, Nexus VPN-gated). Follow-up ticket bundle (final reviewer’s triage): auth-provider PRE-EXISTING info!/debug! body logs (real leak surface), ErrorAttr ids on non-success terminals only, py delegation tests CI-only, data_platform AWS env test panic-safety, opl clear_strategies missing portfolio attr, map_err closure-form drift; plus the standing flight_end_to_end one-liner + mando-core standalone uuid/serde fix. develop since moved (6fae64a5, merge of feature/BE-2262-bess-am-poc): branch 1 merge behind, pre-MR rebase possible when the MR gets the go. → BE-3657 error_stack Adoption, mando-ci-lib-only-test-gate-2026-07-22.
  • New CI-gate companion lesson: per-crate task gates miss consumer crates - BE-3657 phase 3 task gates compiled -p mando_lib only, so 2 tuple-syntax construction sites in the mando-simulator consumer crate stayed invisible until the final whole-branch review caught them as a Critical; workspace-wide cargo clippy --release --all-features (no -p) is the only gate that catches cross-crate fallout, so branch-level gates must always be workspace-wide. Recorded next to the --lib gate discovery. → mando-ci-lib-only-test-gate-2026-07-22, BE-3657 error_stack Adoption.
  • mando-cli --datadog guide tested end to end (VERDICT: accurate, every claim verified): docs/datadog-guide.md at tip c2becf1 exercised fully. Verified: all four fail-fast paths (DD_API_KEY unset/empty, unresolvable/unsanitizable username) with the exact documented messages and zero docker interaction; agent renders only with the flag; results row datadog / env dev-local-levander; ONLY mando-* containers tailed (DD_CONTAINER_INCLUDE_LOGS=name:^mando-.* / EXCLUDE .*, unrelated containers ignored); 103 logs shipped to the EU org; teardown holds at both levels (flag-off up removes the agent via --remove-orphans, down cleans everything). GUIDE GAP: env:dev-local-<user> rides DD_ENV HOST metadata and Datadog’s host-tag join takes up to ~10 min for a brand-new agent host (~8 min observed), so fresh logs are initially untagged (query by service:/container_name: only, tag lands retroactively); Verifying section should say “wait a few minutes”. HAZARD: a stale pre-feature ~/.local/bin binary reporting the SAME 0.4.0 version silently swallows --datadog as a service arg and starts a REAL up (the flag-position warning can’t protect; suggest a guide version-check note). Key acquisition: the Alpiq Standard Operations Datadog role LACKS “API Keys Read” (org key page blocked; Personal Settings ddpat_/ddapp_ tokens do NOT work as DD_API_KEY); working route = AWS Secrets Manager secret DdApiKeySecret-pkeeEykkaqu3 via profile bessos-dev (aws sso login + get-secret-value; source: optimization-universe-iac terraform/secret.tf / CI var DATADOG_API_KEY_SECRET_NAME); Logs Search API api.datadoghq.eu/api/v2/logs/events/search works with that key + a ddapp_ app key. Nit: new mando-cli error strings use em dash characters. → 2026-07-23 end-to-end guide test.
  • BE-3657 phase 3 telemetry VERIFIED LOCALLY end to end (real runtime, not tests): phase-3 branch binary (b394e0d4, rebase-descendant of e0184bc3, built in .worktrees/BE-3657-phase3) run as a HOST process against mando up --datadog -p infra (postgres + wiremock + local-dd-agent). Verified: boot mock (MANDO_DEBUG_MOCK_ERROR=true) 4-level report with error.errors (4 full-path codes) + error.details (real file:line + attributes, data_group on the top level) + error.stack box tree incl. the ErrorAttr sub-line and the opaque HttpContext attachment + codekindtype full path + root-cause message + trace correlation; Volue EMS spot single-level fingerprint code|step_path (data_update.load_battery_momentary_data, spot_data.rs:79, static-first boundary message); Volue EMS retrieve TWO-LEVEL auth chain in production (VolueEmsError::AuthenticationError + VolueEmsAuthenticationProviderError::FailedRequest, ems.rs:217 + ems_auth_provider.rs:155 - the final-review auth level confirmed live); Metis full-path error.code with NO error.errors (walled-tower flattening, mixed world as designed); dd_formatter real nested JSON via the shared parse hook. 12 error lines to Datadog EU via DIRECT INTAKE (env:dev-local-levander service:bess-os-service-mando; host processes invisible to the local dd agent’s mando-* filter). Server left RUNNING on localhost:8081 (health 200); mando down + pkill mando_bess to stop. → Local end-to-end verification (2026-07-23).
  • NEW: local mando HOST-PROCESS run recipe captured from the verification gauntlet (pre-existing friction, team-valuable): .cargo/config.toml.example is STALE (MANDO_FLOW_ACTIVE_VERSION dead; wants MANDO_SETUP_ACTIVE_VERSION=V1_4 + MANDO_SETUP_SCHEDULER_DISABLED, prefix MANDO_SETUP, config-crate style; FINGRID_API_KEY must be non-empty or init panics via unwrap “must contain at least one key”); postgres cert loader reads <cwd>/certs/eu-central-1-bundle.pem (workspace-root runs need a certs symlink to mando-bess/certs; one left untracked in .worktrees/BE-3657-phase3); infra pg creds bess_transactional_db/postgres/postgres @ localhost:5432; MIGRATION RACE between mando’s refinery (DuckDbPostgres mode) and the mando-cli migrations container applying the same set (idle-in-transaction lock pileup wedges boot, move_schema_objects blocked; recovery = pg_terminate_backend on the idle holder + drop public/data/bess_os cascade + single-writer relaunch; MANDO_MODE=DuckDb skips mando-side pg migrations but the stale container set lacks the setting table, so use DuckDbPostgres); MANDO_PORT=8081 (wiremock owns 8080), adapter hosts at localhost:8080 for hermetic 404s. → mando-local-host-run-recipe-2026-07-23 (new note).
  • Epilogue to the local e2e: the first Datadog forward silently VANISHED (202-accepted, never indexed) - Datadog’s JSON-message parsing PROMOTES the embedded ddtags from the log line itself; mando emitted env:local (from MANDO_ENVIRONMENT=local), overriding the intake envelope’s env:dev-local-levander, and the org does not index env:local events at all. Fix: rewrite the embedded ddtags to env:dev-local-levander before forwarding. DURABLE RECOMMENDATION recorded in both notes: local mando runs set MANDO_ENVIRONMENT=dev-local-<user> so emitted ddtags align with the local agent’s env tag (embedded ddtags win over host/env tags on JSON lines, so even a containerized local mando fights the agent’s tagging the same way). Isolation stake: the shared Alpiq org carries REAL env:prod telemetry under the same service:bess-os-service-mando, so the per-developer env tag is the ONLY wall from production views. → Epilogue (2026-07-23): the vanishing first forward, mando-cli-datadog-local-logs-2026-07-22.

2026-07-22

  • (evening) Full 4-judge council validated the whole uncommitted mando-cli tree → PASS after one fix round: judges = error-paths, spec-compliance/desired-output, DRY/SOLID, Alpiq-standards; report .agents/council/2026-07-22-vibe-mando-cli.md (repo). The two Important findings were emergent cross-block seams invisible to per-block reviews (process learning): (1) Datadog flag-off remove_file(...).ok() + existence-driven compose inclusion meant a failed delete silently restarts the agent on plain mando up — now non-NotFound delete errors abort with an actionable message; (2) remove_named_volumes rendered Fail rows but exited 0, violating the same session’s exit_on_failure convention — now Result<bool> threaded through volume clear + down --volumes. Also fixed: shared system/path::normalize (dedups preflight/init), runtime/paths.rs extracted from the build_args.rs grab-bag, wiremock fetch status guards, http.rs panic→fallback, install.rs 300s download cap, preflight skips COPY sources escaping build context, teardown warns. Standards verdict: 0 non-test unwraps, 0 unsafe, format-args clean, Datadog feature BE-3482-conformant (no mando-cli-specific standards doc exists — validated against org DD conformance + mando AGENTS.md). Final gate: clippy ZERO, 395 lib + 395 bin + 8 integration + smoke, 0 failures. User-only remaining: junk-file cleanup before commit, manual Datadog acceptance. → Validation pass (evening).
  • (afternoon) mando up --datadog local log-shipping feature landed in the working tree (uncommitted by design — user commits manually): new mando-cli flag that ships a local mando up run’s error/log telemetry to the real Datadog EU UI to verify BE-3482 Datadog Logs and APM Conformance’s dd_formatter JSON end-to-end (no local→Datadog logs path existed — dev ships via awslogs→CloudWatch→forwarder Lambda; ECS DD agent sidecar is metrics/APM only). Usage: export DD_API_KEY=<key>; mando up --datadog (flag before service args); logs tagged env:dev-local-<os-username> (sanitized lowercase), DD_SITE defaults datadoghq.eu, up prints a datadog / env dev-local-<user> status row; flag-off up or mando down stops shipping (up passes --remove-orphans so the agent is removed). Acceptance: Datadog Logs explorer env:dev-local-<you> → service:bess-os-service-mando with error.code/error.fingerprint (service tag from body ddtags overrides infra enrichment; dd_formatter ddtags carries NO env, so agent DD_ENV is authoritative). Mechanism: new template src/runtime/templates/infra/datadog.yml (agent:7, container_name local-dd-agent so the name:^mando-.* include filter can’t match the agent itself; APM/process-agent off; docker.sock + containers mounts) rendered to .infra/datadog.builtin.yaml only when --datadog; compose::assemble includes it existence-driven (optional, NOT in INFRA_BUILTIN_FILES to avoid missing-file warns); flag-off up deletes it. KEY GOTCHA: template MUST use ${DD_API_KEY:-} not ${DD_API_KEY:?} — compose interpolates on EVERY verb, so :? empirically hard-fails mando down/logs in any shell without the var while the agent file exists (verified live, docker compose 28.5.2, both directions); the up-side resolve_datadog_env precheck is the sole guard. Also extracted build_args::atomic_write_with_dir unifying 3 write sites (override_gen’s write_override became atomic as a side benefit). Spec docs/superpowers/specs/2026-07-22-datadog-local-logs-design.md, plan docs/superpowers/plans/2026-07-22-datadog-local-logs.md; triple review caught the :? footgun (Important, fixed + re-verified CLOSED); gate: build clean, clippy zero, 392 lib + 8 integration green. → mando-cli-datadog-local-logs-2026-07-22 (new note).
  • (afternoon) mando-cli audit fix Block 5 (features) landed in the working tree (uncommitted by design — user commits manually): made the two parsed-but-ignored mock up flags real, added a clean-clone preflight guard, recorded the default-profile decision, and corrected a false audit claim. mock up -p/--port <n> now injects WIREMOCK_PORT into the compose child env (template publishes ${WIREMOCK_PORT:-8080}:8080), warns via a compose port check if the container already runs on a different port, and targets all admin calls at the requested port. mock up --dir <path> now waits for WireMock readiness (30s cap), resets mappings, then loads sorted *.json stubs via admin API with per-file Pass/Fail rows and nonzero exit on any failure (bare mock up unchanged). NEW src/runtime/preflight.rs — Dockerfile COPY-source guard wired into up and build (runs after cargo prebuild so it can’t block its own remedy); on a clean clone mando up now fails fast with the exact fixes (mando build mando --mando=artifact --cargo or --mando=pull) instead of a confusing docker COPY error; handles comments (incl. inside line continuations), --from stages, wildcards, $vars, JSON-form COPY, best-effort skips unreadable files. Decision: default profile mando stays →build, NOT flipped to pull (pull.yml resolves registry-less ${MANDO_IMAGE:-mando:dev} to a bare Docker Hub ref and private ECR needs login — flipping would trade one clean-clone failure for another). Audit correction: the “mock reset does NOT clear the request log” claim was WRONG — WireMockBackend::reset DELETEs /requests too; mock-guide.md now truthful-to-code (also fixed mando down wiremock → mando down mando-wiremock). Triple review approved, 0 Critical/Important, 4 minors fixed (fetch_stub_count delegates to list_stubs; new WIREMOCK_CONTAINER_PORT const; down() uses WIREMOCK_SERVICE; comment-inside-continuation parsing + test). Gate: build clean, clippy ZERO, 385 lib + 8 integration + smoke, 0 failed. → Fix status.
  • (afternoon) mando-cli audit fix Block 4 (over-engineering cleanup) landed + session wrap (uncommitted by design — user commits manually): executed the ponytail cut list and cleared the known-red clippy baseline. Deps dropped: bollard, bytes, tar, log, comfy-table. Files deleted: runtime/docker_client.rs, workspace/git/git.rs, network/manager.rs, workspace/projects/{algo,dash,iac}.rs. bollard inspect replaced by docker inspect --format '{{json .}}' shell-out — status --detailed verified byte-identical by reviewer. Single-impl traits collapsed: MockBackend→concrete WireMockBackend, MigrationRunner→FlywayMigrationRunner, ProjectContext→inherent fn, JoinProject inlined. Dead per-project wrapper layer + 5 never-read MandoWorkspaceProject fields deleted; query-feature schema types cfg-gated. Clippy baseline 8 errors → ZERO under -D warnings. Session totals: 4 blocks, 20 audit findings fixed + 4 review-caught issues (incl. an ETXTBSY self-update hazard caught in review); whole tree 43 files changed +815/−998; final gate verified: build clean, clippy zero, 361 lib + 8 integration + smoke all green. Open decisions in repo ledger .superpowers/sdd/progress.md: broken pre-existing --features query build (flight.rs vs mando-core drift), unimplemented simulator-runtime spec, mock --dir/--port stub flags, default-profile clean-clone gap; remaining LOW sweep + doc mismatches unfixed. → Fix status.
  • (afternoon) mando-cli audit fix Block 3 (data safety) landed in the working tree (uncommitted by design — user commits manually): hardened data-safety paths. Bug #15 — flyway Postgres connections now built via one shared pg_config() using tokio_postgres::Config typed setters at all 3 sites (hostile spaces/quotes/backslashes can no longer break libpq keyword parsing). Bug #16 — apply() runs check_flyway before ensure_history_table, no longer mutating a Flyway-managed DB before bailing. Bug #34 — ensure_database only CREATEs on a genuinely empty result, real errors propagate. Bug #17 — git clone token moved off argv into GIT_CONFIG_COUNT/KEY_0/VALUE_0 env (not visible in ps; needs git ≥2.31). Bug #18 — self-update cross-device path stages into the destination dir + atomic rename, fixing both non-atomic overwrite of the running binary and ETXTBSY on Linux tmpfs (reviewer-caught Important on first impl, re-verified CLOSED). Bug #19 — new shared atomic_write (temp with pid+seq, rename) protects .bessstate.json and host-procs.json from torn writes. Bug #10 — workspace lock probe treats EPERM as alive (no stealing live locks from other users); ESRCH predicate shared via kill_stderr_is_esrch. Triple review approved. Gate: 362 lib tests green, clippy unchanged at 8 baseline errors (Block 4, in flight, will clear them). Parked: REPO_PYMANDO_TOKEN still a docker build-arg (build_args.rs:95, pre-existing); .bess-credentials.json write_restricted still non-atomic; locale-sensitive ESRCH match leaves stale locks uncleared under non-English LC_MESSAGES. → Fix status.
  • (afternoon) mando-cli audit fix Block 2 landed in the working tree (uncommitted by design — user commits manually): closed the --volumes/volume clear no-ops, the hardcoded stub-count port, the missing-timeout cluster, and both WireMock findings. Bug #7 — mando down <svc> --volumes now honored (named volumes discovered via docker inspect + removed, anonymous via compose rm -v). Bug #8 — volume clear reordered to compose rm -s -f before volume rm; container discovery for both down and clear now compose ps -aq via a shared service_container_ids helper (was ps -q running-only — the block’s one Important review finding, re-verified CLOSED). Bug #11 — status stub-count resolves WIREMOCK_PORT env (mirrors compose ${WIREMOCK_PORT:-8080}) with 2s/3s timeouts. Bugs 13 — all reqwest clients now carry timeouts via new system/http.rs client(connect,total) helper (wiremock/check 2s/5s, rest 5s/60s, install connect-only 10s to protect slow downloads). Bug #14 — wiremock.enable() renamed verify_service_stubs (never toggled state, no CLI caller; validation-only semantics made truthful vs speculative admin-API toggle). Bug #30 — wiremock.reset() now checks HTTP status on both admin calls. Triple review approved. Accepted minors: WIREMOCK_PORT env-file vs shell-var divergence, rest.rs 60s hard cap on previously-unbounded queries. Gate: 347 tests green, clippy unchanged at 8 known-baseline errors. → Fix status.
  • (afternoon) mando-cli audit fix Block 1 landed in the working tree (uncommitted by design — user commits manually): closed both HIGH findings + the three exit-0-on-failure MEDIUMs. Bug #1 mass-kill — host_process::stop() now rejects pid <= 1 before signaling, guard in the shared stop() so all callers covered (down.rs ×3, up.rs cleanup). Bug #2 orphan leak — up.rs wraps read+merge+record persistence in a result-capturing closure and SIGTERMs all just-spawned pids on ANY persistence failure before propagating. Bugs 5 — migrate/get/pull exit nonzero via a shared exit_on_failure(bool) helper in commands/mod.rs → CommandError::Exit(1). Triple review (quality/bugs/dup) approved, re-review closed residuals. Known-red clippy baseline: 8 pre-existing dead-code errors in schema.rs/backend.rs/project.rs deferred to the over-engineering block. Parked: host-procs.json non-atomic write; command-level exit-code wiring untested. → Fix status.
  • (evening) BE-3657 phase 3 PLANNED + approved by Andras (execution NOT started, gated on MR !580 merging): scope locked to “plumbing + adapters” (rejected: plumbing-only, full workspace breadth); mechanics = seam replication per the Volue pilot (retrieve_ts_data_reported/send_request_reported/volue_error_from_report in adapter/volue/ems/ts_data_retrieve.rs; rejected: native Report signatures, boundary-only wrapping); keeps riding BE-3657 (branch feature/BE-3657-phase3 off develop AFTER !580 merges, one phase = one MR; subagent-driven per house rule). Conformance pass against Balazs’s team spec (architecture-design doc/content/doc/ops/logs-and-apm.md @ main, fetched 2026-07-22) tightened the design: line 157 absolute (ALL tuple/transparent variants in migrated enums restructured); error.details gains the spec’s attributes field (lines 150-155) via a new mando-core ErrorAttr attachment + extraction landing BEFORE any adapter migrates; line 159 http roll-up = outermost-wins pick of the report’s HttpContext attachment with with_report filling http_context when unset (“force with a macro” wording read positionally → pending team-sentences list); auth provider migrations under a hard sensitive-data rule (lines 121-124: no bodies, no tokens, method/url/status only). Adapter order grounded on the branch: auth providers first (alpiq/authentication + volue ems/atp auth) → volue/ems completion (spot + send join the pilot) → volue/atp → alpiq/ebs (SMB, no http roll-up, share/path ErrorAttr) → alpiq mdr/opl/data_platform/et_3000 → fingrid → alpiq/metis → mando group (rest/algo/simulator/ms_teams/microsoft) → alpiq/position_manager. Two vetoable defaults flagged: adapter/mando/archiver (+ postgres submodule) DEFERRED to the repos phase (data-plane, separate CI harness, no step boundary); startup mock deep error PORTED to mainline mando-bess behind MANDO_DEBUG_MOCK_ERROR default-off (one IaC dev env line at deploy, droppable on team objection). New planned mando-core surface (task 1): ErrorAttr(String, serde_json::Value), error_attr_value, report_first_attachment, report_details entries gain attributes, shared error_json_field_value unifying the error.errors/error.details JSON parse across dd_formatter + py-mando/src/log_formatter.rs + py-mando-simulation/src/log_formatter.rs. Spec docs/superpowers/specs/2026-07-22-be3657-phase3-adapters-design.md, plan docs/superpowers/plans/2026-07-22-be3657-error-stack-phase3.md (15 tasks, untracked). → BE-3657 error_stack Adoption.
  • (afternoon) Full mando-cli codebase audit documented (~18.7k lines, four parallel passes, no fixes applied): 37 verified bugs (2 HIGH — unguarded kill -TERM -{pid} mass-kill risk with no pid<=1 guard in host_process.rs:108-133, and orphaned untracked host processes when host_process::record fails in up.rs:262-270; 18 MEDIUM, 17 LOW). Recurring themes: exit-code-0-on-failure (migrate/get/pull), missing reqwest timeouts everywhere, wiremock.enable() a silent no-op. Plus a −280-line / −5-dep over-engineering cut list (drop bollard/bytes/tar/log/comfy-table, collapse single-impl traits), 9 duplication clusters (biggest: compose-context prologue copy-pasted across 8 command files → compose::prepare), the simulator runtime spec being entirely unimplemented on main, and several mock-guide doc/code mismatches. Confirms the uncommitted piped-output fix (opts.rs/compose.rs) is correct but still lacks a commit + regression test. Source: .agents/research/2026-07-22-audit-mando-cli.md. → mando-cli-full-audit-2026-07-22 (new note), mando-cli-v0.4.0-compose-bugs-triage-2026-05-26, mando-cli-v0.4.0-piped-output-invisible-failures-2026-06-25, mando-cli-simulator-runtime-2026-05-30.
  • (afternoon) MR !578 MERGED; BE-3657 rebased onto the merge: Andras merged !578 (squash 46a9be1b “feat: enrich trace spans with error and http attributes”, merge 4a0d297f = new develop tip, 0 review comments). feature/BE-3657 then ran rebase --onto develop 4a0d297f: all 13 commits replayed zero-conflict, head 2baea68a became 22ccae93, patch-id verified byte-identical to the reviewed content; branch CI pipeline 2696324182 on 22ccae93 success. The BE-3657 MR now waits ONLY on Andras’s explicit yes. → BE-3656 APM Span Enrichment, BE-3657 error_stack Adoption.
  • (afternoon) DD_SERVICE env fix appended to BE-3657 (commit ab8cfd11 “fix: read dd service name from env”; Andras asked to sneak it into the upcoming MR): the hardcoded bess-os-service-mando in mando-lib replaced by a DD_SERVICE env read with the old value as fallback (also on empty string); resolve_dd_service() + OnceLock-cached dd_service() in mando-lib/src/app/mod.rs (app-gated), all 6 hardcode sites routed through it (OTel Resource get_tagging, dd_formatter.rs ddtags, 4 raw global::meter literals in service_base.rs x3 + util/http_client_trace.rs that bypassed the old const); inline test covers default/empty/override. Deployment-neutral: IaC bess_os_ecs.tf:218 ALREADY sets DD_SERVICE = "bess-os-service-mando" on the mando container, so this closes the mando-side root cause of the FR unsuffixed service name. Branch pushed, head ab8cfd11 ls-remote verified; clippy clean, 516 lib tests passed / 0 failed. → BE-3657 error_stack Adoption, fr-region-missing-datadog-logs-2026-07-21.
  • (afternoon) Discovered the CI test gate is --lib-only; integration tests/ targets rot silently (new note): .gitlab/scripts/test.sh runs cargo test --all-features --release --lib -- --test-threads=1, so */tests/ targets are NEVER compiled by the pipeline (the separate “Integration Test Linux Dev” job covers only the mando-lib archiver Postgres suite). Concrete rot: mando-bess/tests/flight_end_to_end.rs compile-broken on develop itself (E0603: DataPointUpdateInfo via mando_lib::service private use, since the mando-repository crate split b3ce27a8); local gates must use --lib for CI parity; pending one-liner fix (import from mando_core::model::datapoint) to be its own change on Andras’s yes, NOT part of BE-3657. AGENTS.md updated in 4 places + a new landmine row; mirror resynced. → mando-ci-lib-only-test-gate-2026-07-22 (new note), Mando CI-CD, Mando AGENTS.md Master Guide, Mando AGENTS Guide (mirror).
  • Error-handling merge train completed + vault caught up (covers 07-16..07-22): MR !569 merged (fingerprint activity, squash b3f39605, merge 53912a70, by 07-15), MR !570 merged (pymando-v2 DD conformance, squash fde2425e “feat: pymando logging conformance”, merge 14dfb3ad; conformant wheel py-mando==1.16.1+dev.2691658359.14dfb3ad published to Nexus dev), MR !571 merged (BE-3541 Single Error Emission: squash d1cf9c75 removes logged_at_site/logged(), one boundary emission with HttpContext on the error; develop tip 1c4c33a5); develop since moved to e2ff6b79 via the unrelated BE-3685 merge. BE-3482 deferral list re-ranked (1 merged, 2 in review as !578, 3 verified, only flow.step.timeout_ms open); Agent Context error-handling row corrected (constructor API changed). → BE-3541 Single Error Emission (new note), BE-3482 Datadog Logs and APM Conformance, BE-3482 pymando Branch Review.
  • MR !578 open for BE-3656; BE-3657 phases 1+2 built and pushed: !578 “feat: enrich trace spans with error and http attributes” on head 8f50ef2a (3 commits, zero-conflict rebase onto post-!571 develop 1c4c33a5), pipeline green, reviewers krisztian.fekete1 / gabor.nagy6 / balint.budavoelgyi / jozsef.nagy1, mergeable with 0 review comments as of 07-22. feature/BE-3657 @ 2baea68a (13 commits atop 8f50ef2a, pushed, rebased 07-21; 3 develop-inherited bare-code fallback tests aligned to full-path expectations, squashed into “feat: use full module path in error code”); gates green via the workspace build graph, standalone cargo build -p mando_core broken by a PRE-EXISTING BE-3643 uuid/serde issue; MR only after !578 merges + Andras yes, then rebase --onto develop. Decisions updated in the note: boundary template FLIPPED static-first "{static}: {outermost}[: {chain}]" (07-16), bodies never on spans (spec deviation, raise with team), error.errors/details excluded from span tags, first-error-wins guard, nested http.request/response; plus the pending DD cutover at BE-3657 merge (create facets @error.fingerprint/@error.errors/@error.details.code; inventory monitors keyed on short codes, flat http.*, exact message “Failed running step”). → BE-3656 APM Span Enrichment, BE-3657 error_stack Adoption.
  • Dev prototype verified in Datadog + BE-3613 code complete (VPN-blocked): dev runs throwaway 1.16.1-feat.2682437891.b7d15e2c (BE-3541 + BE-3656 + BE-3657 phases 1+2 + full-path codes + startup mock deep error + nested http + static-first message + flow.step.system); verified nested error.details (4-level chain, real file:line), error.errors array queries (element/wildcard/negation), nested http.{request,response} shape, trace correlation (sample 0dc8d42739a08ac0805808b2d14b6644); mock fires once per task boot, still on the typed error! arm with hardcoded JSON. BE-3613 (algo services onto the conformant wheel) code COMPLETE on bess-optimization (bb0555e/d47212d/71fb146) and bess-forecast-day-ahead (50231b6/927e1e5/e1d5ade + fork-test deletions), author fixed, branches UNPUSHED, blocked ONLY on VPN/Nexus for poetry lock --no-update && poetry install && poetry run pytest; catalog names bess-os-algo-optimization / bess-os-algo-forecast, forecast-as OUT of scope. → BE-3657 error_stack Adoption, BE-3613 Algo Services py-mando Conformance (new note).
  • FR logs now reach Datadog but with unsuffixed service name: mando hardcodes DD_SERVICE = "bess-os-service-mando" (mando-lib/src/app/mod.rs:38) into the per-line ddtags, which wins over the forwarder’s -fr log-group tag enrichment; all infra tags verified correct; fix belongs in mando (read DD_SERVICE env with constant fallback). → fr-region-missing-datadog-logs-2026-07-21.

2026-07-21

  • Implemented the FR Datadog log-forwarder fix + learned the IaC feature-branch dev-apply trick: seven dd-logs-forwarder blocks added to modules/bess_os/cloudwatch.tf on IaC branch bugfix/fr-datadog-log-forwarders (commit c169afc, asset_simulator block count-gated); forwarder module + Lambda policy verified before deploy; terraform_apply:dev is only: [develop], so a temporary CI commit (8592eeb) added the branch regex (drop before merge, mirrors mando AGENTS.md 14.3); watch for develop-based apply reverting the live FR 1.16.1-feat image + asset-simulator drift; bit-tf-modules source clones via ssh only. → fr-region-missing-datadog-logs-2026-07-21.
  • Root-caused why FR (France) region services have NO logs in Datadog (read-only AWS + IaC analysis, fix not applied): the dev-only module "bess_os_fr" (terraform/bess_os_fr.tf, region_code = "fr", account 794038257734) instantiates the extracted modules/bess_os module whose cloudwatch.tf creates the -fr log groups but carries ZERO dd-logs-forwarder blocks; the top-level terraform/cloudwatch.tf has one forwarder module per original log group, so the four original /aws/ecs/bess-os/eu-central-1/dev/* groups each have a subscription filter to the datadog-forwarder Lambda while the four -fr counterparts have none (FR services ARE writing to CloudWatch, last events ~27 min old). The forwarding step was dropped at module extraction. Fix direction: mirror the top-level dd-logs-forwarder instantiations inside modules/bess_os/cloudwatch.tf (sibling exemplar: modules/bess_am/ecs.tf Datadog Log Forwarding section). → fr-region-missing-datadog-logs-2026-07-21.
  • AWS CLI access switched to AWS Identity Center (SSO), replacing saml2aws (verified on Andras’s mac). ~/.aws/config now has [sso-session alpiq-sso] (start URL https://identitycenter.amazonaws.com/ssoins-69878836cb6e09c7, sso_region = eu-central-1, scopes sso:account:access) plus four Developer-role profiles in eu-central-1: bessos-dev 794038257734, bessos-test 071128452852, bessos-int 621553445748, bessos-prod 282467977019 (the same accounts as the AGENTS.md section 14.1 deployment matrix). One aws sso login --sso-session alpiq-sso browser PKCE login covers all four; verified with aws sts get-caller-identity --profile bessos-dev → assumed-role/AWSReservedSSO_Developer_.../andras.lederer@alpiq.com. Gotcha: the RTK shell hook mangles aws output (prints AWS: ? ?), so prefix every aws call with rtk proxy. Gotcha: to enumerate accounts/roles before profiles exist, take accessToken from the newest ~/.aws/sso/cache/*.json and use aws sso list-accounts / list-account-roles --access-token $TOKEN --region eu-central-1. eu-west-1 is a read-only IdC replica (a fallback [sso-session alpiq-sso-dub] could be added, not configured); ~/.saml2aws (AzureAD) left in place but unused. Practical use: dev/prod ECR + ECS access for mando deployments. → AWS CLI Access via Identity Center.

2026-07-16

  • Error-handling night: BE-3656 deployed + verified live, BE-3657 spec’d + phase-1-planned, dev prototype iterated to full-path codes + queryable arrays + flow.step.system. BE-3656 (APM span enrichment) is now DONE/reviewed/pushed with a third review-hardening commit a0735ebb (3 commits total off BE-3541 head dee24ad3; bodies excluded from spans as a deliberate spec deviation, first-error-wins marker guard since set_attribute appends under the 128-attr cap, u64 try_from, shared OTel test helpers); verified live in Datadog (error.* on errored step spans + Error Tracking pickup, http.* on GET/POST client child spans, log output byte-identical); STILL NO MR (needs Andras’s explicit yes). Dev runs throwaway build d17227ac (branch feature/BE-3482-dev-deploy, NEVER merge; IaC pin 1.16.1-feat.2680023002.d17227ac, applied 02:07Z): full-path error codes (error.code = full module path), a startup one-shot 4-level mock deep error fired each boot in a debug.mock_deep_error span (mando-bess/src/debug_error.rs), error.errors/error.details as real JSON arrays via a targeted dd_formatter record_str parse of exactly those two field names, and flow.step.system cherry-picked from !570 (7f3db1ed). Datadog array-attribute queries VERIFIED working (element match, wildcard, object-field, numeric, negation); facets still to create (Andras click): @error.errors, @error.details.code, @error.fingerprint. BE-3657 (full error_stack =0.8.0 adoption) LOCKED and spec’d: principle “leave what we can to error_stack” (mando adds only the ErrorCode full-path attachment via CodedExt at change_context/new sites + extraction fns + error!(report=...) arm); field contract error.code==kind==type==outermost full-path code, error.errors = codes array, error.details = [{code,file,line,message}]; crate’s own tree rendering with codes as attachment sub-lines; boundary message template "{outermost}: {static}: {chain}"; phase-2 annotation direction #[error_meta(event_type, resolution step, business_message, description)] → details enrichment + resolution-doc codegen (Andras’s own later work, phase 1 must not preclude). Branch feature/BE-3657 DOES NOT EXIST YET, start off BE-3656 head a0735ebb; spec/plan under docs/superpowers/{specs,plans}/2026-07-15-*error-stack*; status planned, awaiting go. Three new tooling landmines recorded: Bash cwd resets to the main checkout between calls (a worktree cherry-pick ran on the wrong branch, phantom conflicts), RTK proxy garbles grep/sed output (ground-truth with Read on absolute paths), grep "test result" | tail -1 is a cargo-test false green (shows only the last binary). → BE-3656 APM Span Enrichment, BE-3657 error_stack Adoption, BE-3482 Datadog Logs and APM Conformance.

2026-07-15

  • Implemented BE-3656 APM span enrichment (branch feature/BE-3656, 2 commits a64f9457 + 6d4f4395 stacked on BE-3541 head dee24ad3, pushed, NO MR yet), closing deferral item 2 of BE-3482 (error.*/http.* on trace spans). New SpanEnrichmentLayer in mando-lib/src/app/span_enrichment.rs copies error.*/http.* off ERROR-level tracing events onto the active OTel span as Datadog APM tags; instrumented_http_client records http.method/url/status_code/version + request/response content_length once per logical send on the final outcome (success/4xx/final failure); bodies only via error events per logs-and-apm.md “Only Error” column. Log output byte-identical (8/8 error-macro + 39/39 workflow capture tests); gates green (455 passed, only the 2 known TEST_PG_HOST archiver failures). Four reusable tracing/OTel Layer gotchas recorded: (1) tracing::Span::current() is EMPTY inside Layer::on_event under a with_default scoped subscriber (scoped-dispatch reentrancy guard), read opentelemetry::Context::current().span() instead (correct in both scoped tests and prod via tracing-opentelemetry 0.32 context_activation); (2) opentelemetry_sdk 0.31 in-memory exporter is ::trace::InMemorySpanExporter not ::testing::trace::; (3) tracing-opentelemetry 0.32 already sets Status::error("") on ERROR events, layers must not; (4) OtelData.state is pub(crate), only public write path is OpenTelemetrySpanExt::set_attribute (appends, DD keeps last, 128 attr limit). Spec + plan under repo docs/superpowers/{specs,plans}/2026-07-15-be3656-*. → BE-3656 APM Span Enrichment, BE-3482 Datadog Logs and APM Conformance.
  • Documented a wall-clock-dependent CI flaky test discovered while shepherding MR !570: scheduler::tests::firing::should_fire_inner_job_through_run (mando-bess/src/scheduler.rs:327, introduced by develop 3a8b8f84). The test adds an every-minute cron counter job to a LIVE started scheduler, calls SelfHealingJob::run(), sleeps 100ms, then assert_eq!(counter, 1); if the ~2s window crosses second :00 of a wall-clock minute the real scheduler ALSO fires the job, counter becomes 2, exact-equality assert panics (observed left:2 right:1 at 09:08:00.656, exactly a minute boundary; same SHA passed an hour earlier). Remedy: retried the failed CI job on pipeline 2678118132, passed; fix deliberately NOT in !570 (zero-unrelated-changes rule). Proper fix needs a follow-up ticket: either don’t add the counter job to the started scheduler, or relax to assert!(counter >= 1); secondary latent flake is the 100ms sleep being too short on a loaded runner (would fail with 0). → mando-known-flaky-tests-2026-07-15, Mando CI-CD.

2026-07-14

  • Hand-deployed the BE-3482 error telemetry to dev (throwaway-branch ceremony on BOTH repos), diagnosed a NEW dev crash-loop class, and got the first live telemetry validation. Goal: get the MR !569 content (fingerprint activity + version-fallback panic fix) flowing on dev while the MR awaits review; dev was nominally on pre-conformance 1.15.0-dev.2655042609.3024e20c (live task labels showed 1.15.0-feat.2664534358.627578d7 from an earlier hand-applied feature deploy). Mechanism: throwaway mando branch feature/BE-3482-dev-deploy (NEVER to be merged) = develop tip 51d2b516 + the two !569 cherry-picks + one CI commit (branch added to Publish Service Docker Dev only: AND the PyMando Win Dev needs/dependencies entry commented out; that job is absent on feature branches and a dangling need kills pipeline creation). IaC: branch feature/error-hdl off IaC develop; bumped components.dev_test.mando.version; second commit “ci: enable dev apply from error-hdl branch” adds the branch to terraform_apply:dev only: (apply normally exists ONLY on develop); both commits temporary, strip if ever merged. Crash-loop: first apply crash-looped mando with MigrationError::PatchApplicationFailed: migration V202605131400__create_event_table is missing from the filesystem; root cause: dev’s refinery_schema_history holds U202605131400__create_event_table + U202605141900__create_kinesis_checkpoint_table from the UNMERGED BE-2262 bess-am/kinesis POC branches (deployed to dev in May as a feat build), so any image lacking those files fails refinery validation at boot; the inverse of the BE-1595 drift case, merging develop cannot fix it. Fix: restored both files byte-exact (single canonical blob f45b6dd4 across all source commits, satisfies refinery checksums) as 6bc418cd “fix: restore bess-am migrations present in the dev database” → image 1.16.1-feat.2674882859.6bc418cd → re-bumped pin → Apply complete! (4 added / 19 changed / destroys 5 → 4). STANDING WARNING (open action): any develop-based image crash-loops on dev until the two bess-am migration files land on develop (small MR, recommended) or dev’s refinery history rows are cleaned. Telemetry validated from the crash itself: error.code == error.kind == error.type (spec-verified: architecture-design logs-and-apm.md lines 87-104 mandate three same-value fields for APM/logs/mobile), error.message = root cause, error.stack = source chain (wrapper-embeds-source Display pattern causes cosmetic text duplication), error.fingerprint absent as expected for non-step errors; follow-up idea recorded: emit a bare-code fingerprint from the Rust error! macro outside flow steps (matching py-mando’s log_error), needs one sentence added to the standard. Plan-reading lesson: the version bump’s “17 add / 19 change / 5 destroy” plan was IaC develop drift catch-up (12 genuinely new resources: the bess_os_fr incoming_sync lambda stack + 2 RDS analytics SG rules) plus immutable task-def replacement mechanics (all destroys were “must be replaced” task definitions), not danger. Ceremony doc updated (PyMando Win Dev needs gotcha, IaC-side apply trick, standing-hazard callout, observed pins); BESS AM + BE-3482 notes cross-updated. → Dev Deploy 2026-07-14 (error telemetry), Mando Deployment Ceremony, BE-3482 Datadog Logs and APM Conformance, BESS AM (BE-2262) - mando-bess-am.

2026-07-13

  • Documented the mando deployment ceremony (all facts verified from code in /Volumes/bandi/coding/poc/optimization-universe-iac + mando .gitlab-ci.yml; also recorded as new section 14 of the untracked mando AGENTS.md). Two-repo ceremony: mando CI publishes docker images; optimization-universe-iac (terraform, ECS) decides which version runs where; NEVER deploy from mando alone. Env matrix: 4 envs to 2 component groups in terraform/main.tf (dev+test → dev_test, int+prod → int_prod); IaC branch rules: dev/test plan+apply from develop (plan also on feature/bugfix branches), int from release/*, prod from main; ALL terraform_apply jobs when: manual; ECR: dev_test → 843164609896.dkr.ecr.eu-central-1.amazonaws.com/poc/mando/deploy, int_prod → 748634852998.dkr.ecr.eu-central-1.amazonaws.com/poc/mando; int_prod naming trap reconfirmed (historically INT, not real prod). Deploy-to-dev: merge to mando develop → Publish Service Docker Dev (only: exactly develop + rc/*) pushes {version}-dev.{pipeline_id}.{short_sha} (the + of APP_VERSION becomes - in docker tags, e.g. 1.11.0-dev.2555942905.50575825) → on IaC develop edit terraform/terraform.auto.tfvars.json components.dev_test.mando.version to the tag (observed commit convention “feat: bump component versions”) → terraform_plan:dev runs on push, review the plan artifact (expected change: mando container image in the bess-os ECS task def ONLY) → manually trigger terraform_apply:dev. Feature-branch trick: temporarily add the branch to the only: list of Publish Service Docker Dev, publish, bump tfvars; MUST drop/revert that CI commit before the MR merges (reviewers reject unrelated changes; leaving it in would publish on every future push). Runtime config without rebuild: per-env mando config (MANDO_FLOW_SCHEDULE_* crons, MANDO_FLOW_ACTIVE_VERSION, EBS_SMB_HOST, SIMULATION_MODE, CPU/memory, notification channels) lives in IaC .gitlab-ci.yml .environment-vars:{env} blocks flowing to TF_VAR_*; constraint: MANDO_FLOW_ACTIVE_VERSION (V1_4 in all envs) must name a setup version existing in mando config/flows/manifest.yaml, bump together. Post-deploy verification: GET https://mando.{env-domain}/version equals the bumped tag (also returns component versions); /health 200; swagger loads; Datadog sidecar logs on error.kind + refinery migration lines at boot (failed migration = most common bad deploy); dev runs SIMULATION_MODE=true; scheduled flows land Success/Warning not Fatal; local pre-deploy E2E test/pi1/end-to-end.py. Observed on rc/1.11.0: dev_test at 1.11.0-dev.2555942905.50575825, int_prod at 1.10.0-2533146896.b06b6b59. → Mando Deployment Ceremony.
  • Mined all 21 of Andras’s mando MRs for review patterns (!322 through !558; 17 merged / 2 closed / 2 open; read-only glab API): 108 reviewer notes (krisztian.fekete1 87, gabor.nagy6 13, balazs.mracsko.alpiq 7), distilled into a new “Recurring review feedback” section of /Volumes/bandi/coding/poc/mando/AGENTS.md (untracked). Nine themes: (1) zero unrelated changes in the diff, revert incidental churn / drop unrelated commits (!437/!474/!495/!548/!556); (2) delete ALL unused code before review, the most repeated complaint (frustration quote !345); (3) failures are ERRORS never warnings: mando_core::error! with an error-code enum variant, Fatal fails the execution, NO custom error message strings (the error’s to_string() IS the message, human context in the message field; 15+ “should be error” comments + policy from balazs, !481); (4) placement: route helpers next to routes, mod rows grouped, util code in util, name modules for what they are (!322/!345/!556); (5) completeness before review: no “you forgot this”/“half done”, mirror sibling configs like data groups (!362/!437); (6) green pipeline + no conflicts before assigning review (!345/!474/!558); (7) rustfmt every new file (!556, gabor); (8) keep the Jira ticket in sync with what was actually implemented (!437); (9) idioms: import errors from mando_core, no thiserror:: path prefix, no unwrap (parse at build time or return Result, !345), bind the service once vs per-type match duplication (!556). Config direction: NEW config goes to config::Config over Envconfig (two reviewers, !556); existing adapters keep envconfig. New dictated rules codified in the guide: commits title-only with NO scopes and no descriptions (no-scope rule reviewer-mandated in !322), commit every ~3 tasks in short form, never use em dashes anywhere (entire AGENTS.md purged of them), always subagent-driven development without asking. → Mando MR Review Patterns.
  • Rebuilt AGENTS.md (round 2) — /Volumes/bandi/coding/poc/mando/AGENTS.md fully rebuilt against the true origin/develop tip 92bfe1c8 (2026-07-09, v1.16.0) via 5 re-run researcher agents on a dedicated worktree at .worktrees/develop; local develop fast-forwarded and now tracks origin/develop — the round-1 “guide reflects stale merge-base 8bbd407e” caveat is obsolete, the Jul-8 drift (4 new crates + flow_registry.rs restructure + v1.16.0) is folded into the guide proper incl. recipe E. New verified facts: (1) FlowStepService (mando-flow-step/src/lib.rs) is a construction trait, NOT execute()-style — type Params: ParamMeta; type Response; + async from_config(config: &str, providers: &StepProviders); StepProviders (src/providers.rs) carries flow_repository, data_point_registry, 6 auth providers, simulation_enabled; 36 type_entry! registrations in mando-bess/build.rs; param structs derive #[derive(Clone, Debug, PartialEq, Serialize, Deserialize, ParamMeta)]; (2) config/flows/manifest.yaml = deployment catalog (setup: {version}: flows: {name}: {path, schedule_env, semaphore_group}) — schedule env vars + concurrency groups live in the manifest not code, semaphores capacity 1/group; (3) error redesign PARTIALLY landed (ae6d1098): error.message=root cause, error.stack=source chain, error.fingerprint={code}|{step_path}, ErrorWithStepStatus.logged_at_site double-log guard (new()/logged()); NOT landed: error.trace, .step_context(); docs/error-handling-redesign-plan.md + doc/errors.json NOT tracked; (4) py-mando-simulation split (3559042d, BREAKING): py_mando.SimulationRunner GONE → new package (Python ≥3.11, ships ddtrace, no polars/pandas); both packages init() at import (JsonFormatter + rustls ring), libduckdb preload (ctypes RTLD_GLOBAL/add_dll_directory), build.rs copies lib/libduckdb/1.4.2 → data/platlib; (5) second sanctioned async bridge: pyo3_async_runtimes::tokio::future_into_py for *_async awaitable variants (alongside allow_threads+block_on); (6) rustfmt: import-grouping/comment opts in rustfmt.toml are nightly-only, silently ignored on stable 1.89; tip NOT fmt-clean; grouping maintained by hand; format! positional ~3:1 over inline (inline = review rule for new code); (7) new crates (mando-flow-step/-derive, mando-bess-lib, py-mando-simulation Rust side) have ZERO tests (1 Python integration test); CI gained “PyMando Simulation Linux Dev” (pytest, no --nbval) + simulator docker publish child pipeline; (8) no MR/issue templates, no CODEOWNERS. Process: agentops:researcher subagents can’t SendMessage — reports via scratchpad files or transcript extraction (~/.claude/projects/<project>/<session>.jsonl, longest assistant text); 1 of 5 v2 researchers (style) died early, scope hand-verified. Vault reconciled: Agent Context (tip hash, crate rows, manifest/type_entry!, error row, fmt warning, CI jobs) + guide note provenance rewritten. → Mando AGENTS.md Master Guide.
  • BE-3482 continuation: merge state verified, orphan fix flagged for salvage, fingerprint ACTIVITY component built (unpushed). Verified feature/BE-3482 (DataDog logs/APM core conformance) MERGED via MR !558 as squash ae6d1098 on 2026-07-08 (30 files, +402/-173, merge commit c2b7d401; origin branch deleted); the 2026-07-02 HANDOVER-BE-3482.md is stale on ALL state claims; all 7 delivered field items confirmed on develop tip a01892f9. Orphan commit 478cd993 (Jul 9, “fall back to crate version when LOG_METADATA.version is None”) exists ONLY on the local feature/BE-3482 branch: adds dd_formatter::app_version(), removes 3 panicking unwraps in mando-bess (api_docs.rs, flow/get_versions.rs, route/version_route.rs); it is the fix for the known “develop panics mando-bess locally” issue; salvage = cherry-pick onto a fresh branch off develop as a small MR, do NOT resurrect the old branch (contains a merge commit, review-rejected shape); worktree .worktrees/logs-apm-conformance-core + local branch archivable only after the salvage decision. New work: error.fingerprint ACTIVITY component per the authoritative standard (architecture-design repo logs-and-apm.md: fingerprint = error.code + flow.step.path + specific activity, e.g. Volue EMS data group id) on branch feature/BE-3482-fingerprint-activity (worktree .worktrees/BE-3482-fingerprint, single title-only commit 6f541363): activity is a runtime value so it rides ErrorWithStepStatus (activity: Option<String>, consuming with_activity() builder, all constructors default None so behavior is bit-identical when absent); private fingerprint() helper emits code|path|activity or code|path with no trailing separator; reference wiring in mando-lib/src/service/volue/ems/ts_data_send.rs (every per-data-group error carries the data group name; real win: DA and AS schedule failures in the auction scheduling step no longer collapse into one Datadog error group); 3 new capture tests, clippy 0 warnings, suite green (mando_lib 254 passed, --test-threads=1); NOT pushed, no MR, awaiting Andras. Known limit: at-site-logged variants (VolueEmsError::ApiError) reach the boundary as WARN with no fingerprint at all, so activity is stored but not emitted there until the single-emission migration (own future ticket). Ranked remaining deferrals: single-emission migration (biggest win, unlocks fingerprint+http on at-site errors) > error.*/http.* on trace spans > verify dd.trace_id correlation in DD > flow.step.timeout_ms. → BE-3482 Datadog Logs and APM Conformance.
  • Field-verified: cargo fmt -- <file> does NOT scope (it touched api_docs.rs, which was never named on the command line). New prescription: rustfmt --edition 2021 <file>; every form of cargo fmt is banned. AGENTS.md sections 2.3, 10, 13.5 + the landmines table updated; Mando AGENTS Guide (mirror) resynced; the fmt warning in Agent Context and correction #3 in the guide note amended. → Mando AGENTS.md Master Guide, BE-3482 Datadog Logs and APM Conformance.
  • BE-3482 orphan salvage RESOLVED: folded into the fingerprint branch. Andras chose to fold 478cd993 into feature/BE-3482-fingerprint-activity instead of a separate MR; cherry-picked as bc3727ba (“fix: fall back to crate version when LOG_METADATA.version is None”, scope dropped from the title, author preserved andras.lederer@alpiq.com). The branch now has 2 title-only commits (6f541363 + bc3727ba); gates re-run on the 2-commit branch: clippy exit 0, tests 387 passed / 0 failed across 22 suites (single-threaded); still NOT pushed, MR awaiting Andras. The old worktree .worktrees/logs-apm-conformance-core + local feature/BE-3482 branch are now purely historical and archivable. → BE-3482 Datadog Logs and APM Conformance.
  • New field-verified landmine: the RTK shell hook guts cargo test. The hook rewrites cargo test --all-features --release -- --test-threads=1 so the threads flag becomes a test-name filter: every test is filtered out and the command exits 0 in 0.00s (output signature: 0 passed, N filtered out); two gate runs were false-green this way before detection. Correct invocation on RTK machines: rtk proxy cargo test --all-features --release -- --test-threads=1, and always confirm the summed passed totals are nonzero (exit 0 alone proves nothing). Adjacent shell pitfall: cargo test | grep | tail; echo $? reports tail’s exit, not cargo’s. Recorded in mando AGENTS.md section 10 + the landmines table; Mando AGENTS Guide (mirror) resynced; warning callouts added to Agent Context and the guide note. → Mando AGENTS.md Master Guide, BE-3482 Datadog Logs and APM Conformance.
  • Reviewed feature/BE-3482-pymando read-only: NEEDS REWORK BEFORE MR (13 commits off develop@3024e20c, worktree .worktrees/BE-3482-pymando, judged against mando AGENTS.md; zero mutations made). The conformance work itself is solid and the gates are genuinely green: clippy clean, 366 passed / 0 failed / 35 ignored via rtk proxy, and mando_lib builds with --no-default-features AND with --features python (proving fmt_util needs no app feature). Three blockers (~30 min total): (1) 13/13 commit titles carry scopes, banned per MR !322; (2) 7 orphaned ContextVars + dead set/reset lines in py-mando/python/py_mando/tracing.py after d528b71f removed their only reader; (3) commits 86d7ffd8 + 34e749d5 are off-ticket AND byte-identical duplicates of bc3727ba on the fingerprint branch. Notable should-fixes: error.fingerprint serializes code|None when no step context (tracing.py:109); dead tracing-serde dep in py-mando/Cargo.toml; fmt_util extraction only 2/3 done (py-mando-simulation still duplicates MapVisitor/WriteAdaptor/collect_span_fields verbatim and could import mando_lib already); 2 branch-introduced rustfmt hunks (fmt_util.rs:47 new file, dd_formatter.rs:373); super::version_or full-path calls; test hygiene (hand-rolled _Capture instead of caplog, unrestored mutated globals despite the commit title claiming restore). Rebase analysis: exactly ONE file overlaps develop (dd_formatter.rs) but the conflict is semantic and guaranteed: develop’s ae6d1098 rewrote dd_formatter inline with the same plumbing d0986230 extracts. Landing order: fingerprint MR first (based on the develop tip, unblocks the develop panic), then rebase pymando dropping 86d7ffd8 + 34e749d5; sharpest option also drops d0986230 (entire conflict surface, off-ticket, 2/3 done) for a zero-conflict 9-commit on-ticket MR, refiling the fmt_util extraction as its own ticket that unifies all THREE formatters against develop’s conformant dd_formatter (cost: temporary 7-line level-to-status duplication in py-mando). Path to MR-ready: 9 items; python pytest MUST be re-run before MR (not re-verified since 2026-07-09). Third tooling landmine: rustfmt --check --skip-children is unrecognized on stable and prints nothing (false green); check without it, attribute hunks by the Diff in <path>: headers (rustfmt follows mod declarations), baseline against git show <base>:<path>; recorded in AGENTS.md section 10 + landmines, Mando AGENTS Guide (mirror) resynced. → BE-3482 pymando Branch Review, BE-3482 Datadog Logs and APM Conformance.
  • Fingerprint MR OPENED: MR !569 for feature/BE-3482-fingerprint-activity (“feat: add activity component to error fingerprint”; 2 title-only commits, 6f541363 + the salvaged bc3727ba version-fallback panic fix), pushed to origin, awaiting review. Step 1 of the recommended landing order (fingerprint first, then pymando) is now in flight. → BE-3482 Datadog Logs and APM Conformance.
  • pymando rework EXECUTED: rebuilt from scratch as feature/BE-3482-pymando-v2 (worktree .worktrees/BE-3482-pymando-v2): 15 title-only, no-scope commits off origin/develop tip 51d2b516 (develop advanced twice mid-session, incl. the BE-1595 Arrow Flight merge, which touches py-mando); zero cherry-pick conflicts; 10 files +374/-55. Commit selection: 9b0c4e70 KEPT (on-ticket, py-mando-only, independent); DROPPED 86d7ffd8 + 34e749d5 (version fix rides MR !569) + d0986230 (fmt_util extraction, refile as own ticket); surprise: fe78dc10 needed NO level_to_status workaround since the helper was already inline pre-extraction (d0986230 is what moved it out), so dropping it cost zero extra code. All review blockers/should-fixes executed: 7 orphaned ContextVars deleted (incl. the dead metadata param + run_with_trace call site), dead tracing-serde dep removed, fingerprint outside a trace scope now emits the bare code instead of code|None (new test test_log_error_without_step_context_omits_path), build_ddtags made public, test hygiene real (caplog replaces _Capture, error_log fixture, logger_state fixture that genuinely restores mutated globals, DD_ENV/DD_VERSION monkeypatch-tracked). NEW field flow.step.system: flow.step.connection exists ONLY in mando-lib/src/workflow/mod.rs (step info_span! built from StepMetadata { flow: DataFlow, system }), so the field is ONE line (flow.step.system = metadata.system) from the structured source, no string parsing; verified 14/14 Wrapper steps declare system: "Mando" and externals declare real names (Volue EMS x19, Metis x14, Position Manager x8); 4-case #[test_case] matrix (Receive Volue EMS / Receive Metis / Send Volue EMS / Wrapper Mando) passes; scope tension flagged: one mando-lib commit (52b4d270) on an otherwise py-mando-only branch, trivially splittable if a pure MR is wanted. Gates all green: clippy exit 0 (zero findings in touched files); rtk proxy cargo tests 434 passed / 0 failed / 35 ignored; rustfmt zero branch-introduced hunks; pytest 120 passed / 1 skipped (baseline 114) incl. a runtime JSON conformance proof from the Rust stream. Cargo.lock carries a legitimate extra hunk: develop’s committed lock is STALE (mandarrow-client 1.16.0 vs workspace 1.16.1), plain cargo build regenerates it. Two more AGENTS.md landmines (section 10 + landmines table, mirror resynced): (a) rustfmt base-file checks via scratch-dir copies are FALSE GREEN (mod children unresolvable, 0 hunks reported) AND running rustfmt on a module-declaring file writes into child files (6 unrelated files touched and reverted during the rework); check base files at their real path only; (b) develop’s committed Cargo.lock can be stale, a lock hunk in a diff may be legitimate. Env notes: ~/.local/bin/python3.12 is broken (venvs crash in ensurepip), use /opt/homebrew/bin/python3.12; a transient SIGBUS hit /Volumes/bandi during maturin develop, retry succeeded. v2 NOT pushed, no MR; old v1 branch/worktree untouched as fallback; main checkout untouched. → BE-3482 pymando Branch Review, BE-3482 Datadog Logs and APM Conformance.

2026-07-10

  • Documented the new master conventions guide for AI agents at /Volumes/bandi/coding/poc/mando/AGENTS.md (untracked on disk, deliberately never committed; synthesized 2026-07-10 by 5 parallel researcher agents from the develop branch + CI config + review feedback; 12 sections + 8 recipes: crate map/dependency direction, 10 non-negotiables, toolchain, architecture/layering, Rust style, the mando_core::error! system, mando-bess API conventions, py-mando PyO3 patterns, testing ceremonies, git/CI ceremonies, verified-landmines table, exemplar recipes A–H). Corrections to prior assumptions recorded (each independently verified against origin/develop): (1) mando-bess/build/generated/**.rs (domain.rs + flows) is GENERATED by build.rs from YAML+Askama but TRACKED in git — edit YAML/template → cargo build → commit input+output together, never hand-edit (same for *.out golden files); (2) NO ErrorCode derive and NO mando-lib-macro crate on develop (verified git grep empty) — they live only in .worktrees/ experiments (poc-error-extractor, BE-2023, BE-1595/BE-3482); error.kind/error.code are extracted at runtime by mando_core::error! from the Debug repr; parent poc/CLAUDE.md is stale on this (also references a non-existent mando/CLAUDE.md); naming trap: develop’s mando-flow-step-derive derives ParamEnum/ParamMeta, NOT ErrorCode; (3) CI has NO fmt gate; lint gate = cargo clippy --release --all-features; canonical test = cargo test --all-features --release -- --test-threads=1 (single-threaded MANDATORY, shared in-memory DB pools); (4) doc/errors.json = untracked generated error catalog (rustdoc-JSON shaped), generator not in CI (likely poc-error-extractor worktree); (5) mando-bess JWT token_layer has signature validation DISABLED (insecure_disable_signature_validation) — User extension is audit metadata only, NOT authz; (6) flow versions immutable once released (new behavior → v(N+1); exemplar manual_schedule v3; tip flows: as_auction_update_v2/auction_v4/data_update_v2/intraday_v2/manual_schedule_v3); (7) py-mando async bridge = py.allow_threads(|| pyo3_async_runtimes::tokio::get_runtime().block_on(…)), single PyMandoError, map_*_error fns in mando-lib/src/python/error.rs; (8) toolchain 1.89.0 / MSRV 1.88.0 (parent docs conflated). Provenance pinned: the guide reflects develop @ merge-base 8bbd407e (Jul 1, v1.15.0); origin/develop tip e5259cbd (Jul 8, v1.16.0) already drifted — 4 new crates (mando-flow-step, mando-flow-step-derive, mando-bess-lib, py-mando-simulation) + flow wiring restructured to flow_registry.rs (per-type src/flow/{type}/v{n}/ dirs gone → recipe E stale for tip). Vault reconciled: Agent Context crate table/versions/error-row/flow-list/gates rewritten to verified tip state (was v1.4.11 + listed mando-lib-macro); mando-lib-macro marked status/outdated with correction callout; Mando CI-CD doc debt from 2026-06-25 cleared (samba removal, 1.89.0, gates section). → Mando AGENTS.md Master Guide.
  • Wrote the approved DESIGN SPEC for BE-1597 (status: design approved, NOT implemented; owner Gergely “Geri” Vászon; repos mando + optimization-universe-iac) — the actionable decision companion to yesterday’s research note (evidence wikilinked, not duplicated). Decision = phased: Approach 1 (dev only) NOW, Quack (all envs) on DuckDB 2.0 in Sept 2026. Approach 1 mechanism: SSM AWS-StartPortForwardingSessionToRemoteHost picks the LOCAL port → forward local 4213, browse http://localhost:4213 (literal localhost, NOT 127.0.0.1 — the bundle string-compares "localhost:4213"); no ALB (already internal=true), no target group / health check / /duckdb* prefix / reverse proxy / bundle patching; the bastion hop is required only because ECS Exec is interactive-only. Concrete deliverables: (a) container — add socat to the mando-bess Dockerfile + a docker-entrypoint.sh that runs socat TCP4-LISTEN:4214,fork,reuseaddr TCP6:[::1]:4213 & when DUCK_DB_UI_SERVER=true then exec mando_bess (NO Rust change — env already read at lib.rs:181/:301; no EXPOSE/portMappings in awsvpc; socat ships in the prod image too, ~400KB, the listener just never starts there); (b) terraform — a dynamic "ingress" block INSIDE the bess_os_ecs SG (inline-blocks resource, so a standalone aws_vpc_security_group_ingress_rule is silently reaped) opening port 4214 from the bastion SG on var.environment=="dev", plus DUCK_DB_UI_SERVER = tostring(var.environment=="dev") on the mando container (use var.environment, NOT local.environment_group, which collapses int+prod/dev+test); (c) a duckdb-ui.sh helper resolving the bastion + task IP and opening the SSM port-forward. Blocking pre-check (open Q1): does the dev Fargate task actually have egress to extensions.duckdb.org/ui.duckdb.org? No NAT/VPC-endpoints in IaC (VPC is a base-vpc data lookup owned by another team) — if blocked, Phase 1 can’t work and no terraform fixes it. Production bar: NEVER run the DuckDB UI in int/prod — the frontend is proprietary MotherDuck JS (source unpublished, still no license as of Jul 2026) proxied live from ui.duckdb.org with no auth that can read the live trading cache; the rest of the supply chain (DuckDB core / libduckdb / duckdb-rs / ui-ext SOURCE / quack / ICU / yyjson) is MIT/permissive and safe to vendor. Phase 2 = Quack (core DuckDB-signed MIT ext, 1.5.3+; quack_serve('quack:0.0.0.0:9494', allow_other_hostname=>true)→token, client ATTACH 'quack:host:9494' (TOKEN …)) run with duckdb -ui ON THE LAPTOP — deferred to DuckDB 2.0 (Sept 2026) because Quack is Beta in 1.5.x (one upgrade, not two). Upgrade 1.4.2→2.0 assessed low-risk (in-memory-only sidesteps storage churn; keep the =1.4.2 pin vs the new 1.MAJOR_MINOR_PATCH.x crate scheme). → BE-1597 DuckDB UI — Design.

2026-07-09

  • Researched BE-1597 DuckDB UI exposure (+ the ticket’s 2nd half: remote client access to the in-memory DuckDB). Root cause both prior attempts failed (Andras + Gergely Vászon, Mar 3–4 2026, IaC branches feature/BE-1597*/feature/expose-duckdb-ui + mando feature/BE-1597-add-tunnel-for-duckDB-UI c700fee9 / feature/BE-1597-proxy-duckdbUI-thorugh-mando 0e06c93e, nothing merged): the DuckDB UI JS bundle hardcodes "localhost:4213" === window.location.host ? "duckdb_ui" : "web", so any other origin silently degrades — Attempt A (socat 4214 + ALB target group + /duckdb* listener rule; died on the 8080 /health check, hence temp: disable healthckeck for duckdb commits) and Attempt B (reqwest reverse proxy mando-bess/src/route/duckdb_proxy.rs rewriting <base href> + string-patching the JS bundle) were both un-winnable. Fix = SSM AWS-StartPortForwardingSessionToRemoteHost picks the LOCAL port → forward local 4213, browse http://localhost:4213 (literal localhost, NOT 127.0.0.1) and the origin check passes; no ALB (mando ALB already internal=true). Bastion SSM hop is genuinely required because ECS Exec is interactive-only (no port forwarding). UI ext constraints: only settings ui_local_port(4213)/ui_remote_url/ui_polling_interval(284ms), NO bind-address (binds IPv6 ::1 → socat needed), no auth, proxies the frontend from ui.duckdb.org per session (internet required), ui_remote_url only honored under allow_unsigned_extensions, and the ui ext is not statically linked (verified nm -D on the vendored 1.4.2 .so → only Icu/Json) so CALL start_ui_server() auto-downloads it. License blocker (the important one): DuckDB core / libduckdb / duckdb-rs / libduckdb-sys / the ui ext SOURCE / quack / ICU / yyjson are all MIT/permissive (IP: Stichting DuckDB Foundation) — but the DuckDB UI FRONTEND assets from ui.duckdb.org are PROPRIETARY MotherDuck code, source unpublished, still no published license as of Jul 2026, served with no auth that (per DuckDB docs) “can access the data you load into DuckDB” → fine for dev, NOT acceptable for the int_prod trading service. Ticket’s 2nd half: the ONLY way to remotely connect a client to a running in-memory DuckDB is the Quack remote protocol (core DuckDB-signed MIT ext, DuckDB 1.5.3 / 2026-05-20; quack_serve('quack:0.0.0.0:9494', allow_other_hostname=>true) → auth_token, client ATTACH 'quack:host:9494' (TOKEN …); beta until DuckDB 2.0, Sept 2026); 1.4.2 has no wire protocol at all. Upgrade 1.4.2→1.5.4 = low-risk (in-memory-only sidesteps storage-format churn; =1.4.2 pin Cargo.toml:60 safe against the new 1.MAJOR_MINOR_PATCH.x crate scheme where 1.5.4→1.10504.0; C API additive-only; date_trunc(DATE)→TIMESTAMP and -> lambda-deprecation SQL breaks both clear against mando). Infra gotchas: bastion SSM-only + zero-ingress; bess_os_ecs SG uses INLINE ingress blocks — a standalone aws_vpc_security_group_ingress_rule is silently deleted on next apply, use a dynamic "ingress" block; NO NAT/VPC-endpoints so Fargate egress to duckdb.org UNVERIFIED; int+prod both map to int_prod. → BE-1597 DuckDB UI Exposure Research.

2026-06-25

  • Documented a two-bug visibility fix for mando-cli v0.4.0 found while VALIDATING the [[mando-cli-v0.4.0-mando-bess-binary-missing-2026-06-25|mando up --mando=pull workaround]]: when up output is piped/redirected (remote tester does mando … 2>&1 | tee log), failures were INVISIBLE — captured .mando/compose-up.log was empty and the user saw only a bare docker compose up … failed (exit 1). Two independent, stacking root causes (fixing one is insufficient): (A) the streaming compose runner (src/runtime/compose.rs, run_streaming) pipes child stdout/stderr, but docker compose writes its pull/up progress AND the error to /dev/tty, bypassing the pipe → empty capture. Fix A: add top-level --progress plain to the compose invocation (compose.rs ~L231, Cmd::new("docker").arg("compose").arg("--progress").arg("plain").arg("-p")…) to force plain newline output into the pipe (--ansi never equivalent). (B) CliBuffer::add_log_line (src/opts.rs:57) forwarded lines via indicatif::MultiProgress::println, which is a silent no-op on non-TTY (returns Ok, writes nothing) → dropped ALL streamed docker output AND the failure-tail replay block (compose.rs:263-272). Fix B: fall back to eprintln! when std::io::stderr().is_terminal() is false (use std::io::IsTerminal; TTY → multi.println with eprintln! fallback on err, non-TTY → eprintln! directly). Validated on macOS: redirected (non-TTY) up --mando=pull went from ~35 silent lines (bare failed (exit 1)) to 138 lines showing the real pull access denied for mando … 'docker login' error + --- last 40 log lines --- tail + --- full log: .mando/compose-up.log pointer. Reusable gotchas: (1) docker compose’s progress writer targets /dev/tty — any pipe-captured invocation must pass --progress plain/--ansi never or output+errors vanish from the captured stream; (2) indicatif::MultiProgress::println is a silent no-op on non-TTY — any CLI using it for log passthrough must fall back to eprintln! when stderr isn’t a terminal, else piped/CI/tee output disappears. Tester’s now-visible failure: pull access denied → needs docker login registry.gitlab.com (GitLab PAT, read_registry scope); mando-cli’s “registry auth” line is GitLab API creds (git/glab credential helper), NOT docker login — a misleading-but-important distinction (separate credential stores). Status: both fixes implemented + built + validated on macOS, NOT committed; linux/amd64 build for gabi (WSL) pending. → mando-cli-v0.4.0-piped-output-invisible-failures-2026-06-25.
  • Documented the BE-1595 Arrow Flight dev deploy + hardening as four durable, reusable notes (distilled from a multi-session dev deploy of the Flight streaming feature: mando server + py-mando wheel + optimization/forecast consumer images). (1) Flight → REST graceful degradation (F4 capability-gap fallback) — the DuckDB/passthrough repo only streams within its cache window; a retrieve_stream over a range predating the cache (Permanent storage) returns RepoError::MethodNotSupported (repo_passthrough.rs:335). Originally repo_error_to_status (mando-bess/src/flight.rs) mapped it to gRPC FailedPrecondition and the client only fell back to REST on is_retryable(), so it propagated as Fatal and crashed forecast (fetches historical data). Fix (commit 7926278c): map MethodNotSupported → gRPC Unimplemented; add ClientError::is_unimplemented() (mandarrow-client/src/error.rs); client falls back on is_retryable() || is_unimplemented() (py-mando/src/polars.rs). KEEP CyclicDependency as FailedPrecondition (real error, REST can’t fix). Net: optimization fully on Flight (cache-window fetches), forecast logs falling back to REST then completes. Resolves the “Known limit” from be-1595-flight-execution-id-parity-2026-06-17 (flipped that note’s callout to [!done]). → be-1595-flight-rest-graceful-degradation-2026-06-24. (2) Wheel ↔ server version skew — the #1 deploy hazard: wheel (mandarrow-client Flight wire types + bess client Python API like BessOptClient.send(update_id=...)) and server deploy independently, but a develop merge bumped shared DataPointUpdateInfo (renamed fetch_time→update_time, added update_id) breaking old-wheel↔new-server (send() got an unexpected keyword argument 'update_id'). Fix (commit 73cbdcf3): tolerant wire types — #[serde(default)] on every optional/added field, #[serde(alias="old")] on renames, NEVER #[serde(deny_unknown_fields)] on boundary structs, + compat tests (old-shape JSON must still deserialize). Two hard constraints: can’t retroactively fix released versions (deploy mando + wheel from the same build on a wire change); send-side signature mismatch only resolves by rebuilding consumers on the matching wheel. → be-1595-flight-wire-type-version-skew-2026-06-24. (3) Consumer wheel-lock deploy script (deploy-arrow-consumers.ps1 / base64 iex one-liner) — runs on Windows on corp net (Nexus must resolve). Gotchas: clone via token URL (https://oauth2:$Token@…) + GIT_TERMINAL_PROMPT=0/GCM_INTERACTIVE=Never to stop the GitLab credential popup hanging; the base64 one-liner is a frozen snapshot so a stale clipboard silently re-locks the WRONG wheel — always re-copy fresh + verify the echoed wheel; dev images gated behind the python-docker-publish component (Publish Docker Dev only on develop/rc) so the script appends a local rules-override for feature/* (relatedly needs the optional: true .Publish→Test patch, be-1595-publish-docker-dev-feature-branch-test-need-2026-06-16). → be-1595-arrow-consumer-lock-script-2026-06-24. (4) Dev deploy runbook (recurring facts) — migration drift → mando crash-loop (V…__… is missing from the filesystem), fix by merging develop so branch migrations ⊇ DB’s; IaC at optimization-universe-iac .worktrees/mando-arrow terraform terraform.auto.tfvars.json (mando + optimization-algo + forecast-algo image pins); terraform_apply:dev always red on a pre-existing customer-portal S3 403 HeadObject even though terraform prints Apply complete! (harmless, not ours); runtime switch MANDO_FETCH_STRATEGY=flight (default rest) + MANDO_FLIGHT_HOST/PORT on algo task defs; verify in Datadog EU env:dev services bess-os-service-mando/bess-os-algo-optimization/bess-os-algo-forecast. → be-1595-arrow-flight-dev-deploy-runbook-2026-06-24.
  • Documented the NEXT mando-cli v0.4.0 mando up failure after the 2026-05-26 compose-runtime fixes (commit 2233959): on a fresh clone the default profile mando-mocked-algos aborts docker compose build at #14 [mando 6/6] COPY target/release/mando_bess … → failed to compute cache key: "/target/release/mando_bess": not found (collateral: bess-trader-dashboard build shows CANCELED). Reported by remote QA gabi (gabriel.vasile1) on WSL. Root cause (verified in source): mando/Dockerfile is a THIN runtime image (FROM debian:13.1-slim AS runner, :17 COPY target/release/mando_bess …, CMD ["mando_bess"]) that copies a PRE-compiled binary — no Rust build stage; mando/bess-service.yaml:23-32 context_includes ships target/release/mando_bess (a CI-only assumption that cargo build --release already ran). The default profile maps mando→build runconfig (runprofile.rs:225; mando-full→build :237, mando-fast-dev→build-dev :249) — NO builtin profile maps mando to pull/artifact. The build runconfig (templates/runconfig/build.yml:10-12) emits a real build: section and NOTHING in the up flow compiles mando_bess (the host-build --cargo flag is gated to the artifact runconfig only, cli.rs:55). Net: any tester on a fresh clone hits it; works in CI only because CI pre-builds the binary. Workaround (no code change): mando up --mando=pull — the pull runconfig (templates/runconfig/pull.yml) swaps build: for image: ${MANDO_IMAGE:-…} + docker compose pull; on gabi’s box resolves to registry.gitlab.com/alpiq_cicd/.../mando:1.10.0-2533146896.b06b6b59 (authed → pulls the released image); per-service override syntax at cli.rs:78. Durable fix (decision pending, NOT implemented): (1) flip default mando→pull (runprofile.rs:225) — OPEN RISK: confirm a clean checkout’s default MANDO_IMAGE points at the GitLab registry, NOT a local mando:dev (the ref is NOT hardcoded in mando-cli source; comes from workspace config/env), else pull 404s; (2) add a preflight guard in up: build/build-dev runconfig + missing target/release/mando_bess → fail early with an actionable message. Status: diagnosed, fix pending user decision. → mando-cli-v0.4.0-mando-bess-binary-missing-2026-06-25.
  • HARDENED the remotefs-smb to smb migration EBS upload path after a two-reviewer (correctness+style) pass — three ebs.rs fixes: (1) replaced panic-prone Resource::unwrap_file() (PANICS if the path resolves to an existing directory, server-state-dependent) with fallible resource.try_into()::<File> (smb Resource: TryInto<File>, err (smb::Error, Self)) — never unwrap_file() in prod; (2) fixed a workgroup/empty-username auth bug — old if wg.is_empty(){user}else{format!("{wg}\\{user}")} produced malformed "WORKGROUP\" (trailing \) when workgroup set but username empty AND dropped anonymous-on-empty-username, now a 3-arm match (wg.is_empty(), user.is_empty()) → empty username ⇒ String::new() (anonymous) else WORKGROUP\user; (3) added file.flush().await? (inherent pub async fn flush(&self), NOT a trait) before close() for durability. Plus a formatting finding: rustfmt.toml sets nightly-only opts (group_imports=StdExternalCrate, imports_granularity=Module) but there’s NO fmt gate in .gitlab-ci.yml and ALL sibling adapters fail cargo +nightly fmt --check (mdr/opl/et_3000/data_platform) — the real bar is stable cargo fmt; do NOT nightly-format individual files (breaks import consistency vs siblings); rustfmt --check outside the repo root panics. Re-verified: cargo build -p mando_lib clean, cargo test -p mando_lib 205/32 ignored/0, clippy + stable fmt clean (pkg is mando_lib, underscore). → remotefs-smb to smb migration.
  • MIGRATED the SMB client off GPL: replaced remotefs-smb + companion remotefs (=0.3.1) with the pure-Rust smb crate =0.11.2 (github.com/afiffon/smb-rs) in mando-lib. Why: remotefs-smb → pavao → pavao-sys FFI-binds the system libsmbclient, and pavao/pavao-sys are GPL-3.0 (dynamic-link-only) which Alpiq cannot use; smb is pure Rust (no libsmbclient) so the GPL transitive dep is gone entirely (cargo tree -p mando_lib | grep -iE 'remotefs|pavao' now empty). Blast radius = ONE call site: the Alpiq EBS (Energy Balance System) adapter mando-lib/src/adapter/alpiq/ebs.rs, which is upload-only (mando generates .xlsx bid-templates and pushes them to an SMB/CIFS share); EbsClient::upload_file’s public signature was preserved so the service-layer callers (service/alpiq/ebs.rs, …/ebs_clear_bids.rs) needed no changes. Full cleanup chosen (no dead GPL footprint): bumped rust-toolchain.toml 1.88.0 → 1.89.0 (forced — see MSRV gotcha), swapped workspace Cargo.toml + mando-lib/Cargo.toml deps, rewrote the upload path and deleted both #[cfg(target_family=…)] create_client builders (pure-Rust ⇒ no unix/windows split), removed pavao=off from default RUST_LOG (app/mod.rs), and dropped samba-libs + libsmbclient apt installs from Dockerfile + mando-simulator/Dockerfile. API recipe (low-level + async, vs remotefs’s high-level RemoteFs trait): default features (sign,encrypt,compress,async,std-fs-impls,netbios-transport) — NOT kerberos(reqwest)/quic; Client::new(ClientConfig::default()); UncPath::from_str(r"\\host\share") (needs use std::str::FromStr); share_connect(&unc, &user, password) where a workgroup is mapped by format!("{workgroup}\\{username}") (parsed by sspi::Username::parse, accepts DOMAIN\user); unc.with_path(&path) consumes self (call share_connect(&unc,…) before moving unc); create_file(&path, &FileCreateArgs::make_overwrite(FileAttributes::new(), CreateOptions::new())) uses disposition OverwriteIf which collapses the old exists()+remove_file()+create_file() dance into one call (smb has no explicit delete); res.unwrap_file() → File::write_at(&[u8], u64)->usize (loop for partial writes) → File::close(). Import gotcha: write_at is the smb::WriteAt TRAIT (must be in scope); FileAttributes/CreateOptions live in smb-fscc but are re-exported at the smb root (pub use smb_fscc::*). Gotchas: (1) MSRV blocker — smb 0.11.2 is edition 2024 and it + all 8 sub-crates declare rust-version="1.89.0", so 1.88.0 hard-errors (“requires rustc 1.89.0”) → toolchain bump; re-verify smb MSRV before any future bump. (2) ~295 crates added pulling the SMB3 signing+encryption stack at RustCrypto release-candidate versions (aes-gcm 0.11.0-rc.1, aead 0.6.0-rc.2, ccm 0.6.0-pre.0) + sspi/NTLM (picky/rsa/curve25519) — mandatory for SMB3, can’t trim much; conscious tradeoff (pure-Rust+no-GPL vs large pre-release-crypto tree). (3) cargo build -p py_mando link failure is a RED HERRING (pre-existing) — undefined libpython symbols from pyo3_ffi is standard pyo3 extension-module cdylib behavior (verified identical on clean baseline; smb compiles fine in py_mando closure); build the Python bindings with maturin develop, not cargo build. (4) cargo package name is mando_lib (underscore), not mando-lib. Verify (all green): cargo build -p mando_lib clean; whole-workspace build green except the pre-existing py_mando maturin link (gotcha 3); cargo test -p mando_lib = 205 passed / 32 ignored / 0 failed; cargo clippy -p mando_lib clean. OPEN acceptance step: the #[ignore]d live tests local_hourly_test/local_quarter_hourly_test (env EBS_SMB_HOST/USERNAME/PASSWORD/SHARE/WORKGROUP/SUB_FOLDER) need running against a real SMB share to confirm the new crate authenticates+uploads end-to-end — could not be validated in dev; flag before merge. Doc debt: Mando CI-CD + Agent Context still list samba-libs/libsmbclient + Rust 1.88.0 (now 1.89.0) — flagged, not yet reconciled. → remotefs-smb to smb migration.
  • FIXED the mando-codegen expand_variants python_field casing bug from the BESS AM (BE-2262) - mando-bess-am note (was: min1.Mean instead of all-lowercase min1.mean for multi-segment Named variants, affecting BESS AM 1-min aggregate datapoint names). Fix lowercases the value segment at mando-codegen/src/util.rs:41 → format!("{pf}.{name_lower}.{}", v.to_lowercase()). Commit cc4720b2 “fix: lowercase value segment of variant python_field” on feature/BE-2262-bess-am-poc, pushed to MR !512. Verified cargo test -p mando_codegen now 51/51 passing (was 50/1). Residual (unrelated): two PRE-EXISTING rustfmt drifts remain in that same file (~lines 50 and 390, author’s code), left untouched — flagged by nightly rustfmt. Flipped the note’s > [!warning] callout to > [!done]. → BESS AM (BE-2262) - mando-bess-am.

2026-06-24

  • Documented BESS AM (BE-2262) as durable reference knowledge (newly discovered from mando git history, not previously in the vault). BESS AM = BESS Asset Management: a headless Kinesis stream processor (new crate mando-bess-am/mando_bess_am) for per-second WAGO battery-telemetry events (real-time SoC/SoH) on the FI/Valkeakoski/Beskar asset, authored almost exclusively by Gergely Vászon (ext). Pipeline: WAGO Box → AWS IoT Core → Kinesis (bess-am-events) → mando-bess-am → Kinesis (bess-os-events) → mando-bess. The service consumes the raw bess-am-events stream, persists raw per-second events to Postgres schema bess_am (event table), computes 1-minute aggregates (Mean/Max/Min/Last/StdDev/Count/Sum) via a windowing + grace-period closure mechanism (service/window_closure), and forwards results to bess-os-events which mando-bess ingests. Introduces a NEW Event-typed data-point class (real-time telemetry) distinct from the existing TimeSeriesDouble/TimeSeriesDoubleMatrix/StaticData time-series flows — the flow-engine list (Trading/Manual Schedule/Auction/Intraday/Data Update/AFRR) does NOT include AM. Code fingerprints: crate modules pipeline/service/aggregation/service/forward/service/window_closure/startup/metrics; mando-lib/src/adapter/kinesis/ (consumer/producer/pipeline/checkpoint/config); mando-core/src/model/{event,wago}.rs; mando-bess/src/kinesis.rs; mando-bess/config/parts/battery_online.yaml (Event datapoints BATTERY_SOC_ONLINE/BATTERY_SOH_ONLINE under Asset/FI/Valkeakoski/Beskar/Battery/.../Online, keyed to Kinesis external IDs [bess-am-events, OnlineSOC/OnlineSOH]); container.bess-am.Dockerfile; config/bess-am.yaml. Infra: schema bess_am, migrations create_bess_am_schema/create_event_table/create_kinesis_checkpoint_table/add_event_latest_historization; env prefix BESS_AM_*; Kinesis streams bess-am-events & bess-os-events; CI job “Publish BESS-AM Docker Dev” (ECR tag bess-am-<version>); local-dev doc/kinesis-local-setup.md + MiniStack (ministackorg/ministack). Related tickets: BE-2341 (generic Kinesis producer in mando-lib, no standalone MR, folded into the BE-2262 MR) and BE-2132 (interval/Event datapoint groundwork AM builds on). MR landscape (2026-06-24): core MR !512 feat: add kinesis consumer and event processing (feature/BE-2262-bess-am-poc → develop) is OPEN, requested_changes, reviewers krisztian.fekete1/gabor.nagy6/balint.budavoelgyi/andras.lederer/jozsef.nagy1, empty description, created 2026-05-19 / updated 2026-06-17; !524 feat: add basic interval functionality (feature/BE-2132-interval-poc-2, closes BE-2132, OPEN/not_approved); !500 same title (feature/BE-2132-interval-poc, OPEN/DRAFT, stale, superseded by !524); branch feature/BE-2262-bess-am-poc-build-test + commit 5a739baa “feat: separate out bess-am” splits AM into its own crate/image but has NO own MR (downstream of !512). Nothing BESS AM has merged to develop yet — still in flight, blocked on review. Added pointers from Agent Context (new crate row + dedicated BE-2262 section) and Alpiq BESS (Key Concepts). → BESS AM (BE-2262) - mando-bess-am.
  • Appended an Update 2026-06-24 section to the BESS AM note with two facts verified from the mando repo. (1) origin/develop merged into the !512 branch feature/BE-2262-bess-am-poc (was 46 commits behind), pushed fast-forward f9a1f81d..80a17e78 (merge commit 80a17e78); conflicts resolved in mando-lib/src/repo_postgres.rs (kept BOTH new DataSchemaPostgres defaults — branch’s has_override() + develop’s latest_pk_columns()), py-mando/bess-csv/defaults.csv (kept the 3 BESS AM Event rows, took develop’s newer EUR/MW/h FINGRID units), and the generated domain.rs files (regenerated via build.rs, converged CRLF→LF to match develop — branch had committed them CRLF); post-merge compiles clean (cargo check, 0 errors), 428 lib unit tests pass. (2) Pre-existing mando-codegen bug: expand_variants mis-cases python_field for multi-segment Named variants — a Min1/Mean variant yields market.da.price.min1.Mean instead of all-lowercase market.da.price.min1.mean (lowercases the first path segment but not later ones); caught by failing test mando-codegen/src/util.rs → util::tests::expand_variants::named_variant_generates_datapoint (~line 291); affects BESS AM 1-minute aggregate names (Min1/Mean, Min1/Max, Min1/StdDev, …); NOT caused by the develop merge (mando-codegen byte-identical before/after) — originated in the branch’s own feat: separate out bess-am commit 5a739baa; fix = lowercase every segment of the variant path; left unfixed/separate per decision, merge push didn’t change it. → BESS AM (BE-2262) - mando-bess-am.

2026-06-22

  • Captured a reusable clippy tooling gotcha discovered while designing the mando error-handling redesign: for clippy’s disallowed_macros lint (mando bans tracing::error!/log::error! via clippy.toml to force logging through the mando_core::error! wrapper in mando-core/src/error.rs), an #[allow(clippy::disallowed_macros)] placed at or around the call site — on the tracing::error! invocation, on a wrapping #[allow] { ... } block, or on the enclosing fn/match arm/let — is silently ineffective on a newer clippy (probe reported 1.95); the lint still fires and clippy additionally warns the attribute is “unused, since it’s applied to a macro invocation”. The only robust placement is a module-root inner attribute #![allow(clippy::disallowed_macros)] at the file that DEFINES the wrapper macro, because disallowed_macros resolves the lint level at the lexical site where the banned tokens physically appear (the macro definition, not the call site) — so one allow in mando-core covers all cross-crate callers (mando-lib, mando-bess). Caveat: the workspace pins toolchain 1.88.0 (rust-toolchain.toml) and the macro on develop currently uses the block-level #[allow] form which presumably passes CI on 1.88; the probe used a newer clippy, so the block form is fragile across clippy versions while the module-root #![allow] can’t be worse on 1.88 and is confirmed on newer clippy — verify clippy-clean under the pinned toolchain. General lesson: for clippy lints that fire on macro expansions, #[allow] at the expansion/call site often no-ops; put the #![allow] at the macro’s definition site. → clippy-disallowed-macros-allow-placement-2026-06-22.

2026-06-17

  • Implemented Flight execution_id parity on feature/BE-1595 to fix a dev regression: enabling MANDO_FETCH_STRATEGY=flight hard-broke the optimization/forecast algo runners (they always fetch with an execution_id) because py-mando’s flight guard HARD-ERRORED "Flight strategy does not support access_token/headers/execution_id; use REST". Root cause: REST applies execution-bound manual overrides server-side (look up flow execution by execution_id, inject each manual_overrides entry into matching DataPointFilters where manual_override.is_none()), but the Flight path had no equivalent and the client refused any execution_id/headers. Fix: (1) extracted the REST override block (was inline in mando-lib/src/app/route/query_data_route.rs) into shared mando_lib::app::execution_override::apply_execution_overrides(&FlowRepository, Uuid, &mut [DataPointFilter]) + pure inject_overrides; both REST get_data and the Flight server call it (DRY). (2) Added execution_id: Option<Uuid> to BOTH client QueryTicket (mandarrow-client/src/ticket.rs, +uuid dep) and server FlightTicket (mando-bess/src/flight.rs), #[serde(default)] for mixed-deploy safety. (3) Flight server applies overrides at the HANDLER layer: do_get resolves ticket.execution_id and mutates ticket.data_points BEFORE retrieve_stream (mirrors REST, NO trait change); MandoFlightService gained an Arc<FlowRepository> threaded serve→spawn_flight_server, and lib.rs restructured so each DB-mode branch builds flow_repository once and shares the Arc with both flight server + get_app. (4) Client guard (py-mando/src/polars.rs fetch_with_strategy) relaxed to error ONLY on non-empty access_token; execution_id+headers no longer block. headers dropped on flight path (trace stitching stays REST-only — deferred); access_token REST-fallback safety net also deferred (still errors). KEY correctness fact (verified): retrieve_stream honors DataPointFilter.manual_override identically to REST (shared get_retrieve_sql + :manual_override binding, repo.rs:497/repo_duckdb.rs:911; materializable path calls retrieve), so handler-layer injection is sufficient. Known limit: manual-override generation_time predating the DuckDB cache window for Permanent/DataPlatform DPs → streamed path returns MethodNotSupported where REST serves from permanent storage (repo_passthrough.rs:335). Deploy note: client changes are in the py-mando wheel, so this needs rebuilding the mando image AND the wheel AND the optimization/forecast consumer images. Plan: docs/superpowers/plans/2026-06-17-flight-execution-id-parity.md. → be-1595-flight-execution-id-parity-2026-06-17.
  • Found two latent FlowRepositorySqlite bugs while writing the test seed (local Sqlite dev/test path only, NOT Postgres prod; worth a ticket). Both in mando-lib/src/workflow/repository/flow_repository_sqlite.rs: (1) create_flow (:111) uses lazy stmt.query(params![...]) for an INSERT and drops the Rows without iterating → the SQL may never execute and the row never persists (rusqlite footgun; should use .execute()/.insert()). (2) flow_execution.started_at schema default is current_timestamp (stores TEXT) but get_execution reads column 2 as i64 (Utc.timestamp_micros), so a row created without an explicit integer started_at cannot be read back (InvalidColumnType TEXT vs i64). → be-1595-flow-repository-sqlite-bugs-2026-06-17.

2026-06-16

  • Documented how to actually turn ON the Arrow Flight client path in the BESS consumers (BE-1595). Runtime switch is py-mando env var MANDO_FETCH_STRATEGY (default rest; Flight only when == flight, case-insensitive; flight.rs:16, read per-fetch at polars.rs:224-236), plus MANDO_FLIGHT_HOST/PORT/PROTOCOL. Whole path is #[cfg(feature="flight")] but the dev wheel ships with -F flight, so enablement is purely the env var — no rebuild. CRITICAL gotcha: on a Flight fetch error py-mando SILENTLY falls back to REST (polars.rs:234), so no-errors ≠ Flight-in-use; must confirm via logs/metrics, and setting the var where the gRPC port is unreachable is a noisy no-op. Per-service ECS topology decides the action: bess-os-algo-optimization + bess-os-algo-forecast share the SAME ECS task as bess-os-service-mando (bess_os_ecs.tf, reach mando over localhost) → enabling = add 3 env vars (MANDO_FETCH_STRATEGY=flight, MANDO_FLIGHT_HOST=localhost, MANDO_FLIGHT_PORT=tostring(var.mando_flight_port)); DONE on feature/mando-arrow (commit d3db63e). bess-os-dashboard-trader is a SEPARATE task (trader_dashboard_ecs.tf, reaches mando over local.mando_domain HTTPS) — the Flight gRPC port (50051) is only a host port inside the bess-os task with NO NLB/target-group/SG path for the dashboard, so Flight there is NOT an env flip (needs gRPC-over-network plumbing); left on REST. Server side already in place: mando exposes var.mando_flight_port (default 50051) via MANDO_FLIGHT_PORT + gRPC port mapping (bess_os_ecs.tf ~211-213, ~307) + SG rule (security_group.tf). → be-1595-enabling-arrow-flight-consumers-2026-06-16.
  • CI/CD gotcha while deploying BE-1595 Arrow Flight to dev: enabling Publish Docker Dev on feature/* in the BESS Python consumer repos fails the pipeline ('Publish Docker Dev' job needs 'Test' job, but 'Test' does not exist) because .Publish’s needs: hard-depends on Test, whose rules only fire on MR/develop/rc/main/release — not feature/*. Fix: add optional: true to the Test need in .Publish. Applies to bess-optimization + bess-forecast-day-ahead; bess-trader-dashboard’s .Publish has no Test need so it’s unaffected (automation must guard on presence of - job: Test). Tradeoff: dev image ships without the test suite on feature branches (vs. adding /^feature/ to Test rules, which would block the image on unrelated failures) — optional chosen for disposable arrow-deploy branches. → be-1595-publish-docker-dev-feature-branch-test-need-2026-06-16.

2026-06-15

  • mando session: implemented Per-Flow Error Context Store (“OEF v2”) on feature/BE-3117 (worktree, NOT yet committed; replaces the committed OEF v1 on the same branch). Goal: emit EXACTLY ONE aggregate business ERROR per flow run retaining 100% of error data, replacing OEF v1’s dd_formatter ERROR→WARN demotion + thin repo-read aggregate (lost fidelity, coupled on magic strings). Mechanism, all in mando-lib: (1) ErrorRecord struct in tracing/tracing.rs — Clone + HAND-WRITTEN serde::Serialize that redacts HTTP req/resp bodies when http_context.sensitive; carries timestamp/step/error_kind/error_code/error_source(full anyhow {:#} chain)/message/http_context/validation_results/event_type/call_site/extra. (2) FLOW_ERROR_CONTEXT = tokio::task_local! Arc<Mutex<Vec<ErrorRecord>>>, sibling to TASK_CONTEXT. (3) record_error! macro in tracing/error.rs (#[macro_export]): in-flow-scope → build record (kind/code via mando_core::error::kind_code), push to store, emit INFO breadcrumb (never raw bodies); out-of-scope → fall back to mando_core::error! (ERROR). TT-muncher (__record_error_parse!/__record_error_emit!) because naive $(...)? optional-named-field hits local ambiguity. (4) Flow boundary in workflow/flow.rs: store created INSIDE the spawned task (task-locals don’t propagate into a detached tokio::spawn — same class as BE-1842 Datadog Observability); scoped FLOW_ERROR_CONTEXT.scope(store, TASK_CONTEXT.scope(ctx, service.oneshot(param)).instrument(span)); after scope returns (OUTSIDE span) flush_flow_errors(trace_id,&store,completed)->bool drains (mem::take) + emits ONE tracing::error! with flat facets flow.summary=true/flow.exec_id/flow.error.count/flow.error.codes/flow.error.failed_steps/event_type=Integration + full flow.errors JSON. FlowCompletionGuard holds same Arc, flushes on Drop (completed=false) for panic/cancel; mem::take+completed+empty-store early-return ⇒ no double emit. (5) Step-level: StepResult::log Err arm records via local record_step_error! using ErrorRecord::from(&ErrorWithStepStatus) (reuse precomputed kind/code), gated on !logged_at_site; From impl lives in workflow/mod.rs (struct fields module-private). dd_formatter ERROR→WARN demotion REMOVED. Invariant — record exactly once per flow, gated by ErrorWithStepStatus.logged_at_site (successor to the logged_at_failure_site flag of flow-step-log-message-dropped-2026-05-26): a site that records itself via record_error! MUST return logged_at_site=true (via message_logged/mark_logged) so step log() takes WARN and does NOT re-record; errors not logged at site are recorded by the step path (fires only when !logged_at_site). Gotcha fixed: 4 MarketNotFound sites (intraday/v2 + manual_schedule/v3 energy_bids_step & open_position_notification_step) had a PRE-EXISTING double-log (site error! + step step_error!, both ERROR); fixed via mark_logged() ⇒ one record. DD-facing: flow.failed_steps → flow.error.failed_steps (new flow.error.* flat hierarchy, continues BE-2272); event_type=Integration preserved. Constraint: record_error! in mando-lib NOT mando-core (mando-core can’t depend on mando-lib; needs ErrorRecord/FLOW_ERROR_CONTEXT/HttpErrorContext). Status: TDD two-stage review per task; all green (mando_core+mando_lib+mando_bess tests, py_mando compiles, clippy clean on touched files); uncommitted. Non-blocking follow-up: no E2E failing-flow test through the real spawned task; three out-of-scope infra error! calls (DelayCalculationFailed, cancel_queued_steps, PostFlowServiceFailed) can co-emit by design. → BE-3117 Per-Flow Error Context Store.

2026-06-02

  • mando-cli session: simulator env contract realigned to real images + arbitrary-extras passthrough (commit c3f0af7 on feature/simulator-runtime; 8 files, +440/−67; 813 tests pass, no new clippy; sole author andras.lederer, no co-author trailer). §A of mando-cli-simulator-runtime-2026-05-30 is now SUPERSEDED. Real orchestrator contract: SIMULATOR_DATABASE_{HOST,NAME,USERNAME,PASSWORD,SCHEMA,MAX_CONNECTIONS} (replaces POSTGRES_*/SIMULATOR_DB_SCHEMA); new SIMULATOR_START_DATE/_END_DATE/_SHUTDOWN_AT_END; per-runner SIMULATOR_<NAME>_HOST defaulting to simulator-<name>:<port> (replaces SIMULATOR_RUNNERS CSV); repo URL/branch/commit_hash MOVED off orchestrator onto the runners. Runners read unprefixed SIMULATOR_REPO_URL/BRANCH/COMMIT_HASH inside the container (assumption — verify when runner image lands); workspace .env keeps prefixed ${SIMULATOR_<NAME>_*} override convention. Schema renamed bess_simulation → simulator. Healthcheck reverted Python → curl; Gurobi reverted from file mount → GUROBI_LIC env var. Recommended orchestrator image: registry.gitlab.com/.../simulator:1.11.0-feat.2569583284.7d69a070. New extras passthrough: filter_extras in src/runtime/templates.rs:~116 filters any workspace .env key against MANAGED_SIM_ENV_KEYS const (templates.rs:~56); survivors auto-injected as KEY: "${KEY}" into every simulator service (sorted alpha, deterministic); each emits tracing::info!(env_var, simulator, "passing env var to simulator"). bootstrap.rs::ensure_all_generated reads .env via Adapter::Dotenv and threads filtered set into SimulatorGenCtx. sim-postgres aliasing pattern (broadly reusable): aliased the postgres image’s native env (POSTGRES_USER/PASSWORD/DB) to the orchestrator’s SIMULATOR_DATABASE_* so one .env override controls both — kills the silent-auth footgun where two services had independent credential defaults. Plus pg_isready healthcheck. Audit polish patterns kept: (1) derive UI summary strings from the actual invocation-args function (pre-fix bug: success row literally said "up -d --build --remove-orphans" while up_compose_args had omitted --build for sim → silent drift; fix: derive both from up_compose_args output); (2) single-source the image tag (SIM_IMAGE_TAG="dev") and service name (SIM_ORCHESTRATOR_SERVICE="mando-simulator") via placeholders (PH_SIM_IMAGE_TAG, PH_SIMULATOR_SERVICE) threaded by the renderer at templates.rs:354-355; (3) volume.rs error string uses COMPOSE_PROJECT_NAME/SIMULATOR_PROJECT_NAME consts; (4) -b/--build help text reworded (“no-op on dev, primarily for sim”); (5) MANDO_SIMULATOR_IMAGE-without-pull Warn row added to the simulator plan render. → mando-cli-simulator-env-contract-2026-06-02.

2026-05-30

  • mando-cli session: simulator runtime landed on feature/simulator-runtime (commit b244f2e, +1280/−240, 27 files, 2 new). Adds a second first-class Docker Compose stack mando-sim coexisting with the dev mando project — neither evicts the other. Six runners (forecast/optimization/execution/market/asset/post-delivery-market) generated from a single data-driven Rust list SIMULATOR_RUNNERS; own Postgres (sim-postgres, host port 5433, DB bess_simulation); dropped host_project() (mando-simulator is a service, only Project::SimulatorRunner is a real cloned repo + new Project::DEV subset excludes it); RunProfile gains compose_project/layers(); env merged via fill_build_args (not --env-file); Python healthcheck; Gurobi license as file mount. Two-stack ergonomics centralized in new src/runtime/service_stack.rs::select_profile — shared resolver for logs/exec/volume. Supersedes GitLab MR !2 / feature/BE-2256 (balint) which was hardcoded YAML. Defines the §A orchestrator↔runner env contract (CLI-defined, services implement): runner env (SERVER_PORT, APP_NAME, SIMULATION_MANDO_HOST/PORT, GITLAB_TOKEN, NEXUS_INDEX_URL, optimization-only GRB_LICENSE_FILE mount, GET /health); orchestrator env (per-runner SIMULATOR_<NAME>_REPO_URL/BRANCH/COMMIT_HASH, comma-separated SIMULATOR_RUNNERS, Postgres + SIMULATOR_DB_SCHEMA). Services (mando_simulator crate, simulator-runner image, six sim repos) do NOT exist yet — CLI leads the contract. Multi-perspective audit found + closed: HIGH (run_capture empty-files guard for spurious status row), 3 functional bugs (volume clear resolved against dev files, get not alias-aware, pull/status loud on un-cloned runner), 6 DRY violations (added shared SIMULATOR_PROJECT_NAME/PROFILE_NAME/sim-service-name consts; removed redundant SimRunner.repo; volume clear joined the single-resolver flow). Fresh-eyes re-review confirmed all closed. Process lesson: per-task verification used cargo test --bin mando which skips tests/ integration tests — masked a compile break in tests/up_compose_smoke.rs (RunProfile literals missing new compose_project field). Final-review caught it. Going forward: full cargo test, never --bin mando for green-light. Full cargo test: 785 passed, 1 ignored (docker-requiring smoke). Clippy unchanged from main baseline. Authored solely by andras.lederer (no co-author). End-to-end mando up -p simulator deferred until service images land. → mando-cli-simulator-runtime-2026-05-30.

2026-05-26

  • Investigation captured (diagnosed, not fixed): ErrorWithStepStatus::log(status, message) returned from any flow step silently drops message at the parent log site. StepResult::log() (mando-lib/src/workflow/mod.rs:150-229) destructures the Log variant with .. (lines 155-159), dropping message; only the generic wrapper string + flow.step.status + flow.step.execution_time reach tracing::error!/warn! (lines 196-225). Display impl (mod.rs:443) is #[error("status: {status}")] — message also dropped from stringification. status_or_error (mod.rs:231-239) collapses Log → Ok(status), losing it again. Top-level catch at mando-bess/src/workflow/flow.rs:289 only sees "status: Error". Error(anyhow) arm is correctly logged via error! (mod.rs:160-171) — only ::Log is broken. Tests at mod.rs:548-605 assert level + wrapper string only, never message payload — how the regression shipped. Affects all environments. Possible overlap with follow-up commits c945514e, 4c543cb1, 567373a5, 63d6fa69, 8f9297e4 on feature/BE-2272 — diff before patching. Secondary: mando-lib/src/app/dd_formatter.rs:122-124 record_error uses value.to_string() (Display only), but niche path. → flow-step-log-message-dropped-2026-05-26.
  • mando-cli session: triaged Gabi’s 2026-05-25 bug report against v0.4.0 compose-runtime rewrite (commit 6ba0d61). Both reported bugs CONFIRMED real. (1) src/runtime/templates/runconfig/build.yml:11 + build_dev.yml:18 use context: . — Compose resolves relative paths from the compose-file’s parent dir, so context becomes <project>/runconfig/ (no Dockerfile). Fix: context: ... (2) Mocked runconfig — mocked.yml only defines <service>-mocks, but up.rs:192 passes bare slug; reporter’s diagnosis was incomplete — the exact “Must specify either image or build” error originates in render_override (templates.rs:298-325) which emits a malformed <service>: stub per docker_target into .mando/override.builtin.yaml. The stub is normally dormant via profiles: ["{run_tag}"] (templates.rs:317) — that’s a load-bearing invariant. Cleanest fix: rename positional arg AND skip mocked entries in override generation. Smoke-test round (8de8f64) missed both: Bug 1 masked by image cache; Bug 2 not exercised with mando-mocked-algos set as default profile against fresh checkout. Fixes not yet committed. → mando-cli-v0.4.0-compose-bugs-triage-2026-05-26.

2026-05-22

  • mando-cli session: documented the local macOS (Apple Silicon) cross-compile recipe for producing a Linux x86_64 / WSL release binary of mando v0.4.0. Target x86_64-unknown-linux-musl (static-pie). Two gotchas captured: (1) Docker pulls the arm64 image on Apple Silicon → ring 0.17 C build fails with cc1: unrecognized command-line option -m64 → fix is --platform linux/amd64; (2) optional query feature has path deps into ../mando/.worktrees/BE-1595/* that Cargo reads during resolution even when disabled → must mount the poc/ parent dir. Verified binary in ubuntu:24.04 + alpine (mando --version → mando cli 0.4.0). Distinct from the CI build mirror; cross-linked both ways. → mando-cli-wsl-linux-build.

2026-05-18

  • Investigation captured (diagnosed, not yet fixed): Calculated and Virtual DPs leak rows past to in all four retrieval methods (retrieve/retrieve_at/retrieve_history/retrieve_client). Root cause in MandoServiceBase::handle_data_point_types (mando-lib/src/service_base.rs:94-159) — Virtual/Calculated branches lack a final filter_data_frame_by_range after Polars transformations. Two leak mechanisms: (A) convert_to_metadata upsampling explodes 1 row → N (convert_resolution.rs:39-95); (B) evaluate_expression Full-join/concat-group_by produces union of dep timestamps (evaluation.rs:62-72, 175-178). EvaluationMetaData.range is plumbed but only consumed by FillMissing. Proposed fix: trim per-DP at final Virtual/Calculated branches using evaluation_metadata[&dp_id].range. → calculated-virtual-dp-range-cutoff-bug-2026-05-18.

2026-05-06

  • mando-cli session: 5 fixes shipped + 1 design shelved.
    • 0be3458 feat: mando mock down with idempotent teardown (404 from remove_container = success). Pins canonical 7-step pattern for docker-backed lifecycle commands. → mando-cli-mock-down-idempotent-2026-05-06.
    • f8a54bf fix: WireMock healthcheck targets /__admin/health (200) instead of /__admin (302→404) using curl -fsS. Diagnostic technique: docker inspect --format '{{json .State.Health}}' (wget exit 8 = HTTP error). → mando-cli-mock-down-idempotent-2026-05-06.
    • 68bcc63 fix: mando status made read-only and bounded under 2s. New connect_readonly (single connect + 2s timeout, no retries, no ensure_database) and table_exists helpers in db/flyway.rs; sets statement_timeout = '2s' post-connect. Status commands must be pure reads. → mando-cli-status-readonly-2026-05-06.
    • 6b1f7c7 feat: yaml-driven build context to stop COPY-everything hangs. New build.context_includes: Vec<String> on ServiceBuildDef + new runtime/build_context.rs::build_filtered_tar used by both commands/build.rs and runtime/runner.rs. Caught + fixed runner.rs hard-coded "Dockerfile" regression in same commit. → mando-cli-build-context-filter-2026-05-06.
    • 302be50 feat: shipped context_includes defaults for all 5 app services in src/config/defaults/*.yaml. → mando-cli-build-context-filter-2026-05-06.
    • SHELVED: profile-driven build variants (dev runtime-only Dockerfile + cargo build --release pre-step vs release multi-stage chef Dockerfile). Captured design + open questions; no code shipped. → mando-cli-build-variants-shelved-2026-05-06.
  • Parallel-release CI restructure shipped to mando-cli-github-build-mirror (a939117 on master): split monolithic gitlab-release job into init-gitlab-release → build matrix (each matrix job uploads + links its own binary) → release + gitlab-finalize (checksums only). Linux/macOS no longer block on Windows aarch64. New “Parallel release flow (2026-05)” section in the doc.

2026-05-05

  • BE-2272 branch feature/BE-2272 (renamed from prior bugfix/BE-2023) — continuation of the BE-1842 Datadog Observability arc; flattens DD log JSON.
  • Removed the span.* namespace from formatter output: flow.exec_id, flow.context, step.name, step.connection now sit at the document root alongside error.* / http.* (symmetric DD facet layout).
  • Single-file change in mando-lib src/app/dd_formatter.rs (+295/-16): dropped serialize_entry("span", ...), added MapVisitor: tracing::field::Visit to collect event fields into serde_json::Map<String, Value>, span-fields-first / event-fields-second merge with explicit event-wins precedence.
  • Removed magic name injection in collect_span_fields (was outermost span name; unused in DD dashboards).
  • 11 unit tests added with a reusable capture harness (tracing::subscriber::with_default + custom MakeWriter over Mutex<Vec<u8>>); pattern reusable for future dd_formatter changes.
  • 262 workspace tests pass, 0 regressions; scope strictly contained to the formatter.
  • Plan in repo: docs/superpowers/plans/2026-05-05-flatten-log-fields-to-root.md.
  • Open follow-ups: DD dashboard column migration (@span.X → @X), execution.id vs flow.exec_id naming unification, dead ErrorCode derive arms in mando-lib-macro.
  • Branch state: local-only on feature/BE-2272, uncommitted.

2026-05-04

  • dc7b4259 chore: bumped Cargo.lock for py-mando after pulling in thiserror dep.
  • cd97fc35 fix: converted py-mando error logs to mando_core::error! macro so Python-binding errors carry typed error.kind (parity with Rust pattern from MR !481).
  • 923603f0 refactor: removed inline step.name/step.connection event fields now that the step span carries them — children inherit via dd_formatter root→leaf scope walk.
  • b8d3278d fix: added step.name and step.connection onto the step span at mando-lib/src/workflow/mod.rs:350 so child events inherit them in Datadog (see BE-1842 Datadog Observability).
  • c945514e fix: log step errors at the failure site to preserve real error.kind instead of generic wrapper at the catch boundary.
  • 4c543cb1 fix: downgraded parent flow error logs to warn when the child step has already logged the error (deduplicates Datadog noise).
  • ab622e29 fix: instrumented every tokio::spawn call with tracing spans so async tasks no longer drop trace context.
  • 5618603e fix: removed per-layer FilterFn from the OTel layer — the filter was suppressing events and breaking span field inheritance (root cause of BE-1842 Datadog Observability regressions).
  • 117f7b58 fix: foundation commit on bugfix/BE-2023 — deduped step error logging, upgraded OTel deps, threaded execution_id through FlowInfo.
  • All 9 commits are follow-ups to MR !481 (feat: error handling redesign, BE-2023) addressing reviewer feedback (Balazs Mracsko, Krisztian Fekete) and Datadog defects; iterative debugging captured in screenshots under /Volumes/bandi/coding/poc/mando/ (datadog-tab2-broken.png, dd-doublelog-1.png, dd-current-state.png, dd-log-expanded.png, etc.). Context: Agent Context.
  • Initialized activity log.

Graph View