Levandor

dionysus Topics

4 min read

For Agents

Living index of themes for this project. Each H2 is a topic; bullets are wikilinks to related notes. Updated by obsidian-documenter when documenting work. Read by historian at bootstrap. Topics kept alphabetical.

Architecture & Decisions

Backtest-Safety & Bitemporal Model

  • dionysus — backtest-safety section (two time axes, as_of, why-retrofit-is-brutal)
  • dionysus-architecture-decisions — decision 10: bitemporal from day 1 (the existential look-ahead-bias fix)
  • dionysus-phase1-ingest-skeleton — bitemporal single knowledge-clock realized: first_seen_at DEFAULT transaction_timestamp() read back in-tx → outbox knowledge_ts
  • dionysus-phase2-findings — drafted Phase 2 plan has a bitemporal clock SPLIT to fix: Qdrant payload knowledge_ts = Utc::now() (line 1635) vs Postgres transaction_timestamp() (line 1664) — different instants for the same enrichment, breaks as_of parity between vector store and source of truth

Consistency & Messaging

  • dionysus-architecture-decisions — transactional outbox (closes the dual-write gap); per-event Nats-Msg-Id; NATS JetStream subject design + terminal news.failed
  • dionysus-phase1-ingest-skeleton — outbox + single-leader relay built (FOR UPDATE SKIP LOCKED); async-nats 0.49 double-await gotcha; post-build audit: relay must DRAIN-TO-EMPTY per cycle (one batch/cycle = unbounded backlog — HIGH prod bug, 1204→100 → 1749→1749); NATS subject-injection slug (subject_token())
  • dionysus-phase2-findings — drafted Phase 2 enricher bugs (fix on execution): news.enriched emitted even when confirm-searchable fails (only warn!, violates spec §8); unbounded retry storm (status IN ('detected','failed'), no attempt_count/terminal stop); rows stranded in enriching on error/crash (no sweeper/lease reclaim)

Cost

Data & Ingestion

  • dionysus-architecture-decisions — GDELT-spine hybrid + tiered full-text (live + CC-NEWS), filtered v1 corpus, fast/slow path; durable GDELT gotchas (scale, raw entities, format/license)
  • dionysus-phase1-ingest-skeleton — GKG 2.1 parser with empirically-verified column indices; frozen canonical-URL rule (percent-encoding equivalence); live-smoke-verified ingest; post-build audit: GDELT is HTTP-ONLY (broken TLS cert — verify, don’t assume https); gdelt_file_ts from filename; prepare skips non-https junk URLs; warn on empty priority_terms
  • dionysus-phase2-findings — the “who is the article about” entity pipeline is MISSING: gkg.rs parses persons/orgs but matches_priority (ingest.rs:171) uses them only as a boolean then discards; PreparedArticle (ingest.rs:26) has no entity fields; articles (0001) has no entity columns; NO article_mentions table anywhere (grep=0) despite spec §6; drafted Phase 2 plan writes Qdrant entity_ids: [] + indexes the empty field → entity-filtered retrieval (Elon-tweet use case) non-functional

Embeddings & Enrichment Provider

  • dionysus-phase2-findings — CONSTRAINT: Anthropic has NO embeddings API (Messages API is generation-only) — the article-vectorization step CANNOT use a Claude OAuth token; embeddings must come from OpenAI text-embedding-3-small (dim≤1024) or Voyage voyage-3-lite (Anthropic’s recommended partner). A Claude token can only serve a GENERATION step (e.g. entity salience). Provider choice not yet locked; Embedder trait is swappable
  • dionysus-architecture-decisions — embeddings via swappable API trait (no Python in the engine); cost ~cheap (the VM is the recurring cost, not embeddings)

Implementation & Build

  • dionysus-phase1-ingest-skeleton — Phase 1 (Ingest Skeleton) build report: what shipped, 37 tests, live smoke, + the durable reusable learnings the review caught; post-build security/correctness/DRY audit (final commit 21332bd): real bugs found + fixed; meta-lesson — re-validate audit suggestions against reality (curl caught the bad https change)
  • dionysus — current status (Phase 1 complete + post-build-audited) + the Phase 2+ backlog
  • dionysus-phase1-ingest-skeleton — DRY: shared dev-only dionysus-testsupport crate; a Cargo dev-dependency cycle is FINE (dev crate not linked into the release binary)
  • dionysus-phase2-findings — adversarial re-review of the drafted Phase 2 plan: cost_cap_usd_cents plumbed but never enforced (dead config); fake semaphore concurrency (per-row body runs inline/sequential, never spawned); enrich_pool built with default size not the spec’d max-5; no total .timeout() on the embed HTTP client (only connect_timeout); subject_token duplicated between enrich.rs and ingest.rs (DRY)

Deployment & Infra

  • dionysus — v1 managed-first (Qdrant Cloud + Supabase + NATS on existing e2-small) vs Phase-2 self-host
  • dionysus-architecture-decisions — phase-gated infra (decisions 4 & 5), the phase-gated @deploy contract, kind: job 15-min limit, Empire zero-backup/DR

Networking & Security

  • dionysus-architecture-decisions — decision 8: host tailscaled (not the unaudited embedded crate), honest egress (serving vs ingest plane)
  • dionysus-phase1-ingest-skeleton — post-build security audit: GDELT plaintext-ingest hardening — validate_gkg_url host-allowlist + redirect::Policy::limited(0) (MitM/SSRF); 512 MiB decompression-bomb cap (unzip_single); NATS subject-injection slug; residual plaintext-MitM accepted as GDELT’s infra limitation

Observability

  • dionysus-architecture-decisions — SigNoz via OTel (reuses empire pipeline), span-per-stage, trace context through NATS, fetch-hit-rate SLI
  • dionysus-phase1-ingest-skeleton — OTel 0.32 dual-provider gotcha (wire BOTH tracer + meter or metrics silently no-op); SIGTERM-flush follow-up; post-build audit: record detect_latency on error cycles too (outcome=error|ok) so outages aren’t invisible; poll interval → MissedTickBehavior::Skip (no poll-storm)

Storage & Transport

  • dionysus-architecture-decisions — bitemporal Postgres + Qdrant storage split, NATS JetStream bus, partition article_entities
  • dionysus-phase1-ingest-skeleton — schema 0001_fastpath.sql (articles/outbox/ingest_checkpoints); testcontainers/Supabase Postgres-version gotchas (PG16 parity, pgcrypto-denied, sqlx Migrator self-locks)

Graph View