dionysus — Phase 1 Ingest Skeleton (COMPLETE)

Phase 1 (Ingest Skeleton) is BUILT, reviewed, audited, and committed

Working all-Rust walking skeleton: polls GDELT 2.0 → parses GKG 2.1 → ingests into a bitemporal Postgres (fast-path subset) in ONE transaction with a transactional outbox → a single-leader relay publishes news.detected to NATS JetStream → observable via OpenTelemetry (traces + metrics). 37 tests pass (unit + Docker-testcontainers integration), build + clippy clean. After the build was committed + pushed, a 3-lens adversarial audit (security / correctness / DRY) ran over the whole codebase and found + fixed real issues (a production relay-drain backlog bug + 4 security hardenings) — see Post-build security & correctness audit (2026-06-09). Re-smoke-verified against LIVE GDELT after the fixes: fetched a real file, ingested 1749 articles, relayed 1749 to NATS (caught up — was 1204 → 100 before the drain fix). Final commit 21332bd, pushed to wowjeeez/dionysus.

This note records WHAT Phase 1 delivered and the durable, reusable learnings the review + post-build audit caught. The design rationale lives in dionysus-architecture-decisions; the project map is dionysus. Plan of record: docs/superpowers/plans/2026-06-08-dionysus-phase1-ingest.md (don’t duplicate — read it for task-level detail). Spec: docs/superpowers/specs/2026-06-08-dionysus-news-engine-design.md.

What Phase 1 is

The minimum end-to-end vertical slice that proves the spine: feed → parse → bitemporal persist + outbox (1 tx) → relay → bus → telemetry. It is intentionally a subset of the full design — the fast path only, on the bitemporal store, with news.detected as the single published event. The slow path (full-text, embeddings, Qdrant), enrichment SCD-2, signals, entity resolution, and the as_of API are all Phase 2+ (see Phase 2+ backlog).

Executed via subagent-driven development — a fresh implementer per task, then a spec-review pass and a quality-review pass per task.

For Agents

Phase 1 ground truth (verified, not planned — reflects the post-build audit):

• Crates: Cargo workspace — dionysus-core (lib) + dionysus-ingest (bin), plus a dev-only dionysus-testsupport crate (shared testcontainer fixtures; NOT in the release binary — see audit finding #18).
• Migration: 0001_fastpath.sql creating articles, outbox, ingest_checkpoints.
• Tests: 37 passing (unit + testcontainers integration on PG16).
• Live smoke (post-fix): real GDELT file → 1749 articles ingested → 1749 relayed to NATS JetStream (caught up; was 1204 → 100 before the relay-drain fix #12).
• Final commit: 21332bd, pushed to wowjeeez/dionysus (post-build security/correctness/DRY audit fixes).
• Build + clippy clean.

Code structure

Phase 1 module layout (2 crates)

graph TD
    subgraph core["dionysus-core (lib)"]
        CFG["config"]
        DB["db pool +<br/>sqlx migrations"]
        URL["canonical URL rule"]
        AID["article_id"]
        PARSE["GKG 2.1 parser"]
        TX["fast-path ingest tx<br/><i>(persist + outbox, 1 tx)</i>"]
    end
    subgraph ingest["dionysus-ingest (bin)"]
        HTTP["GDELT HTTP client"]
        RELAY["outbox relay"]
        OTEL["OTel telemetry"]
        LOOP["poll loop"]
    end
    ingest --> core
    style core fill:#264653,stroke:#2a9d8f,color:#fff
    style ingest fill:#2d2d2d,stroke:#888,color:#fff

dionysus-core (library)

Module	Responsibility
config	Env-driven configuration
db	Postgres pool + sqlx::migrate!() migration runner (advisory-locked by Migrator itself — see gotcha #3)
canonical URL	The frozen dionysus URL-canonicalization rule (see gotcha #4)
article_id	Deterministic article identity derived from the canonical URL
GKG parser	GKG 2.1 CSV parser with empirically verified column indices (see gotcha #5)
fast-path ingest tx	Writes articles + outbox rows in one transaction; reads back first_seen_at inside the tx for the outbox knowledge_ts (see gotcha #8)

dionysus-ingest (binary / service)

Module	Responsibility
GDELT HTTP client	Fetches the GDELT 2.0 file over http:// only (GDELT’s data server has a broken TLS cert — gotcha #13), with a host-allowlist (validate_gkg_url, gotcha #14), a 512 MiB decompression cap (unzip_single, gotcha #15), redirect::Policy::limited(0), and connect_timeout per the empire gotcha
outbox relay	Single-leader publisher: SELECT ... WHERE published=false ORDER BY created_at FOR UPDATE SKIP LOCKED, publishes news.detected to NATS JetStream, marks published — all under lock for the whole tx (see gotcha #6). Now drained to empty per cycle (loop until caught up — gotcha #12)
OTel telemetry	Wires both a tracer provider and a meter provider (see gotcha #7)
poll loop	Drives fetch → parse → ingest → relay on a cadence; interval set to MissedTickBehavior::Skip (gotcha #16); records detect_latency on error cycles too (gotcha #17)

Schema — 0001_fastpath.sql

Table	Role
articles	Fast-path article rows; first_seen_at DEFAULT transaction_timestamp() — the single knowledge clock (gotcha #8)
outbox	Transactional-outbox rows (carry knowledge_ts); drained by the single-leader relay
ingest_checkpoints	Resumable cursor — commits in the same tx as the outbox row

Durable learnings (reference gotchas)

For Agents

These were surfaced by the review process during the Phase 1 build and are reusable across the empire — treat them as constraints, not opinions. Several are latent production bugs caught before they shipped. They complement the design-time gotchas in Durable gotchas (reference).

1. CREATE EXTENSION pgcrypto fails on Supabase’s least-privilege app role

A latent prod-deploy bug: the migration originally created pgcrypto for gen_random_uuid(), but Supabase’s application role lacks superuser, so CREATE EXTENSION is denied at deploy time. Fix: gen_random_uuid() is built into core Postgres since PG13 — no extension needed. Pin testcontainers to a prod-parity PG16 so the function is available, and never rely on superuser-only extension creation in migrations.

2. testcontainers-modules 0.12 Postgres::default() = postgres:11-alpine

Postgres::default() does NOT give a recent Postgres — it pins postgres:11-alpine. Fix: always set the tag explicitly — .with_tag("16-alpine") — for test/prod parity. PG-version gaps cause real bugs (e.g. INT4 vs i64 decode mismatches, and the gen_random_uuid() availability above).

3. sqlx::migrate!().run() already takes its own advisory lock

sqlx’s Migrator already acquires its own advisory lock by default when running migrations. Fix: do NOT wrap migrate!().run() in a manual pg_advisory_lock / pg_advisory_unlock sandwich — it’s redundant, and the manual unlock’s ? can discard a successful migration’s result (early-return on the unlock error throws away the Ok). Let the Migrator lock; call run() directly.

Supersedes a design-time assumption

dionysus-architecture-decisions and DEPLOY.md describe “sqlx advisory-lock migrations.” That’s correct — but the locking is built into sqlx, so the app must not add its own. The single-migrator requirement still holds.

4. URL canonicalization must preserve percent-encoding equivalence

/AB and /%41%42 are the same URL and must canonicalize identically. The frozen dionysus canonical-URL rule:

1. http → https
2. lowercase host, strip www. / m. / amp. subdomains
3. strip default ports
4. drop tracking params (utm_*, fbclid, gclid, mc_cid, mc_eid) by filtering the RAW query (preserve encoding — do not decode-then-reencode the whole query)
5. strip a /amp segment + trailing slash
6. lowercase the fully-unreserved-decoded path, with residual percent-hex uppercased

The percent-encoding equivalence is the subtle part: decoding only unreserved octets and uppercasing residual hex makes /AB and /%41%42 collapse to the same canonical form while leaving genuinely-encoded reserved characters intact. This rule feeds article_id, so getting it wrong silently splits or merges article identity.

5. GDELT GKG 2.1 column indices (0-based, empirically verified)

Verified against a live GDELT file + the GKG codebook:

Field	0-based index
SourceCommonName	3
DocumentIdentifier	4
V2EnhancedPersons	12
V2EnhancedOrganizations	14
V1.5Tone	15 (first comma-separated field)

Persons / orgs are ;-separated Name,Offset pairs.

The original plan guessed these indices WRONG

The Phase 1 plan guessed 4 / 5 / 13 / 14 / 17 — all off. The correct indices above were established empirically against a real file. Re-verify column indices against a live file whenever touching the parser; GKG documentation alone is not sufficient.

6. async-nats 0.49 JetStream — double-await + single-leader relay

Publish: publish_with_headers(...).await?.await? — a double-await. The first await? resolves the publish call (outer Result); the second await? resolves the returned PublishAckFuture (the broker ack). Dropping the second await means you never confirm the ack. Single-leader relay (no double-publish): SELECT ... WHERE published=false ORDER BY created_at FOR UPDATE SKIP LOCKED, holding the row locks for the whole transaction while publishing + marking published=true. SKIP LOCKED lets only one relay instance own a given row, so concurrent relays cannot double-publish.

7. OpenTelemetry Rust — wire BOTH providers or metrics silently no-op

Resolved versions: opentelemetry / opentelemetry_sdk 0.32, opentelemetry-otlp 0.32, tracing-opentelemetry 0.33. Gotcha: you must wire both a tracer provider and a meter provider, and set both as global. If you only set the tracer provider, metrics silently no-op (no error, just no metrics). Reads OTEL_EXPORTER_OTLP_ENDPOINT from env (consistent with the empire’s auto-injected OTel config — see 9. Observability = SigNoz via OTel (reuse the empire pipeline)).

8. Bitemporal single knowledge-clock

articles.first_seen_at DEFAULT transaction_timestamp() is the one knowledge-time authority. It is read back inside the same transaction that inserted the row, and that value flows into the outbox row’s knowledge_ts. Using transaction_timestamp() (stable for the whole tx, unlike clock_timestamp()) guarantees the article and its outbox event carry the identical knowledge timestamp — one DB authority, no skew. This is the concrete day-1 realization of 10. Bitemporal from day 1 (NEW).

Post-build security & correctness audit (2026-06-09)

A post-build re-validation pass found + fixed REAL issues — not theater

After Phase 1 was committed + pushed, a 3-lens adversarial audit (security / correctness / DRY) was run over the whole codebase. It caught a HIGH-severity production bug (the relay-drain backlog) and four security hardenings on the plaintext GDELT ingest path, plus five smaller correctness/robustness gaps and a DRY extraction. All fixed + pushed to wowjeeez/dionysus @ 21332bd; 37 tests still green; live-smoke re-validated (articles=1749 relayed=1749). These continue the durable-gotcha numbering above (#12+) and are reusable across the empire.

For Agents

The audit itself surfaced a meta-lesson worth keeping: an audit agent “helpfully” switched the GDELT fetch from http:// to https:// — which would have broken the live fetch (GDELT’s cert is broken; see #13). It was caught by manual curl verification, not by trusting the audit’s suggestion. Re-validate audit suggestions against reality before applying them.

12. Outbox relay MUST drain to empty per cycle, not one batch (HIGH — real prod bug)

The poll loop called relay::drain_once once per cycle. drain_once publishes at most batch_size (= 100) outbox rows. But each poll ingests a whole GDELT file (~1200-1750 articles → that many outbox rows). So the outbox grew ~1100+ rows every cycle, unboundedly, and news.detected fell further behind on every cycle — a slow-motion backlog that never recovers. The first smoke run showed the symptom in plain sight: articles=1204 relayed=100.

Fix: drain in a loop until drain_once returns fewer than batch_size rows (i.e. the outbox is caught up for this cycle). Re-smoke after the fix: articles=1749 relayed=1749 — confirmed caught up.

Durable lesson — the transactional-outbox drain rule

With a transactional outbox, the relay must drain-to-empty each cycle, not publish one batch. A single bounded batch per cycle is a backlog generator the moment producers outpace batch_size. This applies to every outbox relay in the empire (it’s the same pattern dionysus, and future services, will reuse).

13. GDELT is HTTP-ONLY — its data server has a BROKEN/mismatched TLS cert

curl https://data.gdeltproject.org/... fails with:

SSL: no alternative certificate subject name matches target host name 'data.gdeltproject.org'

The cert presented by GDELT’s data server does not match the host. reqwest + rustls (the empire’s TLS stack) rejects it — so any https:// GDELT fetch fails hard. The audit agent’s well-meaning “upgrade to https” change would have broken live ingest; manual curl caught it.

Durable lesson — verify, never assume, that a service serves valid HTTPS

GDELT v2 fetches MUST use http://. Do not assume a data provider serves working TLS — verify with curl against the real host. When you’re forced onto a plaintext path, defend it (next three findings) rather than papering over it with unverified TLS.

14. MitM/SSRF hardening on the plaintext ingest path (validate_gkg_url + no redirects)

The service fetches a GKG URL that it parses out of GDELT’s plaintext lastupdate.txt — a classic SSRF/redirect surface (whatever that file says, the service would fetch). Hardening added:

• Host-allowlist — validate_gkg_url requires the followed URL’s host to be exactly data.gdeltproject.org (and scheme http) before any fetch.
• redirect::Policy::limited(0) on the reqwest client — the server cannot redirect the fetch elsewhere.

Residual risk (accepted, documented)

Residual plaintext-MitM on the wire is GDELT’s own infrastructure limitation (it doesn’t serve a valid cert), not something dionysus can fix client-side. The allowlist + no-redirect + decompression cap (#15) bound the blast radius; the residual is recorded, not “solved.”

15. Decompression-bomb cap on the GDELT zip (unzip_single, 512 MiB)

unzip_single inflated the GDELT .zip with no size limit → a malicious/corrupt archive could OOM the process (DoS). Fix: read through a capped reader —

file.by_ref().take(MAX_DECOMPRESSED_BYTES + 1).read_to_end(&mut buf)?;
// error out if buf.len() > MAX_DECOMPRESSED_BYTES   (cap = 512 MiB)

The +1 lets the code detect “exceeded the cap” rather than silently truncating. Erroring (not truncating) is the correct DoS posture.

Durable lesson — always cap decompression of untrusted archives

Any code that inflates a remotely-fetched compressed payload needs an explicit decompressed-size cap with an error-on-exceed. “Read the whole thing into a Vec” is an OOM-DoS waiting to happen. (Stream-unzip + per-field caps are an even stronger Phase-2 follow-up — see the deferred list — but the cap covers the main risk.)

16. NATS subject injection from raw GDELT source_common_name

The subject was built as news.detected.<source>.<id> with <source> = the raw GDELT SourceCommonName — unsanitized. NATS treats . * > (and spaces) as subject-tree control characters, so a hostile/odd source name could pollute or inject into the subject hierarchy (e.g. wildcard tokens, extra levels).

Fix: a subject_token() slug applied only to the subject token:

• lowercase
• keep [a-z0-9-], replace everything else with _
• cap length at 64
• empty → unknown

The real, unmodified source is preserved in the NATS payload and the DB — only the routing token is slugged.

Durable lesson — sanitize any external string used in a NATS subject

External data must never flow raw into a subject. Slug it for routing; keep the real value in the payload/DB. (Same class of bug as SQL/log injection — the subject tree is a namespace an attacker can otherwise write into.)

17. detect_latency recorded on error cycles too (outages must be visible)

detect_latency was only recorded on the happy path, so a cycle that errored out was invisible in metrics — exactly when you most need the signal. Fix: record detect_latency on error cycles as well, with an outcome=error|ok attribute, so SigNoz shows failed cycles and their latency.

18. Poll interval set to MissedTickBehavior::Skip (no poll-storm)

tokio::time::interval’s default MissedTickBehavior is Burst — if a cycle runs longer than the interval, it fires the missed ticks back-to-back to “catch up”, causing a poll-storm against GDELT. Set to Skip so a long cycle just resumes on the next aligned tick.

19. Determinism + input-validation gaps (filename timestamp, junk URLs, empty filter)

Three smaller correctness fixes:

• gdelt_file_ts from the FILENAME — derived from the file’s YYYYMMDDHHMMSS prefix instead of a nondeterministic Utc::now() fallback. (This also closes the “should derive from the filename” follow-up that was previously in the Phase 2+ backlog.)
• prepare skips non-https:// document_identifiers — GKG rows whose DocumentIdentifier isn’t an https:// URL are junk for our purposes and are skipped.
• Startup warns on empty filter — if priority_filter_enabled is set but priority_terms is empty, the service would silently ingest zero rows. It now logs a startup warning. (A misconfiguration that’s otherwise invisible until you wonder why nothing’s being ingested.)

20. DRY — shared dionysus-testsupport dev-crate (and a Cargo dev-cycle note)

The two crates’ tests duplicated PG + NATS testcontainer boilerplate and a copy of the priority_off helper. Extracted into a dev-only crate dionysus-testsupport exposing fresh_pool() / priority_off() / start_nats(), with the PG_IMAGE_TAG pinned in one place (one source of truth for the PG16-parity tag — ties directly to gotchas 2). Deleted dionysus-core/tests/common/mod.rs and the duplicated helpers. Also removed now-orphaned percent-encoding / serde workspace deps.

For Agents — a Cargo dev-dependency cycle here is FINE

There is a cycle on paper: dionysus-core [dev-dependency] → dionysus-testsupport → dionysus-core [lib]. Cargo allows dev-dependency cycles — the dev crate is compiled only for tests and is NOT linked into the release binary (verified). Do not “fix” this cycle; it’s the intended way to share test fixtures without polluting the shipped artifact.

What’s NOT in Phase 1 (deferred)

Phase 1 is the fast-path skeleton only. The slow path, enrichment SCD-2, signals, entity resolution, the as_of API, CC-NEWS backfill, and all deploy wiring are Phase 2+ — the full carry-forward list lives in Phase 2+ backlog.

Deferred items recorded by the post-build audit (Phase 2 backlog, NOT blockers)

The audit explicitly accepted and recorded these as non-blockers (each has a reason it’s safe to defer):

• Stream-unzip + per-field / per-mention size caps — stronger than the single decompression cap (#15), which already covers the main OOM risk.
• deny.toml / CI cargo audit gate — the only current advisories are dev-only (tokio-tar, rustls-pemfile via testcontainers) and are not in the shipped binary; a CI gate is still worth adding.
• Relay holds its tx open across NATS publishes — a Phase-2 resilience refactor to a fetch → release lock → publish → mark pattern (so a slow broker doesn’t pin a DB transaction).
• Checkpoint pre-check — skip re-ingesting an unchanged GDELT file (compare against the last checkpoint before doing the work).
• RFC-4122 variant bits on article_id — the UUID is currently a raw blake3-128 content hash (not a spec-compliant UUIDv-anything); accepted as-is for now.

Related

• dionysus — project overview; current status now reflects Phase 1 complete + the Phase 2+ backlog
• dionysus-architecture-decisions — the design decisions Phase 1 implements (bitemporal, outbox, host tailscaled, all-Rust)
• polymarket-fetch — sibling empire service; origin of the connect_timeout gotcha baked into the GDELT client
• levandor-infra — the @deploy project that owns the Phase 2 deploy wiring
• LOG
• TOPICS