Levandor

dionysus Phase 2 — Verified Findings (embeddings provider + entity-pipeline gap)

9 min read

For Agents

Two verified current-state findings about dionysus as of Phase 1 (built) + the drafted Phase 2 enrichment plan (docs/superpowers/plans/2026-06-09-dionysus-phase2-enrichment.md). These are observations, NOT final decisions — they bound what Phase 2 must fix, they do not prescribe the only fix. Plus a list of HIGH-severity bugs an adversarial re-review found in the drafted plan, to fix before/when executing it. Verified against the codebase at /Users/levander/coding/dionysus and spec v2 (docs/superpowers/specs/2026-06-08-dionysus-news-engine-design.md).

Two findings shape the dionysus enrichment (Phase 2) design: (1) the embedding step cannot use a Claude/Anthropic token because Anthropic ships no embeddings API, and (2) the north-star “who is this article about” feature has no data pipeline today — GDELT entities are parsed then thrown away, no article_mentions table exists, and the drafted Phase 2 plan writes an always-empty entity_ids payload to Qdrant.

Finding 1 — Claude/Anthropic has no embeddings API

Constraint: the embedding step CANNOT use a Claude token

Anthropic ships no embeddings model and no embeddings endpoint (verified 2026-06-09 via the claude-api skill). Everything goes through the Messages API, which is generation only. A Claude OAuth2 token (Authorization: Bearer, sk-ant-oat01-...) can therefore serve only a generation step — never embeddings.

What this means for dionysus

Pipeline stepCan a Claude/Anthropic token serve it?
Article vectorization (embedding → Qdrant)No — needs a real embedding provider
Entity salience / “who is this about” extractionYes (it’s a generation task)
Summarization / any text-generation enrichmentYes (generation)

The article-vectorization (embedding) step must come from a dedicated embedding provider. Spec §14 already names the two candidates:

  • OpenAI text-embedding-3-small (dim ≤ 1024; the drafted plan pins dimensions=512). This is what the Phase 2 plan currently wires (embed.rs OpenAIEmbedder, embed_model_version = "openai/text-embedding-3-small/512").
  • Voyage voyage-3-lite — Anthropic’s recommended embedding partner (Anthropic points at Voyage precisely because it has no first-party embeddings model).

Not a final decision

The candidate provider is not yet locked. The constraint is hard (no Anthropic embeddings exist); the choice between OpenAI and Voyage is still open. The Embedder trait in the plan (crates/dionysus-core/src/embed.rs) is deliberately swappable, so the provider can change without touching the enricher.

Implication for the two-token mental model

If dionysus ever holds a Claude token, it is for a generation enrichment (e.g. entity salience extraction — see Finding 2), and a separate embedding-provider key (OpenAI or Voyage) for vectorization. They are two different credentials for two different steps; they are not interchangeable.

Finding 2 — The “who is the article about” north-star has no data pipeline

Constraint: entity-filtered retrieval is currently NON-FUNCTIONAL

The core use case — correlating news about a person/entity (the Elon-tweet-frequency predictor) — has no working data path as of Phase 1 + the drafted Phase 2 plan. GDELT entities are parsed, used once for a boolean priority check, then discarded. No table stores them. The drafted Phase 2 plan writes entity_ids: [] (always empty) into Qdrant, so vectors cannot be filtered by entity.

Evidence (verified in code)

1. GDELT persons/orgs are parsed but only used for priority filtering, then dropped.

  • crates/dionysus-core/src/gkg.rs parses persons (COL_PERSONS = 12) and organizations (COL_ORGANIZATIONS = 14) into Vec<String> on GkgRecord.
  • crates/dionysus-core/src/ingest.rs matches_priority() (the persons/orgs chain is at ingest.rs:171) consumes them only to return a bool — does the record match a priority term? — and nothing propagates them further.

2. PreparedArticle carries no entity fields.

  • crates/dionysus-core/src/ingest.rs:26 — PreparedArticle { id, canonical_url, source, published_at, gdelt_record_id, language }. No persons, no organizations, no mentions.

3. The articles table has no entity columns.

  • crates/dionysus-core/migrations/0001_fastpath.sql — articles columns are id, canonical_url, source, published_at, first_seen_at, gdelt_record_id, gdelt_file_ts, language, dedup_cluster_id. No entity/mention column.

4. There is NO article_mentions table anywhere in the codebase.

  • grep -rilc "article_mentions" over the whole repo returns zero matches in every file.
  • Yet spec §6 defines it (article_mentions — article_id, normalized_mention = the raw GDELT string), and the fast path was specified to “record raw mentions.” That step was never built in Phase 1.

5. The drafted Phase 2 plan writes an always-empty entity_ids and indexes on it.

  • docs/superpowers/plans/2026-06-09-dionysus-phase2-enrichment.md line 1637 — the Qdrant point payload sets "entity_ids": [] (literal empty array).
  • Line 1698 — the news.enriched outbox payload also sets "entity_ids": [].
  • Line 1055 — ensure_collection() builds a keyword field index on entity_ids — i.e. an index over a field that is always empty. The plumbing for entity-filtered ANN exists, but it filters on nothing.
  • The plan’s “Resolved Ambiguity #4” says it intends to store raw persons/organizations as-is, but the actual code it specifies writes []. So even the raw-mention fallback is not wired.

Why this matters

dionysus’s entire reason to exist for the first consumer is “retrieve the news that is about entity X in window W.” With entity_ids always empty and no article_mentions table:

  • Qdrant ANN can find semantically similar articles, but cannot filter “about Elon Musk” at the vector layer.
  • Postgres has no mention rows to join on either.
  • The Elon-tweet-frequency correlation (the motivating consumer) is currently impossible to serve.

Not a final decision — this is the gap to close

This documents the current state, not a chosen design. Closing it likely needs at minimum: (a) carry persons/orgs through PreparedArticle, (b) build + populate the spec §6 article_mentions table in the fast-path tx, (c) populate entity_ids in the Qdrant payload from those mentions (raw strings now; resolved entity_ids after Phase 3 entity resolution + the versioned alias map). The exact shape is open. Note GDELT entities are raw strings, not disambiguated KB IDs — see GDELT entities are RAW strings, not disambiguated KB IDs. A generation step (Claude-token-eligible per Finding 1) for entity salience (“who is this primarily about”, not just mentioned) is a candidate enrichment, separate from the embedding step.

Phase 2 plan re-review — HIGH-severity bugs to fix before/when executing

A full adversarial re-review of the drafted plan (docs/superpowers/plans/2026-06-09-dionysus-phase2-enrichment.md) found these issues. They are plan bugs to fix on execution, not yet code (the plan isn’t built).

HIGH severity

Four HIGH-severity correctness/spec bugs in the drafted enricher

Each would ship a real defect if the plan is executed verbatim.

  • Unbounded retry storm on the failed path. drain_enrichment_batch re-claims rows with status IN ('detected', 'failed') (plan line 1581) but nothing increments an attempt count or transitions to a terminal state — a permanently-failing article (dead URL, embed always errors) is re-attempted forever every cycle. The 0002_enrichment.sql migration has no attempt_count column and there is no terminal stop. Fix: add attempt_count, increment on failure, stop at a max (terminal failed/dead).
  • Rows stranded in status='enriching' on any per-row error or crash. The claim UPDATE sets status='enriching' (line 1576), but every downstream ? (embed at line 1627, Qdrant upsert at 1642, tx commit) returns Err and never resets the row — and a process crash mid-enrich leaves it enriching too. There is no sweeper / lease-timeout reclaim, so stranded rows are never re-driven. (The Phase 2 backlog already lists a “reconcile sweeper” — it is a prerequisite, not optional, given this.) Fix: lease-timeout reclaim (enriching older than N → back to detected) + reset on error.
  • news.enriched emitted even when confirm-searchable fails — violates spec §8. At line 1644 confirm_searchable is checked, but a false result only logs warn! (line 1646) and execution continues to write the enriched revision + news.enriched outbox row (lines 1660–1712). Spec §8 specifies upsert-then-confirm: a vector that isn’t searchable must NOT be announced as enriched. Fix: on confirm failure, retry (the plan’s own “Resolved Ambiguity #3” says 3× with 500 ms), then treat as a Qdrant error — do not emit news.enriched.
  • Bitemporal clock split — two different “knowledge” timestamps for the same fact. The Qdrant payload sets knowledge_ts: Utc::now() (line 1635) while the Postgres enrichment row uses transaction_timestamp() (line 1664, read back via RETURNING knowledge_ts). These are different instants for the same enrichment, which breaks as_of reconstruction parity between the vector store and the source of truth. dionysus’s whole bitemporal premise is a single knowledge clock — see 10. Bitemporal from day 1 (NEW) and the Phase 1 realization in dionysus-phase1-ingest-skeleton. Fix: stamp the Qdrant payload with the same transaction_timestamp() value read back from Postgres, not a fresh Utc::now().

MEDIUM / correctness-and-DRY

  • cost_cap_usd_cents plumbed but never enforced. Threaded from config (plan line 2073) into EnrichConfig (carries it at line 1543), but drain_enrichment_batch (lines 1568–1719) never reads cfg.cost_cap_usd_cents — no token accounting, no cap check. It’s dead config. Fix: actually account embed tokens and stop at the cap (or delete the knob).
  • Fake semaphore concurrency (sequential loop). A Semaphore::new(cfg.concurrency) is created (line 1601) and a permit acquired per row (line 1605), but the per-row body (fetch → embed → upsert → commit, lines 1617–1715) runs inline and .awaited before the next iteration — no tokio::spawn. So enrichment is fully sequential regardless of concurrency; the semaphore does nothing. Fix: spawn per-row tasks holding the permit, or drop the concurrency pretense.
  • enrich_pool not capped at 5. Plan “Resolved Ambiguity #5” specifies enrich_pool = max 5 connections, but main.rs builds it with db::connect(&config.db_url) (line 2005) — the same helper as fast_pool (line 1998), inheriting its default pool size. The intended cap is not applied. Fix: build enrich_pool with an explicit max_connections(5).
  • No total timeout on the embed HTTP call. OpenAIEmbedder::new sets only .connect_timeout(...) (plan lines 711–714); embed() (called at line 1627) has no .timeout(...) — a stalled response after connect can hang the (sequential) enricher indefinitely. Contrast fetch.rs which does set .timeout(15s) (line 1193). Fix: add a request .timeout(...) on the embedder client.
  • subject_token duplicated (DRY). Defined in enrich.rs (plan line 1722) and already present in ingest.rs from Phase 1 — identical NATS subject-slug function. Fix: extract to one shared location in dionysus-core and call it from both.
  • dionysus — project overview + Phase 2+ backlog (slow path / enrichment / entity resolution)
  • dionysus-architecture-decisions — GDELT raw-string entities; bitemporal-from-day-1 (single knowledge clock); the why behind each choice
  • dionysus-phase1-ingest-skeleton — Phase 1 build: where gkg.rs parses persons/orgs, the single-knowledge-clock pattern realized, 0001_fastpath.sql schema
  • AiTrading — the empire vision; the Elon-tweet-frequency predictor is the motivating first consumer
  • LOG
  • TOPICS

Graph View

Backlinks