EU/HU Tender Scraping Topics

For Agents

Living index of themes for the EU/HU public-tender scraping project. Each H2 is a topic; bullets are wikilinks to related notes. Updated by obsidian-documenter when documenting work. Read by historian at bootstrap. Topics kept alphabetical.

Status at a glance (Phases 2+3 built 2026-06-15)

Direction: Hungary/EKR-only + a “does Levandor qualify to bid?” feature. The old EU-wide / pricing-intelligence thesis is superseded (see Pricing Intelligence (SUPERSEDED — out of scope)). Phase 1 shipped to master (gates green): TED removed end-to-end, tenders.source is 'ekr'-only, defense-sector OR-gate, eligibility prose captured into jsonb columns. Phases 2 & 3 built on master (NOT pushed) — see qualification-phase-2-3: editable company_profile singleton + Settings form (Phase 2); on-demand tender-qualify edge fn (DEPLOYED) + queue/verdict UI cloning /disclosure (Phase 3). Code-reviewed clean, web tests + build green. ⚠️ NOT live end-to-end — 2 operator blockers: ANTHROPIC_API_KEY not set in Supabase secrets (real assessments error) + web SPA deploy parked on wrangler login. Also: tenders table needs a real EKR ingestion verification run. Pending: Phase 4 null-criteria PDF/TED-notice enrichment; push Phases 2+3. Design docs are in the repo: docs/superpowers/specs/2026-06-09-hungarian-tender-qualification-design.md, docs/superpowers/plans/2026-06-09-tender-hu-only-phase1.md. Build narrative: LOG.

Architecture Divergence (spec vs shipped)

As of the 2026-06-09 refocus, the source-strategy-spec is also superseded on its core thesis

The spec describes an EU-wide, 5-source, pricing-model design. The shipped-and-refocused system is Hungary/EKR-only with a qualification feature instead. Treat both spec docs as historical intent, not as-built. Current intent lives in the in-repo docs (docs/superpowers/specs/2026-06-09-hungarian-tender-qualification-design.md, docs/superpowers/plans/2026-06-09-tender-hu-only-phase1.md).

The code diverged from the spec on every major axis. Read the specs as intent, not as-built.

Axis	Spec said (source-strategy-spec / implementation-guide)	SHIPPED (post 2026-06-09 refocus)
Storage	Standalone SQLite + Drizzle + FTS5	Supabase / Postgres (migrations 003+004 + 20260609192522_tender_hu_only, tenders_view, FTS search_vector)
Runner	Local process / CLI	Supabase edge function tender-sync (120s budget, service-role, 7-day lookback) + cli/backfill.ts
Sources	5 (TED API, TED Bulk XML, EKR Eljárástár, EKR Szerződéstár, CoRe HTML)	1: EKR Eljárástár JSON only (TED removed end-to-end; tenders.source CHECK = 'ekr')
Stages	normalize, dedupe, enrich-pricing, score	normalize + dedupe only (pricing stages out of scope)
Pricing model	per-buyer/per-CPV model from awarded values	none — superseded by the qualification feature

• source-strategy-spec - the strategy/thesis (undercut the AWARDED value, not the ESTIMATE); describes the 5-source, pricing-model design that was only half-built.
• implementation-guide - source/API validation (TED v3, ISO alpha-3 country codes, EKR specifics) verified 2026-04-26.
• LOG - the per-commit build narrative showing where reality landed.

Company Profile (qualification baseline)

Phase 2 — DONE (on master, not pushed)

Editable company_profile singleton (id integer pk default 1 check (id = 1), migration 20260615093853_company_profile applied live) + Settings CompanyProfileForm, hooks useCompanyProfile / useUpdateCompanyProfile (upsert {id:1}), en/hu i18n. The baseline the qualification assessment compares each tender’s eligibility criteria against.

Gotcha: focus-refetch clobbered unsaved edits → hydrate-once guard

const [hydrated,setHydrated]=useState(false); if(data&&!hydrated){setHydrated(true);setForm(data)} — initialize form state from server data exactly once so TanStack focus-refetches don’t overwrite in-progress edits.

• qualification-phase-2-3 - Phase 2 full writeup.
• LOG - Phase 2 DONE (2026-06-15 entry); original scope (2026-06-09 refocus entry).

Data Sources

EKR-only as of 2026-06-09 — TED removed

The single source is now EKR Eljárástár JSON. TED was deleted across the pipeline package, the tender-sync edge fn, and the web UI; tenders.source CHECK is 'ekr'-only (the ted_publication_number / ted_url fields were kept because EKR also populates them). See LOG (2026-06-09 refocus).

• LOG - DONE: TED removal + EKR-only source; new config.ts centralizes CPV codes, tag map, and the defense buyer allowlist/keywords.
• implementation-guide - historical: validated TED Search API v3 (POST /v3/notices/search, ITERATION pagination, HUN not HU) and EKR specifics. The TED adapter is now removed; EKR specifics remain relevant.
• source-strategy-spec - historical source taxonomy (TED Bulk XML, EKR Szerződéstár, CoRe scrape) — none of these are in scope under the refocus.

Defense Sector (DONE)

Shipped in Phase 1 (2026-06-09)

A defense-sector OR-gate inside the EKR source: cpvOk || defenseOk before yield, so defense-sector buyers are ingested + tagged defense_sector even with non-IT / null CPV (cpvCodes relaxed to allow empty). Buyer detection via an allowlist/keywords + accent-folding isDefenseBuyer in config.ts (catches Magyar Honvédség, Védelmi Beszerzési Ügynökség).

Defense ≠ CPV-35 in EKR

EKR has no CPV-35 / military procurement codes. This gate captures defense-sector civilian buyers only — not weapons tenders. Also, the seed roots HM EI Zrt and Katonai Nemzetbiztonsági Szolgálat are unconfirmed against live buyer names — validate on first ingestion.

• LOG - defense OR-gate + config.ts allowlist (2026-06-09 refocus entry).

Gotchas & Drift

Upstream APIs drift and silently break ingestion

Commit 7bb267e already had to adapt the pipeline once to TED Search API v3 and changed EKR API response shapes. A future re-run of the sync may need another drift fix. Treat any “no new tenders” symptom as a possible upstream-schema break, not an empty result.

• LOG - records the 7bb267e drift fix.

Pricing Intelligence (SUPERSEDED — out of scope)

Dropped on 2026-06-09. This is no longer the plan.

The “undercut the AWARDED contract value, not the padded ESTIMATE” thesis was superseded by the refocus to a qualification feature (see Qualification (LLM eligibility assessment)). The pieces below were never built and are now out of scope — kept here only as historical context:

• EKR Szerződéstár contracts ingestion (apikey=PSZT, ~135k awarded contracts; schema/contract.ts exported-but-unused)
• CoRe (Nyilvános Elektronikus Szerződéstár) scrape for price benchmarking
• TED Bulk XML historical backfill (TED itself is now removed)
• enrich-pricing stage (per-buyer/per-CPV historical-award model)
• score stage
• HUF → EUR normalization via ECB rates (grossValueEur / netValueEur fields)

• Strategy: why “undercut by 10%” is the wrong primitive - historical rationale for the dropped pricing model.

Qualification (LLM eligibility assessment — the new strategic core)

Replaces pricing intelligence as the value thesis

The new goal: “does Levandor qualify to bid on this tender?” — an on-demand LLM assessment of each tender’s eligibility criteria vs the editable company profile. Mirrors the CRM’s /disclosure analyzer pattern (queue + inline verdict UI).

Phase 1 capture — DONE

Free-text eligibility prose is now captured: the 6 alkalmassagiKovetelmenyReszletek fields + documents flow types → adapter → zod → upsert into new tenders.eligibility_criteria / tenders.documents jsonb columns (migration 20260609192522_tender_hu_only).

Phase 3 assessment — DONE (on master, not pushed)

On-demand tender-qualify edge fn (DEPLOYED) + queue/verdict UI cloning /disclosure. Tables tender_qualification_request (queue) + tender_qualification (one verdict/tender, upsert on tender_id), RLS authenticated SELECT/INSERT + service_role ALL (migration 20260615163827_tender_qualification). Edge fn: verifyAuth bearer-or-secret → newest-pending → claim-race → load tender+company_profile → empty-criteria short-circuit to unknown (no LLM) → else Claude claude-opus-4-8 thinking:adaptive + structured output. Frontend: lib/tender-qualification.ts, hooks (request + 8s polling w/ 15-min bound + verdict reads), TenderQualifyButton + QualificationVerdictPanel + eligibility-prose on TenderOverviewTab, en/hu via t().

Phase 3 LLM landmines

1. Structured-output schema: VERDICT_SCHEMA must avoid minimum/maximum/minLength/pattern/recursion or the raw fetch 400s.
2. Adaptive thinking: parse the content[] block where type==='text', NOT content[0] (the thinking block is placed first).

Blockers — NOT live end-to-end

1. ANTHROPIC_API_KEY not set in Supabase secrets → every real LLM assessment lands status='error'. Fix: supabase secrets set ANTHROPIC_API_KEY=... --project-ref mkofmdtdldxgmmolxxhc (same secret backs _shared/llm.ts).
2. Web SPA deploy parked on wrangler login / CLOUDFLARE_API_TOKEN → frontend not live. Edge fn not self-verifiable (browser-driven invoke); empty-criteria short-circuit untestable (all 101 live tenders have criteria).

Eligibility criteria are free-text Hungarian prose

The captured alkalmassagiKovetelmenyReszletek fields are unstructured Hungarian text — Phase 3’s LLM assessment consumes them against the company profile.

• qualification-phase-2-3 - Phase 3 full writeup (edge fn pipeline + landmines + blockers).
• LOG - Phase 3 DONE (2026-06-15 entry); eligibility-capture DONE Phase 1 (2026-06-09 refocus entry).

Schema & Database

• LOG - migrations 003_tender_pipeline.sql (tenders, cpv_codes) + 004_tender_tracking.sql (tender_tracking lifecycle enum watching → evaluating → bidding → decided, tenders_view, FTS search_vector) + 20260609192522_tender_hu_only (purged 42 TED rows + sync rows, rewrote source CHECK to 'ekr'-only, added eligibility_criteria / documents jsonb columns, recreated tenders_view). Applied live to project mkofmdtdldxgmmolxxhc. ⚠️ tenders table is now empty — EKR ingestion needs a verification run. Zod schemas schema/tender.ts (used) + schema/contract.ts (exported, unused — pricing dropped).
• qualification-phase-2-3 - migrations 20260615093853_company_profile (singleton company_profile, id integer pk default 1 check (id = 1)) + 20260615163827_tender_qualification (tender_qualification_request queue + tender_qualification one-verdict-per-tender, RLS authenticated SELECT/INSERT + service_role ALL). Both applied live to mkofmdtdldxgmmolxxhc. ⚠️ Note: 101 live tenders all have eligibility_criteria (0 null).

Status & Roadmap

Roadmap (Phases 2+3 built 2026-06-15)

• DONE (Phase 1, pushed to master): TED removal + EKR-only; defense-sector OR-gate; eligibility-criteria capture; live migration 20260609192522_tender_hu_only. Gates green.
• DONE (Phase 2, on master, NOT pushed): editable company_profile singleton (check id=1, migration 20260615093853_company_profile live) + Settings form; hydrate-once gotcha. See qualification-phase-2-3.
• DONE (Phase 3, on master, NOT pushed): on-demand tender-qualify edge fn (DEPLOYED, claude-opus-4-8 + structured outputs) + queue/verdict UI cloning /disclosure; migration 20260615163827_tender_qualification live. See qualification-phase-2-3.
• PENDING — Phase 4: null-criteria PDF / TED-notice enrichment.
• PENDING: push Phases 2+3.
• DROPPED: pricing intelligence (superseded).
• ⚠️ Operator blockers (not live): ANTHROPIC_API_KEY not set in Supabase secrets; web SPA deploy parked on wrangler login / CLOUDFLARE_API_TOKEN. Plus tenders table still needs an EKR ingestion verification run.

• LOG - DONE: discovery + tracking UI merged and routed (pre-refocus); Phase 1 refocus shipped. DROPPED: pricing intelligence.

Tracking UI (CRM)

• LOG - shipped web/src/pages/tenders/ (list + [tenderId] detail), Tender*.tsx components (Table, Row, Sheet, FilterSidebar, MetricsBar, DetailHeader, Overview/Activity/Related tabs), hooks useTenders / useTenderTracking / useTenderSync / useTenderSyncLog / useRelatedTenders, routes /tenders + /tenders/:tenderId wired into App.tsx + Sidebar.