Levandor

sqlx Runtime Type-Decode Bug Class + db-tests Remediation

3 min read

A real prod bug: a sqlx::query_scalar::<_, i64>("SELECT 1 …") type-decode mismatch that COMPILES FINE, is invisible to the offline test suite, and was masked by .ok().flatten() — silently halting all multi-wallet money routing fleet-wide for ~40 minutes. Plus the testcontainers db-tests suite that catches this bug class.

Bug class: sqlx runtime type-decode mismatch is invisible to compile + offline tests

A query that decodes a Postgres value into the wrong Rust type does NOT fail to compile and is NOT caught by sqlx’s offline/prepare checking in the normal build. It only fails at RUNTIME against a real Postgres. If the caller swallows the Result (.ok(), .unwrap_or(false)), the failure becomes a silent wrong answer. “Verified by reading the code” cannot catch this — only a real-Postgres test can.

Symptoms

  • The multi-wallet consumer silently SKIPPED EVERY ENTRY, fleet-wide.
  • ~40-minute real-money routing halt (no trades placed) with no error surfaced.
  • Every source read back as “not routed to any wallet.”

Root Cause

wallet_handles_source in crates/position_manager/src/consumer.rs used:

sqlx::query_scalar::<_, i64>("SELECT 1 FROM pmv2_source_wallets WHERE …")

Postgres types the integer literal 1 as int4, not int8. sqlx then fails to decode int4 into a Rust i64 at runtime. The call chain used .ok().flatten(), which converted the decode Err into None → treated as false → “source not routed” for every source. So the routing gate the multi-wallet fleet depends on was hard-false the moment it went live.

Two compounding faults:

  1. Type mismatch — SELECT 1 is int4; decoding as i64 fails at runtime.
  2. Error swallow — .ok().flatten() erased the DB error into a benign-looking false instead of logging/propagating it.

Fix (commit c9bbffb)

  • Stop scalar-decoding entirely: use sqlx::query(...).fetch_optional(...).await?.is_some() — existence is answered by row presence, no value decode.
  • Log DB errors instead of swallowing them, so a future query fault is visible rather than silently false.

Remediation: db-tests regression suite (commit 69a1cb8)

A feature = "db-tests" testcontainers suite: crates/position_manager/tests/wallet_fak_db.rs.

  • Spins up a real Postgres via testcontainers, applies all real migrations, and exercises the queries through their actual public fns (not re-implemented SQL).
  • Anchor test wallet_handles_source_regression: a mapped source → true, an unmapped source → false. Fails on the old query_scalar::<_, i64> code, passes on the fix. This is the guard for the whole bug class.

Running / CI

Run just the db-test: cargo test --features db-tests --test wallet_fak_db. A full cargo build --features db-tests is currently blocked by pre-existing drift in ops_commands.rs (Command::Enable / Command::Disable), so use the targeted --test form. Recommend wiring db-tests into CI — this is the only layer that catches runtime type-decode bugs.

Lesson

DB queries “verified by reading” are a review blind spot. Two durable rules:

  1. Prefer existence checks (fetch_optional(...).is_some()) over query_scalar::<_, i64>("SELECT 1 …") — avoid decoding literal-typed columns.
  2. Never .ok()/.unwrap_or(false) a DB result on a control-flow gate; log the error. A swallowed DB error on a routing gate = a silent fleet-wide halt.

Graph View

Backlinks