position_manager’s self-watch feed (crate pmv2-onchain-watch, src/feed.rs, prod pin 719dc32) connects to the DRPC ogws endpoint then dies ~200 ms later, respawning every ~5 s with zero events delivered. Root cause: a rustls CryptoProvider panic on the first wss:// handshake inside the spawned feed task, because PM’s full dependency graph compiles rustls 0.23 with both the ring and aws-lc-rs features and never installs a default provider. The panic kills only the spawned task; run_feed’s &mut JoinHandle select arm resolves with a JoinError, logs feed task ended, and respawns. Same bug class as the earlier cohort_algo “Detector rustls CryptoProvider fix.” Backfill keeps working because it rides a separate reqwest path that never exercises provider resolution.
The key gotcha — a panicked spawned task RESOLVES the parent's JoinHandle select arm
A panic inside a tokio::spawn’d task makes the parent’s _ = &mut handle => select arm resolve (with a JoinError) — it is NOT only a clean-return signal. The original control-flow analysis assumed that arm only fires when the feed task returns normally, so it never looked for a panic. With panic = "unwind" (the default), the panic unwinds the spawned task, the JoinHandle becomes ready with Err(JoinError{panic}), the select arm fires, and run_feed logs feed task ended; backing off before respawn and respawns after respawn_backoff (5 s). The panic bypasses all tracing, so the crate emits zero logs about the actual failure — only the generic respawn line. This was the missing mechanism.
Symptoms
position_manager self-watch feed connects to DRPC ogws:
wss://lb.drpc.org/ogws?network=polygon&dkey=... (see polygon-wss-provider-drpc)
~200 ms later logs feed task ended; backing off before respawn (or feed channel closed; ...)
Respawns every ~5 s (= FeedConfig::respawn_backoff default), delivering zero events
Backfill still works: POLYGON_RPC_URLeth_getLogs path (separate reqwest call, see watcher-onchain-gap-backfill) keeps delivering fills in ~5 s batches
Probing the same ogws endpoint with python-websockets looks perfectly healthy
The split — backfill fine, live WSS dead — is the tell: the server is healthy, the bug is client-side and specific to the WSS/TLS path.
Root cause
A rustls panic on the first wss:// handshake inside the spawned feed task.
tokio-tungstenite 0.29 (tls.rs:126) builds its client TLS config via rustls ClientConfig::builder(), which resolves the process-default CryptoProvider.
rustls 0.23’s get_default_or_install_from_crate_features() only auto-picks a provider when exactly one of the ring / aws_lc_rs cargo features is enabled. With both enabled and none installed, it panics:
Could not automatically determine the process-level CryptoProvider … make sure exactly one of the ‘aws-lc-rs’ and ‘ring’ features is enabled
position_manager’s Cargo.lock compiles rustls 0.23.40 with BOTHaws-lc-rs (pulled transitively by quinn-proto, via reqwest http3) andring, and PM never calls install_default — so the panic fires on the first handshake.
The panic unwinds (default panic = "unwind"), killing only the spawned feed task. run_feed’s _ = &mut handle => arm resolves with a JoinError, logs feed task ended, and respawns after respawn_backoff (5 s default).
This single mechanism explains all three observations:
Observation
Explained by
~200 ms to failure
panic fires on the first handshake, immediately after connect
~5 s respawn cadence
FeedConfig::respawn_backoff default = 5 s
zero crate logs about the cause
panic bypasses all tracing; only the generic respawn line prints
backfill unaffected
eth_getLogs rides a separate reqwest path that never hits rustls provider resolution this way
For Agents — why a standalone-crate repro looks healthy
A single-provider build (ring only) does NOT panic and will NOT reproduce this. The dual-provider condition (ring + aws-lc-rs both compiled, none installed) only materializes in the full consumer dependency graph — quinn-proto / reqwest http3 drags in aws-lc-rs. So building the pmv2-onchain-watch crate on its own, or any service whose graph happens to resolve to a single provider, will show a healthy feed and hide the bug. The bug is a property of the aggregate dependency graph, not of the crate’s own code.
Why python-websockets looked fine
python-websockets uses a different TLS stack (OpenSSL). It never exercises rustls’s process-level provider resolution, so it connects cleanly. The DRPC ogws server is healthy — the failure is entirely client-side rustls.
Same class as the earlier cohort_algo fix
This is the same bug class as the earlier “Detector rustls CryptoProvider fix” in cohort_algo. Both of these install a provider as the first line of main():
cohort_algo/src/main.rs:57 — let _ = rustls::crypto::ring::default_provider().install_default();
position_manager was missing this install. The lesson generalizes: any binary whose dependency graph can pull in two rustls providers must install one explicitly, or the first rustls handshake panics.
Fix shipped this session (NOT committed)
Made the crate itself robust so it can’t depend on the consumer’s main() doing the right thing:
ensure_crypto_provider() in src/feed.rs — idempotent via std::sync::Once; installs ring as the process default only if none is installed. Called at the top of connect_once, before connect_async.
Cargo.toml — added rustls = { version = "0.23", default-features = false, features = ["ring", "std"] } to the crate. Keeps the crate single-provider (ring only), so the crate’s own builds stay panic-free.
TDD:
Unit test connect_path_installs_process_default_crypto_provider (red → green) — asserts the provider is installed on the connect path.
Integration regression test tests/feed_teardown_repro.rs — mock ogws server: a healthy ACK holds one connection open; a server-side Close drives the respawn. Locks in the teardown/respawn behavior.
Operator action — the crate fix only helps PM after a pin bump + redeploy
The crate change is inert for PM until one of:
PM bumps its pmv2-onchain-watch git pin — currently rev = 719dc32 in position_manager/crates/position_manager/Cargo.toml — and redeploys; or
Apply the immediate one-line hotfix in position_manager/crates/position_manager/src/bin/position-manager.rsmain():
let _ = rustls::crypto::ring::default_provider().install_default();
Caveat: end-to-end “panic gone” can only be observed in PM’s dual-provider graph. The crate itself is single-provider and never panicked, so a crate-level test will show green either way — it does not prove PM is fixed.
Lessons
For Agents — debugging takeaways
A tokio::spawn panic resolves the parent’s &mut JoinHandle select arm (with JoinError) — treat that arm as “task ended for any reason,” not “task returned cleanly.” If a select-arm-driven respawn loop spins with no error logs, suspect a panic in the spawned task swallowed by the JoinHandle.
rustls dual-provider panics are an aggregate-graph property.ring + aws-lc-rs both compiled with none installed → panic on first handshake. A single-provider standalone repro looks healthy and hides it. Check the consumer’sCargo.lock, not the crate’s.
A different TLS stack (OpenSSL via python-websockets) is not a valid client repro for a rustls provider bug — it never exercises the failing code path. Server-healthy + one-client-stack-healthy does not clear the rustls client.
Make libraries robust to the consumer’s main() — an idempotent ensure_crypto_provider() (via Once, install-only-if-absent) inside the connect path removes the dependency on every binary remembering the install_default line.