Behind tailscale serve, babylon’s /mcp endpoint returned 403 Forbidden while every plain axum route (/healthz, /readyz, /provision) returned 200. Root cause: rmcp 1.7’s StreamableHttpService runs a DNS-rebinding Host-header check as the very first step of request handling, and its default allowed_hosts allow-list is loopback-only — so any tailnet hostname is rejected before auth/method/content-type are even considered. Fixed by pin 164b49d (wowjeeez/babylon main) adding a BABYLON_ALLOWED_HOSTS config. See also the broader v1 Build Complete (2026-06-09) rmcp gotchas.
Known Issue (fixed at
164b49d)Any rmcp
StreamableHttpServicedeployment reached via a non-loopbackHost(Tailscaleserve, a reverse proxy, a real domain) returns 403 on/mcpout of the box, becauseStreamableHttpServerConfig.allowed_hostsdefaults to["localhost","127.0.0.1","::1"]. The fix below is reusable for any rmcp-1.7 service.
Symptoms
- Behind
tailscale serve,POST /mcp→ 403 Forbidden. - The plain axum routes mounted outside the rmcp service all returned 200:
/healthz,/readyz,/provision. - This split (only
/mcpgated) is the diagnostic tell — see Why only /mcp. - Server log, emitted before any tool/auth logic ran:
WARN rmcp::transport::streamable_http_server::tower: rejected request with
disallowed Host header (possible DNS rebinding attempt)
host=NormalizedAuthority { host: "babylon.taild4189d.ts.net", port: None }
Root Cause
rmcp 1.7’s StreamableHttpService performs a DNS-rebinding protection check as the first thing in request handling — in
transport/streamable_http_server/tower.rs::validate_dns_rebinding_headers — before method / Accept / Content-Type / auth are evaluated.
The check consults StreamableHttpServerConfig.allowed_hosts, which defaults to ["localhost", "127.0.0.1", "::1"]. Any Host not in that list is a hard 403. The tailnet name babylon.taild4189d.ts.net is not loopback ⇒ rejected.
For Agents — rmcp 1.7 API facts (verified against the SDK)
StreamableHttpServerConfigis#[non_exhaustive]— you cannot struct-literal-construct it with all fields; start from::default()and mutate via builders.- Field:
pub allowed_hosts: Vec<String>.- Builder
.with_allowed_hosts(hosts)— REPLACES the whole list (does not append).- Builder
.disable_allowed_hosts()— clears the list → allow-all.- Internal
host_is_allowed: an empty list ⇒ allow all; an allowed entry without a port matches any request port.
Why only /mcp was gated
/healthz, /readyz, and /provision are plain axum handlers mounted outside the rmcp service, so they never touch validate_dns_rebinding_headers. Only the nest_service("/mcp", svc) path runs through rmcp’s tower layer — which is why the perimeter looked half-broken (health green, MCP 403). If you ever see “infra checks pass but /mcp 403s,” suspect this check first.
Fix
Pin 164b49d on wowjeeez/babylon main adds a BABYLON_ALLOWED_HOSTS config (comma-separated).
Config parsing. A de_csv_vec serde helper accepts either a TOML array (config file) or a CSV string (env var), so BABYLON_ALLOWED_HOSTS=a,b,c and allowed_hosts = ["a","b","c"] both work.
Wiring — crates/babylon-server/src/serve.rs, start from the default config (which already contains the loopback entries) and then:
| Configured value | Action | Resulting behaviour |
|---|---|---|
any entry is a lone * | disable_allowed_hosts() | allow-all (empty list) |
non-empty (no *) | with_allowed_hosts(loopback_defaults + configured) | loopback plus the configured hosts |
| empty / unset | leave default untouched | loopback-only (backward compatible) |
Quick Reference — make /mcp reachable behind Tailscale
The deployment must set:
BABYLON_ALLOWED_HOSTS=babylon.taild4189d.ts.netEmpty/unset keeps the old loopback-only behaviour, so existing dev setups are unaffected. Use
BABYLON_ALLOWED_HOSTS=*only to deliberately disable the rebinding check (allow-all).
Important wiring details
- Extend the loopback defaults — don’t replace them — so loopback dev (
http://127.0.0.1:8787/mcp) keeps working alongside the tailnet name.with_allowed_hostsreplaces, so passdefaults + configured, not justconfigured.- Because an allowed entry without a port matches any port, you only list the hostname, not
host:port.
Testing note / reusable technique
To regression-test a Host-header-dependent path against a server bound to loopback, override DNS resolution in the client so the Host header carries the tailnet name while the TCP connection lands on 127.0.0.1:
let client = reqwest::Client::builder()
.resolve("babylon.taild4189d.ts.net", "127.0.0.1:PORT".parse()?)
.build()?;
// URL uses the tailnet hostname -> Host header = tailnet name,
// but the socket connects to loopback.- Positive test: with
BABYLON_ALLOWED_HOSTSset to the tailnet name, run a full MCPregisterhandshake through that client — it must succeed (no 403). - Negative test: with no config, the same request must return 403.
- Guard validated by reverting the
serve.rswiring and confirming the positive test fails with the exact 403 — i.e. the test genuinely exercises the guard, not a tautology.
For Agents — generalizable lesson
reqwest::Client::builder().resolve(name, addr)is the clean way to decouple theHostheader from the connection target in tests. Use it for any test that must drive a host-header / vhost / rebinding-protection code path against a loopback-bound server, without/etc/hostsedits or a real DNS name.
Status
- Pinned at
164b49d; gated —clippy -D warningsclean,cargo fmtclean, 50/50 tests (up from 43 at Code audit + hardening (2026-06-09)). - Handed to @deploy (Round 98 in
AGENT_HANDOFF.md) to redeploy and setBABYLON_ALLOWED_HOSTS=babylon.taild4189d.ts.netonpolymarket-infra. See babylon-deploy-notes for the host/env wiring. - Phase 1 round-trip (
/babylon:init code) unblocks once the env is live and the new pin is deployed — see babylon-migration.
Related
- babylon — project overview; the v1 Build Complete (2026-06-09) section collects the other hard-won rmcp 1.7 gotchas (object-root
outputSchema, rustc ≥ 1.88 floor, schemars 1.0, transport wiring) - babylon-design-decisions — why
tailscale serve(not an embedded node) is the v1 trust boundary, which is why this Host check bites - babylon-deploy-notes — where
BABYLON_ALLOWED_HOSTSmust be set on thepolymarket-infradeploy - babylon-migration — Phase 1 cutover that this fix unblocks