Site logo WebRTC & Real-Time Media Architecture

Monitoring STUN Binding Success Rates

A STUN node fails quietly. There is no connection to drop, no 5xx to count, and no session to expire — a Binding Request that gets no answer simply produces one fewer candidate in one browser, and the call either survives on a host pair or falls back to relay while the user notices nothing except a slower join. This guide is part of the STUN Server Deployment Strategies section of the WebRTC Protocol Stack & Signaling Servers guide, and it answers one operational question: what do you measure, from where, and at what threshold do you page someone when a region stops answering on UDP 3478?

Context & Trade-offs

The only metric that means anything is server-reflexive candidate yield: the fraction of gathering attempts that produced at least one srflx candidate, measured on the client, segmented by region and by network. Everything else is a proxy. A process-liveness check tells you turnserver is running; it does not tell you that a security-group edit two hours ago dropped inbound UDP while leaving your TCP health check green. A packets-received counter on the node tells you what arrived; it cannot count what a blackholed path silently discarded upstream.

Server-side telemetry is worth collecting but is structurally blind to the failure you care about. Coturn’s Prometheus exporter is relay-oriented — allocations, sessions, traffic bytes — and a STUN-only listener creates none of those. Its interesting counters sit near zero whether the node is perfectly healthy or completely unreachable, so a dashboard built on them looks identical in both states. Client-side yield is the inverse: it observes the exact path a real user takes, including the carrier, the corporate firewall, and the routing layer that chose the node.

Yield is not a single number, though, and treating it as one is how teams end up either paging at 3 a.m. for a locked-down enterprise office or missing a genuine regional outage inside statistical noise. Segment it before you threshold it.

Population Healthy srflx yield Notes
Consumer broadband, same-region node 98–99.5% The reference population; anything lower is your node
Mobile / CGNAT 97–99% Bindings refresh in under 30 s, so slow answers also cost you
Enterprise with UDP egress filtering 0–40% Correct behaviour, not an outage — these clients need TURN
Fleet-wide, all networks blended 96–98% Useful as a trend, useless as a page trigger

The practical consequence: build the alert on the reference population and exclude the rest explicitly. Sessions with iceTransportPolicy: 'relay' never gather reflexive candidates at all, and sessions that produced zero host candidates never got as far as a Binding Request — both belong outside the denominator, not inside it as silent failures.

srflx yield decomposition across 10,000 gathering attempts A full-width bar shows 96.8 percent of sessions producing a server-reflexive candidate and a narrow failing remainder of 3.2 percent. The remainder is expanded into a second bar split into UDP egress blocked at 2.1 percent, Binding timeout at 0.7 percent, and excluded sessions at 0.4 percent, each annotated with the operational verdict. Where the failing 3.2% goes — srflx yield per 10,000 gathering attempts srflx present — 96.8% (9,680 sessions) 3.2% UDP egress blocked — 2.1% timeout 0.7% excluded 0.4% Real users, not your node — route them to TURN over TCP 443 The SLO signal — node, security group, or path on 3478/udp Relay-only policy or zero host candidates — keep out of the denominator
Yield is only actionable once the structurally-zero populations are separated out.

Yield alone still misses half the story, because a node can answer every request and be too slow to be useful. Track time-to-first-srflx alongside it: measured from setLocalDescription(), a same-region self-hosted node should land a p50 under 30 ms and a p95 under 150 ms. A p95 crossing 400 ms almost always means the routing layer is steering some clients to a distant node, which is a placement problem covered in Choosing STUN Server Regions for Latency rather than a reachability one. On mobile paths the latency matters twice over: NAT bindings can refresh in under 30 s, so a reflexive candidate discovered late may describe a mapping that no longer exists by the time the remote peer checks it.

One more split is worth carrying in the data from day one: address family. A dual-stack client gathers an IPv6 host candidate that frequently needs no reflexive lookup at all, so on IPv6-heavy networks a portion of sessions legitimately produce a usable path with a thinner IPv4 reflexive record. If you collapse both families into one yield number, an IPv6 rollout at a large carrier reads as a slow multi-week degradation of your STUN node. Recording the family per candidate costs one field and makes that shift self-explanatory instead of a week of investigation.

Minimal Runnable Implementation

Two instruments, deployed together. The first is a per-session record emitted by every client; it is the SLO. The second is a synthetic prober that answers the question the SLO cannot — is the node down, or did the population change? — within seconds instead of within a traffic window.

// Attach BEFORE createOffer() so t0 is honest. One record per PeerConnection.
const probe = { t0: performance.now(), srflx: [], errors: [], host: 0 };

pc.addEventListener('icecandidate', (e) => {
  if (!e.candidate) return;               // null == end of gathering, not a candidate
  const c = e.candidate;
  if (c.type === 'host') probe.host++;    // 0 host => gathering never really ran; exclude later
  if (c.type === 'srflx') {
    probe.srflx.push({
      url: c.url || 'unknown',            // Chrome only: WHICH iceServers entry answered
      ms: Math.round(performance.now() - probe.t0),
      family: (c.address || '').includes(':') ? 'ipv6' : 'ipv4'
    });
  }
});

// The signal most teams never wire up: the server did not answer at all.
pc.addEventListener('icecandidateerror', (e) => {
  // 701      = no response from the STUN/TURN server (socket-level timeout)
  // 300..699 = a real STUN error response came back, so the node IS alive and routable
  probe.errors.push({ url: e.url, code: e.errorCode, text: e.errorText, port: e.port });
});

pc.addEventListener('icegatheringstatechange', () => {
  if (pc.iceGatheringState !== 'complete') return;
  navigator.sendBeacon('/metrics/stun', JSON.stringify({
    region: EDGE_REGION,                  // stamped by your edge, never client-supplied
    asn: EDGE_ASN,                        // separates "our node is down" from "this office blocks UDP"
    relayOnly: pc.getConfiguration().iceTransportPolicy === 'relay', // denominator exclusion
    hostCandidates: probe.host,
    srflxCount: probe.srflx.length,
    firstSrflxMs: probe.srflx.length ? probe.srflx[0].ms : null,
    stunErrors: probe.errors              // empty array on a clean gather
  }));
});

Three details carry the whole design. RTCPeerConnectionIceErrorEvent.errorCode === 701 is the difference between unreachable and broken: 701 means nothing came back on the socket, while any 300–699 code means a STUN error response arrived, which proves the path and the process are both alive. candidate.url is Chrome-specific but invaluable when iceServers lists more than one entry — without it you cannot attribute a candidate to a node, and a self-hosted outage masked by a public fallback looks like perfect health. And the event-driven emit costs nothing: there is no need to poll getStats() at 1 s intervals for this, because gathering is a one-shot phase and every fact you need arrives as an event.

The prober is deliberately dumb and deliberately hosted somewhere your production VPC is not — a probe that originates inside the same security group validates a path no user ever takes.

# /etc/stun-probe.conf — synthetic Binding prober, one per external vantage point
[probe]
targets=stun-use1.example.net:3478,stun-euw1.example.net:3478
family=udp4              # probe UDP explicitly; a TCP connect to 3478 proves nothing
interval=30              # seconds between Binding Requests per target
timeout=1500             # ms — ~50x a healthy 30 ms RTT, so only true silence trips it
retries=1                # one retransmit, then the probe is a failure
verify_mapped=true       # fail if MAPPED-ADDRESS is RFC 1918 (external-ip misconfigured)
[alert]
consecutive_failures=3   # 3 x 30 s = 90 s to page, short enough to beat most call volume

Keep the label set on the aggregated series small. Region, endpoint, address family, and a coarse network class are enough to answer every question the triage path below asks; adding raw ASN or user identifiers as labels multiplies series count without changing a single decision. Store the fine-grained fields on the raw beacon records instead, where you can query them after an alert fires. Sampling is safe too — reflexive yield is a population statistic, and 10% of sessions in a busy region still resolves a 1% shift long before a human reads the graph, while a low-traffic region needs every record it can get and should stay unsampled.

Binding request timeline: healthy answer versus retransmission to timeout Two lanes on a shared four-second time axis. The upper lane shows a single Binding Request answered at plus thirty-four milliseconds, yielding a server-reflexive candidate. The lower lane shows five retransmissions at zero, two hundred fifty, seven hundred fifty, one thousand seven hundred fifty and three thousand seven hundred fifty milliseconds against a blackholed region, ending in an icecandidateerror with code 701. One gathering transaction: answered node vs. silent region healthy node — us-east-1 BINDING RESPONSE at +34 ms srflx emitted; firstSrflxMs = 34 silent region — eu-west-1, 3478/udp blackholed TX1 TX2 TX3 TX4 TX5 RTO backoff 250 → 500 → 1000 → 2000 ms, no response to any icecandidateerror — errorCode 701, srflxCount 0 host candidates still gather, so iceGatheringState reaches 'complete' regardless 0 1 s 2 s 3 s 4 s
A dead node costs the client seconds of retransmission before it reports anything.

That backoff is why a silent region is expensive even when the call ultimately succeeds: the agent keeps retransmitting on a doubling RTO while the user waits, and the connection completes on whatever host or relay pair happened to work. It is also why the 1500 ms prober timeout is generous — a healthy node answers two orders of magnitude faster, so anything past that is silence rather than slowness.

Reproduction Steps & Debugging Log Patterns

  1. Deploy the prober against every regional endpoint and let it run for an hour to establish that all vantage points report a MAPPED-ADDRESS matching each node’s real public IP.
  2. Ship the client instrumentation behind a sampling flag and confirm the reference population settles in the 98–99.5% band. If it does not, fix the measurement before trusting the alarm.
  3. Blackhole one region exactly as a bad security-group rule would: iptables -I INPUT -p udp --dport 3478 -j DROP on that node. Do not stop the process — the whole point is that the process stays up.
  4. Watch what does not happen: a TCP health check on 3478 still succeeds if you left the TCP listener enabled, systemd still reports the unit active, and the node stays in rotation. This is the silent failure, reproduced.
  5. Confirm the prober pages within 90 s and that per-region yield for that endpoint collapses to near zero in the next reporting window, while the other regions stay flat.

A client in the affected region logs this shape — note that gathering still completes, which is why iceGatheringState is useless as a health signal:

// t+2ms    candidate host 192.168.1.24        typ host
// t+3ms    candidate host fe80::1c2b          typ host
// t+250ms  (no srflx yet — TX2 retransmit)
// t+3990ms icecandidateerror url=stun:stun-euw1.example.net:3478 errorCode=701
//          errorText="STUN binding request timed out"  address=0.0.0.0 port=0
// iceGatheringState: complete          // completes cleanly with zero srflx
// beacon: { region:'euw1', srflxCount:0, firstSrflxMs:null, hostCandidates:2 }

Compare that against a rate-limited or misconfigured node, which produces an error code in the 300–699 range instead of 701 — the node answered, it just refused. Cross-check any ambiguous case in a live dump; the candidate table and its timings are covered in Reading chrome://webrtc-internals Dumps.

Triage tree for a server-reflexive yield alert A yield alert branches on whether an off-network prober still receives a Binding Response. If no, the causes are a blocked UDP path or a misbound coturn process. If yes, the causes are a network population that blocks UDP or a polluted denominator from relay-only sessions. Triage: which failure the yield drop actually is ALERT srflx yield < 95% for 10 min Off-net prober still getting a Binding Response? no yes Node is not answering on 3478/udp from outside Server is fine the denominator moved security group or iptables DROP on udp 3478 coturn bound to wrong interface or external-ip drift one ASN blocks UDP needs TURN, not STUN segment by network relay-only sessions or 0 host candidates polluting the ratio
The prober's verdict splits the alert into two entirely different investigations.

Common Implementation Mistakes

FAQ

What success rate should I actually page on?

Page on the reference population only: consumer and mobile clients in a region, excluding relay-only and zero-host sessions. Below 95% sustained for 10 minutes is a real regression; below 90% is an outage. Keep the fleet-wide blended number on a dashboard as a trend line but never as a trigger — it moves whenever your user mix moves.

Should I trust the client metric or the synthetic prober?

Both, for different jobs. The prober is fast and unambiguous — 90 seconds to detect, and its verdict immediately tells you whether the node or the population changed — but it only tests the paths you thought to test. Client yield is slow and noisy but tests every real path, including the carrier that started filtering UDP last night. Fire the page from the prober, and use yield to size the blast radius and to catch the failures the prober cannot see. The same two-signal pattern applies to media SLOs, as in Alerting on Freeze and Packet-Loss SLOs.

Do the server’s own metrics have any value here?

Yes, but as corroboration rather than detection — they distinguish “no packets arrived” from “packets arrived and we dropped them on a quota”. Scrape them alongside the rest of your real-time fleet using the approach in Exporting SFU Metrics to Prometheus and Grafana, and remember that a STUN-only listener generates none of the allocation counters a relay does.

Related: return to STUN Server Deployment Strategies, and read Self-Hosting Coturn STUN vs Public STUN for the node you are monitoring and WebRTC over CGNAT for why mobile yield behaves differently from broadband.

Choosing STUN Server Regions for Latency Signaling Self-Hosting Coturn STUN vs Public STUN Servers Signaling WebRTC Protocol Stack & Signaling Servers: Architecture, Implementation & Production Configs Signaling Connection Recovery & ICE Restart: Surviving Network Changes Mid-Call Signaling Disconnected vs Failed ICE Connection States Signaling Handling Wi-Fi to Cellular Network Handover Signaling