SFU Observability & Metrics: Instrumenting a Media Server You Can Debug

The hardest support ticket in real-time video is “my video froze for about three seconds around 14:02” — one subscriber, one publisher, one moment, inside a process that forwarded four hundred thousand packets in that window and kept no memory of any of them. This guide is part of the Media Server Architecture: SFU & MCU guide, and it covers the instrumentation that turns that ticket into a query: which counters a forwarding server must keep per participant, how to record them without slowing the packet path, how to export them without detonating your time-series database, and how to join a server-side counter to a client-side getStats() sample so the two describe the same three seconds.

The implementation goal is a metrics and logging pipeline where every quality complaint resolves to a named event — a layer demotion, a keyframe request storm, a send-loop stall — attributable to a specific (room, publisher, subscriber) triple, with numbers you trust because you verified them against the browser’s own view of the same stream.

Step 1 — Define the metric set and the SLOs it serves

Start from the failure you want to detect, not from the counters your media library happens to expose. An SFU has exactly four ways to disappoint a user: the picture freezes, the picture is permanently blurry, audio breaks up, or joining takes too long. Every metric you keep should be defensible as evidence for one of those. That discipline gives a small, opinionated set.

Throughput, per direction. Inbound bitrate per publishing track and outbound bitrate per subscribed track, sampled as a byte-counter delta over a 1 s window — the same cadence the browser side uses, which matters later when you correlate. Inbound tells you what the publisher actually delivered (a browser silently dropping its top simulcast layer under CPU pressure shows up here as a flat counter). Outbound tells you what you chose to forward.

Loss, jitter, RTT. These arrive from the far end, not from your own bookkeeping. RTCP Receiver Reports carry fraction-lost, cumulative packets lost, interarrival jitter, and the timestamps you need for RTT; transport-wide feedback packets carry per-packet arrival times when the peer negotiated them, which is a much richer signal than an 8-bit loss fraction every second — the distinction is worked through in Transport-CC vs REMB Feedback.

Repair and control traffic. PLI and FIR rate per publisher, NACK rate per subscriber, and RTX bytes as a fraction of primary bytes. A room where PLI rate climbs above roughly one per publisher per 10 s is a room that is spending its bitrate on keyframes rather than motion.

Forwarding decisions. Layer switch count per subscriber per minute, current forwarded rid, and the reason code for the last switch. Without the reason code you cannot distinguish healthy adaptation from a flapping selector.

Server internals. Forwarding latency — the microseconds between a packet entering the receive path and leaving the send path — plus CPU per active forwarded stream and send-loop lag. A healthy SFU forwards in the 1–3 ms range end to end, so a p99 that drifts to 15 ms is a real regression even while no user has complained yet.

Turn that set into objectives before you write any code, because the objective decides the aggregation. “99% of subscriber-minutes have zero freezes longer than 500 ms” needs a per-minute boolean per subscriber, which is a very different data shape from “p95 forwarding latency under 5 ms”, which needs a latency histogram. Encode the objectives as data so the exporter, the dashboards, and the alerts all read the same definition.

// SLO definitions drive what gets aggregated, at what resolution, and for how long.
// Keep them in one module so dashboards and alerts cannot drift apart.
const SLOS = [
  {
    id: 'subscriber_freeze',
    // A "good minute" is a subscriber-minute with no freeze longer than 500 ms.
    unit: 'subscriber_minute',
    source: 'server',                 // derived from forwarded-frame gaps, not the client
    objective: 0.99,                  // 99% of subscriber-minutes must be good
    window: '28d',                    // rolling error budget window
    burnAlert: { fastHours: 1, fastFactor: 14 }  // page when burning 14x budget for 1 h
  },
  {
    id: 'downlink_loss',
    // Loss is read from RTCP RR; 2% sustained is where Opus FEC stops hiding it.
    unit: 'subscriber_minute',
    source: 'rtcp',
    threshold: 0.02,
    objective: 0.995,
    window: '28d'
  },
  {
    id: 'forward_latency',
    // Server-internal only: receive path to send path, excludes the network.
    unit: 'packet',
    source: 'server',
    quantile: 0.99,
    thresholdMs: 5,                   // an SFU that forwards in 1-3 ms has headroom here
    window: '7d'
  }
];

// Every metric registered must name the SLO it serves, or it does not get registered.
// This is the cheapest defence against a metrics endpoint that grows without bound.
function register(metric) {
  if (!metric.slo || !SLOS.some(s => s.id === metric.slo)) {
    throw new Error(`metric ${metric.name} serves no declared SLO`);
  }
  return registry.add(metric);
}

The freeze objective is the one worth arguing about, and the argument is covered in depth by Alerting on Freeze and Packet-Loss SLOs, which works through burn-rate windows and the difference between a freeze the server caused and a freeze the last mile caused.

Step 2 — Instrument the forwarding path

The forwarding path is the one place in the process where you cannot be casual. A single 720p stream at 30 fps is roughly 1,500 packets per second; a node forwarding 5,000 streams handles millions of packets per second across its worker threads. Anything you do per packet is multiplied by that number, so the rule is: on the hot path, only increment preallocated integers and read a monotonic clock. No map lookups keyed by strings, no label formatting, no allocation, no logging.

Allocate a stats struct per forwarded stream when the stream is created, hang it off the stream object, and touch only that struct while packets flow. A separate 1 s tick — the same tick that drives your /metrics snapshot — walks the live streams, converts the raw counters into rates, and does all the expensive work of naming, labelling, and formatting.

The RTCP handler is the second instrumentation site, and it runs at a far lower rate — a Receiver Report per stream every few seconds — so it can afford to do arithmetic. This is where downlink loss, jitter and RTT enter the system, and where you should compute the per-interval loss fraction yourself rather than trusting the 8-bit fraction lost field, which quantises badly at low loss rates.

// Per-stream stats struct, allocated once when the forwarded stream is created.
// Plain numbers on a fixed-shape object: no Map, no string keys on the hot path.
function newStreamStats() {
  return {
    bytesIn: 0, bytesOut: 0, packetsOut: 0,
    pliSent: 0, nackRecv: 0, rtxBytes: 0,
    layerSwitches: 0, currentRid: 'f', lastSwitchReason: 'init',
    // Fixed-width latency histogram in microseconds; index chosen by a branch,
    // never by a division, so the hot path stays predictable.
    latBuckets: new Uint32Array(8),
    // RTCP-derived, updated a few times per second at most.
    lossFraction: 0, jitterMs: 0, rttMs: 0,
    prevLost: 0, prevExpected: 0
  };
}

const LAT_BOUNDS_US = [100, 250, 500, 1000, 2000, 4000, 8000]; // 8th bucket = overflow

// Hot path: called once per forwarded packet. Keep it branch-light and allocation-free.
function onPacketForwarded(s, bytes, tInNs, tOutNs) {
  s.bytesOut += bytes;
  s.packetsOut += 1;
  const us = Number(tOutNs - tInNs) / 1000;      // BigInt monotonic clock delta
  let i = 0;
  while (i < LAT_BOUNDS_US.length && us > LAT_BOUNDS_US[i]) i++;
  s.latBuckets[i] += 1;                          // histogram, not an average
}

// Cold path: one Receiver Report per stream every few seconds. Arithmetic is fine here.
function onReceiverReport(s, rr, nowNtp) {
  const dLost = rr.cumulativeLost - s.prevLost;
  const dExpected = rr.extendedHighestSeq - s.prevExpected;
  s.prevLost = rr.cumulativeLost;
  s.prevExpected = rr.extendedHighestSeq;
  // Compute loss over the real interval instead of reading the coarse 8-bit field.
  s.lossFraction = dExpected > 0 ? Math.max(0, dLost / dExpected) : 0;
  s.jitterMs = (rr.jitter / s.clockRate) * 1000;  // RTP timestamp units -> ms
  // RTT = now - LSR - DLSR, all in 1/65536 s NTP units.
  s.rttMs = ((nowNtp - rr.lsr - rr.dlsr) / 65536) * 1000;
}

// Record the reason alongside the switch: a count with no reason cannot be triaged.
function onLayerSwitch(s, toRid, reason) {
  s.layerSwitches += 1;
  s.currentRid = toRid;
  s.lastSwitchReason = reason;   // 'bwe_drop' | 'bwe_recover' | 'resize' | 'layer_lost'
}

The switch reason matters more than the switch count. A subscriber that demotes on bwe_drop and recovers 20 s later on bwe_recover is the system working; a subscriber alternating between the two every 4 s is a hysteresis bug in the selector, and the mechanics of getting that boundary right belong to Switching Layers Without Visible Glitches. Similarly, separate PLI counts by trigger, because keyframes requested on a deliberate layer promotion are expected cost while keyframes requested after packet loss are damage — a split explored in Keyframe Request Strategies in an SFU.

Step 3 — Export and aggregate without exploding cardinality

Here is where most media-server observability projects die. The natural instinct is to label every metric with participant_id, because that is exactly the granularity at which you debug. That instinct produces a time series per participant per metric per stream direction per layer, and a time-series database stores each unique label combination as its own object in memory. Twenty nodes, 2,000 concurrent participants each publishing one stream and subscribing to eight, times a dozen metrics, is well past a million active series — and because participant ids churn every call, yesterday’s million are still resident in the index.

The resolution is a strict separation. Metrics carry only bounded labels: node, region, codec, rid, direction, and bucketed room size. Unbounded identifiers live in logs and traces, where storage is linear in events rather than quadratic in label combinations, and where you can sample. Anything you genuinely need per participant in a dashboard becomes a top-K query over logs, not a metric.

Enforce the boundary in configuration rather than in code review. An allowlist of permitted label keys, a refusal list for the identifiers people keep trying to add, and a hard series cap that logs and drops rather than exhausting memory will survive staff turnover in a way that a wiki page will not.

; sfu.conf - observability section, read once at startup
[metrics]
listen = 127.0.0.1:9091        ; scrape endpoint on loopback, never the media NIC
path = /metrics
tick_ms = 1000                 ; matches the 1 s getStats cadence used on clients
; Only these label keys may be attached to any registered metric.
labels = node,region,codec,rid,direction,room_size_bucket
; Identifiers that are refused as labels; the exporter throws at registration.
forbidden_labels = participant_id,session_id,ssrc,room_id,user_agent
max_series = 200000            ; hard cap: log the offending name and drop past this
room_size_bucket = 2,5,10,25,50,100   ; bucket edges, so room size stays bounded

[histograms]
; Forwarding latency in microseconds. An SFU forwards in 1-3 ms, so bucket tightly
; at the low end - wide buckets hide exactly the regression you are looking for.
forward_latency_us = 100,250,500,1000,2000,4000,8000
; Send-loop lag: anything past 20 ms becomes downstream jitter for every subscriber.
send_lag_ms = 1,2,5,10,20,50

[logging]
format = json
; These keys go on every log line so a single query can reconstruct one session.
correlation_keys = session_id,participant_id,room_id,node,rid
; Per-packet-ish events must be sampled or the log bill exceeds the media bill.
sample_layer_switch = 1.0      ; rare enough to keep in full
sample_nack = 0.01             ; 1% is plenty to characterise a distribution
retention_days = 14

Two aggregation choices are worth calling out. First, export histograms rather than averages for anything latency-shaped; a mean forwarding latency of 2 ms tells you nothing about the 1% of packets that took 40 ms and caused the visible stutter. Second, pre-aggregate per-participant data into distributions at the node before export — “how many subscribers currently sit on the quarter-scale layer” is a bounded gauge, while “which layer is participant p-4471 on” is a log query. The concrete exporter wiring, recording rules and dashboard panels are the subject of Exporting SFU Metrics to Prometheus and Grafana, and the log-side join key is designed in Tracing One Participant Across SFU Logs.

Step 4 — Verification: prove the pipeline end to end

An unverified metrics pipeline is worse than none, because it produces confident wrong numbers during an incident. Verify in three passes: inject a known fault and confirm the metric moves, compare server counters against the client’s own view of the same stream, and confirm the correlation id actually joins the two.

Inject a known fault. Shape one subscriber’s link to 400 kbps for 10 s while a publisher sends three simulcast layers. You should see, in order: RTCP loss rising on that subscriber within one report interval, a bwe_drop layer switch within about 1 s, a PLI to the publisher, a keyframe, and outbound bitrate for that subscriber settling near the lower layer’s ceiling. If any link in that chain is missing from your dashboards, the instrumentation has a hole exactly where an incident will be.

Compare against the client. Poll getStats() on the subscriber at the same 1 s cadence and compare inbound-rtp.bytesReceived against the server’s bytesOut for that stream over the same window. Agreement within a few percent is expected; a persistent gap means packets are leaving the server and dying on the path, which is a network finding, not a metrics bug. The client-side reading technique — which reports matter and how they relate — is covered in Interpreting getStats() for Congestion Signals.

Confirm the join. The correlation id must be minted at exactly one place — the signaling layer, when a participant joins — and then travel everywhere: into the SFU’s per-stream struct, onto every log line, into the client’s stats beacon, and ideally into a trace span. Verifying the join is a two-query test: pick a random session id from the log, and confirm you can retrieve both the client’s stats samples and the server’s forwarding events for the same second.

// Verification harness: reconcile the client's inbound view with the server's
// outbound counters for one subscribed stream, over a shared 1 s window.
async function reconcile(subscriberPc, serverSnapshot, sessionId) {
  const stats = await subscriberPc.getStats();
  const client = { bytes: 0, freezes: 0, decoded: 0 };

  for (const r of stats.values()) {
    if (r.type === 'inbound-rtp' && r.kind === 'video') {
      client.bytes = r.bytesReceived ?? 0;
      client.freezes = r.freezeCount ?? 0;      // Chrome/Firefox; absent on Safari
      client.decoded = r.framesDecoded ?? 0;
    }
  }

  // serverSnapshot came from the SFU's /metrics or admin API for the SAME sessionId.
  const drift = client.bytes > 0
    ? Math.abs(serverSnapshot.bytesOut - client.bytes) / client.bytes
    : 1;

  // Over 5% drift means bytes left the server and never arrived: a path problem,
  // not an accounting problem. Under 5% and the two views agree.
  return {
    sessionId,
    drift: +(drift * 100).toFixed(2),
    verdict: drift > 0.05 ? 'PATH_LOSS_OR_MISMATCHED_WINDOW' : 'AGREES',
    // Cross-check the causal chain the fault injection should have produced.
    chain: {
      lossSeen: serverSnapshot.lossFraction > 0.02,
      switched: serverSnapshot.lastSwitchReason === 'bwe_drop',
      keyframed: serverSnapshot.pliSent > 0,
      clientRecovered: client.freezes >= 1 && client.decoded > 0
    }
  };
}

Run this reconciliation continuously against a small sampled fraction of live sessions — one in a thousand is enough — so the pipeline is validated by production rather than by a test you ran once. Because the sampled comparison also captures node identity, it doubles as a way to spot a single misbehaving node in a pool, which is the signal that should feed the scale-out decisions in Autoscaling SFU Nodes on CPU and Bandwidth.

Edge Cases & Browser Quirks

• Safari omits the stats you most want. WebKit’s inbound-rtp has historically not reported freezeCount, totalFreezesDuration or framesDropped consistently, and iOS builds lag desktop Safari. Derive freeze detection server-side from forwarded-frame gaps as the primary signal and treat client freeze counters as a bonus, or your Safari cohort will look artificially healthy.
• Firefox reports jitter in seconds where you may expect milliseconds. Firefox’s jitter in inbound-rtp is expressed in seconds per spec, and several dashboards silently plot it as milliseconds, making Firefox appear 1000× better than Chrome. Normalise units at ingest, not in the dashboard query.
• Chrome’s qualityLimitationReason flips to cpu before bitrate falls. On a thermally throttled laptop the publisher will drop its top simulcast layer with no network cause at all. Without recording the publisher’s limitation reason, a server-side “top layer missing” alert points at the network and wastes an hour; the client dump reading that confirms it is described in Reading chrome://webrtc-internals Dumps.
• RTCP report intervals are not fixed. The reporting interval scales with session bandwidth and participant count, so a large room’s Receiver Reports may arrive every 4–5 s rather than every second. Any rate you compute from RTCP must divide by the actual elapsed time between reports, never by an assumed 1 s.
• Clock skew between client and server breaks naive joins. Client wall clocks can be seconds off. Join on the correlation id and on the server’s receive timestamp for the stats beacon, not on the client’s own timestamp field.
• Relayed participants inflate RTT. A participant on a TURN relay pays an extra 20–40 ms one-way, so their RTT baseline is legitimately 40–80 ms higher than a direct peer’s. Label the candidate pair type on the session’s log line, or your RTT alert will fire on every relayed user.

Common Implementation Mistakes

• Labelling metrics with participant or session ids. The single most common cause of an observability stack that costs more than the media tier. Bounded labels in metrics, unbounded ids in logs and traces — no exceptions, enforced at registration.
• Averaging latency instead of bucketing it. Means hide the tail, and in real-time media the tail is the entire user experience. Export histograms and alert on quantiles.
• Counting layer switches without recording why. A switch counter alone cannot distinguish correct adaptation from a flapping selector, so the metric generates alerts nobody can action. Always carry a reason code.
• Formatting strings on the packet path. A single template literal or map lookup per packet costs more than the forwarding work itself at millions of packets per second. Increment integers on the hot path and name them on the tick.
• Trusting the RTCP fraction lost byte at low loss. It quantises to roughly 0.4% steps, so genuine 0.5% loss reads as noise. Compute loss from the cumulative counters across the actual report interval.
• Sampling client stats at a different cadence than the server. Comparing a 5 s client sample against a 1 s server window produces drift that looks like packet loss. Fix both at 1 s and align on the server’s receive time.
• Instrumenting only aggregate room health. Room-level averages stay green while one subscriber freezes, which is precisely the ticket you cannot close. Keep per-subscriber granularity somewhere, even if that somewhere is sampled logs.

FAQ

How much overhead should observability add to an SFU? Under 2% of CPU and no measurable change to forwarding latency. If your instrumentation is integer increments into a preallocated per-stream struct plus a 1 s aggregation tick, it disappears into the noise. If p99 forwarding latency moves at all when you enable metrics, you are doing work on the packet path that belongs on the tick.

Can I just label metrics by participant during an incident and remove it after? No — the series created during that window persist in the database index for the full retention period, and incidents are exactly when the system is least able to absorb a cardinality spike. Get per-participant detail from sampled logs and traces, which you can raise the sampling rate on temporarily without permanent cost.

Should freeze detection be server-side or client-side? Server-side as the authoritative signal, client-side as corroboration. The server knows precisely when it stopped forwarding decodable frames to a subscriber and why; the client only knows the picture stopped, and reports it inconsistently across browsers. Use the client’s freezeCount to validate that your server-side derivation matches perceived experience.

What is the minimum viable metric set if I am starting today? Five: outbound bitrate per subscriber, RTCP-derived loss fraction, PLI rate per publisher, layer switches with reason, and forwarding latency as a histogram. Those five, labelled only by node and rid and joined to logs by a session id, will diagnose the large majority of quality complaints before you add anything else.

Related: this sits under Media Server Architecture: SFU & MCU — pair it with Exporting SFU Metrics to Prometheus and Grafana for the export wiring, Tracing One Participant Across SFU Logs for the correlation id design, Alerting on Freeze and Packet-Loss SLOs for burn-rate alerting, and Selective Forwarding Unit Design for the forwarding internals these counters describe.