Site logo WebRTC & Real-Time Media Architecture

Alerting on Freeze and Packet-Loss SLOs

A freeze is the only media defect users report unprompted: bitrate charts stay green, RTT looks fine, and someone still writes “it froze for three seconds”. This guide is part of the SFU Observability & Metrics guide, and it settles one decision — how to turn freezeCount, totalFreezesDuration and downlink loss into an objective you can page on, without waking anyone because a single participant sat too far from their access point.

Context & Trade-offs

The two counters that matter live on inbound-rtp in the subscriber’s getStats(). freezeCount increments when the inter-frame delay exceeds the larger of three times the recent average inter-frame interval or that average plus 150 ms; totalFreezesDuration accumulates the seconds spent in that state. Both are monotonic totals, so every useful number is a delta between two samples. Reading them alongside the loss and jitter fields is the same discipline described in Interpreting getStats() for Congestion Signals — the difference here is that you are not diagnosing one call, you are building a population statistic.

The naive alert is a threshold on the instantaneous value: page when median freeze duration across the fleet exceeds 200 ms, or when downlink loss exceeds 2%. Both fail in the same way. At 5,000 concurrent participants there is always someone freezing — a laptop thermally throttling, a phone walking out of Wi-Fi range, a browser tab backgrounded on a busy machine. A threshold on the aggregate fires constantly at low traffic and never at high traffic, because the same absolute number of unhappy users is a different fraction of a different denominator.

The fix is to make the unit of measurement a participant-minute and the statistic a ratio. A participant-minute is good when the subscriber accumulated less than 500 ms of freeze in that minute and its downlink loss stayed under 2%; otherwise it is bad. The 500 ms figure is not arbitrary — below roughly 400 ms a gap reads as a stutter rather than a stop, and above 500 ms viewers describe it as the video “stopping”. The 2% loss threshold is the point at which Opus in-band FEC stops concealing audio gaps, a boundary worked through in Tuning Opus Bitrate and FEC for Lossy Networks.

State the objective as a fraction: 99% of participant-minutes are good, measured over a rolling 28 days. That converts directly into an error budget. At 5,000 concurrent participants the fleet produces 300,000 participant-minutes per hour, 7.2 million per day, and roughly 201 million over 28 days. One percent of that is a budget of about 2.0 million bad participant-minutes — which sounds enormous until you notice that a single node stalling its send loop for an hour, affecting 400 subscribers, spends 24,000 of them in one incident.

Keep freeze and loss as two separate objectives with two separate budgets, even though a single minute can fail both. They fail for different reasons and on different timescales: loss is continuous, visible in RTCP within one report interval, and often recoverable by dropping a simulcast layer, whereas a freeze is discrete, arrives seconds late in the client beacon, and usually means a keyframe never landed. Folding them into one composite number produces an alert whose remediation is ambiguous, which is the same as no alert. Two budgets also let you set different objectives — 99% for freeze, 99.5% for loss is a reasonable starting pair, because sustained loss is both rarer and more directly your fault.

Now put the bad Wi-Fi link in the same units. One participant freezing continuously for a full hour contributes 60 bad participant-minutes: 0.003% of the 28-day budget, and 0.02% of that hour’s 300,000. It is arithmetically incapable of threatening the objective, and any alert that fires on it is measuring the wrong thing.

Error budget depletion across one day A chart of remaining 28-day error budget from 100 percent down to 95 percent across 24 hours. The line drifts gently downward at about a third of the sustainable rate. At hour six a single participant freezing for a whole hour produces a notch too small to see, annotated as 0.003 percent of budget. At hour fourteen a node send-loop stall burns 14.4 times the sustainable rate for one hour and drops the line by 2.1 percent. A dashed reference line shows the one times burn rate that would exhaust the budget exactly on schedule; the actual line finishes the day above it. 28-day error budget remaining, hour by hour 100% 99% 98% 97% 96% 95% one bad Wi-Fi link, 60 freeze-minutes 0.003% of budget — no visible notch node n-07 send-loop stall 14.4x for 1 h = 2.1% of budget 400 subscribers, 24 000 bad minutes steady state burns 0.3x — the fleet is never freeze-free 0 4 8 12 16 20 24 h budget remaining 1x reference — exhausts the budget exactly on schedule
The day ends above the 1x reference, so the objective is safe — yet the 1 h burn rate during the stall still deserved a page, which is why alerts read the slope and not the remaining balance.

Burn rate is that slope expressed as a multiple: the observed bad fraction divided by the allowed bad fraction. At a 99% objective, an hour in which 14.4% of participant-minutes were bad is a 14.4x burn — it consumes about 2% of a 28-day budget in sixty minutes. Alerting on multiwindow burn rates (a fast pair at 14.4x, a slower pair at 6x and 3x) gives you both quick detection of sharp incidents and eventual detection of slow bleeds, without the flapping of a raw threshold.

Two guards convert this from a formula into something safe to page on. The first is a minimum denominator: refuse to evaluate the ratio unless the window holds at least 500 participant-minutes. Without it, a quiet night with 10 concurrent participants means one unhappy user is 10% of a 5-minute window — a 10x burn rate produced by one person’s router. The second is a per-participant clamp: no single participant may contribute more than 5% of a window’s numerator. Together they make the alert structurally incapable of firing on an individual, which is exactly the property you want, because individual last-mile failures — including the mid-call path changes covered in Handling Wi-Fi to Cellular Network Handover — are permanent background noise, not incidents.

Minimal Runnable Implementation

// One 60 s bucket per (sessionId, subscribed track), fed by a client beacon that
// batches ten 1 s getStats() samples, plus the SFU's own forwarding-gap counter.
const FREEZE_BUDGET_MS = 500;   // >500 ms of freeze in a minute marks the minute bad
const LOSS_THRESHOLD  = 0.02;   // 2% downlink loss: where Opus FEC stops concealing
const MIN_DENOMINATOR = 500;    // never evaluate a ratio on a thinner sample
const MAX_SHARE       = 0.05;   // one participant may be at most 5% of the numerator

// prev/now are inbound-rtp snapshots 60 s apart. serverGapMs is the longest gap the
// SFU recorded between forwarded decodable frames for the same stream in that minute.
function classifyMinute(prev, now, serverGapMs) {
  // totalFreezesDuration is in SECONDS per spec; freezeCount is a plain counter.
  const freezeMs = ((now.totalFreezesDuration ?? 0) - (prev.totalFreezesDuration ?? 0)) * 1000;
  const froze = now.freezeCount !== undefined
    ? freezeMs > FREEZE_BUDGET_MS
    // Safari reports neither counter: fall back to a decoded-frame deficit.
    : (now.framesDecoded - prev.framesDecoded) < now.nominalFps * 60 * 0.5;

  const dLost = now.packetsLost - prev.packetsLost;
  const dRecv = now.packetsReceived - prev.packetsReceived;
  const loss  = dLost + dRecv > 0 ? dLost / (dLost + dRecv) : 0;

  return {
    bad: froze || loss > LOSS_THRESHOLD,
    // Attribution decides who gets paged, so compute it here, not in the query.
    // A server gap over 200 ms means WE stopped forwarding decodable frames.
    cause: serverGapMs > 200 ? 'server' : 'last_mile',
    freezeMs, loss
  };
}

// Aggregate one evaluation window (5 m, 1 h, 6 h, 1 d) into a burn rate.
function burnRate(minutes, objective = 0.99) {
  const total = minutes.length;
  if (total < MIN_DENOMINATOR) return { burn: 0, verdict: 'INSUFFICIENT_SAMPLE', total };

  // Clamp each participant's contribution before summing, so no individual
  // can move the ratio no matter how catastrophic their own link is.
  const cap = Math.max(1, Math.floor(total * MAX_SHARE));
  const perParticipant = new Map();
  for (const m of minutes) {
    if (!m.bad) continue;
    perParticipant.set(m.participantId, (perParticipant.get(m.participantId) ?? 0) + 1);
  }
  let numerator = 0;
  for (const count of perParticipant.values()) numerator += Math.min(count, cap);

  const badFraction = numerator / total;
  return {
    burn: badFraction / (1 - objective),   // 14.4x means 2% of a 28 d budget per hour
    verdict: 'EVALUATED',
    total,
    participants: perParticipant.size,
    // Only shared-cause degradation is worth a page; the rest is a per-user ticket.
    serverShare: minutes.filter(m => m.bad && m.cause === 'server').length / (numerator || 1)
  };
}

The evaluator that consumes those numbers is a small state machine, and writing it as one keeps the alert honest about hysteresis: a burn rate that crosses 14.4x for nine seconds is noise, and a page that clears the instant the ratio dips is worse than no page at all.

Alert evaluator state machine Five states. Steady is the resting state below one times burn. Crossing three times over one day moves to Watch, which files a ticket. Crossing six times over six hours moves to Burning, which escalates. Crossing 14.4 times on both the one hour and five minute windows moves to Paging. Paging returns to Steady only after the short window stays below threshold for ten minutes. Any state moves to Suppressed when the window holds fewer than 500 participant-minutes, and returns when traffic recovers. Burn-rate states: only one of them pages SUPPRESSED sample below 500 p-min STEADY burn under 1x WATCH ticket, no page BURNING escalate in hours PAGING wake someone 3x / 1 d 6x / 6 h 14.4x / 1 h + 5 m burn falls under 1x 5 m window under 14.4x for 10 minutes — auto-resolve, no flapping overnight traffic denominator recovers
Suppression is a state, not a silence rule — the evaluator records that it declined to judge, so a quiet window never looks like a healthy one.

Reproduction Steps & Debugging Log Patterns

  1. Stand up the pipeline against a 200-participant staging room and confirm the denominator guard: with 200 participants a 5-minute window holds 1,000 participant-minutes, so it evaluates, while a 20-participant room holds 100 and must report INSUFFICIENT_SAMPLE.
  2. Shape one participant’s downlink to 250 kbps with 8% loss for 10 minutes. Expect their own minutes to classify bad with cause: 'last_mile', expect the burn rate to move by roughly 1%, and expect no state change.
  3. Stall one SFU node’s send loop for 60 s — a SIGSTOP on the worker is the bluntest reliable way — and watch every subscriber on that node accumulate freeze duration simultaneously. This is the shape a real page should have: many participants, one node, cause: 'server'.
  4. Confirm the clamp by replaying step 2 with the same participant contributing 40% of raw bad minutes in a thin window; the clamped numerator must cap their contribution at 5%.
  5. Resolve the stall and verify the evaluator holds PAGING for the full 10-minute recovery window before returning to STEADY.

The alerting rules themselves belong in configuration, next to the objective definitions, so the thresholds are reviewable without reading evaluator code:

; alerts.conf - burn-rate rules, loaded by the evaluator at startup
[slo.subscriber_freeze]
objective       = 0.99          ; 99% of participant-minutes must be good
window          = 28d           ; rolling error-budget window
bad_if_freeze_ms  = 500         ; cumulative freeze in one participant-minute
bad_if_loss      = 0.02         ; downlink loss fraction over the same minute

[burn.page]
short_window    = 5m            ; both windows must breach before paging
long_window     = 1h
factor          = 14.4          ; ~2% of a 28 d budget consumed per hour
min_denominator = 500           ; participant-minutes required to evaluate at all
max_participant_share = 0.05    ; clamp: one person cannot carry the numerator
require_rooms   = 3             ; shared cause, not one meeting going badly
require_server_share = 0.5      ; over half the bad minutes must be server-attributed
resolve_after   = 10m           ; hysteresis on the way down

[burn.ticket]
short_window    = 30m
long_window     = 6h
factor          = 6
min_denominator = 2000
route           = queue         ; never paged, reviewed the next working day

A healthy evaluation and a real incident look completely different in the log, and the difference is in the denominator and the spread, not in the freeze numbers themselves:

// 03:14 quiet night — guard holds, nothing fires
// {"slo":"subscriber_freeze","window":"5m","total":140,"verdict":"INSUFFICIENT_SAMPLE"}
// 09:40 one bad link — evaluated, clamped, ignored
// {"window":"5m","total":24800,"raw_bad":61,"clamped_bad":61,"burn":0.25,"rooms":1}
// 14:02 node stall — the shape that deserves a page
// {"window":"5m","total":25100,"clamped_bad":3610,"burn":14.4,"rooms":38,
//  "server_share":0.94,"top_node":"n-07","state":"STEADY -> PAGING"}

Note what the middle line proves. Twenty-four thousand eight hundred participant-minutes with 61 bad ones is a 0.25% bad fraction against a 1% allowance — a quarter of the sustainable rate, logged and forgotten. The evaluator still wrote the line, because a suppressed or ignored evaluation that leaves no record is indistinguishable from a broken pipeline during the postmortem where someone asks why nothing fired.

The rooms and top_node fields are what make the page actionable at 2 a.m.: 38 rooms on one node is an infrastructure fault, while 38 rooms across 12 nodes is a regression in the build. From the top_node value, the single-participant walk-through in Tracing One Participant Across SFU Logs turns the aggregate back into a specific stream you can inspect.

Common Implementation Mistakes

Routing a burn-rate breach A decision tree. A 14.4 times breach on both windows first checks whether the window holds 500 participant-minutes; if not it is suppressed. Then it checks whether affected participants span three or more rooms; if not it becomes a per-user ticket for last-mile trouble. Then it checks whether the damage is concentrated on one node; if so the node is drained and infrastructure is paged, otherwise the media on-call is paged for a fleet-wide fault. A breach is not yet an incident — route it before you ring anyone burn 14.4x on 1 h AND 5 m freeze or loss SLI 500+ participant-minutes? SUPPRESS sample too thin to judge 3+ distinct rooms affected? TICKET last-mile, one meeting concentrated on one node? DRAIN + PAGE infra shift rooms off n-07 PAGE media on-call fleet-wide: suspect the build yes yes no no no yes Two of the four outcomes are pages; the other two exist so that the pages stay believable.
The same 14.4x number routes four different ways depending on how many rooms and nodes it touches.

FAQ

Should the SLI come from client getStats() or from server-side forwarding gaps?

Both, for different jobs. The client’s freezeCount and totalFreezesDuration are the ground truth for what the user saw, so they define the objective. The server’s forwarding gap decides attribution, and attribution decides routing — a bad minute with no server gap is a last-mile event that belongs in a ticket queue, never on a pager.

What if my traffic is too small to ever reach 500 participant-minutes?

Then a ratio-based page is the wrong instrument at that scale, and you should say so rather than lowering the guard. Alert instead on absolute counts with a floor — for example, at least 15 distinct participants degraded within 5 minutes — and keep the burn-rate objective purely as a monthly quality report.

Does a 500 ms freeze threshold hide shorter stutters from us entirely?

No — the threshold decides what counts against the budget, not what gets recorded. Keep the full totalFreezesDuration distribution as a histogram so a fleet-wide shift from a 120 ms median gap to a 380 ms one is visible on a dashboard long before any minute crosses 500 ms. That drift is one of the better early-warning signals you have, and it is exactly the kind of change a codec or scheduler regression produces.

How do burn-rate alerts interact with autoscaling?

They should not compete. A rising burn rate concentrated on the busiest nodes usually means capacity, not a bug, and it belongs to the control loop described in Autoscaling SFU Nodes on CPU and Bandwidth; page only when the burn persists after the new nodes have taken traffic, typically 3–5 minutes later.

Related: return to SFU Observability & Metrics, or read Bandwidth-Aware Layer Selection in an SFU and Sharding Rooms Across SFU Nodes.

Exporting SFU Metrics to Prometheus and Grafana Guide Tracing One Participant Across SFU Logs Guide Media Server Architecture: SFU & MCU Design, Scaling & Production Patterns Guide Load Balancing & Scaling SFUs Guide Autoscaling SFU Nodes on CPU and Bandwidth Guide Sharding Rooms Across SFU Nodes Guide