Keyframe Request Strategies in an SFU

An SFU has no encoder, so the only way it can hand a subscriber a decodable starting point is to ask the publisher for one — a Picture Loss Indication (PLI) or Full Intra Request (FIR) travelling upstream over RTCP. This guide is part of the Selective Forwarding Unit Design guide, and it settles one decision: which events actually justify that upstream request, and how to collapse the resulting fan-in so that ten subscribers joining inside 300 ms cost the publisher one intra frame instead of ten.

Context & Trade-offs

Three events legitimately create demand for an intra frame. A new subscriber starts receiving a source: its decoder has no reference frames at all, so every forwarded delta frame is undecodable and the tile stays black until an intra arrives. A layer or source switch: the router begins forwarding a different encoding — a simulcast upswitch driven by the thresholds in Bandwidth-Aware Layer Selection in an SFU, an active-speaker pin, a screen-share promotion — and the new bitstream references frames the subscriber never received. A decoder error: packet loss punched a hole that NACK retransmission could not fill before the jitter buffer’s play-out deadline, and the subscriber emits its own PLI to the SFU.

Those three are not equally urgent, and only the first two are unambiguously the SFU’s problem. A subscriber-originated PLI should first be answered locally: if the missing packets are still in the per-egress send buffer, retransmit them and send nothing upstream. Escalate only after roughly one RTT of failed recovery, because a keyframe is a far more expensive fix than a handful of retransmitted packets.

The cost is the whole reason this page exists. At 720p and a 1.5 Mbps target, a delta frame averages 4–7 KB while an intra frame runs 40–60 KB — roughly eight times larger. Injected into a stream that is already sized to its estimate, the pacer spreads that intra over 100–200 ms, during which the publisher’s instantaneous send rate reaches 2–3x steady state. One keyframe per second lifts the mean uplink rate by 25–30%. Sustained, that overshoot shows up as queueing delay in the very transport-wide feedback the sender’s estimator reads, and the mechanism described in Bandwidth Estimation & Congestion Control responds by lowering the publisher’s allowed bitrate. A keyframe storm is therefore self-defeating: it degrades the stream it was meant to repair.

PLI and FIR are not interchangeable. PLI (RTCP payload-specific feedback, format 1) is advisory: “I lost a picture, please refresh.” FIR (format 4) is a command carrying its own sequence number, intended for mixers that need a full encoder reset when a source is repurposed. Repeat a FIR without incrementing that sequence number and receivers ignore it as a duplicate. Use PLI for everything routine and reserve FIR for a wedged decoder that has ignored two PLIs, which also keeps you clear of Firefox’s habit of dropping FIRs it judges redundant.

One structural fact makes the rate limit load-bearing rather than defensive: browser encoders in a WebRTC sender run effectively open-ended groups of pictures. They emit an intra at stream start and then only when asked, precisely because a conversational stream has a feedback channel. The SFU’s request policy therefore is the publisher’s keyframe schedule — there is no periodic safety net quietly refreshing the stream every two seconds the way a broadcast encoder would. That cuts both ways. Ask too often and you own the bitrate overshoot; ask too rarely and a subscriber whose request was throttled has nothing else to wait for. This is also why the budget belongs to the publisher rather than the room: the resource being protected is one encoder’s uplink, and it should be spent on the sources that actually have starving subscribers, not shared with an unrelated participant whose subscribers are churning.

Minimal Runnable Implementation

The gate below sits between the router and the publisher’s RTCP sender. It applies three filters in order: deduplicate by (source, layer), debounce with both a leading and a trailing edge, then spend a token from a per-publisher bucket so a pathological room cannot exceed a fixed keyframe budget.

// Per-publisher keyframe gate: dedup -> debounce -> token bucket -> RTCP.
class KeyframeGate {
  constructor(publisher, opts = {}) {
    this.publisher = publisher;
    this.debounceMs = opts.debounceMs ?? 500;  // at most one intra per key per window
    this.capacity = opts.capacity ?? 3;        // burst: joins arrive in clumps, not evenly
    this.refillMs = opts.refillMs ?? 2000;     // sustained ceiling: 1 keyframe / 2 s
    this.tokens = this.capacity;
    this.lastRefill = Date.now();
    this.windows = new Map();                  // key -> { openedAt, pending }
    this.firSeq = 0;                           // FIR seq MUST increment or receivers ignore it
    this.stats = { asked: 0, sent: 0, deduped: 0, throttled: 0 };
  }

  refill(now) {
    const earned = Math.floor((now - this.lastRefill) / this.refillMs);
    if (earned > 0) {
      this.tokens = Math.min(this.capacity, this.tokens + earned);
      this.lastRefill += earned * this.refillMs; // keep the remainder, don't drift
    }
  }

  // key is a stable string per (sourceSsrc, layerIndex); reason is used only for logging/FIR.
  request(key, reason, now = Date.now()) {
    this.stats.asked++;
    const w = this.windows.get(key);
    if (w && now - w.openedAt < this.debounceMs) {
      w.pending = true;                        // an intra is already in flight for this key
      this.stats.deduped++;
      return false;                            // the trailing flush covers late arrivals
    }
    this.refill(now);
    if (this.tokens < 1) {
      this.stats.throttled++;                  // budget exhausted: the room is churning
      return false;                            // caller must not retry in a tight loop
    }
    this.tokens -= 1;
    this.windows.set(key, { openedAt: now, pending: false });
    this.emit(key, reason);
    setTimeout(() => this.flush(key), this.debounceMs); // trailing edge
    return true;
  }

  emit(key, reason) {
    // PLI for everything routine; FIR only for a decoder that ignored earlier PLIs.
    if (reason === 'decoder-reset') this.publisher.sendRtcpFir(key.ssrc, this.firSeq++);
    else this.publisher.sendRtcpPli(key.ssrc);
    this.stats.sent++;
  }

  flush(key) {
    const w = this.windows.get(key);
    if (!w) return;
    this.windows.delete(key);
    // Someone asked mid-window: they joined after the intra was already sent, so serve them now.
    if (w.pending) this.request(key, 'trailing-edge');
  }
}

Make the thresholds configuration rather than constants, because a webinar with 500 passive viewers and a six-person standup want different budgets:

[keyframe]
; one intra per source per window; under ~300 ms the encoder never reaches steady state
debounce_ms = 500
; burst allowance so a ten-person join wave is not serialised over twenty seconds
bucket_capacity = 3
; sustained ceiling per publisher, per layer
bucket_refill_ms = 2000
; replay a cached intra to a joiner instead of asking upstream, if it is this fresh
keyframe_cache_max_age_ms = 2000
; escalate PLI to FIR only after this many unanswered requests
fir_after_unanswered_pli = 2

The cache line is the highest-leverage entry. Holding the most recent intra per layer costs about 40–60 KB of memory per layer per publisher and removes the upstream request entirely for any subscriber that joins within the window — the SFU replays the cached frame into that subscriber’s sequence space, then continues from live packets. It only works if you renumber the injected packets correctly, which is the same rewriting discipline covered in Rewriting RTP Header Extensions When Forwarding.

Reproduction Steps & Debugging Log Patterns

1. Publish a three-layer 720p stream and subscribe one client. Poll getStats() at 1 s intervals on the publisher and record outbound-rtp.pliCount, firCount, keyFramesEncoded and bytesSent as your baseline.
2. Join ten subscribers within 300 ms, all onto the same layer. Correct behaviour: pliCount rises by exactly 1 and keyFramesEncoded by 1. A rise of 10 means the gate is keyed per subscriber rather than per source.
3. Watch bytesSent in the 1 s bucket containing that intra. Expect roughly a 25–30% bump over the neighbouring seconds — that is the keyframe, not a bug.
4. Script 200 layer switches per second against one publisher for 10 s. The bucket should cap output near 5 PLIs and the throttled counter should climb steadily rather than the process spinning.
5. Drop 5% of packets on one subscriber’s downlink. Verify NACK retransmissions rise first and pliCount stays flat, confirming loss is repaired from the send buffer before any escalation.

Instrument the gate itself rather than inferring its behaviour from stats alone: log one line per request() with the coalescing key, the reason, the outcome (sent, deduped, throttled) and the remaining token count. Those four fields turn every keyframe question into a grep, and they let you replay a bad minute offline without a packet capture. The contrast between a healthy and a mis-keyed gate is immediate:

// Healthy gate log for step 2 — ten joins, one intra.
// [pub=P1 key=ssrc7:l1] request reason=subscriber-join  -> PLI sent, tokens=2
// [pub=P1 key=ssrc7:l1] request reason=subscriber-join  -> deduped (in-window, 8 more)
// [pub=P1 key=ssrc7:l1] window closed pending=true      -> trailing PLI, tokens=1
// [pub=P1] keyFramesEncoded 41 -> 42, bytesSent/s 158k -> 201k

// Pathological gate log — the key includes the subscriber id, so nothing ever coalesces.
// [pub=P1 key=ssrc7:l1:subA] request -> PLI sent, tokens=2
// [pub=P1 key=ssrc7:l1:subB] request -> PLI sent, tokens=1
// [pub=P1 key=ssrc7:l1:subC] request -> throttled (tokens=0)   // budget burned in 40 ms
// [pub=P1] keyFramesEncoded 41 -> 44, bytesSent/s 158k -> 297k

The diagnostic tell of a mis-keyed gate is sent tracking asked almost one-for-one with deduped near zero. The tell of an over-tight budget is a throttled counter that never returns to zero and subscribers reporting long black tiles; cross-check against subscriber-side framesDecoded stalling while packetsReceived climbs, and read the congestion side of the same capture using the patterns in Interpreting getStats() for Congestion Signals.

Common Implementation Mistakes

• Keying the debounce by subscriber. The coalescing key must be (source, layer). Include the subscriber id and every join generates its own PLI, which is exactly the storm the gate exists to prevent.
• Leading-edge-only debounce. Dropping in-window requests without a trailing flush leaves any subscriber who joined 50 ms after the intra was sent frozen until the publisher’s next periodic keyframe — often several seconds. Always re-fire once at window close if pending is set.
• Escalating a subscriber PLI straight upstream. A loss gap that the per-egress send buffer can retransmit costs a few kilobytes; a keyframe costs forty. Wait roughly one RTT for NACK recovery before spending a token.
• Reusing the FIR sequence number. A FIR whose sequence number did not increment is treated as a duplicate and ignored, so the wedged decoder stays wedged while your logs claim a request was sent.
• Rate limiting per room instead of per publisher. A shared bucket lets one churning source starve every other publisher’s legitimate requests. Budget per publisher, and ideally per layer within it.
• Requesting a keyframe on an SVC downswitch. Lower spatial and temporal layers are a decodable subset of the same bitstream, so no intra is needed; gate the request on whether the source is simulcast or SVC, and align the cut point with Switching Layers Without Visible Glitches.

FAQ

PLI or FIR — which should the SFU send?

PLI for essentially everything: joins, layer switches and unrecoverable loss. FIR is a command aimed at full encoder state resets and carries a sequence number that must increment on every send; reserve it for a decoder that has ignored two PLIs. Chrome answers both, while Firefox may discard a FIR it considers redundant, so a FIR-first strategy produces silent no-ops.

How long can a joining subscriber wait for its first frame?

Budget 200–500 ms from subscription to first decodable frame. A cached intra replayed from the server hits the low end because nothing leaves the machine; an upstream PLI adds one server-to-publisher RTT plus encoder latency, so a 500 ms debounce puts the worst case near 700 ms. Beyond about 1 s users read it as a broken tile, which is the threshold worth wiring into the SLOs in Alerting on Freeze and Packet-Loss SLOs.

Can rate limiting leave a subscriber frozen forever?

Not if the bucket has a trailing edge and every publisher still emits periodic keyframes. A throttled request is recorded, not discarded outright, and the next refill or the encoder’s own periodic intra resolves it within a couple of seconds. What you must avoid is a caller that retries in a tight loop on refusal — that turns a bounded delay into a busy spin without producing a single extra frame.

Related: this deep-dive sits under Selective Forwarding Unit Design; pair it with Simulcast-Aware Forwarding for the room-wide layer policy that generates most of these requests, and with SFU vs MCU Topologies for why a forwarding server has to ask the publisher at all.