Simulcast-Aware Forwarding

A Selective Forwarding Unit earns its name by never re-encoding media: it receives one set of RTP streams from each publisher and forwards a subset to each subscriber untouched. Simulcast turns that forwarding decision into the core of the product. When a publisher sends three independently encoded resolutions of the same camera — a high, a medium, and a low spatial layer, each its own RTP stream with its own SSRC — the server must choose, per subscriber, which one of those streams to relay, and switch between them as that subscriber’s downlink changes. This guide is part of the Media Server Architecture: SFU & MCU guide, and it covers the exact mechanics of that forwarding path: reading the RID/MID that identifies each layer, mapping a subscriber’s estimated bitrate to a layer, switching cleanly on keyframe boundaries, and rewriting the RTP headers so the receiver never notices the source changed.

The goal is concrete: build a forwarder that can promote a subscriber from the low layer to the high layer when their bandwidth recovers, and demote them again when it drops, with no decoder corruption, no frozen frame, and a switch latency bounded by one keyframe interval. Everything below assumes you have already terminated DTLS-SRTP, demultiplexed RTP, and parsed RTCP feedback — the forwarding logic sits one layer above that.

Step 1 — Read RID and MID to identify the incoming layers

Simulcast layers arrive as separate RTP streams that all belong to the same logical track. The browser tags each one with an RTP Stream Identifier (RID) carried in the urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id header extension, and associates them with a media section through the MID extension (urn:ietf:params:rtp-hdrext:sdes:mid). The publisher’s SDP declares which RIDs exist and in what order via a=simulcast:send and a=rid: lines; your job is to parse those and then match each inbound packet to a layer by reading its RID extension.

Chrome sends the RID only on the first few packets of each stream and on every keyframe, not on every packet — so cache the SSRC-to-RID binding the moment you first observe it, and never assume a later packet will re-advertise the RID. Once you know ssrc → rid, and rid → layer index from the SDP, you have the mapping the selector needs.

// Parse the publisher's simulcast layers from the offer SDP, then bind SSRCs at runtime.
// rid order in `a=simulcast:send h;m;l` is high→low by convention but NOT guaranteed —
// resolve the real spatial size from each a=rid line's max-width/max-height if present.
function parseSimulcastLayers(sdpMediaSection) {
  const rids = [];
  for (const line of sdpMediaSection.split(/\r?\n/)) {
    const m = line.match(/^a=rid:(\S+) send/); // a=rid:h send max-width=1280;max-height=720
    if (m) rids.push(m[1]);
  }
  return rids; // e.g. ['h', 'm', 'l'] — index 0 is the top layer
}

const ssrcToRid = new Map();   // learned from the RID header extension on early/keyframe packets
const ridToIndex = new Map();  // 'h'->0, 'm'->1, 'l'->2 from parseSimulcastLayers order

function onRtpPacket(pkt) {
  const rid = pkt.getHeaderExtension('rtp-stream-id'); // present early + on keyframes only
  if (rid && !ssrcToRid.has(pkt.ssrc)) {
    ssrcToRid.set(pkt.ssrc, rid); // bind once; later packets omit the RID
  }
  const layerIndex = ridToIndex.get(ssrcToRid.get(pkt.ssrc));
  routePacket(pkt, layerIndex); // hand to the per-subscriber selector
}

Binding layers when BUNDLE and RTX are in play

A production forwarder never sees a tidy set of three SSRCs on the wire. With BUNDLE every audio and video stream in the session shares one 5-tuple, so demultiplexing is a two-stage lookup: the MID extension tells you which m-section — which transceiver — a packet belongs to, and only within that section does the RID distinguish the three simulcast layers. Skip the MID stage and a screen-share track’s RID h will collide with the camera track’s RID h, because RID values are only unique per m-section, not per session.

Retransmission doubles the SSRC count again. When the publisher negotiates a=rtcp-fb:96 nack alongside a=fmtp:97 apt=96, each layer gets a paired RTX stream with its own SSRC, so a three-layer video publisher occupies six video SSRCs on one transport. RTX packets carry the original sequence number in the first two bytes of the payload and frequently carry no RID extension at all; you bind them through a=ssrc-group:FID <media> <rtx> when the SDP declares SSRCs, and otherwise by observing which media SSRC each RTX stream repairs. Treat RTX SSRCs as unknown layers and you will either forward duplicate media or discard legitimate repairs.

With RID-based simulcast, though, Chrome emits no a=ssrc lines for the layers at all — the SDP names RIDs and nothing else, so a binding table preloaded from the offer is empty and the mapping exists only at runtime. The publisher-side configuration that produces these lines is walked through in Simulcast with Three Quality Layers in Chrome. The table must therefore be built lazily and must also expire: age out an ssrc → rid entry after roughly 5 s of silence, because a publisher that renegotiates or restarts its encoder can recycle an SSRC into a different layer, and a stale binding routes high-layer packets into a low-layer subscriber’s stream with no error raised anywhere in the pipeline.

Step 2 — Identify spatial and temporal layers within each stream

Simulcast gives you spatial layers — distinct resolutions, each its own RTP stream. Inside each of those streams the encoder usually also produces temporal layers: a base layer at, say, 7.5 fps and additional frames that lift it to 15 and 30 fps, all in one RTP stream and distinguishable only by reading the codec payload. Temporal scalability is what lets you drop a subscriber’s frame rate without a full layer switch, shedding 30–50% of a stream’s bitrate by forwarding only the lower temporal IDs.

How you read the temporal ID depends entirely on the codec. VP8 exposes a TID field plus a picture-id in its payload descriptor; AV1 carries a Dependency Descriptor header extension that encodes the full spatial/temporal dependency graph, which is the same structure the Configuring AV1 SVC Layers in WebRTC workflow relies on. Mishandling this distinction is the single most common source of decoder corruption, so the forwarder must know, per codec, exactly which bytes carry the temporal ID and which frames are safe to drop.

// Extract the temporal ID per codec. Dropping a higher-TID frame is always safe;
// dropping a base-layer (TID 0) frame breaks every frame that depends on it.
function temporalId(pkt, codec) {
  if (codec === 'VP8') {
    // VP8 payload descriptor: T bit signals presence of TID in the extension byte
    const d = pkt.payloadDescriptor;
    return d.hasTID ? d.tid : 0;        // 0 = base layer, must always be forwarded
  }
  if (codec === 'AV1') {
    // AV1 reads the temporal_id straight from the Dependency Descriptor extension
    return pkt.dependencyDescriptor.temporalId;
  }
  return 0; // H.264 simulcast here is treated as spatial-only (no temporal shaping)
}

Temporal shedding is the cheaper adjustment and should always be tried first. Dropping a high temporal ID needs no keyframe and no decoder reset, so it takes effect on the very next frame instead of after a keyframe round trip, and it holds resolution constant — a frame-rate dip is far less noticeable than a resolution jump, which reads as the picture visibly softening. The two mechanisms also work at different scales. With layers at 1/2 and 1/4 scale, high to medium sheds roughly 70% of the bitrate (about 1700 kbps down to 500 kbps) and medium to low another 64% (500 kbps to 180 kbps); the 30% and 50% temporal steps sit inside those gaps, turning a coarse 3-rung ladder into 7–9 usable rungs and letting the selector track a wobbling estimate without a spatial switch every few seconds.

One VP8 caveat governs where you are allowed to make the cut. In the default three-layer pattern only TID 2 frames are truly non-reference; TID 1 frames are referenced by later TID 1 frames through the golden-frame buffer. Start dropping TID 1 in the middle of a 0-2-1-2 cycle and the next TID 1 frame decodes against a picture the subscriber never received. Apply a change to the temporal cap at a pattern boundary — the frame where the next TID 0 arrives — so the dependency chain you are cutting is already closed.

Step 3 — Map subscriber bitrate to a layer and switch on a keyframe

Each subscriber has an estimated downlink bitrate from REMB or transport-wide congestion control feedback — the same estimate produced by the bandwidth estimation pipeline. The selector maps that estimate onto a target layer using each layer’s measured send bitrate plus a safety margin, then commits to the switch only when a usable decoder-refresh point arrives. The full threshold table, debounce timing, and keyframe-request logic are worked out in Forwarding Simulcast Layers by Subscriber Bandwidth; the broader policy that balances every subscriber against the publisher’s available layers lives in Bandwidth-Aware Layer Selection in an SFU.

The non-negotiable rule: you may only begin forwarding a new spatial layer starting at a keyframe of that layer. Inter-coded frames reference earlier frames of the same stream; splice a P-frame from the high layer onto a decoder that was watching the low layer and you get a green-block smear or a hard freeze until the next keyframe. When you decide to upswitch, send an RTCP Picture Loss Indication (PLI) or Full Intra Request (FIR) to the publisher for the target layer’s SSRC, keep forwarding the old layer until the requested keyframe arrives, and only then cut over. Downswitching to a lower layer that is already flowing can often happen on its next existing keyframe without a request, since lower layers are cheaper for the publisher to refresh frequently. The perceptual side of that cutover — resolution jumps, buffer timing, and how long the eye tolerates the transition — is worked through in Switching Layers Without Visible Glitches.

// Per-subscriber forwarder. Switches are pending until a keyframe of the target layer lands.
class SubscriberForwarder {
  constructor(publisher, sendPli) {
    this.publisher = publisher;
    this.sendPli = sendPli;        // (ssrc) => emit RTCP PLI/FIR upstream
    this.currentLayer = 2;         // start conservative on the low layer
    this.pendingLayer = null;
  }

  requestLayer(target) {
    if (target === this.currentLayer || target === this.pendingLayer) return;
    this.pendingLayer = target;
    // ask the publisher for a fresh keyframe on the layer we want to switch into
    this.sendPli(this.publisher.ssrcForLayer(target));
  }

  forward(pkt, layerIndex, isKeyframe) {
    if (this.pendingLayer !== null && layerIndex === this.pendingLayer && isKeyframe) {
      this.currentLayer = this.pendingLayer; // commit exactly on the keyframe boundary
      this.pendingLayer = null;
    }
    if (layerIndex !== this.currentLayer) return; // drop every other layer's packets
    this.rewriteAndSend(pkt);
  }
}

Hysteresis — why promotion is slow and demotion is instant

The mapping from estimate to layer must never be a bare comparison, because the estimate is noisy at the 1 s granularity you poll it at. Make the thresholds asymmetric: require the estimate to exceed the target layer’s measured send bitrate by about 30% and hold there across four consecutive samples before promoting, but demote on the first sample that falls below the current layer’s bitrate. That mirrors the cost asymmetry — sitting one layer too low costs sharpness most viewers never consciously notice, while sitting one layer too high fills the subscriber’s downstream queue, and the resulting queueing delay, loss and freezing is immediately obvious.

One subtlety trips up nearly every first implementation: while a subscriber is pinned to the 180 kbps low layer, its congestion controller has no traffic to probe with, so the estimate plateaus just above the current send rate and the promotion threshold is never crossed — the stream stays at 320x180 on a link that could carry 5 Mbps. Breaking the deadlock requires active probing, padding packets or an RTX burst that briefly lifts the send rate above the current layer so the estimator can observe whether delay increases. That is one of the practical differences explored in Transport-CC vs REMB Feedback: transport-wide feedback reports per-packet arrival times, so a 200–300 ms probe confirms headroom, whereas REMB’s receiver-side estimate reacts more slowly and needs the probe sustained for a second or more.

Oscillation also has a direct bitrate cost. A 720p keyframe is typically 5–10x the size of a P-frame, so every upswitch injects a burst of tens of kilobytes into a link you have just judged marginal; a selector that flaps at 1 Hz can manufacture the very loss that triggers its next demotion.

# Per-subscriber layer selector thresholds — tune these before touching the estimator
upswitch_margin        = 1.3    # estimate must exceed target layer's measured bitrate by 30%
upswitch_hold_samples  = 4      # sustained across 4 samples of 1 s getStats polling
downswitch_margin      = 1.0    # demote as soon as the estimate drops under the current layer
downswitch_hold_ms     = 0      # no debounce on the way down — congestion is not a false alarm
temporal_first         = true   # shed TID before attempting a spatial demotion
pli_retry_ms           = 1000   # one PLI, then retry after 1 s
pli_max_retries        = 3      # abandon the pending upswitch after 3 attempts
layer_dead_ms          = 1000   # no packets on a layer SSRC for 1 s -> treat the layer as absent

Step 4 — Rewrite RTP SSRC, sequence number, and picture-id, then verify

From the subscriber’s decoder’s point of view there is a single continuous RTP stream. But behind the selector you are splicing packets from streams that each have their own SSRC, their own monotonically increasing sequence numbers, and their own timestamp and picture-id baselines. Forward them raw and the receiver sees the SSRC change (it tears down and rebuilds the stream), sees a sequence-number discontinuity (it reports massive packet loss), and sees the picture-id jump (VP8 reference picture selection breaks). The forwarder must therefore present one outgoing SSRC and rewrite every header field to a continuous, gap-free sequence.

Maintain per-subscriber offsets: a fixed output SSRC, a running sequence-number translation that closes the gap left by every dropped packet, and a picture-id translation for VP8. At each switch you snapshot the last values you emitted and rebase the new layer onto them. The header extensions riding alongside these fields — RID, MID, abs-send-time, transport-wide sequence number — need their own rewrite pass, which is covered in Rewriting RTP Header Extensions When Forwarding. Verify the result by pulling the subscriber’s inbound-rtp stats — framesDecoded should keep climbing across a switch, freezeCount should not increment, and pliCount upstream should show exactly one request per upswitch, not a storm.

// Rewrite headers so the splice is invisible. One output SSRC per subscriber.
rewriteAndSend(pkt) {
  const t = this.translation;                 // { outSsrc, seqOffset, lastSeq, picIdOffset }
  pkt.ssrc = t.outSsrc;                        // collapse N source SSRCs into one
  pkt.sequenceNumber = (pkt.sequenceNumber + t.seqOffset) & 0xffff; // gap-free, 16-bit wrap
  t.lastSeq = pkt.sequenceNumber;

  if (pkt.codec === 'VP8') {
    // rebase VP8 picture-id so reference selection stays monotonic across a layer switch
    pkt.payloadDescriptor.pictureId = (pkt.payloadDescriptor.pictureId + t.picIdOffset) & 0x7fff;
  }
  this.send(pkt);
}

// On commit, recompute offsets so the NEW layer continues from the last emitted values.
rebaseOnSwitch(firstPktOfNewLayer) {
  const t = this.translation;
  t.seqOffset = (t.lastSeq + 1 - firstPktOfNewLayer.sequenceNumber) & 0xffff;
  t.picIdOffset = (t.lastPicId + 1 - firstPktOfNewLayer.payloadDescriptor.pictureId) & 0x7fff;
}

What to instrument on the forwarding path

Subscriber-side getStats() reports the outcome but not the decision, so the selector needs its own telemetry. Export, per subscriber and per publisher track, a gauge for the current spatial index and temporal cap, counters for committed switches and PLIs emitted, and a histogram of the wall time between sending a PLI and committing on the resulting keyframe. Add one derived ratio — bytes forwarded over bytes received for that publisher — which sits between 1/3 and 1/10 in a healthy conference; a value approaching 1 means the selector is handing the top layer to everyone and the thresholds are wrong.

Fewer than 2 committed switches per minute per subscriber is normal; more than 6 means the hysteresis window is too tight for that population’s networks. Keyframe wait p95 should stay under 300 ms — beyond that the fault is usually a publisher deferring intra requests under thermal load rather than your forwarder. The selector itself must stay inside the 1–3 ms of forwarding overhead an SFU is allowed to add, so time the per-packet path and treat drift toward the 80–200 ms a transcoding path costs as a hot-loop bug, typically a per-packet allocation. Wiring these series into dashboards and alert rules is covered in Exporting SFU Metrics to Prometheus and Grafana.

Edge Cases & Browser Quirks

The keyframe thundering herd

The nastiest failure here is correctness-preserving and still takes down the call. Picture 50 subscribers watching one publisher when a shared bottleneck clears — a congested Wi-Fi access point drains, or a transit path reroutes. Every estimate rises inside the same 1 s window, every selector independently decides to promote, and every one sends a PLI for the high layer’s SSRC. libwebrtc honours intra requests eagerly, so the publisher emits back-to-back keyframes and its uplink spikes 3–5x above target for a second or more, re-congesting the path that just recovered. Estimates collapse, all 50 selectors demote, and the room settles into a synchronised sawtooth.

The fix is coalescing rather than rate limiting. Because the commit rule is per-subscriber and stateless — commit on the next keyframe of the target layer — one keyframe satisfies any number of pending upswitches for free. Put a token bucket keyed by (publisher, layer) in front of the upstream request path, allow at most one intra request per layer per 500 ms, and let every pending switch resolve against whichever keyframe arrives. Spreading the promotion decision itself over a random 100–300 ms desynchronises the herd before it ever reaches the bucket.

• VP8 picture-id width. VP8’s picture-id is either 7-bit or 15-bit depending on the M bit in the payload descriptor; Chrome emits the 15-bit form, but a naive parser that assumes 7 bits will mis-rebase on every switch and corrupt reference selection. Always read the M bit before masking.
• AV1 dependency descriptor vs VP8 picture-id. AV1 does not use a picture-id at all — its frame relationships live in the Dependency Descriptor header extension, a template-based structure. A forwarder that reuses VP8 rewriting logic for AV1 will drop frames the decoder needed; AV1 requires rewriting the frame_number inside the descriptor instead, and respecting its declared decode targets.
• Chrome RID advertisement timing. Chrome (since roughly M90) sends the RID extension only on the initial packets and on keyframes. If your SSRC binding logic waits for a steady-state packet, it will never bind the low layer on a quiet camera. Bind on the very first packet that carries the extension.
• Safari simulcast support. Safari only gained reliable send-side simulcast for VP8/H.264 in recent versions and historically advertised RIDs it would not actually encode under thermal pressure, silently collapsing to one layer. Detect a missing layer at runtime (no packets on its SSRC for >1 s) and fall the selector back to the layers actually arriving rather than forwarding a dead SSRC; the sender-side symptoms and how to confirm them are catalogued in Debugging Missing Simulcast Layers.
• Firefox temporal layers. Firefox’s VP8 simulcast historically produced fewer temporal layers than Chrome for the same encoding request, so a temporal-ID drop policy tuned on Chrome can over-shed frames on Firefox. Read the actual TIDs present rather than assuming three temporal layers exist.
• NACKs for packets that never existed downstream. The subscriber NACKs sequence numbers in your rewritten space, and some of those numbers were consumed by frames you shed temporally or by a layer you have since switched away from. Answer NACKs from a small per-subscriber cache of packets you actually forwarded — 500 ms or roughly 200 packets is enough — keyed by output sequence number. Never look the number up in the inbound stream: it belongs to a different numbering space, and retransmitting the wrong packet is worse than silently dropping the request.
• Resolution-based layer matching. scaleResolutionDownBy produces non-integer dimensions for many source resolutions, and Chrome rounds the result to even values — a layer requested as 1280x720 ÷ 3 can arrive as 424x240 rather than the 426x240 you computed. A forwarder that identifies layers by their expected resolution will fail to match; identity comes from the RID and nothing else, with resolution treated purely as reporting metadata.

Common Implementation Mistakes

• Switching mid-GOP. Cutting to a new spatial layer on a P-frame instead of waiting for its keyframe is the classic green-smear bug. Always hold the pending switch until the target layer’s keyframe arrives.
• Forwarding the RID-bearing packets but not caching the binding. If you only read RID per-packet you lose the layer identity the instant Chrome stops sending it. Cache ssrc → rid on first sight.
• Leaving SSRC unrewritten. Forwarding the source SSRC straight through makes the subscriber tear down and rebuild its receiver on every switch, adding a visible freeze. Collapse to one output SSRC.
• PLI storms on upswitch. Re-requesting a keyframe every packet while waiting for one inflates upstream bitrate and can knock the publisher’s own encoder into a degraded state. Send one PLI, then wait with a timeout before retrying, following the dedup and pacing rules in Keyframe Request Strategies in an SFU.
• Dropping temporal base-layer frames. Shedding TID 0 frames to save bandwidth breaks every dependent frame. Only ever drop the highest temporal IDs first.
• Ignoring the publisher’s actual send bitrate. Mapping a subscriber to the high layer because the SDP declared 1700 kbps, when the publisher is thermally throttled to 600 kbps, forwards a starved stream. Drive the mapping off measured per-layer bitrate.
• Treating a silent layer as a congested layer. When the publisher’s encoder disables its top layer under CPU overuse, the layer simply stops producing packets. A selector with a pending upswitch will wait forever for a keyframe that is never coming, and the subscriber stays pinned at 320x180 on a fast link. Bound the pending state with the three-retry limit, mark the layer absent after 1 s of silence, and re-evaluate the ladder against the layers actually arriving.
• Sharing one output SSRC across two publisher tracks. The output SSRC identifies a subscriber’s view of one publisher. Reusing it when the subscriber switches which participant they are watching — a speaker-change in a spotlight layout — makes the receiver continue an old timestamp and picture-id lineage against completely unrelated content. Allocate one output SSRC per subscriber per source track and renegotiate rather than recycling.

FAQ

Does simulcast-aware forwarding re-encode video? No. The defining property of this design is that media is forwarded byte-for-byte at the codec level; only RTP header fields and codec-specific descriptors are rewritten. There is no transcode, no pixel work, and therefore no per-stream CPU cost beyond packet rewriting — which is exactly why an SFU scales where an MCU does not. The topology trade-off is detailed in SFU vs MCU Topologies.

How fast can a subscriber switch from the low layer to the high layer? Switch latency is bounded by how quickly the publisher produces a keyframe for the target layer after your PLI. With a sane keyframe-on-request path that is typically one round trip plus encoder latency — tens to low hundreds of milliseconds. You keep forwarding the old layer the entire time, so the subscriber sees continuous video, just at the old quality until the cutover.

Why not just forward all layers and let the client pick? That defeats the purpose: forwarding every layer to every subscriber sends the full aggregate bitrate down each link, which is precisely the congestion simulcast exists to avoid. The server-side selection is what keeps each subscriber’s downlink matched to one layer.

How does this differ from SVC forwarding? With SVC the layers are encoded with dependencies inside a single stream rather than as independent simulcast streams, so the forwarder drops the unwanted enhancement layers of one stream instead of choosing between separate streams. The decision between the two encodings is covered in Choosing Simulcast vs SVC for Large Conferences.

What happens when a subscriber joins mid-call? A joining subscriber’s decoder has an empty reference buffer, so the first packet you forward has to be part of a keyframe or the picture never resolves — feeding it P-frames produces grey or green output that some decoders will not recover from before the next intra anyway. Start the subscriber on the low layer, request one keyframe for it, and hold the outbound stream silent until that keyframe lands; low-layer keyframes are the cheapest for the publisher to produce and typically arrive well inside a second. Only after the subscriber is decoding cleanly should the normal promotion path be allowed to run, otherwise the first estimate sample fires an upswitch before there is a stable picture to switch away from.

How much of the payload does the forwarder actually parse? Only the payload descriptor — for VP8 the first 1–6 bytes carrying the extension flags, picture-id and TID, and for AV1 the Dependency Descriptor riding in the RTP header extension. The compressed frame data is never touched, which is why the forwarder is indifferent to resolution, codec profile and bitrate, and why the whole path adds 1–3 ms of latency against the 80–200 ms a transcoding server spends per stream. It also means the forwarder cannot repair a stream it does not understand: an unknown codec can be relayed spatially but gets no temporal shedding at all, because the bytes that would tell you which frames are safe to drop are opaque.

Related: build the per-subscriber threshold logic in Forwarding Simulcast Layers by Subscriber Bandwidth, place it inside the broader Selective Forwarding Unit Design and its Bandwidth-Aware Layer Selection in an SFU policy, and revisit the client-side encoding setup in Simulcast & SVC Implementation.