Selective Forwarding Unit Design

A Selective Forwarding Unit (SFU) is the component that lets WebRTC scale past the mesh: instead of every participant encoding a separate stream for every other participant, each publisher sends one upstream to the server, and the server forwards that media — unmodified — to every subscriber who wants it. This guide is part of the Media Server Architecture: SFU & MCU guide, and its goal is concrete: build the forwarding core of an SFU that ingests RTP from publishers, routes packets to subscribers without transcoding, terminates RTCP, requests keyframes on demand, and allocates bandwidth per subscriber so one slow link never starves the room.

The defining property — the reason an SFU costs a fraction of a transcoding MCU per stream — is that it never decodes media. It moves encrypted RTP payloads between transports, rewriting only the routing-layer fields (sequence numbers, timestamps offsets, SSRCs) it must. That makes an SFU CPU-cheap and bandwidth-heavy, and it pushes all the hard decisions into which packets to forward and when to ask the publisher for help. The four steps below build that core; the edge cases and mistakes sections cover where browsers diverge and where naive implementations leak.

Step 1 — Publisher and subscriber transport handling

Every participant connects to the SFU over an RTCPeerConnection, but the SFU plays the role the browser usually delegates to a remote peer. Each connection terminates DTLS-SRTP locally: the server decrypts inbound SRTP from publishers and re-encrypts outbound SRTP to subscribers, because the routing-layer rewrites in later steps require touching RTP headers in the clear. A clean design separates two transport roles per peer — an ingest path that receives a publisher’s tracks, and one or more egress senders that deliver forwarded media to subscribers — even though both live on the same ICE/DTLS connection.

The connectivity tier is identical to the browser-to-browser case: the SFU advertises ICE candidates, gathers server-reflexive addresses, and falls back to relays on restrictive networks. Because the server sits in a known data center it usually offers only host candidates plus a TURN allocation; the candidate-filtering and trickle timing rules from ICE Candidate Gathering & Filtering apply unchanged, and a server-side relay still matters for subscribers behind symmetric NAT.

// Per-peer state: one ingest set of tracks, many egress senders.
class PeerSession {
  constructor(peerId, pc) {
    this.peerId = peerId;
    this.pc = pc;                       // RTCPeerConnection (or native equivalent)
    this.publishedTracks = new Map();   // Map<ssrc, IngestTrack>  — what this peer sends in
    this.egressSenders = new Map();     // Map<sourcePeerId, SubscriberSender> — what it receives
  }

  // Called when DTLS completes and SRTP keys are derived.
  onSrtpReady(decryptCtx, encryptCtx) {
    this.decrypt = decryptCtx;          // unwrap inbound SRTP from this publisher
    this.encrypt = encryptCtx;          // wrap outbound SRTP to this subscriber
  }
}

Keep ingest and egress decoupled so a publisher leaving tears down only the senders fed by it, not the subscriber’s whole connection. In max-bundle mode every track for a peer rides one 5-tuple, so a single ICE failure on the egress side affects all forwarded streams to that subscriber — budget ICE-restart retries at the usual maximum of 3 with a 3–5 s fallback before declaring the subscriber’s transport dead.

Step 2 — RTP packet routing without transcoding

This is the heart of the SFU. A publisher’s decrypted RTP packet arrives; the router decides which subscribers should receive it, rewrites the minimum set of header fields each subscriber’s stream requires, re-encrypts, and sends. No decode, no re-encode — the payload bytes are copied verbatim. The rewrites are mandatory because each subscriber sees a single continuous RTP stream even though the SFU may switch which publisher layer or source feeds it over the call’s lifetime.

Three fields demand per-subscriber rewriting. SSRC is remapped to a stable value the subscriber negotiated, so layer or source switches never look like a new stream. Sequence numbers must stay contiguous from the subscriber’s view: when you drop packets (a layer the subscriber isn’t receiving) or switch sources, you maintain a per-subscriber offset so there are no gaps that trigger spurious NACKs. RTP timestamps likewise need an offset when switching sources so the subscriber’s jitter buffer does not see a discontinuity.

// Forward one inbound RTP packet to a subscriber, rewriting routing fields only.
function forwardPacket(pkt, sub) {
  // pkt.payload is the opaque encoded media — never inspected or modified.
  const out = pkt.cloneHeaderOnly();          // copy header; share payload buffer

  out.ssrc = sub.outboundSsrc;                // stable SSRC for this subscriber
  out.sequenceNumber = sub.nextSeq(pkt);      // contiguous seq via per-sub offset
  out.timestamp = pkt.timestamp + sub.tsOffset; // align timebase across source switches

  // Re-mark the extension carrying transport-wide sequence number for congestion control.
  out.setTransportWideSeq(sub.twccSeq++);     // SFU owns the TWCC sequence per egress

  const srtp = sub.encrypt(out);              // re-wrap as SRTP for this subscriber
  sub.transport.send(srtp);
}

The nextSeq helper is where most bugs live. It must produce a strictly increasing sequence with no holes from the subscriber’s perspective, even as the router drops packets belonging to layers that subscriber is not currently consuming. The clean implementation tracks a (lastForwardedSeq, offset) pair and recomputes the offset only at switch boundaries, never per packet.

Step 3 — Keyframe (PLI/FIR) requests and RTCP termination

A subscriber can only start (or recover) decoding at a keyframe. Whenever the router begins forwarding a new source or a new simulcast layer to a subscriber, that subscriber needs a fresh keyframe — but the SFU has no decoder to generate one, so it asks the publisher to emit one by sending a Picture Loss Indication (PLI) or Full Intra Request (FIR) over RTCP. The critical design decision is aggregation: if 200 subscribers each switch into a layer in the same instant, the SFU must not forward 200 PLIs upstream. It coalesces them into at most one PLI per source within a short debounce window (commonly 500 ms–1 s), because a keyframe is large and bursty and each one spikes the publisher’s bitrate.

This makes the SFU a full RTCP terminator: it does not pass RTCP through. Subscriber-side NACKs, PLIs, and receiver reports terminate at the server; publisher-side feedback the SFU generates itself. The SFU runs its own NACK responder per egress (retransmitting from a small per-subscriber send buffer) and translates subscriber loss into upstream PLIs only when retransmission cannot recover the gap.

// Aggregate keyframe requests so the publisher sees at most one PLI per debounce window.
class KeyframeRequester {
  constructor(publisher, debounceMs = 500) {
    this.publisher = publisher;
    this.debounceMs = debounceMs;
    this.lastSent = 0;
    this.pending = false;
  }

  request(reason) {                            // reason: 'layer-switch' | 'subscriber-join' | 'loss'
    const now = Date.now();
    if (now - this.lastSent < this.debounceMs) {
      this.pending = true;                     // coalesce; a PLI is already in flight
      return;
    }
    this.lastSent = now;
    this.pending = false;
    this.publisher.sendRtcpPli();              // single PLI upstream for all interested subscribers
  }
}

Tune the debounce against keyframe cost: too long and a joining subscriber stares at a frozen frame for over a second; too short and a churny room hammers the publisher’s encoder with intra frames, inflating its outbound bitrate and undoing the bandwidth savings that justified the SFU.

Step 4 — Verification and bandwidth allocation per subscriber

Forwarding correctness and per-subscriber bandwidth allocation are verified together because they share one signal: the transport-wide congestion control (transport-cc / TWCC) feedback each subscriber returns. The SFU reads each subscriber’s available outgoing bitrate from that feedback and caps what it forwards to that subscriber accordingly, so a participant on a 600 kbps cellular link receives a lower layer while a subscriber on fiber receives the full stream from the same publisher upstream. This per-subscriber budgeting is the foundation that the bandwidth-aware layer selection deep-dive builds switching logic on top of, and it leans on the same estimator theory documented in Bandwidth Estimation & Congestion Control.

Verify the data path by polling getStats() on both sides at 1 s intervals and reconciling counters:

// Verify forwarding integrity: bytes in roughly equal bytes out across active subscribers,
// and no egress stream is accumulating loss the SFU should have masked with NACK.
async function auditForwarding(publisher, subscribers) {
  const inStats = await publisher.pc.getStats();
  let inboundBytes = 0;
  for (const r of inStats.values()) {
    if (r.type === 'inbound-rtp' && r.kind === 'video') inboundBytes = r.bytesReceived;
  }

  for (const sub of subscribers) {
    const outStats = await sub.pc.getStats();
    for (const r of outStats.values()) {
      if (r.type === 'outbound-rtp' && r.kind === 'video') {
        // availableOutgoingBitrate from transport drives the per-sub cap.
        console.log(
          `sub=${sub.peerId}`,
          `sent=${r.bytesSent}`,
          `nackRecv=${r.nackCount}`,         // climbing NACKs → egress loss, check send buffer depth
          `layer=${sub.currentLayer}`
        );
      }
      if (r.type === 'transport') {
        sub.availableBitrate = r.availableOutgoingBitrate; // the per-subscriber budget
      }
    }
  }
}

A correct SFU shows inbound bytes on a publisher roughly tracking the sum of what it forwards (one upstream fanned to N subscribers), each subscriber’s nackCount bounded by what the per-egress send buffer can retransmit, and availableOutgoingBitrate per subscriber matching the layer the router chose to forward. Divergence between any two of those is the fastest way to localize a routing or allocation bug.

Edge Cases & Browser Quirks

• Chrome simulcast SSRC ordering. Chrome signals simulcast layers with the a=rid / a=simulcast SDP attributes and assigns SSRCs in a deterministic but version-dependent order. Older Chrome (pre-M90) used legacy SSRC-multiplexed simulcast; modern Chrome uses rid-based. The router must map rid → SSRC from the answer, not assume layer order from SSRC numbering.
• Firefox keyframe response latency. Firefox occasionally answers a PLI more slowly than Chrome and can ignore a FIR it considers redundant. Prefer PLI for routine keyframe requests and reserve FIR for hard decoder resets; debounce both on the same timer so the two never race.
• Safari and rid restrictions. Safari (WebKit) historically restricted simulcast to specific codecs and limited the number of usable layers; some versions reject more than two rid layers. Detect the negotiated rid set from the actual SDP answer per peer rather than configuring a fixed three-layer assumption.
• RTX/NACK SSRC pairing. Browsers negotiate a separate RTX SSRC for retransmissions. The SFU must forward and rewrite the RTX stream’s SSRC and sequence space independently, or NACK-driven recovery silently breaks on the subscriber side.
• Transport-cc extension negotiation. If transport-wide-cc is absent from a subscriber’s negotiated SDP, you have no per-subscriber bitrate signal and bandwidth allocation falls back to receiver reports only — far coarser. Confirm the extension is present on every egress before relying on TWCC-based capping.

Common Implementation Mistakes

• Forwarding RTCP through the SFU. Passing subscriber NACKs straight to the publisher floods the publisher and breaks per-subscriber recovery. Terminate all RTCP at the server and regenerate upstream feedback deliberately.
• One PLI per subscriber. Failing to aggregate keyframe requests turns a join storm into a keyframe storm that spikes the publisher’s encoder bitrate. Coalesce to one PLI per source per debounce window.
• Leaving sequence-number holes. Dropping packets for an unforwarded layer without maintaining a per-subscriber sequence offset creates gaps that trigger endless spurious NACKs and stall the jitter buffer. Always rewrite to a contiguous sequence.
• No keyframe on source/layer switch. Switching the source feeding a subscriber without requesting a keyframe leaves the decoder unable to start, producing a frozen or green frame until the next periodic intra. Request a keyframe at every switch boundary.
• Allocating bandwidth from publisher stats. Capping forwarded bitrate using the publisher’s upstream estimate instead of each subscriber’s transport-cc feedback sends the full stream to a congested subscriber and tail-drops it. Budget per subscriber, never per publisher.
• Sharing one send buffer across subscribers. A single retransmission buffer cannot serve subscribers with different loss patterns. Keep a small per-egress buffer so NACK responses are correct for each subscriber’s sequence space.

FAQ

Why does an SFU not need to decode media to forward it? Forwarding only requires rewriting RTP routing fields — SSRC, sequence number, timestamp — which sit in the unencrypted RTP header after SRTP is unwrapped. The encoded payload is copied byte-for-byte, so the server never instantiates a codec. That is precisely why an SFU costs a fraction of an MCU per stream and scales to far more concurrent participants on the same hardware.

How many PLIs should reach a publisher when 100 subscribers join at once? At most one per debounce window — typically one PLI per source every 500 ms to 1 s. Aggregation is mandatory because a keyframe is large and bursty; un-coalesced PLIs would force the publisher’s encoder to emit back-to-back intra frames and spike its outbound bitrate well beyond steady state.

Does the SFU re-encrypt media, or can it forward SRTP as-is? It re-encrypts. Each peer negotiates its own DTLS-SRTP context, so the SFU decrypts inbound SRTP from the publisher, performs its header rewrites in the clear, and re-encrypts per subscriber with that subscriber’s keys. End-to-end encryption schemes (insertable streams / SFrame) add a second media-layer encryption the SFU cannot read, but the transport SRTP is always per-hop.

What happens to a subscriber on a slow link receiving from a fast publisher? The router forwards a lower simulcast or SVC layer to that subscriber based on its transport-cc estimate, while other subscribers receive higher layers from the same upstream. The publisher encodes once; the SFU selects per subscriber. The switching thresholds and hysteresis are detailed in bandwidth-aware layer selection.

Related: this guide sits under Media Server Architecture: SFU & MCU; continue with bandwidth-aware layer selection in an SFU for the switching logic, compare topologies in SFU vs MCU Topologies, see how layers are chosen in Simulcast-Aware Forwarding, and ground the publisher side in Simulcast & SVC Implementation.