Selective Forwarding Unit Design

A Selective Forwarding Unit (SFU) is the component that lets WebRTC scale past the mesh: instead of every participant encoding a separate stream for every other participant, each publisher sends one upstream to the server, and the server forwards that media — unmodified — to every subscriber who wants it. This guide is part of the Media Server Architecture: SFU & MCU guide, and its goal is concrete: build the forwarding core of an SFU that ingests RTP from publishers, routes packets to subscribers without transcoding, terminates RTCP, requests keyframes on demand, and allocates bandwidth per subscriber so one slow link never starves the room.

The defining property — the reason an SFU costs a fraction of a transcoding MCU per stream — is that it never decodes media. It moves encrypted RTP payloads between transports, rewriting only the routing-layer fields (sequence numbers, timestamps offsets, SSRCs) it must. That makes an SFU CPU-cheap and bandwidth-heavy, and it pushes all the hard decisions into which packets to forward and when to ask the publisher for help. The four steps below build that core; the edge cases and mistakes sections cover where browsers diverge and where naive implementations leak.

Step 1 — Publisher and subscriber transport handling

Every participant connects to the SFU over an RTCPeerConnection, but the SFU plays the role the browser usually delegates to a remote peer. Each connection terminates DTLS-SRTP locally: the server decrypts inbound SRTP from publishers and re-encrypts outbound SRTP to subscribers, because the routing-layer rewrites in later steps require touching RTP headers in the clear. A clean design separates two transport roles per peer — an ingest path that receives a publisher’s tracks, and one or more egress senders that deliver forwarded media to subscribers — even though both live on the same ICE/DTLS connection.

The connectivity tier is identical to the browser-to-browser case: the SFU advertises ICE candidates, gathers server-reflexive addresses, and falls back to relays on restrictive networks. Because the server sits in a known data center it usually offers only host candidates plus a TURN allocation; the candidate-filtering and trickle timing rules from ICE Candidate Gathering & Filtering apply unchanged, and a server-side relay still matters for subscribers behind symmetric NAT.

// Per-peer state: one ingest set of tracks, many egress senders.
class PeerSession {
  constructor(peerId, pc) {
    this.peerId = peerId;
    this.pc = pc;                       // RTCPeerConnection (or native equivalent)
    this.publishedTracks = new Map();   // Map<ssrc, IngestTrack>  — what this peer sends in
    this.egressSenders = new Map();     // Map<sourcePeerId, SubscriberSender> — what it receives
  }

  // Called when DTLS completes and SRTP keys are derived.
  onSrtpReady(decryptCtx, encryptCtx) {
    this.decrypt = decryptCtx;          // unwrap inbound SRTP from this publisher
    this.encrypt = encryptCtx;          // wrap outbound SRTP to this subscriber
  }
}

Keep ingest and egress decoupled so a publisher leaving tears down only the senders fed by it, not the subscriber’s whole connection. In max-bundle mode every track for a peer rides one 5-tuple, so a single ICE failure on the egress side affects all forwarded streams to that subscriber — budget ICE-restart retries at the usual maximum of 3 with a 3–5 s fallback before declaring the subscriber’s transport dead.

Why each hop terminates DTLS separately

Engineers new to server-side media routing often ask why the SFU cannot simply relay the publisher’s SRTP packets untouched and skip the crypto work entirely. The blocking constraint is the SRTP authentication tag. SRTP authenticates the RTP header along with the payload, so the moment you change a sequence number or an SSRC the tag no longer verifies and every receiver drops the packet as a forgery. Header rewriting and per-hop encryption are therefore the same decision: if you rewrite, you must re-authenticate, and re-authenticating requires the egress key.

That key comes from a DTLS handshake the SFU runs independently with every peer. The server publishes its certificate fingerprint in each SDP it sends and takes the a=setup:passive role for browser-initiated connections, letting the browser drive the handshake as the DTLS client. One handshake costs a single round trip once ICE has a valid pair, which is why connection setup time is dominated by candidate gathering rather than crypto — and why a relayed subscriber pays the usual 20–40 ms one-way TURN penalty on top of the handshake. The certificate and cipher-suite mechanics, including how fingerprints bind the media path to the signalling channel, are covered in DTLS-SRTP Security & Encryption.

Per-peer termination has an operational consequence worth planning for: the SFU owes every peer independent ICE consent traffic. Each connection needs its own STUN binding refresh inside the 30 s consent interval, so a node holding 2,000 peer connections is emitting a steady few thousand consent checks per 30 s window purely to keep NAT bindings alive. That is negligible bandwidth but non-negligible timer pressure, and it is a common reason a naive implementation starts dropping connections when its event loop stalls under load.

Step 2 — RTP packet routing without transcoding

This is the heart of the SFU. A publisher’s decrypted RTP packet arrives; the router decides which subscribers should receive it, rewrites the minimum set of header fields each subscriber’s stream requires, re-encrypts, and sends. No decode, no re-encode — the payload bytes are copied verbatim. The rewrites are mandatory because each subscriber sees a single continuous RTP stream even though the SFU may switch which publisher layer or source feeds it over the call’s lifetime.

Three fields demand per-subscriber rewriting. SSRC is remapped to a stable value the subscriber negotiated, so layer or source switches never look like a new stream. Sequence numbers must stay contiguous from the subscriber’s view: when you drop packets (a layer the subscriber isn’t receiving) or switch sources, you maintain a per-subscriber offset so there are no gaps that trigger spurious NACKs. RTP timestamps likewise need an offset when switching sources so the subscriber’s jitter buffer does not see a discontinuity. Everything below the header — the encoded payload — is copied byte-for-byte, while the extension block sitting between them has its own remapping rules that rewriting RTP header extensions when forwarding works through field by field.

// Forward one inbound RTP packet to a subscriber, rewriting routing fields only.
function forwardPacket(pkt, sub) {
  // pkt.payload is the opaque encoded media — never inspected or modified.
  const out = pkt.cloneHeaderOnly();          // copy header; share payload buffer

  out.ssrc = sub.outboundSsrc;                // stable SSRC for this subscriber
  out.sequenceNumber = sub.nextSeq(pkt);      // contiguous seq via per-sub offset
  out.timestamp = pkt.timestamp + sub.tsOffset; // align timebase across source switches

  // Re-mark the extension carrying transport-wide sequence number for congestion control.
  out.setTransportWideSeq(sub.twccSeq++);     // SFU owns the TWCC sequence per egress

  const srtp = sub.encrypt(out);              // re-wrap as SRTP for this subscriber
  sub.transport.send(srtp);
}

The nextSeq helper is where most bugs live. It must produce a strictly increasing sequence with no holes from the subscriber’s perspective, even as the router drops packets belonging to layers that subscriber is not currently consuming. The clean implementation tracks a (lastForwardedSeq, offset) pair and recomputes the offset only at switch boundaries, never per packet.

The per-egress send buffer and the NACK deadline

Because the SFU terminates RTCP, it also owns retransmission. When a subscriber NACKs sequence 41,207, no one upstream knows or cares — the SFU must have that packet cached in the subscriber’s own sequence space and resend it. The naive reading of “per-subscriber buffer” is that every egress keeps a private copy of every packet, which is ruinous at scale: a 2.5 Mbps high layer buffered for 500 ms is roughly 156 KB, so 400 subscribers on one node would hold about 62 MB of duplicated video that is byte-identical across all of them.

The refinement is to share the payload but never the sequence space. Cache each ingest packet once per publisher layer, and store per egress only a compact mapping from the subscriber’s rewritten sequence number back to the cached ingest packet plus the header deltas needed to reconstruct it. A NACK then costs a map lookup, a header rebuild, and a fresh SRTP encryption — a few microseconds of work against an object that already exists in memory. The rule the earlier warning encodes is about identity, not storage: two subscribers watching the same layer are at different sequence numbers, so a shared buffer keyed by ingest sequence will answer one of their NACKs with the wrong packet.

Buffer depth should be a time window, not a packet count, and the window that matters is round-trip time plus the receiver’s jitter buffer depth — commonly 200–500 ms in total. A retransmission that lands after the decoder has already advanced past that frame is pure waste: it consumes egress bandwidth on a congested link at exactly the moment the link is struggling. Enforce a deadline, drop NACKs for packets older than the window, and escalate to a keyframe request instead when the gap is unrecoverable. Feeding the retransmission-versus-keyframe ratio into a dashboard is what makes this tunable in production rather than guesswork, which is one of the counters worth wiring into Alerting on Freeze and Packet-Loss SLOs.

Step 3 — Keyframe (PLI/FIR) requests and RTCP termination

A subscriber can only start (or recover) decoding at a keyframe. Whenever the router begins forwarding a new source or a new simulcast layer to a subscriber, that subscriber needs a fresh keyframe — but the SFU has no decoder to generate one, so it asks the publisher to emit one by sending a Picture Loss Indication (PLI) or Full Intra Request (FIR) over RTCP. The critical design decision is aggregation: if 200 subscribers each switch into a layer in the same instant, the SFU must not forward 200 PLIs upstream. It coalesces them into at most one PLI per source within a short debounce window (commonly 500 ms–1 s), because a keyframe is large and bursty and each one spikes the publisher’s bitrate.

This makes the SFU a full RTCP terminator: it does not pass RTCP through. Subscriber-side NACKs, PLIs, and receiver reports terminate at the server; publisher-side feedback the SFU generates itself. The SFU runs its own NACK responder per egress (retransmitting from a small per-subscriber send buffer) and translates subscriber loss into upstream PLIs only when retransmission cannot recover the gap.

// Aggregate keyframe requests so the publisher sees at most one PLI per debounce window.
class KeyframeRequester {
  constructor(publisher, debounceMs = 500) {
    this.publisher = publisher;
    this.debounceMs = debounceMs;
    this.lastSent = 0;
    this.pending = false;
  }

  request(reason) {                            // reason: 'layer-switch' | 'subscriber-join' | 'loss'
    const now = Date.now();
    if (now - this.lastSent < this.debounceMs) {
      this.pending = true;                     // coalesce; a PLI is already in flight
      return;
    }
    this.lastSent = now;
    this.pending = false;
    this.publisher.sendRtcpPli();              // single PLI upstream for all interested subscribers
  }
}

Tune the debounce against keyframe cost: too long and a joining subscriber stares at a frozen frame for over a second; too short and a churny room hammers the publisher’s encoder with intra frames, inflating its outbound bitrate and undoing the bandwidth savings that justified the SFU. Choosing between PLI and FIR, debouncing per source versus per layer, and surviving a mass-join storm are worked through in keyframe request strategies in an SFU.

Rewriting sender reports so lip-sync survives

RTCP termination has a consequence that is easy to miss until users complain that voices are ahead of mouths. Receivers do not synchronise audio and video from RTP timestamps alone — those are per-stream clocks with arbitrary random origins, ticking at 48,000 Hz for Opus and 90,000 Hz for video. The correlation comes from the RTCP Sender Report, which pairs a wall-clock NTP timestamp with the RTP timestamp that was current at that instant. The receiver interpolates both streams onto that shared wall clock and holds one back until the other catches up.

Step 2 rewrote RTP timestamps with a per-source offset. If the SFU generates Sender Reports for its egress streams without applying the same offset to the SR’s RTP timestamp field, it hands the subscriber a mapping that is wrong by exactly that offset. The symptom is characteristic: media plays perfectly, no loss, no freezes, but audio and video drift apart by a fixed amount — often hundreds of milliseconds — and the drift changes abruptly at the moment of a source switch rather than creeping in gradually. Because the audio and video offsets are computed independently, the two errors do not cancel.

The fix is to treat SR generation as part of the rewrite path rather than a separate bookkeeping task: whenever the router updates a subscriber’s tsOffset, the SR generator for that egress must read the same value. Verify it by comparing getStats() on the subscriber, where healthy synchronisation shows the audio and video inbound-rtp reports converging on consistent estimatedPlayoutTimestamp values; a persistent split between them points straight at the SR mapping. The same NTP-to-RTP correlation is what makes multi-track alignment possible offline, as Synchronising Audio and Video in Recordings explains from the capture side.

Receiver reports need the inverse treatment. A subscriber’s RR describes loss and jitter in the SFU’s rewritten sequence space, not the publisher’s, so its fraction-lost and extended-highest-sequence fields are meaningless upstream. Translate them into the SFU’s own view of egress health and discard them rather than relaying — the publisher should only ever see feedback the server constructed from the ingest link’s actual condition.

Step 4 — Verification and bandwidth allocation per subscriber

Forwarding correctness and per-subscriber bandwidth allocation are verified together because they share one signal: the transport-wide congestion control (transport-cc / TWCC) feedback each subscriber returns. The SFU reads each subscriber’s available outgoing bitrate from that feedback and caps what it forwards to that subscriber accordingly, so a participant on a 600 kbps cellular link receives a lower layer while a subscriber on fiber receives the full stream from the same publisher upstream. This per-subscriber budgeting is the foundation that the bandwidth-aware layer selection deep-dive builds switching logic on top of, and it leans on the same estimator theory documented in Bandwidth Estimation & Congestion Control.

Verify the data path by polling getStats() on both sides at 1 s intervals and reconciling counters:

// Verify forwarding integrity: bytes in roughly equal bytes out across active subscribers,
// and no egress stream is accumulating loss the SFU should have masked with NACK.
async function auditForwarding(publisher, subscribers) {
  const inStats = await publisher.pc.getStats();
  let inboundBytes = 0;
  for (const r of inStats.values()) {
    if (r.type === 'inbound-rtp' && r.kind === 'video') inboundBytes = r.bytesReceived;
  }

  for (const sub of subscribers) {
    const outStats = await sub.pc.getStats();
    for (const r of outStats.values()) {
      if (r.type === 'outbound-rtp' && r.kind === 'video') {
        // availableOutgoingBitrate from transport drives the per-sub cap.
        console.log(
          `sub=${sub.peerId}`,
          `sent=${r.bytesSent}`,
          `nackRecv=${r.nackCount}`,         // climbing NACKs → egress loss, check send buffer depth
          `layer=${sub.currentLayer}`
        );
      }
      if (r.type === 'transport') {
        sub.availableBitrate = r.availableOutgoingBitrate; // the per-subscriber budget
      }
    }
  }
}

Splitting one subscriber’s budget across many publishers

availableOutgoingBitrate is a single number for the whole bundled transport, but a subscriber in a twelve-person room is receiving eleven streams over it. The allocator’s real job is dividing one budget among many sources, and the order in which it does so determines whether degradation looks graceful or catastrophic.

Reserve audio first and never let video bid against it. Audio is inelastic — Opus at typical voice settings of 24–32 kbps degrades badly below that floor, and a call where you cannot hear anyone is a failed call regardless of picture quality. Eleven concurrent audio streams cost roughly 350 kbps before any video is allocated, and that reservation should be deducted from the budget as a fixed cost. Distribute the remainder by visible importance rather than equally: the active speaker and any large tiles get high-layer allocations, thumbnails get the 1/4-scale layer, and participants scrolled out of view get nothing at all until they scroll back. Equal division is the failure mode that makes every tile mediocre at once.

Leave 10–15% of the estimate unallocated. The estimator is a moving target, and an allocator that fills the pipe exactly will overshoot on every downward revision. Overshoot on the egress path shows up as latency rather than loss, because packets queue in the SFU’s socket buffer and in intermediate routers before anything is dropped — so the first sign of a too-aggressive allocator is climbing round-trip time and jitter with packetsLost still near zero. Whether that estimate arrives as transport-wide feedback or as a coarser receiver-side REMB changes how quickly the allocator can react, a trade-off examined in Transport-CC vs REMB Feedback.

Finally, keep allocation decisions rate-limited independently of the 1 s stats poll. Reading the estimate every second is fine; acting on every reading is not, because each upgrade triggers a layer switch, each layer switch triggers a keyframe, and a keyframe costs the publisher a bitrate spike that can itself push a marginal subscriber back into congestion. A subscriber whose budget oscillates around a layer boundary will otherwise generate a keyframe every second indefinitely.

A correct SFU shows inbound bytes on a publisher roughly tracking the sum of what it forwards (one upstream fanned to N subscribers), each subscriber’s nackCount bounded by what the per-egress send buffer can retransmit, and availableOutgoingBitrate per subscriber matching the layer the router chose to forward. Divergence between any two of those is the fastest way to localize a routing or allocation bug. Ad-hoc polling is fine while building; once the server is live, push the same counters into time-series storage as shown in Exporting SFU Metrics to Prometheus and Grafana.

Edge Cases & Browser Quirks

• Chrome simulcast SSRC ordering. Chrome signals simulcast layers with the a=rid / a=simulcast SDP attributes and assigns SSRCs in a deterministic but version-dependent order. Older Chrome (pre-M90) used legacy SSRC-multiplexed simulcast; modern Chrome uses rid-based. The router must map rid → SSRC from the answer, not assume layer order from SSRC numbering.
• Firefox keyframe response latency. Firefox occasionally answers a PLI more slowly than Chrome and can ignore a FIR it considers redundant. Prefer PLI for routine keyframe requests and reserve FIR for hard decoder resets; debounce both on the same timer so the two never race.
• Safari and rid restrictions. Safari (WebKit) historically restricted simulcast to specific codecs and limited the number of usable layers; some versions reject more than two rid layers. Detect the negotiated rid set from the actual SDP answer per peer rather than configuring a fixed three-layer assumption.
• RTX/NACK SSRC pairing. Browsers negotiate a separate RTX SSRC for retransmissions. The SFU must forward and rewrite the RTX stream’s SSRC and sequence space independently, or NACK-driven recovery silently breaks on the subscriber side.
• Transport-cc extension negotiation. If transport-wide-cc is absent from a subscriber’s negotiated SDP, you have no per-subscriber bitrate signal and bandwidth allocation falls back to receiver reports only — far coarser. Confirm the extension is present on every egress before relying on TWCC-based capping.
• Extension rewriting can push packets over path MTU. Chrome sizes its RTP packets close to a 1200-byte payload budget already. If the egress negotiation includes an extension the ingest side did not carry, appending it grows the packet by a few bytes and can cross the path MTU, producing IP fragmentation or silent drops on tunnelled networks. Compute the worst-case extension block per egress at negotiation time and reduce your forwarding MTU accordingly instead of discovering it as unexplained loss on one network only.
• Extension ID mismatch between peers. Header extension IDs are negotiated per peer connection, so the numeric ID for abs-send-time on the publisher’s ingest is frequently not the ID the subscriber negotiated. Forwarding the ingest ID unchanged makes the receiver parse the extension as a different one entirely — the failure is usually invisible in stats and shows up as a bandwidth estimator that never converges.

Common Implementation Mistakes

• Forwarding RTCP through the SFU. Passing subscriber NACKs straight to the publisher floods the publisher and breaks per-subscriber recovery. Terminate all RTCP at the server and regenerate upstream feedback deliberately.
• One PLI per subscriber. Failing to aggregate keyframe requests turns a join storm into a keyframe storm that spikes the publisher’s encoder bitrate. Coalesce to one PLI per source per debounce window.
• Leaving sequence-number holes. Dropping packets for an unforwarded layer without maintaining a per-subscriber sequence offset creates gaps that trigger endless spurious NACKs and stall the jitter buffer. Always rewrite to a contiguous sequence.
• No keyframe on source/layer switch. Switching the source feeding a subscriber without requesting a keyframe leaves the decoder unable to start, producing a frozen or green frame until the next periodic intra. Request a keyframe at every switch boundary, and time the cutover as switching layers without visible glitches describes.
• Allocating bandwidth from publisher stats. Capping forwarded bitrate using the publisher’s upstream estimate instead of each subscriber’s transport-cc feedback sends the full stream to a congested subscriber and tail-drops it. Budget per subscriber, never per publisher.
• Sharing one send buffer across subscribers. A single retransmission buffer cannot serve subscribers with different loss patterns. Keep a small per-egress buffer so NACK responses are correct for each subscriber’s sequence space.
• Recomputing the timestamp offset on every packet. Deriving the offset from the current wall clock per packet instead of latching it once at the switch boundary introduces jitter into the timestamps themselves, which the receiver’s jitter buffer then tries to absorb — inflating playout delay for a problem the network never caused. Latch at the boundary; carry the constant.
• Running the forwarding loop alongside signalling and garbage collection. The forwarding budget is roughly 1–3 ms per packet path; a stop-the-world pause or a synchronous JSON parse on the same thread blows straight through it and shows up as bursty jitter across every subscriber at once. Isolate the media path from request handling, and treat a rise in forwarding latency variance as a scheduling bug rather than a network one.
• Assuming one node is the whole architecture. A forwarding core that is correct on a single server still needs a story for room placement and node failure once concurrency grows; design the routing table so a room’s participants can be steered to a chosen node from the start, as Load Balancing & Scaling SFUs sets out.

FAQ

Why does an SFU not need to decode media to forward it? Forwarding only requires rewriting RTP routing fields — SSRC, sequence number, timestamp — which sit in the unencrypted RTP header after SRTP is unwrapped. The encoded payload is copied byte-for-byte, so the server never instantiates a codec. That is precisely why an SFU costs a fraction of an MCU per stream and scales to far more concurrent participants on the same hardware.

How many PLIs should reach a publisher when 100 subscribers join at once? At most one per debounce window — typically one PLI per source every 500 ms to 1 s. Aggregation is mandatory because a keyframe is large and bursty; un-coalesced PLIs would force the publisher’s encoder to emit back-to-back intra frames and spike its outbound bitrate well beyond steady state.

Does the SFU re-encrypt media, or can it forward SRTP as-is? It re-encrypts. Each peer negotiates its own DTLS-SRTP context, so the SFU decrypts inbound SRTP from the publisher, performs its header rewrites in the clear, and re-encrypts per subscriber with that subscriber’s keys. End-to-end encryption schemes (insertable streams / SFrame) add a second media-layer encryption the SFU cannot read, but the transport SRTP is always per-hop.

What happens to a subscriber on a slow link receiving from a fast publisher? The router forwards a lower simulcast or SVC layer to that subscriber based on its transport-cc estimate, while other subscribers receive higher layers from the same upstream. The publisher encodes once; the SFU selects per subscriber. The switching thresholds and hysteresis are detailed in bandwidth-aware layer selection.

Does every new publisher force a renegotiation with every subscriber? Not if you plan the transceiver layout in advance. Adding an m-line requires a fresh offer/answer exchange, so a room where people join constantly will spend its life renegotiating. The common mitigation is to pre-allocate a pool of inactive transceivers on each subscriber’s connection at join time — say six video and six audio slots — and bind an arriving publisher to a free slot by switching the forwarded SSRC into it. That converts a signalling round trip into a local routing change, at the cost of negotiating capacity a subscriber may never use. Renegotiate only when the pool is exhausted.

How much CPU does forwarding one stream actually cost? The per-packet work is an SRTP decrypt, a header rewrite, an SRTP encrypt, and a socket write, which lands in the ~1–3 ms range end to end and is dominated by crypto and syscall overhead rather than the rewrite itself. That is why SFU capacity is usually bounded by network interface throughput and packet rate long before it is bounded by processor time — the opposite of a transcoding server, where an 80–200 ms decode/encode cycle per stream makes CPU the first wall you hit.

Related: this guide sits under Media Server Architecture: SFU & MCU; continue with bandwidth-aware layer selection in an SFU for the switching logic, compare topologies in SFU vs MCU Topologies, see how layers are chosen in Simulcast-Aware Forwarding, and ground the publisher side in Simulcast & SVC Implementation.