Site logo WebRTC & Real-Time Media Architecture

Server-Side Recording & Composition

Recording a multi-party WebRTC session is not “save the stream to a file” — there is no single stream to save. A Selective Forwarding Unit relays each participant’s RTP independently, so what arrives at the recorder is a set of separately-encoded, separately-timed media flows that must be decoded, aligned, laid out, mixed, and re-encoded before anything resembling a watchable file exists. This guide is part of the Media Server Architecture: SFU & MCU guide, and it covers the server-side pipeline end to end: deciding between per-track and composite recording, decoding RTP on the server, building a compositor and an audio mixer, encoding to MP4/WebM, and keeping audio and video in sync across participants who join and leave at arbitrary times.

The audience is engineers running an SFU who now need durable recordings — for compliance, replay, transcription, or highlight reels — and who have discovered that browser-side MediaRecorder cannot capture what it never receives: the other participants’ original, full-quality streams. Server-side recording moves the capture to where every stream already converges. The goal of the pipeline below is a single, correctly-synchronized output (or a clean set of per-track outputs) produced without dropping frames or drifting audio over a one-hour call.

Server-side recording and composition pipeline RTP streams from the SFU enter a jitter buffer and decoder, then split into a video compositor and an audio mixer driven by a shared RTP timestamp clock, which feed an encoder that writes an MP4 or WebM file to storage. SFU RTP per peer Jitter buffer + decoder depacketize Video compositor grid / speaker Audio mixer sum + resample Encoder H.264 / Opus File MP4 / WebM shared RTP clock
RTP from the SFU is depacketized and decoded, split to a video compositor and an audio mixer driven by one timestamp clock, then re-encoded and muxed into a stored MP4/WebM file.

Step 1 — Choose per-track recording vs composite

The first decision dictates the entire pipeline cost, and it is irreversible without re-recording. There are two strategies.

Per-track recording writes each participant’s RTP to its own file (or its own track inside a container) with no decoding and no re-encoding. You essentially dump the depacketized media — VP8/H.264 frames and Opus packets — straight to disk alongside a timing manifest. CPU cost is near zero, quality is bit-exact, and a four-person call costs four cheap recorders. The downside: the output is not directly watchable as one video; you defer composition to an offline batch job or to the playback client.

Composite recording decodes every track, lays the video out into a single canvas, mixes all audio into one track, and re-encodes the result live. The output is one finished file. The cost is real: decoding plus encoding N video streams in real time is the single most expensive operation an SFU host performs, and it does not scale per-room the way forwarding does.

Dimension Per-track Composite
Server CPU minimal (no transcode) high (decode + encode N streams)
Output N files + manifest one watchable file
Quality bit-exact source re-encoded (generation loss)
Layout flexibility deferred to playback fixed at record time
Best fit compliance, archival, post-edit replay, sharing, livestream egress

A common production pattern is per-track for the durable archive plus an optional composite egress only for rooms that request it. If you need composite output but not in real time, record per-track and run the compositor as an offline job — same code path, no real-time deadline. The layout and synchronization mechanics below apply identically; only the deadline changes.

Sizing the composite recorder budget

Composite recording is an MCU bolted onto an SFU, and it inherits the MCU cost curve described in SFU vs MCU Topologies. Forwarding costs roughly 1–3 ms of processing per packet; decoding, scaling, mixing and re-encoding puts you back in the 80–200 ms transcode regime, and moves the work from a per-packet memcpy to a per-pixel operation whose cost scales with resolution times frame rate times participant count. Budget against that product, not against the room count.

Concretely: software-decoding one 720p30 VP8 stream costs roughly a third of a modern core, scaling and colour-converting it into a tile costs a little more, and software-encoding the single 720p30 composite costs about a full core on its own. A six-person room therefore lands near three to four cores for one output file, while forwarding that same room costs a small fraction of one. Two levers move the number materially: dropping the composite from 1280×720 to 960×540 cuts the encoder’s pixel rate by about 44% for output that is still perfectly legible on replay, and going from 30 fps to 24 fps removes another 20%. Hardware encoders (NVENC, Quick Sync, VA-API) take encoding off the CPU entirely, but only pay off if decoded frames stay in GPU memory for the scale-and-composite step — a pipeline that copies every frame back to system RAM to draw it and uploads it again spends more on PCIe transfers than it saved.

Because the cost is per output stream rather than per participant, resist any request for per-participant composites: that multiplies encode cost by N for output nobody watches.

Choosing a recording strategy A decision tree: if no single watchable file is required, record per-track only. If one is required but not during the call, record per-track and composite offline. If it is required live, a fixed layout gives live composite on a dedicated encoder node, while a flexible layout means per-track plus client-side composition at playback. Need one watchable file? Needed during the call? Fixed layout acceptable? Per-track only bit-exact, near-zero CPU Composite offline same code, no deadline Compose at playback layout chosen per viewer Live composite dedicated encoder node no yes no yes no yes Only the bottom-right leaf pays real-time decode plus encode for every room.
Three questions separate the cheap recording strategies from the one that costs a decode-plus-encode pass per participant in real time.

Step 2 — Decode RTP on the server

A recorder is an RTP receiver that never renders to a screen. It joins the room as a subscriber (often a special “recorder” peer the SFU treats like any other), terminates DTLS-SRTP, and receives the same forwarded RTP that a browser would. From there the server-side path diverges from a browser: you must reconstruct frames yourself.

The incoming RTP needs a jitter buffer before anything else. Packets arrive out of order and with variable spacing; a recorder that depacketizes in arrival order produces corrupt frames. Size the buffer for the worst-case network jitter you expect (typically 50–200 ms of depth) and reorder by RTP sequence number before handing complete frames to the decoder. Unlike a live renderer, a recorder can afford a deeper buffer — you are writing to disk, not to a human’s eyes, so trading 100–200 ms of added buffering for fewer dropped frames is almost always correct.

// Server-side RTP receiver feeding a jitter buffer, then a decoder.
// Runs in a Node SFU process; mediasoup-style PlainTransport delivers raw RTP.
const jitterBuffer = new JitterBuffer({ maxDepthMs: 150 }); // deeper than a live renderer

rtpReceiver.on('rtp', (packet) => {
  // packet: { sequenceNumber, timestamp, payload, marker, ssrc }
  jitterBuffer.push(packet);                  // reorder by sequenceNumber internally
});

// Drain complete frames on a fixed cadence aligned to the stream clock rate
jitterBuffer.on('frame', (frame) => {
  // frame.rtpTimestamp is in the codec clock (90000 Hz for video, 48000 Hz for Opus)
  decoder.decode(frame.payload, frame.rtpTimestamp, (rawFrame) => {
    // rawFrame is YUV (video) or PCM (audio); tag it with a normalized wall-clock ts
    pipeline.submit(frame.ssrc, normalizeTimestamp(frame.rtpTimestamp, frame.ssrc), rawFrame);
  });
});

The decoder itself is codec-specific. For video you decode VP8, H.264, or AV1 to raw YUV; the codec your participants negotiated is the codec you must decode, which is one more reason to pin codecs at the application layer as discussed in VP8 vs H.264 vs AV1 Codec Selection. For audio you decode Opus to PCM at 48 kHz. Critically, request a keyframe from each sender when the recorder joins (an RTCP PLI/FIR) — without a recent keyframe the decoder cannot start, and the first seconds of a recording will be black or smeared. Because that request costs every subscriber a bitrate spike, throttle it the way the SFU already throttles subscriber-driven requests, following the patterns in Keyframe Request Strategies in an SFU.

Loss concealment on the recorder’s own path

The recorder is a subscriber, so it has its own network path and its own packet loss — and that is where recording differs fundamentally from live viewing. A live viewer tolerates a corrupted frame because it disappears in 33 ms; a recorder writes that corruption into a file people will watch repeatedly. Every artifact you fail to repair at capture time is permanent, which justifies spending far more on repair than a real-time renderer would.

Three mechanisms carry that weight. Negotiate NACK/RTX on the recorder’s transport with a longer retransmission window than a viewer would use: at a 150 ms jitter buffer instead of a viewer’s 50 ms, a retransmission arriving 120 ms late is still in time to be placed correctly, so one round trip of repair is effectively free. For video, a lost packet invalidates the whole frame and the damage propagates through the rest of the group of pictures, so track concealed frames per sender and issue an RTCP PLI once concealment crosses a small threshold — rate-limited to one request per 2 s per sender, or a recorder glitch becomes a bitrate spike for every live participant. For audio, enable Opus in-band FEC, which piggybacks a low-bitrate copy of the previous 20 ms frame on the next packet and recovers isolated losses with no retransmission at all; the bitrate and packet-loss-percentage tuning behind that redundancy is covered in Tuning Opus Bitrate and FEC for Lossy Networks.

The cheapest fix is topological: put the recorder on the same host, rack, or at minimum the same VPC as the SFU forwarding to it. A recorder reachable only across the public internet at 1% loss bakes 1% loss into every archive, and no buffer depth changes that. Colocation also keeps the TURN relay’s 20–40 ms one-way penalty off the recorder’s path.

Step 3 — Lay out video and mix audio

Composition splits cleanly into two independent subsystems sharing one clock: a video compositor and an audio mixer.

The compositor maintains a canvas at the target resolution (1280×720 is a sane default for a grid) and, on each output frame tick — typically 30 fps, one tick every 33.3 ms — draws the most recent decoded frame from each active participant into its assigned rectangle. Layout is a pure function of the active participant set: a 2×2 grid for up to four, a speaker-plus-thumbnails view driven by audio energy, or a fixed presenter layout. The compositor must hold a “last known frame” per participant, because senders run at different and variable frame rates; if participant B sent its last frame 80 ms ago, you redraw B’s last frame rather than leaving a hole. Active-speaker layouts read short-term audio energy (RMS over a 100–300 ms window) to pick the main tile, and should debounce switches by 1–2 s so the layout does not flicker on every cough.

The audio mixer runs on its own clock at 48 kHz. It resamples every decoded PCM stream to a common rate, aligns each to the output timeline by its normalized timestamp, and sums the samples, applying a soft limiter or per-source attenuation to prevent clipping when several people speak at once. Mixing is where echo and double-talk artifacts surface; the same device-side hygiene from Audio/Video Track Management matters here, because a participant whose echo cancellation failed will inject that echo into the mix for everyone in the recording. Silence-suppressed (DTX) Opus streams send no packets during silence, so the mixer must fill those gaps with silence on the output timeline rather than stalling.

Output clock ticks versus uneven input streams A timeline over 300 milliseconds divided into 33.3 millisecond output ticks. Peer A delivers a video frame on every tick. Peer B stops sending for three ticks and the compositor redraws its held frame. The audio mixer receives no Opus packets during a DTX gap and pads silence. The composite output emits one frame on every tick regardless. Output clock — one tick every 33.3 ms at 30 fps 0 ms 100 200 300 Peer A video Peer B video hold hold hold Audio mix DTX DTX DTX Composite Peer B stalls for three ticks — the compositor redraws its held frame instead of leaving a black tile. Opus DTX sends nothing — the mixer writes 60 ms of explicit silence so the audio timeline never compresses.
The output clock, not the inputs, drives the timeline: every tick emits exactly one composited frame, filling sender gaps with a held frame and DTX gaps with padded silence.
// Compositor output tick: 30 fps. Draw each active participant's latest frame.
const TARGET_FPS = 30;
const FRAME_INTERVAL_MS = 1000 / TARGET_FPS;     // 33.3 ms per output frame

function renderCompositeFrame(outputPtsMs) {
  const layout = computeLayout(activeParticipants);  // e.g. 2x2 grid rectangles
  canvas.clear();

  for (const p of activeParticipants) {
    // Use the most recent decoded frame at or before this output PTS.
    // Senders run at different / variable fps, so reuse the last frame on a gap.
    const frame = p.frameQueue.latestAtOrBefore(outputPtsMs) ?? p.lastDrawnFrame;
    if (frame) {
      const rect = layout[p.id];
      canvas.drawScaled(frame, rect.x, rect.y, rect.w, rect.h);
      p.lastDrawnFrame = frame;                  // hold for the next gap
    }
  }
  encoder.pushVideoFrame(canvas.snapshot(), outputPtsMs); // PTS in output timeline ms
}

Aspect ratios, mid-stream resolution changes, and layout transitions

Two properties of the incoming frames break naive compositors, and both show up the first time a phone joins a desktop call. The first is aspect ratio: a laptop sends 16:9, an iPad may send 4:3, a handheld in portrait sends 9:16. Stretching a 9:16 frame to fill a 16:9 tile widens faces, and it is the most-reported visual defect in composited recordings. Pick a policy per tile class rather than globally — centre-crop small grid tiles, where losing the outer thirds of a portrait frame is invisible and a letterbox would waste most of the tile, and letterbox the large speaker tile, where cropping a shared slide destroys content. Fill the bars with the panel colour so the tile boundary stays visible.

The second is that a sender’s resolution is not constant. An encoder downscaling under congestion can drop from 1280×720 to 640×360 mid-sentence, and Chrome does so with no signaling event the recorder can subscribe to — the decoder simply starts emitting smaller frames. A compositor that configures its scaler once, on the first frame, crashes or silently produces garbage at that moment. Read dimensions off every decoded frame and rescale per frame, caching the scaler keyed on input geometry so the common case stays cheap.

Layout changes need the same care from the other direction. Recompute the layout whenever the active participant set changes, but keep the output canvas geometry and encoder configuration fixed for the whole recording: changing encoded resolution mid-stream forces a codec reconfiguration, and a meaningful share of players — including some hardware-accelerated mobile decoders — stall or show a black frame at that discontinuity. Cross-fading tile positions over 200–300 ms rather than snapping them costs one extra draw per tile and makes joins read as intentional rather than as a glitch.

Step 4 — Encode, mux, and verify

The composited canvas and the mixed audio buffer feed an encoder. Encode video to H.264 or VP8/VP9 and audio to Opus or AAC, then mux into a container: WebM for VP8/VP9 + Opus, MP4 for H.264 + AAC. Write a fragmented container (fMP4 or streamable WebM) so a crash leaves a playable file up to the last fragment instead of an unfinalized, unplayable blob — finalize the moov atom incrementally rather than only on clean shutdown.

The encoder is fed by presentation timestamps (PTS) derived from the shared output clock, not from arrival time. Video PTS advances by exactly FRAME_INTERVAL_MS per frame; audio PTS advances by samples-over-sample-rate. As long as both are derived from the same monotonic output clock, the muxer interleaves them correctly and the file plays in sync regardless of what the network did upstream; the normalization maths for turning per-sender RTP timestamps into that one timeline is worked through in Synchronising Audio and Video in Recordings.

// Verification pass after the recording finalizes — run this in CI and post-record.
const probe = await ffprobe(outputPath);              // shell out to ffprobe -show_streams
const v = probe.streams.find((s) => s.codec_type === 'video');
const a = probe.streams.find((s) => s.codec_type === 'audio');

console.assert(Math.abs(v.duration - a.duration) < 0.25,    // A/V drift under 250 ms
  `A/V duration drift too large: v=${v.duration}s a=${a.duration}s`);
console.assert(v.avg_frame_rate.startsWith('30'),          // encoder held target fps
  `unexpected fps: ${v.avg_frame_rate}`);
console.assert(probe.format.duration > 0 && a.channels >= 1,
  'file is unplayable or has no audio');

Verify three things on every recording: total duration matches the call duration within a second, audio and video durations agree within 250 ms (the sync budget), and the file opens in a standard player without a remux. Sample the output mid-call, not just at the start — drift accumulates, so a file that is in sync at second 5 can be 400 ms out at minute 30 if the audio and video clocks were ever allowed to diverge.

Storage, crash recovery, and watching a recording while it runs

Fragment size trades recoverability against overhead. Two-to-four-second fragments bound worst-case loss on a crash to a few seconds while keeping container overhead under a percent; sub-second fragments buy little extra safety and multiply index entries and upload requests. Upload each fragment as it closes rather than at the end — a one-hour 720p recording is roughly a gigabyte, and a process that only uploads on clean shutdown turns every OOM kill and node preemption into total data loss. Keep the last few fragments on local disk so a transient upload failure retries without stalling the encoder, and write a manifest of fragment order, start PTS, and duration so a restarted process resumes the same recording instead of starting a second file.

The recorder also needs a health signal, because its failure mode is silent: nobody notices a broken recording until someone asks for the file. The measurement that matters is encoder lag — the wall-clock cost of compositing and encoding one frame against the 33.3 ms you have. Once it consistently exceeds the frame interval, the input queue grows without bound and the process dies of memory exhaustion. Bound the queue and degrade on a policy you chose rather than one the allocator chooses for you.

// Recorder health loop: watch encode cost against the real-time budget.
const FRAME_BUDGET_MS = 1000 / 30;            // 33.3 ms per composited frame
let overBudgetStreak = 0;

function onFrameEncoded(encodeMs, queueDepth) {
  recorderMetrics.observe('encode_ms', encodeMs);       // export as a histogram
  recorderMetrics.gauge('queue_depth', queueDepth);     // frames waiting to encode

  overBudgetStreak = encodeMs > FRAME_BUDGET_MS ? overBudgetStreak + 1 : 0;
  if (overBudgetStreak > 90) {                          // ~3 s of missed deadlines
    degradeOutput({ fps: 15 });                         // shed load deliberately
    recorderMetrics.increment('degradations_total');
  }
  if (queueDepth > 120) dropOldestFrames(queueDepth - 120); // hard ceiling: ~4 s
}

Export those counters next to the forwarding metrics so one dashboard covers both the media path and the archive path, following the labelling conventions in Exporting SFU Metrics to Prometheus and Grafana. Carry the room and recording id on every series: when someone reports a bad file weeks later, replaying those series for that recording is the only reconstruction you get.

Edge Cases & Browser Quirks

Common Implementation Mistakes

FAQ

Should I record on the SFU host or on a dedicated recorder node?

Dedicate a node for composite recording. Live decode-plus-encode competes for CPU with packet forwarding, and an overloaded SFU drops media for live participants — which is far worse than a delayed recording. Per-track recording is cheap enough to colocate, but composite transcoding belongs on separate capacity, often a different machine class with hardware encoders — scale that pool on its own encoder-CPU signal rather than the forwarding metrics covered in Autoscaling SFU Nodes on CPU and Bandwidth.

How do I keep audio and video in sync over a long call?

Derive both audio PTS and video PTS from one monotonic output clock and normalize every input by its RTP timestamp, not arrival time. Pad Opus DTX silence explicitly so the audio timeline never compresses. Then verify with ffprobe that the two stream durations agree within roughly 250 ms, sampling mid-call rather than only at the start.

Can’t the browser’s MediaRecorder do this instead?

Only for the local participant’s own composited DOM, and only at the quality and layout that one client sees. It cannot capture other participants’ original full-resolution streams, it dies if that one client’s tab closes, and it gives you no server-side archive. Server-side recording at the Selective Forwarding Unit Design is the only place every full-quality stream converges.

What codec and container should I default to?

H.264 + AAC in fragmented MP4 is the most broadly playable output and benefits from hardware H.264 encoders. Choose VP9/Opus in WebM when you want better quality per bitrate and control the playback environment. Match the encoder to your audience’s players, not to the codecs your participants happened to send.

What resolution and frame rate should the composite output use?

720p at 24–30 fps covers almost every use. Recordings are watched, not interacted with, so the frame rate that matters is the one that keeps motion smooth, and 24 fps saves roughly a fifth of the encoder budget with no perceptible difference on talking-head content. Drop to 540p for grid-heavy rooms where no tile exceeds 640×360 anyway, and reserve 1080p for single-presenter screen shares where text legibility is the point. Whatever you pick, fix it for the whole recording.

How should I record a simulcast sender without decoding everything?

For per-track recording, subscribe the recorder to exactly one spatial layer — the highest — and write those RTP packets verbatim; keeping all three layers triples storage for footage you will never composite from. For SVC, capture the full bitstream, since the layers form one dependent stream and dropping enhancement layers at capture time is irreversible. Record the layer decision in the manifest either way, so a downstream compositor can tell a deliberate low layer from a participant who was congested throughout.

Related: return to Media Server Architecture: SFU & MCU, then read the deep-dive on Compositing Multi-Party Recordings Server-Side, and cross-reference Selective Forwarding Unit Design and Simulcast-Aware Forwarding for how the source streams reach the recorder.

Load Balancing & Scaling SFUs Guide Selective Forwarding Unit Design Guide SFU Observability & Metrics: Instrumenting a Media Server You Can Debug Guide SFU vs MCU Topologies Guide Simulcast-Aware Forwarding Guide Media Server Architecture: SFU & MCU Design, Scaling & Production Patterns Guide