Synchronising Audio and Video in Recordings

A recorder receives two independent RTP streams per participant, each stamped in its own codec clock with its own random starting value, and nothing in those timestamps says the two were captured at the same instant. This guide is part of the Server-Side Recording & Composition guide, and it resolves one problem: how to turn per-SSRC RTP timestamps into a single wall-clock timeline that still holds lip-sync at minute 55 of an hour-long call, and what to do at mux time when a few tens of milliseconds of skew survive anyway.

Context & Trade-offs

An RTP timestamp is a 32-bit counter that ticks at the payload’s clock rate — 90 000 Hz for video, 48 000 Hz for Opus — starting from a value the sender chose at random. Two SSRCs from the same browser tab, capturing the same person, share no base and no rate. Subtracting one from the other is meaningless. The only thing that relates them is the RTCP sender report, which carries a 64-bit NTP timestamp and the RTP timestamp that the sender’s clock read at that same instant. One SR per SSRC gives you one anchor point; the mapping falls straight out of it.

The first trade-off is latency versus completeness. RTCP is budgeted at roughly 5% of session bandwidth, so a sender report lands about every 5 seconds and the first one can be several seconds into the stream. A recorder that starts writing immediately has no mapping for its earliest packets. You either buffer 5 seconds of media and map it retroactively, or you start writing with a provisional offset and accept a correction later — which means rewriting timestamps you have already muxed. Buffering is almost always the right call: 5 seconds of decoded audio is trivial, and it removes an entire class of “first ten seconds are out of sync” bugs.

The second trade-off is the nominal rate. Treating 90 000 as exact is fine for a 90-second clip and fatal for a 90-minute one. Consumer capture clocks are typically 20–100 ppm off nominal, and drift is linear: at 50 ppm a video track gains or loses 180 ms per hour against wall time, against a lip-sync budget of roughly 250 ms of A/V offset before viewers notice. Because each participant has their own crystal, in a six-person recording you are correcting six independent slopes, not one. The fix is to stop trusting the nominal rate and measure the real one by least-squares regression across the accumulated sender-report pairs — 12 pairs is a minute of observation and already resolves the slope to a few ppm.

One detail catches most implementations by surprise: when the recorder subscribes through an SFU rather than directly to the sender, the sender reports it receives may not be the sender’s own. A forwarding node that rewrites sequence numbers and timestamps — which it must do when it switches a subscriber between simulcast layers, since each layer is a separate encoder with its own timestamp base — has to regenerate the sender report to match the stream it actually emitted. If it forwards the original SR unchanged alongside rewritten RTP, the anchor pair describes a timeline that no longer exists and every mapped timestamp is wrong by whatever offset the rewrite introduced. Confirm which of the two your server does before you trust a single anchor, and treat a fitted rate that lands hundreds of ppm from nominal as evidence of a rewrite mismatch rather than a bad crystal.

Re-anchoring naively to the newest sender report looks attractive — it is one line of code and the residual error is small — but it moves the mapping in discrete steps every 5 seconds, which shows up as a repeating micro-stutter in the muxed file. Regression with a slew limit gives you the accuracy without the steps: compute the ideal offset, then move the applied offset toward it no faster than about 1 ms per second so a correction is spread over time rather than injected as a jump.

Minimal Runnable Implementation

// Per-SSRC clock mapper: RTP ticks -> shared wall clock, with measured rate and unwrap.
const NTP_EPOCH_MS = 2208988800000;   // seconds between 1900-01-01 and the Unix epoch, in ms
const TWO_32 = 4294967296;            // 2^32, the RTP timestamp modulus
const MAX_PPM = 250;                  // reject fits implying a crystal worse than 250 ppm

function ntpToMs(hi, lo) {
  // 64-bit NTP is 32.32 fixed point: integer seconds in hi, fraction of a second in lo
  return hi * 1000 + (lo / TWO_32) * 1000 - NTP_EPOCH_MS;
}

class ClockMapper {
  constructor(nominalRate) {          // 90000 for video, 48000 for Opus
    this.nominalRate = nominalRate;
    this.rate = nominalRate;          // refined from sender reports as they arrive
    this.pairs = [];                  // {wallMs, ticks} anchors, newest last
    this.rollovers = 0;
    this.lastRaw = null;
  }

  unwrap(rtp32) {
    // The 32-bit counter wraps every 13.25 h at 90 kHz — but the base is random, so a
    // stream that started near 2^32 wraps minutes in. Count rollovers explicitly.
    if (this.lastRaw !== null && rtp32 < this.lastRaw && this.lastRaw - rtp32 > TWO_32 / 2) {
      this.rollovers++;
    }
    this.lastRaw = rtp32;
    return this.rollovers * TWO_32 + rtp32;
  }

  onSenderReport(sr) {
    // sr = { ntpHi, ntpLo, rtpTs } straight out of the RTCP SR body
    this.pairs.push({ wallMs: ntpToMs(sr.ntpHi, sr.ntpLo), ticks: this.unwrap(sr.rtpTs) });
    if (this.pairs.length > 24) this.pairs.shift();   // ~2 min window at one SR per 5 s
    if (this.pairs.length >= 4) this.refit();          // need a few pairs before trusting a slope
  }

  refit() {
    // Least squares on ticks-vs-wallMs. The slope IS the sender's true clock rate.
    const n = this.pairs.length;
    const mx = this.pairs.reduce((s, p) => s + p.wallMs, 0) / n;
    const my = this.pairs.reduce((s, p) => s + p.ticks, 0) / n;
    let num = 0, den = 0;
    for (const p of this.pairs) {
      num += (p.wallMs - mx) * (p.ticks - my);
      den += (p.wallMs - mx) ** 2;
    }
    if (den === 0) return;
    const measured = (num / den) * 1000;                        // ticks per second
    const ppm = Math.abs(measured - this.nominalRate) / this.nominalRate * 1e6;
    if (ppm < MAX_PPM) this.rate = measured;                    // ignore obvious outliers
  }

  toWallMs(rtp32) {
    // Anchor on the newest pair, extrapolate with the MEASURED rate, not the nominal one.
    const a = this.pairs[this.pairs.length - 1];
    if (!a) return null;                                        // no SR yet: caller must buffer
    return a.wallMs + ((this.unwrap(rtp32) - a.ticks) / this.rate) * 1000;
  }
}

// Writer side: the timeline must advance even when a participant stops sending.
function fillGap(track, fromMs, toMs) {
  // Audio: emit exact silence for the wall-clock duration, never "wait for the next packet".
  if (track.kind === 'audio') track.writeSilence(Math.round((toMs - fromMs) * 48));  // 48 samples/ms
  // Video: hold the last decoded frame; a black frame is a visible flash on rejoin.
  else track.repeatLastFrame(fromMs, toMs);
}

toWallMs returning null is the case worth designing around rather than papering over. Until the first sender report lands, that SSRC has no position on the shared timeline at all, and there is no honest default — a provisional offset of zero is a guess that will be wrong by however long the sender’s capture pipeline is. Hold those packets in a per-SSRC queue capped at about 6 seconds of media, and drain the queue through the mapper as soon as the first anchor arrives. If the cap is reached with no sender report, that stream is unmappable and should be written as a separate track rather than mixed into a shared timeline you cannot justify. The same queue absorbs the ordinary case of video arriving a few hundred milliseconds before its first RTCP packet, which is otherwise the single most common source of “the first ten seconds are out of sync” reports.

The sequence below is what those calls look like against a real capture: two SSRCs from one participant report independently, the recorder fits each slope separately, and a mid-call drop forces the mapper to start over rather than extend the old fit across the gap.

Reproduction Steps & Debugging Log Patterns

1. Record a 10-minute call in which one participant plays a clapperboard-style stimulus: a short beep with a simultaneous full-frame white flash, once a minute.
2. Run ffprobe -show_entries packet=pts_time,stream_index over the finished file and note the audio and video PTS of each flash-beep pair.
3. Plot the audio-minus-video delta at minute 0, 5 and 10. A flat line is a fixed offset; a straight ramp is uncorrected clock drift; a step is a mishandled gap.
4. Kill one participant’s connection for 3 seconds mid-recording and confirm the padded silence exactly matches the wall-clock gap, not the packet count.
5. Re-run the mapper offline against the captured RTCP and compare its fitted rate to the nominal one — anything over 200 ppm means the fit ate an outlier.

Expected healthy log, one line per sender report:

// ssrc=0x4a1 kind=video pairs=4  rate=90004.1  ppm=+46  offsetMs=+3.1
// ssrc=0x91c kind=audio pairs=4  rate=47999.4  ppm=-12  offsetMs=-0.8
// ssrc=0x4a1 kind=video pairs=12 rate=90004.4  ppm=+49  offsetMs=+3.4   // slope stable
// gap ssrc=0x4a1 from=40000ms to=43210ms filled=3210ms frames_held=96
// gap ssrc=0x91c from=40000ms to=43210ms filled=3210ms silence_samples=154080
// slate check: t=0 delta=+11ms  t=300s delta=+14ms  t=600s delta=+16ms

A failing run reads differently in a diagnostic way. If ppm swings between +40 and -180 on consecutive fits, the regression window is too short or a duplicated sender report is being counted twice. If offsetMs climbs monotonically while ppm stays near zero, the mapper is using the nominal rate and ignoring its own fit. If silence_samples is far below 48 * gap_ms, the writer padded by packet count rather than wall clock, and the audio track will finish short — the same failure that Measuring Audio Latency and Jitter Buffer Delay surfaces on the live side. Tag every one of those lines with the participant identifier so a single stream can be followed end to end the way Tracing One Participant Across SFU Logs describes.

When a small offset survives all of that — typically 20–80 ms of capture-pipeline latency the sender never reported — fix it at mux time rather than by fudging the mapper:

# Shift the audio input 120 ms later relative to video. -itsoffset applies to the
# input that FOLLOWS it, and a positive value delays that input.
ffmpeg -i video.mkv -itsoffset 0.120 -i audio.opus \
  -map 0:v -map 1:a -c:v copy -c:a aac -b:a 128k \
  -af aresample=async=1:min_hard_comp=0.100:first_pts=0 \
  -fps_mode cfr -r 30 -movflags +faststart out.mp4

# aresample async=1 stretches or drops audio to track the video clock, correcting
# residual drift the offset alone cannot; min_hard_comp keeps it from reacting to
# anything under 100 ms, which would otherwise chew on normal jitter.
# -copyts is deliberately absent: let ffmpeg rebase to zero after the offset applies.

Common Implementation Mistakes

• Assuming audio and video RTP timestamps are comparable. They are separate counters at separate rates with separate random bases. Map each SSRC to wall clock through its own sender report before any comparison.
• Using the nominal clock rate for the whole recording. A 50 ppm crystal error is invisible at one minute and 180 ms out at one hour. Fit the real rate from the anchor pairs.
• Ignoring 32-bit rollover. The starting value is random, so a stream that begins near 2^32 wraps within minutes, and a naive mapper computes a timestamp roughly 13 hours in the past.
• Extending a dropped participant’s mapping across the gap. A reconnect brings a new SSRC and an unrelated timestamp base; the old anchors and fitted rate must be discarded, exactly as when the connection transitions through the states covered in Disconnected vs Failed ICE States.
• Padding gaps by packet count. Opus discontinuous transmission means missing packets are not missing time. Pad by wall-clock duration, which is why the DTX behaviour negotiated in Munging SDP to Prefer Opus DTX matters to the recorder.
• Correcting drift with a hard jump. Snapping the offset to the newest anchor every 5 seconds produces periodic stutter; slew the correction at about 1 ms per second instead.

FAQ

How much A/V offset can I actually leave in the file?

Roughly 250 ms end to end before lip-sync reads as broken, and viewers are considerably more forgiving of audio lagging video than leading it. Aim for under 40 ms measured at three points across the file so a mid-length recording never approaches the limit.

What if a sender never emits RTCP sender reports?

Then you have no anchor and must fall back to the SFU’s own receive timestamps, which folds network jitter into the timeline. If the forwarding path rewrites timestamps, keep the abs-send-time and playout-delay extensions intact per Rewriting RTP Header Extensions When Forwarding; an SFU that strips them removes the last usable timing signal.

Should the recorder correct drift, or should ffmpeg?

The recorder, for anything that changes over time. -itsoffset applies one constant shift and cannot follow a ramp, so use it only for a fixed capture-latency offset and leave rate correction to the mapper, with aresample=async=1 as a safety net for the last few milliseconds.

Related: return to Server-Side Recording & Composition, and read Compositing Multi-Party Recordings Server-Side for the layout and mixing stage that consumes these timelines, or Selective Forwarding Unit Design for how the streams reach the recorder in the first place.