Switching Layers Without Visible Glitches

A layer switch that does not corrupt the decoder is only half the job. The other half is that nobody watching should be able to point at the moment it happened. This guide is part of the Simulcast-Aware Forwarding guide, and it deals with one narrow problem: given that the forwarder has already decided to change layers, how do you execute that change so the subscriber sees no freeze, no resolution pop, and no phantom packet loss? Three mechanics carry almost all of the result — how you obtain the decodable start point, how you number the packets you emit, and how you order the steps so the cheap invisible ones happen before the expensive visible ones.

Context & Trade-offs

There are exactly three ways a switch becomes perceptible. The first is a gap: the forwarder stops sending the old layer before the new one is decodable, and the receiver holds its last frame. Chrome’s inbound-rtp freeze accounting treats a frame interval longer than roughly 150 ms above the running average as a freeze, so at 30 fps you have about four frame periods of slack before freezeCount ticks and a human notices. The second is a resolution pop: the picture jumps from 1280x720 to 640x360 and back within a couple of seconds, which reads as flicker even though every frame decoded correctly. The third is a numbering discontinuity: the subscriber sees a hole in the RTP sequence space, NACKs for packets the server deliberately dropped, and inflates its own loss statistics until the retransmission timer gives up.

The first mechanic is the choice between waiting for the publisher’s next keyframe and asking for one. Waiting is free in bandwidth terms and costs latency: with a publisher configured to emit a keyframe every 2 s you wait a mean of 1 s and a worst case of 2 s, during which the subscriber keeps watching the old layer. Requesting inverts that — an RTCP PLI or FIR travels upstream, the encoder produces an IDR, and the first decodable packet of the target layer arrives one round trip plus encoder latency later, typically tens to low hundreds of milliseconds. The cost is a bitrate spike: a keyframe is roughly 3–5x the size of a P-frame at the same resolution, and a publisher fielding requests from many subscribers at once can push its own uplink into congestion, which is why the dedup and pacing rules in Keyframe Request Strategies in an SFU exist at all.

A common misconception is that the downward direction is free because the lower layer is already arriving at the server. It is arriving, but the subscriber’s decoder has never seen it and holds no reference frames for it, so a spatial demotion needs a decodable start point just as much as a promotion does. What actually makes descent cheap is size: a 320x180 keyframe is roughly an eighth of the pixels of a 1280x720 one, so requesting it immediately costs almost nothing upstream. Descend by requesting, always, and never sit on a layer the link can no longer carry while you wait for a cadence keyframe.

The second mechanic is ordering. A subscriber losing a few hundred kbps does not necessarily need a different resolution — it may only need fewer frames. Dropping the top temporal ID of the layer you are already forwarding sheds around 30% of the bitrate, takes effect on the very next frame, needs no keyframe at all, and changes nothing the eye registers as a jump, because the resolution is identical and the motion simply gets slightly less smooth. This is the server-side twin of the client-side choice covered in degradationPreference: Resolution vs Framerate, and it means a well-built forwarder has two or three free rungs to walk down before it has to touch resolution at all.

Minimal Runnable Implementation

// Ordered quality ladder, best first. `sid` is the spatial layer (simulcast rid index),
// `maxTid` is the highest temporal id still forwarded. Consecutive entries sharing a
// `sid` are free steps: they change frame rate only and need no keyframe.
const STEPS = [
  { sid: 0, maxTid: 2, kbps: 1700 }, // 720p 30 fps  - everything
  { sid: 0, maxTid: 1, kbps: 1200 }, // 720p 15 fps  - temporal shed
  { sid: 0, maxTid: 0, kbps: 850  }, // 720p 7.5 fps - temporal shed
  { sid: 1, maxTid: 2, kbps: 500  }, // 360p 30 fps  - spatial move, needs a keyframe
  { sid: 1, maxTid: 0, kbps: 300  }, // 360p 7.5 fps - temporal shed
  { sid: 2, maxTid: 2, kbps: 180  }, // 180p 30 fps  - the floor
];

class GlitchFreeSwitch {
  constructor({ sendPli, ssrcForLayer, outSsrc }) {
    this.sendPli = sendPli;                 // (ssrc) => emit one RTCP PLI upstream
    this.ssrcForLayer = ssrcForLayer;       // (spatialIndex) => the publisher SSRC for it
    this.outSsrc = outSsrc;                 // fixed for the life of the subscription
    this.step = 5;                          // start at the floor and climb when proven
    this.pendingStep = null;                // spatial move awaiting a keyframe
    this.pliDeadline = 0;
    this.pliTries = 0;
    this.outSeq = 40000;                    // assigned, never copied from the source
    this.tsOffset = 0;
    this.lastOutTs = 0;
    this.lastSrcTs = null;
    this.spatialLockUntil = 0;              // dwell timer, spatial moves only
    this.recentSpatial = [];                // timestamps of committed spatial moves
  }

  // Called by the bandwidth policy with a target index into STEPS.
  goTo(next, now = Date.now()) {
    if (next === this.step || next === this.pendingStep) return;
    const sameSpatial = STEPS[next].sid === STEPS[this.step].sid;
    if (sameSpatial) {
      this.step = next;                     // temporal-only: live from the next frame
      return;
    }
    // Upward spatial moves obey the dwell timer; downward ones never do.
    if (next < this.step && now < this.spatialLockUntil) return;
    this.pendingStep = next;
    this.pliTries = 1;
    this.pliDeadline = now + 300;           // 300 ms per attempt, 3 attempts max
    this.sendPli(this.ssrcForLayer(STEPS[next].sid));
  }

  tick(now = Date.now()) {
    if (this.pendingStep === null || now < this.pliDeadline) return;
    if (this.pliTries >= 3) { this.pendingStep = null; return; } // give up, stay put
    this.pliTries += 1;
    this.pliDeadline = now + 300;
    this.sendPli(this.ssrcForLayer(STEPS[this.pendingStep].sid));
  }

  onPacket(pkt, now = Date.now()) {
    const want = STEPS[this.step];
    // Commit exactly on the first keyframe packet of the target spatial layer.
    if (this.pendingStep !== null &&
        pkt.sid === STEPS[this.pendingStep].sid && pkt.isKeyframe) {
      const prev = STEPS[this.step].sid;
      this.step = this.pendingStep;
      this.pendingStep = null;
      this.lastSrcTs = null;                // force a timestamp rebase on this packet
      this.recentSpatial.push(now);
      this.recentSpatial = this.recentSpatial.filter((t) => now - t < 30000);
      // Flap damping: 3+ spatial moves inside 30 s means the estimate is unusable.
      if (this.recentSpatial.length >= 3) this.spatialLockUntil = now + 30000;
      else if (STEPS[this.step].sid < prev) this.spatialLockUntil = now + 2000;
    }
    const cur = STEPS[this.step];
    if (pkt.sid !== cur.sid) return;        // wrong spatial layer: drop, emit nothing
    if (pkt.tid > cur.maxTid) return;       // shed frame: consumes no output seq number
    this.emit(pkt);
  }

  emit(pkt) {
    if (this.lastSrcTs === null) {
      // Rebase once per splice: continue the subscriber clock from the last frame sent,
      // one frame period (3000 ticks at 90 kHz = 33 ms) after it.
      this.tsOffset = (this.lastOutTs + 3000 - pkt.timestamp) | 0;
    }
    this.lastSrcTs = pkt.timestamp;
    pkt.timestamp = (pkt.timestamp + this.tsOffset) >>> 0;
    this.lastOutTs = pkt.timestamp;
    pkt.ssrc = this.outSsrc;                // one SSRC for the life of the subscription
    pkt.sequenceNumber = this.outSeq;       // assigned counter: gap-free by construction
    this.outSeq = (this.outSeq + 1) & 0xffff;
    this.send(pkt);
  }
}

Assigning outSeq from a counter rather than translating the source number through an offset is the detail worth arguing about. The offset approach, described in the parent guide, needs the offset corrected on every single dropped packet, and one missed correction leaves a permanent hole. A counter cannot leave a hole because it only advances on packets that actually go out. The price is that you lose the identity mapping the retransmission path needs, so keep a 512-entry ring buffer of outSeq → (sourceSsrc, sourceSeq) and resolve incoming NACKs through it; the same buffer is where you look up the header extensions you rewrote, following Rewriting RTP Header Extensions When Forwarding.

The dwell timer is the other half of stability, and it is a time constraint rather than a bandwidth one — the kbps entry and exit thresholds that feed goTo() are worked out separately in Forwarding Simulcast Layers by Subscriber Bandwidth. Here the rule is that a committed spatial move locks further upward spatial moves for 2 s, so a subscriber that has just dropped to 360p cannot bounce straight back to 720p on the next optimistic estimate. Downward moves ignore the lock entirely, because holding a layer the link cannot carry is worse than any flicker. The flap damper on top of that is deliberately blunt: three spatial commits inside 30 s means the estimate driving this subscriber is not trustworthy, so pin the ladder for 30 s and let the temporal rungs absorb whatever the link does in the meantime.

Reproduction Steps & Debugging Log Patterns

1. Drive goTo() from step 0 down to step 2 and back with no bandwidth change on the wire. Confirm zero PLIs are emitted and that framesPerSecond in the subscriber’s inbound-rtp stats halves and doubles while frameHeight stays at 720.
2. Force a spatial move from step 2 to step 3 and log every packet decision either side of the commit. The commit must land on isKeyframe === true and nothing else.

// tid=2 pkt sid=0 -> shed          (maxTid=0, no output seq consumed)
// tid=0 pkt sid=0 -> out seq=41287 (old layer still flowing while PLI is in flight)
// PLI sent ssrc=0x7A1C3B90 try=1
// tid=0 pkt sid=1 key=false -> drop (target layer, but not a start point yet)
// tid=0 pkt sid=1 key=true  -> COMMIT step=3, ts rebased +2841390
// tid=0 pkt sid=1 -> out seq=41288 (dense: the commit consumed no numbering)

3. Verify continuity from the receiving side. inbound-rtp.packetsLost must not move across the splice, nackCount must not increment within 500 ms of it, and framesDecoded must keep climbing at the same rate — the joint reading of those three counters is set out in Interpreting getStats() for Congestion Signals.
4. Break it deliberately: comment out the pkt.tid > cur.maxTid early return so shed frames advance the counter, and watch the failure signature appear.

// receiver: packetsLost 0 -> 214 over 4 s, nackCount 0 -> 96
// receiver: freezeCount 0 -> 3, totalFreezesDuration 1.14 s
// server: retransmit lookup miss for seq 41290 (never sent)

5. Starve the publisher’s keyframe path — block PLIs at the publisher — and confirm the retry logic gives up cleanly. You should see exactly three PLI sent ... try=N lines 300 ms apart, then pendingStep=null, and the subscriber should still be watching the old layer with an unbroken sequence run rather than a frozen frame.

Common Implementation Mistakes

• Cutting the old layer the moment the decision is made. Stopping the old stream while the PLI is in flight guarantees a freeze of at least one round trip. The old layer must keep flowing until the target layer’s keyframe has actually been emitted downstream.
• Letting dropped packets consume output sequence numbers. Whether you use offsets or a counter, every packet the forwarder discards must be invisible in the output numbering. Otherwise the receiver NACKs for packets that were never sent and reports loss that never happened.
• Jumping resolution when a temporal step would have done. A forwarder with only three rungs — high, medium, low — is forced into a visible resolution change for every 300 kbps of drift. Interleaving temporal rungs turns most of those into invisible frame-rate changes.
• Copying source timestamps straight through at the splice. The two layers run on independent 90 kHz baselines; forwarding the new one unrebased makes the subscriber’s jitter buffer see a huge time jump and either stall or dump its contents. Rebase once, at commit, then leave the cadence alone.
• Unbounded keyframe retries. A publisher whose keyframe path is broken will happily absorb a PLI per packet. Cap at 3 attempts 300 ms apart, then abandon the move and stay on the layer you have.
• No flap damping. Hysteresis on the bitrate thresholds alone does not stop oscillation when the estimate itself is noisy. Count committed spatial moves in a rolling 30 s window and pin the ladder when the count says the input cannot be trusted.

FAQ

Should I ever wait for a natural keyframe instead of requesting one?

Only for pre-emptive moves nobody is waiting on — for example promoting a subscriber whose bandwidth has been comfortable for 10 s, where an extra second at the old quality costs nothing. Every move driven by a user action or by congestion should request, because waiting turns a bounded delay into one that depends on the publisher’s encoder cadence.

Does temporal shedding need any keyframe handling at all?

No, and that is the whole point of doing it first. Frames of a higher temporal ID are, by construction, referenced by nothing else, so dropping them leaves every remaining frame decodable. The only requirement is that you read the temporal ID correctly for the codec in use — VP8 exposes it in the payload descriptor, while AV1 carries it in the Dependency Descriptor described in Configuring AV1 SVC Layers in WebRTC.

How do I prove in production that switches are actually invisible?

Instrument the pair that matters: a server-side counter of committed spatial moves per subscriber, and the client’s freezeCount and totalFreezesDuration from inbound-rtp at a 1 s polling interval. A healthy forwarder shows spatial moves without any correlated freeze increment; a ratio above a few percent means the cutover is landing early. Wiring those two series into the same dashboard is covered in Alerting on Freeze and Packet-Loss SLOs.

Related: the switching mechanics here sit inside the parent Simulcast-Aware Forwarding guide, take their target layer from Forwarding Simulcast Layers by Subscriber Bandwidth, and share their upstream request budget with Keyframe Request Strategies in an SFU.