Exporting SFU Metrics to Prometheus and Grafana

A media server already counts everything worth knowing; the work is deciding which of those counters becomes a Prometheus series, which becomes a histogram bucket, and which never leaves the log stream at all. This guide is part of the SFU Observability & Metrics guide, and it settles one concrete question: the exact metric set, metric types and scrape configuration to expose from a forwarding node, plus the Grafana panels engineers actually open at 02:00 rather than the ones that photograph well.

Context & Trade-offs

Three decisions decide whether this pipeline earns its keep. The type you assign each signal decides whether the dashboard tells the truth. The labels you allow decide whether the database survives. The relationship between your internal sampling tick and the scrape interval decides whether the three seconds you are investigating exist in the data at all.

Metric type follows from the shape of the signal, never from convenience. Anything that counts occurrences — bytes written to subscriber sockets, PLIs sent upstream, layer switches, NACKs received — is a counter, exported cumulatively with a _total suffix and never pre-divided into a rate. That last rule is the one teams break: exporting sfu_bitrate_bps as a gauge you computed yourself looks friendlier, but it throws away the property that makes counters safe. rate() recognises the reset when a node restarts mid-window and skips it; your hand-computed gauge just draws a cliff, and the on-call engineer spends ten minutes deciding whether the media stopped or the process did.

Gauges are for genuinely instantaneous, bounded levels — streams currently alive, the estimated send ceiling right now. Histograms are for per-item distributions, and they are also the escape hatch from the per-participant labelling temptation. Observing each subscriber’s RTCP-derived loss ratio once per tick into sfu_subscriber_loss_ratio gives you histogram_quantile(0.95, ...) across every subscriber on a node without a single participant label anywhere in the series.

Resist the summary type while you are at it. A summary computes its quantiles inside the exporting process, which means the p99 it reports belongs to that node alone and cannot be combined with any other node’s — averaging twenty p99 values produces a number with no statistical meaning, which is nonetheless what most dashboards end up plotting. Histogram buckets are counters, so sum by (le) across the fleet is exact, and histogram_quantile on top of that is the only fleet-wide tail latency you can defend in a post-mortem.

The label budget is arithmetic, not taste. Six metrics labelled by node, dir, rid and codec across twenty nodes is roughly 120 to 500 active series and a scrape body around 12 KB that transfers in single-digit milliseconds. Add participant_id and the same six metrics become well over a million series; at roughly 3 KB of head-block memory per active series that is several gigabytes of resident state, the exposition body grows to tens of megabytes, and the default 10 s scrape_timeout starts killing scrapes before the body finishes. The failure is not gradual — the endpoint works fine in staging with four participants and takes the database down the first Monday morning.

The third decision is sampling. Internally you tick at 1 s, matching the getStats() cadence clients use so the two views line up; Prometheus scrapes at 15 s by default. Those cadences are compatible only because counters and histograms are cumulative: fifteen ticks of increments accumulate between scrapes, so the integral is exact even though the shape is lost. A gauge has no such protection. A 3 s stall that begins and ends between two scrapes simply never happened.

The corollary is a query rule. rate() needs at least four samples inside its window to be stable, so at a 15 s scrape use [1m] on incident dashboards and [5m] in alert rules where a single failed scrape should not blank the panel. Do not reach for a 1 s scrape interval to “see more” — it multiplies storage fifteenfold and still cannot resolve a 200 ms glitch. That resolution belongs in logs and traces.

One naming note before the code. The convention is base units, so the strictly idiomatic name is sfu_forward_latency_seconds with bucket edges at 0.0001 through 0.008. Microseconds are easier to reason about when you are staring at a forwarding path that should complete in 1–3 ms, but if you keep them you must set the panel unit explicitly in Grafana, because the auto-detection reads the suffix and will happily label a 2,000 µs bar as 2,000 seconds.

Minimal Runnable Implementation

// One module owns every metric: name, type, help text, label keys and buckets.
// Registration validates label keys against an allowlist, so a stray participant
// id fails at process start rather than at 03:00 during an incident.
const ALLOWED = ['node', 'region', 'codec', 'rid', 'dir', 'kind', 'reason', 'trigger'];

const M = {
  // Counters: cumulative, base units, _total suffix, never pre-divided. rate() needs
  // the raw counter to spot the reset when this process restarts mid-window.
  bytes:    counter('sfu_forwarded_bytes_total', 'Bytes written to subscriber sockets'),
  pli:      counter('sfu_pli_sent_total', 'PLIs sent upstream, split by trigger'),
  switches: counter('sfu_layer_switch_total', 'Forwarded-layer changes, split by reason'),

  // Gauge: an instantaneous level with a bounded label set. Nothing transient goes
  // here — a gauge only ever exists at the instant a scrape reads it.
  streams:  gauge('sfu_active_streams', 'Forwarded streams currently alive'),

  // Histogram in microseconds, bucketed tightly at the low end: a healthy SFU
  // forwards in 1-3 ms, so wide buckets would hide the regression you are hunting.
  latency:  histogram('sfu_forward_latency_us', 'Receive path to send path',
                      [100, 250, 500, 1000, 2000, 4000, 8000]),

  // The cardinality escape hatch. One observation per subscriber per tick turns a
  // per-participant value into a distribution you can take quantiles of, with no id.
  loss:     histogram('sfu_subscriber_loss_ratio', 'RTCP downlink loss per subscriber',
                      [0.005, 0.01, 0.02, 0.05, 0.1, 0.2])
};

let exposition = '';   // rendered once per tick, served to every scraper unchanged

// 1 s tick. The packet path only bumped integers in a preallocated per-stream
// struct; all naming, labelling and text formatting happens here, off that path.
setInterval(() => {
  M.streams.set({ node: NODE, kind: 'video' }, streams.size);

  for (const s of streams.values()) {
    const l = { node: NODE, dir: 'out', rid: s.currentRid, codec: s.codec };
    M.bytes.inc(l, s.bytesOut - s.bytesOutPrev);   // export the delta, keep it cumulative
    s.bytesOutPrev = s.bytesOut;
    M.latency.adopt({ node: NODE }, s.latBuckets); // fold the per-stream buckets in
    M.loss.observe({ node: NODE }, s.lossFraction);// one sample per subscriber per tick
  }

  // Render the whole body once. A scrape then costs a string write, never a walk
  // of 4 000 live streams — so a scrape storm cannot reach the media path.
  exposition = registry.render();
}, 1000);

http.createServer((req, res) => {
  if (req.url !== '/metrics') { res.writeHead(404).end(); return; }
  res.writeHead(200, { 'Content-Type': 'text/plain; version=0.0.4' });
  res.end(exposition);        // at most 1 s stale, far below any sane scrape interval
}).listen(9091, '127.0.0.1'); // loopback plus a sidecar, never the media NIC

Two details in that body repay attention. The le="+Inf" bucket is not optional: omit it and histogram_quantile returns NaN for every quantile, silently, on a panel that still renders axes. And the counter values are cumulative since process start, which is why the panel query wraps them in rate() — plotting the raw counter produces a permanently rising line that looks identical whether the node is healthy or dead.

Reproduction Steps & Debugging Log Patterns

1. Start the exporter and count what it produces: curl -s localhost:9091/metrics | grep -vc '^#'. On a node with 4,000 forwarded streams the answer should be under 200. If it scales with participant count, a label leaked in.
2. Point a scrape job at it with the limits below, then confirm up{job="sfu"} is 1 and scrape_duration_seconds sits under 0.05.
3. Shape one subscriber’s link to 400 kbps for 10 s while its publisher sends three simulcast layers, and watch the panels react in order: loss quantile, layer switch, PLI, outbound bitrate settling on the lower layer.
4. Restart the node mid-window and confirm rate(sfu_forwarded_bytes_total[1m]) dips rather than plunging negative — that is the counter-reset handling you gave up if you exported a gauge.
5. Query the recording rules directly and check they agree with the raw expressions they replace. A rule that silently evaluates to no data because its sum by (le, node) grouping dropped a label is the classic way a dashboard looks fine while an alert quietly never fires.

; prometheus-job.conf — the scrape-side settings that actually matter.
; Written here in ini form; these map one-to-one onto the job's keys.
[job:sfu]
scrape_interval = 15s          ; 15x coarser than the exporter tick, and that is fine
scrape_timeout  = 10s          ; a body that cannot render in 10s is already broken
; Hard defences. Prometheus drops the whole scrape and marks the target down rather
; than ingesting a cardinality explosion — a loud failure beats a slow one.
sample_limit             = 20000   ; ~100x headroom over a healthy node
label_limit              = 8       ; refuse a metric that grew a seventh label key
label_value_length_limit = 64      ; a 36-char UUID as a label value is the smell
; Relabel away anything that slipped past the exporter allowlist.
metric_relabel_drop = participant_id|session_id|ssrc

[recording_rules]
; Pre-compute the two expressions every panel and alert shares. Evaluating them once
; per 15s beats re-evaluating a 20-node histogram_quantile on every dashboard refresh.
sfu:forward_latency_us:p99 = histogram_quantile(0.99, sum by (le, node) (rate(sfu_forward_latency_us_bucket[5m])))
sfu:subscriber_loss:p95    = histogram_quantile(0.95, sum by (le, node) (rate(sfu_subscriber_loss_ratio_bucket[5m])))

A healthy scrape and a leaking one are trivially distinguishable at the endpoint, before Prometheus ever sees them:

// healthy: series count is flat while participants churn
// 09:00  streams=1820  series=136  scrape_duration=0.004s  body=12KB
// 09:30  streams=3960  series=138  scrape_duration=0.005s  body=12KB

// leaking: series tracks participants, and the scrape starts timing out
// 09:00  streams=1820  series=21840  scrape_duration=0.41s  body=1.9MB
// 09:30  streams=3960  series=47520  scrape_duration=1.30s  body=4.1MB
// 09:41  prometheus: "sample limit exceeded" -> up{job="sfu"} 0, all panels blank

The panels that survive contact with a real incident are few, and they are chosen so that the first three answer “is it us?” before anyone argues about the network. Forwarded bitrate by node answers whether media is still moving. Subscribers grouped by forwarded rid is the single best quality panel in the set: when a room’s top-layer subscriber count collapses into the quarter-scale layer, everyone in that room is looking at a soft picture, and the selection logic behind that shift is covered in Forwarding Simulcast Layers by Subscriber Bandwidth. Forwarding-latency quantiles from the histogram separate a server regression from a last-mile one in about five seconds. PLI rate split by trigger exposes keyframe storms, which is why the trigger label is worth carrying — the taxonomy is set out in Keyframe Request Strategies in an SFU.

Panel hygiene matters more than panel count. Aggregate with sum by (node) rather than a bare sum, or one sick node averages into invisibility behind nineteen healthy ones — the same per-node visibility that the scale-out logic in Autoscaling SFU Nodes on CPU and Bandwidth depends on. Set bitrate panels to bits per second so the axis matches what people say out loud, pin every panel to the shared time range, and keep the log table adjacent so the jump from “the p95 loss quantile moved” to “these eleven sessions” is one click rather than a new tab.

Common Implementation Mistakes

• Exporting a pre-computed rate as a gauge. It reads nicely in a browser and destroys reset detection, unit clarity and the ability to re-window the query later. Export the counter; let rate() do arithmetic.
• Building the exposition body inside the HTTP handler. Two Prometheus replicas plus a curious engineer means three concurrent walks of every live stream. Render on the tick, serve a cached string.
• Bucketing latency by round numbers. Buckets at 1 ms, 10 ms and 100 ms put every healthy sample in the first bucket, so p99 is unmeasurable exactly where an SFU lives. Bucket tightly across 100 µs to 8 ms.
• Omitting the +Inf bucket or changing bucket edges live. The first makes quantiles NaN; the second makes historical comparisons meaningless, because the old and new series cannot be summed.
• Alerting straight off raw panel queries. Dashboard expressions use [1m] for responsiveness and go blank on one missed scrape; alert rules need [5m] and a for clause, which is the subject of Alerting on Freeze and Packet-Loss SLOs.
• Scraping the metrics endpoint over the media interface. Bind to loopback and let a sidecar expose it; a scrape arriving on the media NIC competes with packet forwarding for the same socket buffers.
• Deleting a metric the moment a stream ends. Removing the series makes the last increments unqueryable and leaves a gap that looks like an outage. Let the gauge fall to zero and leave counters in place until the process exits; staleness handling already covers the rest.

FAQ

Should the SFU push metrics instead of being scraped?

Scrape unless the node is short-lived. Pull gives you up as a free liveness signal and a natural back-pressure story: a struggling node serves a stale cached body rather than queueing pushes. Use a push gateway only for job-shaped work such as a recording composition that exits before the next scrape.

Can I keep per-participant metrics if retention is short?

No. Retention shortens how long a series is queryable, not how long it occupies the index — the head block holds every series touched in the current window regardless. Per-participant granularity belongs in logs joined by a correlation id, which is designed in Tracing One Participant Across SFU Logs.

What happens to my dashboards when a node is drained or replaced?

Its series go stale five minutes after the last successful scrape and then stop contributing, which is why sum by (node) panels briefly show a phantom node during a rolling deploy. Add the deploy revision as a bounded label if you need to tell versions apart, and accept the short overlap rather than trying to delete series on shutdown.

How do I reconcile a Grafana panel with what the browser reports?

Sample getStats() on the client at the same 1 s cadence, compare inbound-rtp.bytesReceived against the node’s sfu_forwarded_bytes_total delta over the same window, and expect agreement within a few percent; a persistent gap is path loss, not an accounting bug. Which client-side readings are trustworthy is covered in Interpreting getStats() for Congestion Signals.

Related: return to SFU Observability & Metrics for the full instrumentation design, or read Media Server Architecture: SFU & MCU for the forwarding internals these series describe.