Comparing two designs on the same seed
Run a streaming simulation with seed 1 and again with seed 2 and you get two different sets of sampled numbers from the same design. Run it once with the shield on and once with it off using different seeds, and the difference you see is the shield plus whatever the dice did. Separating those two is most of the work in comparing delivery designs.
A seed makes randomness repeatable
The simulator's random numbers come from Rng in random.ts, a small mulberry32 generator. It has one piece of state, a 32-bit integer, so the same seed produces the same sequence on any machine. Every random decision in a run (which title a viewer asks for, how long a service takes) draws from it in order.
That is what makes a run a pure function. The Media Network Lab's default seed is 1337 (SIM_SEED), and the runner's contract is that state at substep k depends only on the scenario, the fault schedule and k. Scrubbing the timeline backwards works because the lab replays from the seed instead of storing history.
For comparison, the useful property is that two designs can face the same random draws. If design A and design B each pull a different number of values, the streams drift apart after the first divergence, and the audience stops being identical. The draw order is fixed, so the same design replays exactly. Designs that differ can still consume different draws, which is why the noise gets reported below.
Hold everything but one thing
In the package, compareRuns in learn/compare.ts takes a seed, two scenarios, a duration and an optional fault schedule. It applies withSeed to both scenarios, runs each headless for the same simulated time, applies the same schedule to both sides, and measures them the same way. It also carries an axis label (shield, cdnCount, latencyMode and so on), and its own doc comment says the label is only a label: the package cannot prove the two scenarios differ on that axis alone.
The Media Network Lab enforces that at the UI, though it does not call compareRuns. Its net/compare.ts runs each side with runScenario and decides significance with the package's deltaOf. Compare mode offers four axes (single against multi-CDN, shield against no shield, VOD against live, and HLS against LL-HLS against WebRTC), and each option is a function that applies one design choice to the lab's state. The LL-HLS option sets both the mode and the segment length, and the CDN axis adds a brownout fault to both sides. A and B are the same state with one option applied to each. The bar's tooltip says the same seed, audience and schedule apply on both sides.
One run is a sample
A delta between two runs is a sample of a difference, not the difference. compareRuns reports each metric as a, b and delta = b - a, and flags it significant only when the interval delta ± sqrt(hA² + hB²) excludes zero, where h is each side's 95% half-width. A zero delta is never significant.
Where the half-widths come from depends on the tier. For the sampled discrete-event region, the confidence module cuts a time series into 10 batches and applies a Student-t interval to the batch means, because neighbouring samples in a run are correlated and a naive standard error would be too small. Startup percentiles are the exception: they use an order-statistic interval on the samples instead. Across seeds there is replicationCI, which takes one value per seed. The aggregate tier is deterministic, so it carries no interval and counts as significant whenever the delta is non-zero.
In the lab a row goes grey when the delta is zero or its size is at most sqrt(hA² + hB²), the same test as above, and the legend calls those rows "within noise". A grey row is a result. It says this change did not measurably move that number.
Off the main thread
Fidelity costs time. Here the runner steps the simulation in 16.67 ms substeps and the lab plays at 1x, 10x or 60x, so the page cannot run that on the thread that paints it.
The runner API is the same for all three hosts. pickSimHost picks an Electron utility process when the desktop bridge exists, then a Web Worker, then an in-process runner for tests and server rendering. The runner posts a snapshot about every 100 ms (SNAPSHOT_EVERY_MS). In the Worker the snapshot's Float64Array buffer is transferred without a copy. Electron's message ports can only transfer ports, so there it is cloned, which is fine at that rate.
The lab compares on a single seed today. replicationCI is exported from the package but nothing in the lab calls it yet, and running the same comparison over a set of seeds is the obvious next step.