Speculative decoding is one of the genuinely elegant ideas in inference optimisation: run a small fast model to guess several tokens ahead, then verify all of them with one forward pass of the large model. When the guesses are good you get several tokens for the price of one. The published speedups are substantial and well replicated.
They are also measured on hardware with memory to spare. A browser agent runs on a laptop that is simultaneously rendering pages, and the draft model's weights have to live somewhere. We wanted to know whether the idea survives that constraint.
It works, where there is room
On the M4 Max with 36 GB, a 1B draft paired with a 9B target improved sustained throughput 2.1×, which is squarely in line with the literature. On the Ryzen machine with 32 GB the gain was 1.6×. Both of those machines have room for the draft model's weights without disturbing anything else.
On the 18 GB M3 Pro the picture inverts. A 9B target at Q4_K_M plus a 1B draft plus the browser's own footprint exceeds what the machine can hold, and the system begins paging. Measured throughput fell 38% below the non-speculative baseline. The draft model did not fail to help; it actively made things worse by consuming the headroom that kept the target model resident.
Acceptance rate is a task property, not a model property
We expected acceptance rate to be dominated by how well matched the draft and target models are. It is not. Across five pairings the variation by task class was far larger than the variation by pairing. Summarisation accepts 0.71 of drafted tokens on average, because prose is predictable and a small model guesses common continuations well. Structured extraction accepts 0.34, because the tokens that matter are exactly the unpredictable ones — a date, an identifier, a figure — and those are precisely what a 1B model gets wrong.
Speculative decoding accelerates the tokens that were easy anyway. For a browser agent, the expensive tokens and the important tokens are the same tokens.
| Task class | Acceptance | Effective speedup | Share of agent turns |
|---|---|---|---|
| Summarisation | 0.71 | 1.9× | 34% |
| Question answering | 0.58 | 1.6× | 28% |
| Draft composition | 0.63 | 1.7× | 14% |
| Structured extraction | 0.34 | 1.2× | 18% |
| Tool-call formatting | 0.29 | 1.1× | 6% |
Weighting by how often each task actually occurs gives a blended effective speedup of about 1.6× on a machine with headroom. That is a real gain and we ship it there. But it is materially below the headline figure, and it is smallest on the task class where latency hurts most, because extraction is the one where the user is waiting for a specific fact rather than reading a stream.
What we ship
Dart enables speculative decoding dynamically rather than as a setting. At model load we compute the combined resident footprint of target, draft, and a reserve for the browser, and enable speculation only if the total sits below 60% of system memory. On machines that clear the bar it is on by default; on machines that do not, the draft model is never loaded and the user sees no option, because an option that is always the wrong choice is not a feature.
Method
Limitations
The 60% memory threshold is calibrated on four machines and is almost certainly not the right number everywhere; it is a heuristic that avoided the cliff in our sample rather than a principled bound. Our task-class shares come from alpha session telemetry, which reflects an early and technical user population and would likely shift toward summarisation in general use, which would flatter these numbers. We did not evaluate tree-based or multi-candidate speculation, which may have different acceptance characteristics on extraction specifically, and that is the most promising direction we have not yet measured.
This work predates and motivated DBR-2026-05; the quantisation study reuses its harness and the same four machines. Both are available together: research@dartbrowser.com.
Related work
Speculative decoding was introduced by Leviathan et al. [1] and independently by Chen et al. [2], with Medusa [3] and tree-based variants extending the idea to multiple candidates. Those results are established and we reproduce them where memory permits. What the literature has had little reason to examine is the case where the draft model’s residency is itself the scarce resource, because server deployments rarely face it. Our contribution is a negative result in that regime and the task-conditional acceptance breakdown, which suggests the technique is least helpful precisely where an interactive agent needs it most.
Artefacts
Everything below is published or available on request. A number nobody can reproduce is an advertisement, not a result.
References
- Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. ICML.
- Chen, C., Borgeaud, S., Irving, G., et al. (2023). Accelerating Large Language Model Decoding with Speculative Sampling. arXiv.
- Cai, T., Li, Y., Geng, Z., et al. (2024). Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. ICML.
- Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR.
- Gerganov, G., et al. (2023–). llama.cpp. Software.
- Bhardwaj, M. (2026). Quantisation trade-offs for on-device browser agents. Dart Browser Research, DBR-2026-05.
Bhardwaj, M. (2025). Speculative decoding on consumer hardware: when the draft model pays for itself. Dart Browser Research, DBR-2025-01.
