Quantisation trade-offs for on-device browser agents

A browser agent is not a chatbot. When someone asks a chat interface a question, they will tolerate a two-second pause because the reply streams and reading absorbs the wait. When someone clicks “summarise these tabs” in a browser, they are blocked on a foreground task, and the perceived cost of a pause is much higher. That single difference reshapes which local models are actually viable.

We wanted a defensible answer to a question we kept being asked by alpha testers: what should I actually run? The marketing answer — “any GGUF that fits in memory” — is technically true and practically useless. So we built a harness, picked four machines that match the hardware distribution in our alpha, and measured fourteen configurations against the constraints a browser agent actually has.

The four constraints

We treat a configuration as viable only if it clears all four of the following. These thresholds came from watching session recordings, not from a spec sheet.

• Time to first token under 900 ms. Past roughly one second, testers began clicking elsewhere, and the agent output arrived in a context they had already left.
• Sustained throughput above 18 tokens per second. Below this, a 400-token summary takes longer than reading the tabs would have.
• Peak resident memory under 60% of system RAM. The browser still has to render pages. A model that starves the compositor produces a janky browser, which testers blamed on the browser rather than the model.
• Extraction accuracy within 5 points of the FP16 baseline. An agent that cites the wrong figure is worse than no agent.

The throughput picture is roughly what you would expect from parameter count and memory bandwidth, with one exception. The 14B model on the 16 GB machine does not degrade gracefully; it falls off a cliff. Once the working set exceeds available unified memory, the operating system begins paging, and throughput drops by more than an order of magnitude rather than tapering. There is no useful middle ground to tune into.

Time to first token is the constraint that binds

Throughput gets the attention, but in our measurements time-to-first-token (TTFT) was the constraint that eliminated the most configurations. TTFT is dominated by prompt processing, and a browser agent's prompts are long: the semantic index retrieves passages from open tabs, history, and local documents, so a typical agent turn carries 2,000–6,000 tokens of context before the user's question.

Three findings stand out. First, the 3B models clear the budget everywhere, including on the oldest machine we tested. Second, the 7–9B class clears it on everything with unified memory and an NPU or a modern GPU, which describes most machines bought since 2023. Third, the 2.3× penalty we measured when a model spills to swap is not recoverable by any amount of prompt engineering; the only fix is a smaller model or more memory.

Below Q4, the failure mode changes character

The most consequential result was not about speed. Aggressive quantisation is usually discussed as a quality-for-memory trade, with the implication that you lose a little polish. On extraction tasks — pull the effective date out of this contract, find the figure this paragraph cites — that framing is wrong. The degradation is not graceful.

Summarisation holds up reasonably well down to Q3, which is why quantisation benchmarks that lead with summarisation scores paint an optimistic picture. Extraction and citation accuracy fall much faster. At Q2_K, the model still produces fluent, confident, well-formatted answers — it simply attributes them to the wrong source about four times in ten. For an agent whose entire value proposition is “it cites what it actually read,” that is a disqualifying failure, and it is invisible to anyone eyeballing output quality.

A fluent wrong answer is more expensive than a slow right one. We now gate every quantisation on citation accuracy before we look at throughput at all.

Where quality and memory actually trade

Plotting all fourteen configurations makes the shape of the problem clear. There is a steep region below about 4.5 GB where small memory savings cost a great deal of accuracy, and a flat region above about 7 GB where additional memory buys almost nothing. The interesting frontier is narrow. Q4_K_M at 7–9B parameters sits almost exactly at the knee, which is why it is now our default recommendation and the configuration the alpha auto-selects when it detects 16 GB of unified memory.

Method

Each configuration was run 40 times per task class on an otherwise idle machine, with the browser open and five representative tabs loaded, because measuring inference on a quiescent system overstates what a browser agent will actually see. We report medians; the interquartile range was under 8% for every configuration except those that spilled to swap, where it exceeded 60%. Prompt contexts were sampled from real alpha sessions with participant consent, then scrubbed of identifying content and fixed as a static set so every configuration saw identical inputs.

Limitations

Our evaluation set is English-only, and we expect extraction degradation under quantisation to be worse in languages with richer morphology — we have not measured this and would not want anyone to assume the numbers transfer. The four machines are drawn from our alpha population, which skews toward recent, well-specified hardware; a median consumer laptop is slower than everything reported here. We measured a fixed point in a fast-moving field, and quantisation methods released after March 2026 are not represented.

The evaluation harness and the scrubbed prompt corpus are available on request for reproduction. Write to research@dartbrowser.com. We are particularly interested in results from hardware slower than anything in our sample.

Related work

Post-training quantisation for large language models is well characterised in the aggregate. Dettmers et al. [1] established that outlier features dominate degradation at 8-bit, and GPTQ [2] and AWQ [3] both show that careful per-channel treatment recovers most of the loss at 4-bit on general benchmarks. What that literature largely measures is perplexity and multiple-choice accuracy, both of which are aggregate scores over generation. Our contribution is narrower and more specific: we measure a task class — extraction with attribution — where the failure is silent, and we measure it against the latency and memory budget of a foreground browser action rather than an offline batch. On that axis the published picture is optimistic by a wide margin.

Artefacts

Everything below is published or available on request. A number nobody can reproduce is an advertisement, not a result.

References

1. Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS.
2. Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR.
3. Lin, J., Tang, J., Tang, H., et al. (2024). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. MLSys.
4. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS.
5. Gerganov, G., et al. (2023–). llama.cpp. Software.
6. Bhardwaj, M. (2026). Retrieval quality against index size for on-device browsing history. Dart Browser Research, DBR-2026-02.