The promise of local AI is simple: your data stays on your device, your inference costs nothing per token, and you are not beholden to any provider’s uptime or pricing changes. The problem has always been hardware. Large language models are memory-bandwidth-bound, and for most of the last three years, “consumer hardware” meant “slow.”
2025 changed this. The Apple M4 family, AMD’s Ryzen AI-MAX with its 96 GB unified memory option, and Qualcomm Snapdragon X Elite have all delivered hardware capable of running 70B-parameter models at speeds that are genuinely useful for interactive agentic tasks. We spent six weeks benchmarking the Dart Agent on this hardware to understand where the lines are today.
The benchmark setup
We tested the Dart Agent against three representative task classes:
Retrieval + synthesis. The agent was given 20 open tabs (varying content length: news articles, research papers, product pages) and asked to answer a question requiring synthesis across at least four sources. We measured time-to-first-token and total time-to-complete-answer.
Document extraction. A 180-page PDF was indexed and the agent was asked to extract and structure specific clauses, compare sections, and produce a summary. We measured indexing time and query-response latency.
Agentic workflow. The agent was asked to navigate to three URLs, extract pricing data from each, and produce a structured comparison. We measured end-to-end task completion time including navigation.
Models tested: Llama 3.3 70B (Q4_K_M quantisation via Ollama), Llama 3.2 11B (Q8), Mistral Small 3.1 22B (Q6_K), and Gemma 3 27B (Q6_K). All benchmarks were run five times and averaged, with the fastest and slowest results excluded.
Apple M4 Max (128 GB unified memory)
The M4 Max is, for now, the best consumer hardware for local LLM inference. With 128 GB of unified memory and memory bandwidth of approximately 546 GB/s, it can run Llama 3.3 70B at Q4_K_M quantisation comfortably in memory without swapping.
Llama 3.3 70B Q4_K_M: 28 tokens/second for generation tasks. At this speed, a 500-token synthesis answer appears in roughly 18 seconds. For the retrieval + synthesis task, end-to-end time including retrieval from the local semantic index: 23 seconds. This is usable. It is not fast, but for background research tasks where you are working in parallel, it is genuinely viable.
For the document extraction task, PDF indexing (180 pages) took 41 seconds. Subsequent query responses on the indexed document averaged 19 seconds. Acceptable for the use case.
Smaller models performed significantly faster: Mistral Small 22B reached 67 tokens/second; Llama 3.2 11B reached 112 tokens/second. For simpler tasks — single-tab summarisation, short-form answers, form-filling — an 11B model at 112 tokens/second is indistinguishable from a cloud API call in terms of perceived latency.
Apple M3 Pro (36 GB unified memory)
The M3 Pro with 36 GB cannot fit Llama 3.3 70B Q4_K_M entirely in memory (~41 GB required). With Ollama’s memory-mapping, the model partially swaps to SSD, and performance drops significantly: approximately 8 tokens/second. At this speed, a 500-token answer takes over a minute. This is not usable for interactive agentic tasks.
The practical ceiling for M3 Pro is the 22B class: Mistral Small 22B at Q6_K fits comfortably in 36 GB and achieves 34 tokens/second. This is the sweet spot for this hardware tier: fast enough to feel responsive, smart enough to handle most research and synthesis tasks well.
For the M4 Pro (48 GB), the picture improves: Llama 3.3 70B at aggressive Q3_K_M quantisation fits in memory and achieves approximately 16 tokens/second. Borderline useful; the 22B class is still the better trade-off for interactive use.
AMD Ryzen AI-MAX (96 GB HBM)
AMD’s AI-MAX with 96 GB of HBM memory is the only x86 consumer platform capable of running 70B models at full speed. Memory bandwidth on the AI-MAX 395 is approximately 256 GB/s — lower than M4 Max but sufficient to achieve 14 to 18 tokens/second on Llama 3.3 70B Q4_K_M, depending on the prompt and generation length.
This is slower than M4 Max but meaningfully faster than any other x86 option. For users who prefer Windows or Linux and need to run 70B-class models locally, AI-MAX is currently the only viable path.
Important caveat: Ollama’s ROCm support on AI-MAX was still maturing at the time of our benchmarks (late 2025). Some models showed instability on long contexts. We expect this to improve as driver support matures.
What this means for the Dart Agent
Our design philosophy for the Dart Agent is that the model is a parameter, not a constraint. You connect the model that fits your hardware and your use case. Based on these benchmarks, our guidance for late 2025 hardware is:
M4 Max / M4 Ultra (96 GB+): Llama 3.3 70B Q4_K_M is the recommended model for research and synthesis tasks. It is slow by cloud API standards but acceptably fast for background-task workflows. Mistral Small 22B for interactive use where latency matters.
M4 Pro (48 GB) / M3 Max (48 or 64 GB): Mistral Small 22B Q6_K as the primary model. Gemma 3 27B Q6_K as an alternative with different strengths. Reserve the 70B tier for overnight or background tasks where latency is not critical.
M3 Pro / M2 chips (36 GB or less): Llama 3.2 11B Q8 or Mistral Small 22B Q4_K for interactive tasks. 70B is not viable at interactive latency on this hardware tier today.
AMD AI-MAX: Llama 3.3 70B Q4_K_M is viable. Performance is below M4 Max but better than any other x86 option. Monitor ROCm stability updates from AMD.
Windows with NVIDIA RTX 4090 (24 GB VRAM): 70B models do not fit. Mistral Small 22B Q6_K at approximately 45 tokens/second is the practical maximum. VRAM is the constraint; unlike Apple unified memory, GPU VRAM is separate and cannot be augmented without buying a new card.
Where we think this goes in 2026
The hardware trajectory is improving faster than the software ecosystem expected. M5, expected in mid-2026, is projected to further increase memory bandwidth. AMD’s next AI-MAX generation and NVIDIA’s GB300 series (targeting consumer availability) will push the envelope further.
Our prediction: by end of 2026, running a 70B-class model at 40+ tokens/second on consumer hardware will be achievable at the $2,500 to $3,500 price point. At that speed, local inference becomes genuinely preferable to cloud APIs for most interactive tasks — not just for privacy, but for latency.
The Dart Agent is designed for this world. BYOM is not a workaround for people who cannot afford cloud APIs; it is the architecture for a future where local inference is faster, cheaper, and more capable than cloud inference for most tasks. We are in the early innings of that transition. The benchmarks above are the 2025 floor, not the ceiling.
Benchmark note: all tests conducted November–December 2025 using Ollama 0.5.1 on macOS 15.2 and Windows 11 24H2. Model quantisations: Q4_K_M, Q6_K, Q8 as indicated. Results are averages across five runs excluding outliers. Individual performance may vary significantly based on system load, thermal state, and model version. We welcome reproductions and corrections.