Snorkel AI study measures the efficiency of local language models

A metric for local AI efficiency

The study, presented by Jon Saad-Falcon and Avanika Narayan of Stanford University and Hazy Research at a Snorkel AI Reading Group, introduces Intelligence per Watt, or IPW. The metric divides task accuracy by power draw, allowing researchers to compare models and hardware based on both capability and energy efficiency.

Snorkel AI says the project is intended to examine how much inference demand could move from data centers to laptops, workstations and other local devices. The researchers focus on chat, reasoning, agentic and coding tasks, while measuring accuracy alongside energy, latency and the compute required to serve each model.

The project is motivated by rapidly rising demand for inference. The source says Google showed demand growing more than 10x year over year at its latest I/O conference. It also says about 77% of that demand involves practical, information-seeking tasks that can be handled by smaller or local models.

Models, hardware and measured workloads

The evaluation covered 20+ local language models from the Qwen, GPT-OSS, Gemma and IBM Granite families. The models included both dense and mixture-of-experts systems, ranging from 1 billion to 200 billion parameters.

Researchers tested the models on 10 local accelerators spanning 2023 to 2025. The hardware came from Nvidia, AMD, SambaNova and Apple. Workloads included ChatGPT-type conversations, general reasoning and standardized professional-work benchmarks such as SuperGPQA and MMLU-Pro.

The study also looks beyond raw accuracy. Its measurements include energy use, latency and power, providing a way to compare the trade-offs involved in running AI locally rather than sending every request to a data center.

Accuracy and efficiency improved rapidly

According to the study, local language models achieved a 3x improvement in accuracy from 2023 to 2025. Intelligence efficiency, defined as the amount of intelligence delivered per watt, improved by about 2.3x each year, or 5.3x over the two-year period. Snorkel AI says better language models, rather than better hardware, drove most of that gain.

When measured as intelligence per joule, or intelligence delivered per unit of energy, the improvement reached 18x over 16 months. In that case, the source says hardware was the primary driver. More memory and compute, mixed-precision formats such as FP4, and memory techniques associated with Apple and Nvidia helped make larger local deployments possible.

The source points to the DGX Spark series, Mac minis and the M5 Pro as examples of hardware enabling deployments that were previously not possible. However, local accelerators still lag enterprise chips because they commonly serve one user and operate at lower batch sizes than cloud systems.

Routing could create hybrid deployments

Snorkel AI says one practical way to use these gains is to route requests between local and cloud models. A routing system that is only 80% accurate could reduce energy consumption by about 80%, compute consumption by upwards of 77% and cost by upwards of 73%, according to the study.

In a comparison between an Apple M4 Max and a B200, the M4 Max delivered about 2x to 3x less intelligence per watt and per joule. The source says bespoke inference accelerators such as SambaNova’s SN40L create a larger gap, particularly for intelligence per joule.

The researchers describe the likely future as “local by default but hybrid by design,” with most queries handled on laptops, workstations or on-premises systems. They say the remaining 5% or less, involving frontier coding, mathematics and science queries, would be sent to data centers. Snorkel AI also previewed Joules to GDP, a companion study measuring the dollar value and expert time produced per joule across 220 tasks in 44 occupations.