Joules-per-Solution for Stochastic and Agentic Inference: Benchmarking Frontier and Agentic LLMs Against the Human Brain

In the paper's words

The joules-per-solution (J/S) metric, introduced in JPCUB P0 as a universal, physics-grounded measure of computational efficiency, assumed deterministic solvers: one run yields one solution. Large language models (LLMs) violate that assumption twice over: they are stochastic samplers whose outputs must be verified, and agentic systems execute graphs of sub-queries whose total token budget dwarfs a single inference. This paper extends J/S to stochastic and agentic inference.

The abstract, verbatim, and its opening sentence is the claim. Every line below is quoted from the text.

Numbers it reports

1. CROP reports 80.6% token reduction with nominal accuracy loss; quantization is shown to inflate reasoning tokens.

   from 2. Related Work

2. Task class LLM J/S (system) Human J/S (20 W) Ratio LLM/human Simple verifiable QA (factoid/MCQ) ~35–70 J 400–600 J ~0.06–0.17 Math word problems (GSM8K-class) ~90–180 J 900–1,800 J ~0.05–0.20 Graduate reasoning (GPQA-class, single pass) ~9,000–12,600 J 6,000–18,000 J ~0.5–2.1 GPQA + self-consistency ($n = 8$) ~55,000 J 6,000–18,000 J ~3–9 Agentic coding (SWE-bench-class) $10^{5}$–$5 \times 10^{6}$ J $1.4 \times 10^{5}$–$4.3 \times 10^{5}$ J ~0.2–36 Sampling transparency (anti-gaming A1): $n$ and $p_{q}$ are stated per row; temperature is not applicable to these derived estimates (it is a measurement-time parameter, required when measurements are reported).

   from 7. Task-Stratified Estimates

3. System Training energy Amortized per solution GPT-3 ~$4.6 \times 10^{12}$ J (1,287 MWh) 46 – $4.6 \times 10^{3}$ J/query (at $10^{11}$–$10^{9}$ lifetime queries) GPT-4-class ~$1.8$–$3.6 \times 10^{14}$ J (50–100 GWh; widely reported industry estimate, no primary disclosure) $1.8 \times 10^{3}$ – $3.6 \times 10^{5}$ J/query Human brain (development) ~$1.3 \times 10^{10}$ J (20 W $\times$ 20 yr, brain-only) ~$1.3 \times 10^{5}$ J/solution (at ~$10^{5}$ lifetime professional solutions) Frontier-model training amortizes to ~$10^{3}$ J/query only at $10^{11}$ lifetime queries; at $10^{9}$ queries it dominates inference (up to $3.6 \times 10^{5}$ J/query).

   from 8. Training-Amortization Asymmetry

4. Token-budget evidence shows accuracy rises sub-linearly in the token budget, with 60–80% budget cuts achieving nominal accuracy loss.

   from 9. The Reasoning-Budget Pareto Frontier

5. Its headline anchors: Llama3-405B consumes approximately 40–60 J per output token in FP16 on 16 H100 GPUs (FP8 reduces this by roughly 30%), MoE models consume about as much as dense models of their active size, and context growth from 2K to 10K tokens raises energy per token by roughly 3$\times$ for large dense models.

   from 2. Related Work

6. "Where Do the Joules Go?" measures 46 models, 7 tasks, and 1,858 configurations on H100 and B200, finding a 25$\times$ energy-per-response spread driven by task type: GPQA problem-solving averages 4,625 J per response versus 184 J for text conversation, because reasoning emits 10$\times$ the output tokens (mean 6,988 vs 717) and longer sequences cap batch size, raising energy per token.

   from 2. Related Work

How it is done

1. P0 surveyed fourteen domain-specific benchmarks and found none provide cross-domain comparability; the J/S metric was offered as the first universal arbiter, with anti-gaming provisions (pre-registration, adversarial validation, Pareto-frontier reporting, component audit, living benchmark protocol).

   from 1. Introduction

2. Because published LLM energy numbers are overwhelmingly GPU-only, we report both columns and use a system multiplier of 1.5 as the JPCUB lower bound.

   from 5. System Boundary and Anti-Gaming

3. A comparison that charges one side's amortized cost against the other's marginal cost is biased by up to three orders of magnitude and is rejected by this protocol.

   from 6. Human Brain Baseline

4. Task canon: verifiable tasks only (math exact-match, code unit-test, QA ground-truth, retrieval relevance); verifier and threshold pre-registered.

   from 10. Measurement Protocol Extension (J/S-LLM)

What would falsify it

1. P6-F1 (falsified if): an independent measurement, under the full J/S boundary and a standardized sampling regime, finds frontier/agentic $J/S \lt 10^{-1}\times$ the expert-human baseline at matched accuracy on $\geq 2$ of 3 canonical hard-reasoning task classes (GPQA-grade, SWE-bench-grade, verified multi-step math).

   from 11. Falsification Conditions and Calibration Register

2. P6-F2 (also falsified if): the $(J/S, \text{accuracy})$ frontier is not monotonically increasing with diminishing accuracy returns — i.e., if sampling or additional reasoning budget reduces J/S at matched accuracy (an efficiency free lunch), or if accuracy does not saturate as the budget grows.

   from 11. Falsification Conditions and Calibration Register

3. Equivalently, falsified if Proposition 1's monotonicity or the concavity of $p_q(b)$ fails under measurement.

   from 11. Falsification Conditions and Calibration Register

Where the claim stops

1. (ii) pass@1 figures are leaderboard-grade approximations, not measurements on identical task instances.

   from 12. Discussion and Limitations

2. (iii) Agentic token counts are assumption-graded (A12) pending instrumented traces.

   from 12. Discussion and Limitations

3. Every limitation is named, quantified, and audit-trailed in ESTIMATES.md.

   from 12. Discussion and Limitations

Where to read

1. 1. Introduction480 words · 2 min
2. 2. Related Work450 words · 2 min
3. 3. Notation and Conventions110 words · 1 min
4. 4. The Metric: J/S for Stochastic and Agentic Inference444 words · 2 min
5. 5. System Boundary and Anti-Gaming99 words · 1 min
6. 6. Human Brain Baseline122 words · 1 min
7. 7. Task-Stratified Estimates299 words · 1 min
8. 8. Training-Amortization Asymmetry116 words · 1 min
9. 9. The Reasoning-Budget Pareto Frontier99 words · 1 min
10. 10. Measurement Protocol Extension (J/S-LLM)99 words · 1 min
11. 11. Falsification Conditions and Calibration Register276 words · 1 min
12. 12. Discussion and Limitations159 words · 1 min