Bottling as an Investment Decision: An Analytical Reanalysis of Cost–Quality Trade-offs in the BOTTLED Benchmark

In the paper's words

Organisations that run millions of similar queries through large language model (LLM) APIs pay per instance, and the decision to invest in a cheaper task-specific artifact is today made by hand, if at all. The BOTTLED benchmark shows that LLM agents can make this investment autonomously — a capability its authors call "bottling" — but that strong zero-shot performance does not reliably transfer: 48 of 60 bottling runs fall below the lower bound of the 95% confidence interval of their model's zero-shot score, and 31 of 60 underperform a small-model distillation baseline. What the reported picture lacks is an explicit decision theory: when is bottling worth the setup cost, and how surprising are the observed failure rates?

The abstract, verbatim, and its opening sentence is the claim. Every line below is quoted from the text.

Numbers it reports

1. Under an independence assumption, the probability of observing 48 or more of 60 runs below a per-model 95% confidence floor is approximately $1.3 \times 10^{-65}$, so bottling failure is a structural, model-level property rather than per-run noise; by contrast, 31 of 60 runs below a distillation baseline is statistically consistent with a coin flip ($z \approx 0.26$, two-sided $p \approx 0.80$).

   from Abstract

2. Bottling can yield substantial savings — on query-product relevance classification, Opus 5 retains about 82% of its zero-shot macro-F1 at roughly 657 times lower reported cost, and recovers about 94% of the macro-F1 of Jev, a "system one" model built for cheap, repetitive inference, at a quarter of Jev's projected full-workload cost.

   from 1. Introduction

3. But strong zero-shot performance does not reliably translate into strong bottling: models with similar zero-shot scores can differ substantially after bottling, 48 of 60 bottling runs score below the lower bound of the 95% confidence interval of their model's zero-shot performance, and 31 of 60 runs underperform the stronger of two small-model distillation baselines with the same token budget.

   from 1. Introduction

4. An exact binomial reanalysis showing that the observed 48-of-60 failure count is inconsistent with independent per-run failures at any plausible rate (tail probability approximately $1.3 \times 10^{-65}$ under the 95% confidence-floor reading; $z \approx 38.4$ by normal approximation), derived in Section 4.2.

   from 1. Introduction

5. Across ten models and three tasks, strong zero-shot task performance does not reliably translate into strong bottling capability; 48 of 60 bottling runs score below the lower bound of the 95% confidence interval of their model's zero-shot performance, and 31 of 60 runs underperform the stronger of two small-model distillation baselines with the same token budget.

   from 2. Background and Related Work

How it is done

1. The BOTTLED benchmark shows that LLM agents can make this investment autonomously — a capability its authors call "bottling" — but that strong zero-shot performance does not reliably transfer: 48 of 60 bottling runs fall below the lower bound of the 95% confidence interval of their model's zero-shot score, and 31 of 60 underperform a small-model distillation baseline.

   from Abstract

2. But strong zero-shot performance does not reliably translate into strong bottling: models with similar zero-shot scores can differ substantially after bottling, 48 of 60 bottling runs score below the lower bound of the 95% confidence interval of their model's zero-shot performance, and 31 of 60 runs underperform the stronger of two small-model distillation baselines with the same token budget.

   from 1. Introduction

3. The supplied abstract states the problem and the experimental design: LLM agents receive an entire unlabelled workload and must complete it under fixed time, compute and LLM API budgets, choosing their own approach such as training a small model or writing a reusable program.

   from 2. Background and Related Work

4. Each run is compared against the lower bound of the 95% confidence interval of its own model's zero-shot performance.

   from 3.3 Statistical reanalysis of the reported counts

What would falsify it

1. If the analysis holds, an operator can read any bottling result as an investment appraisal: a break-even workload, a quality-adjusted cost ratio, and a falsifiable expectation for how failure rates should behave if bottling skill were randomly distributed across runs.

   from 1. Introduction

2. The single most likely value under $H_0$ is $k=1$: with $\ln(0.975) \approx -0.025318$, $(0.975)^{59} = e^{59 \times (-0.025318)} = e^{-1.49376} \approx 0.22459$, so $P(X=1) \approx 60 \times 0.025 \times 0.22459 = 0.33689$ — the null predicts roughly one sub-floor run, not 48.

   from 4.2 Binomial null for the confidence-floor count

3. Reading (b) would not rescue the preservation null; it would relocate the finding from "models fail at bottling" to "zero-shot calibration does not transfer to bottled deployment", which is operationally similar for a decision-maker: in neither case can zero-shot score be used to predict bottled outcome.

   from 6. Discussion

Where the claim stops

1. No new data are collected, simulated, or assumed beyond explicitly labelled modelling assumptions.

   from 3.5 Scope and inputs

2. The setup cost $C_0$ is not reported in the supplied abstract; it is therefore carried symbolically, and all projections that require a numeric $c_L$ or $C_0$ are labelled as projections with their assumptions stated.

   from 3.5 Scope and inputs

3. Reading (b) would not rescue the preservation null; it would relocate the finding from "models fail at bottling" to "zero-shot calibration does not transfer to bottled deployment", which is operationally similar for a decision-maker: in neither case can zero-shot score be used to predict bottled outcome.

   from 6. Discussion

Where to read

1. Abstract258 words · 1 min
2. 1. Introduction616 words · 3 min
3. 2. Background and Related Work1,313 words · 6 min
4. 3. Methods710 words · 3 min
5. 4. Analysis1,165 words · 5 min
6. 5. Results309 words · 1 min
7. 6. Discussion725 words · 3 min
8. 7. Practical Implications and Engineering Pathways708 words · 3 min
9. 8. Conclusion205 words · 1 min
10. Provenance and AI Involvement197 words · 1 min