Probe-Before-Commit: Interactive Diagnostic Probing as a Decision-Theoretic Foundation for LLM Task Routing

In the paper's words

Static capability benchmarks poorly predict how well a given large language model (LLM) will perform on a specific task, yet most routing systems assign tasks to models using exactly such static signals. The gap is that per-task fitness — the probability that a specific model produces an acceptable answer to a specific task — is never measured before commitment. We propose operationalizing intelligent routing as a low-cost interactive probe: before committing a task, the router sends cheap diagnostic queries to candidate models, observes their responses, updates a per-model fitness estimate, and routes according to a decision matrix combining predicted quality, latency, and monetary cost.

The abstract, verbatim, and its opening sentence is the claim. Every line below is quoted from the text.

Numbers it reports

1. We formalize the probe as an expected-value-of-information (EVI) problem nested inside a routing decision, derive the posterior update for binary probe signals, and give a fully worked numerical example in which probe-informed routing raises expected utility from $0.5700$ to $0.6782$ — a net gain of $0.1082$ after probe costs of $ 0.001 per model ( $ 0.003 per task) — with a break-even per-probe cost of $c_p^{*} \approx$ $ 0.00821.

   from Abstract

2. Efficiency-focused work on LLM deployment shows that serving costs and latency constraints are structural: YouZhi identifies KV-cache memory overhead as the bottleneck that "inflates infrastructure costs and throttles scalability" in high-concurrency deployment, and reports that a naive architecture transition degrades Llama3-8B perplexity from $6.1$ to $25.8$ while its layer-adaptive transition limits the degradation to $6.1 \rightarrow 12.9$ [7].

   from 1. Introduction

3. On the other side, work on LLM stochasticity shows that output quality is not a fixed model property: AutoDroid observes a $2.1\%$ accuracy reduction when the temperature of GPT-3.5 is set to $0$ and a $3.8\%$ accuracy boost when it is raised to $0.7$ [1], which motivates defining fitness as a probability rather than a deterministic attribute.

   from 1. Introduction

4. It derives, with full arithmetic, a worked example demonstrating when probing beats static routing and when it barely pays, including the break-even probe cost of $c_p^{*} \approx \$0.00821$ and the precision requirement $\sigma_f \lesssim 0.0379$ (Section 4).

   from 1. Introduction

5. Its quantified results ground the cost side of our decision matrix: the layer-adaptive transition reduces perplexity degradation by $65\%$ relative to the TransMLA baseline on Llama3-8B ($6.1 \rightarrow 12.9$ versus $6.1 \rightarrow 25.8$), and the resulting YouZhi-7B and YouZhi-14B achieve average general-benchmark scores of $66.8$ and $68.0$ [7].

   from 2. Background and Related Work

6. We define per-task fitness as a probability of acceptable output and frame model selection as a utility maximization problem over quality, cost, and latency trade-offs; our utility function in Section 3.2 adopts exactly this structure, with the probe supplying the evidence term in $\mathbb{E}[q_i \mid \text{evidence}]$.

   from 2. Background and Related Work

How it is done

1. It specifies who would use the result, at what readiness level, and what the smallest falsifying experiment is (Sections 6 and 7).

   from 1. Introduction

2. ILR's own protocol estimates each LLM's capability parameter $\gamma_i$ offline from a randomly selected validation set of $1{,}000$ MATH samples before training, and compares this one-time measurement against a periodic-update strategy [5].

   from 2. Background and Related Work

3. A probe is a cheap diagnostic query sent to model $M_i$ before commitment, costing $c_p$ dollars and $\ell_p$ seconds.

   from 3.3 Probe model

4. All numbers below are derived from explicitly stated illustrative assumptions; none are empirical measurements.

   from 4. Analysis

What would falsify it

1. Static capability benchmarks poorly predict how well a given large language model (LLM) will perform on a specific task, yet most routing systems assign tasks to models using exactly such static signals.

   from Abstract

2. The framework gives router builders concrete design targets — a probe price ceiling and a precision floor — and states explicit falsification conditions for the claim that probing beats static routing.

   from Abstract

3. It specifies who would use the result, at what readiness level, and what the smallest falsifying experiment is (Sections 6 and 7).

   from 1. Introduction

Where the claim stops

1. The worked example establishes possibility, not prevalence: under the assumed priors, probe quality, and trade-off weights, probing pays with a wide margin.

   from 6. Discussion

2. Several assumptions could fail in deployment, and each failure mode has a checkable signature.

   from 6. Discussion

3. The assumed discriminative power ($a = 0.9$, $b = 0.3$) is generous.

   from 6. Discussion

Where to read

1. Abstract225 words · 1 min
2. 1. Introduction545 words · 2 min
3. 2. Background and Related Work1,420 words · 6 min
4. 3. Methods479 words · 2 min
5. 4. Analysis812 words · 4 min
6. 5. Results198 words · 1 min
7. 6. Discussion585 words · 3 min
8. 7. Practical Implications and Engineering Pathways722 words · 3 min
9. 8. Conclusion155 words · 1 min
10. Provenance and AI Involvement225 words · 1 min