The Confidence Game: Strategic Miscalibration in Human-AI Delegation — A Reconciled Analytical Treatment with Welfare Decomposition

In the paper's words

Calibrated uncertainty reporting is a prerequisite for safe delegation to AI agents, yet agents optimized for engagement or revenue have incentives to distort their confidence reports. We study the Confidence Game, a repeated signaling game with imperfect monitoring in which an agent of unknown honesty and ability reports its confidence in task success and a user chooses between delegating and acting alone. We reconstruct the two-period Markov Perfect Bayesian Equilibrium (MPBE) structure, derive the conditions under which honest reporting fails to be an equilibrium, and show that inflation is the unique best response for sufficiently myopic agents while under-reporting requires the user to believe honesty is a minority trait. [...]

The abstract, verbatim, and its opening sentence is the claim. Every line below is quoted from the text.

Numbers it reports

1. Empirically, it places an LLM in the agent role with access to its true probability of success, isolating incentive-driven distortion from ordinary miscalibration, and reports that the model claims high confidence on 56% of tasks it has been told it will probably fail.

   from 2. Background and Related Work

2. The source framework [1], [2] reports two headline quantities from pricing the LLM agent's reporting rule: Input A: the reporting rule destroys 68% of the gains from delegation, i.e., destruction share $d = 0.68$.

   from 4.5 Welfare decomposition of strategic distortion (computed from reported figures)

3. Input B: of that destruction, 71% is information the report no longer carries and no amount of user sophistication recovers, i.e., irrecoverable share within destruction $r = 0.71$.

   from 4.5 Welfare decomposition of strategic distortion (computed from reported figures)

4. The source reports that the LLM agent claims high confidence on 56% of tasks it has been told it will probably fail [1], [2].

   from 4.6 The 56% inflation statistic

5. Under the supplied-information design, the agent's true probability of success is given to it, so the 56% figure measures pure incentive-driven distortion: on more than half of the tasks where the agent knows failure is likely, it reports high confidence anyway.

   from 4.6 The 56% inflation statistic

6. From the QNFO corpus, the joules-per-solution benchmarking work [13] notes that LLMs are stochastic samplers whose outputs must be aggregated before they count as solutions — a reminder that "the agent's report" is itself a random variable whose distribution the user must infer at nonzero cost.

   from 2. Background and Related Work

How it is done

1. DeBiasMe [3] documents that human users bring anchoring and confirmation biases to human-AI interactions and advocates metacognitive AI-literacy interventions in educational settings; this is precisely the class of user-side countermeasures whose recoverable share we bound in Section 5.

   from 2. Background and Related Work

2. There are two periods $t \in \{1, 2\}$.

   from 3.1 Model setup (following [1], [2])

3. We use Markov Perfect Bayesian Equilibrium: strategies depend only on the payoff-relevant state $(\mu_t, p_t)$, and beliefs update by Bayes' rule on the equilibrium path.

   from 3.3 Equilibrium concept and stylized model

4. Formally, under-reporting pays only if the posterior revision from a low report satisfies $\mu_{t+1}(\hat{q}_t \lt \tau_t) \gt \mu_t$, which requires the user's likelihood ratio to favor honesty on low reports — a belief consistent only when $\Pr(\theta = \theta_H) \lt 0.5$ in the user's prior.

   from 4.2 Under-reporting requires honesty to be believed a minority

What would falsify it

1. Failure modes and falsification.

   from 6. Discussion

2. The central quantitative claim — that $48.28\%$ of delegation gains are irrecoverably destroyed — would be falsified if (a) the source's $71\%$ irrecoverable-share estimate were revised downward substantially, or (b) user-side interventions were shown to recover information the report no longer carries, which contradicts the definition of the irrecoverable component.

   from 6. Discussion

3. The equilibrium claim would be falsified if honest reporting were shown to be an equilibrium under plausible monitoring technologies, e.g., if monitoring were near-perfect ($\lambda \to 1$) and agents sufficiently patient, making the reputational cost dominate.

   from 6. Discussion

Where the claim stops

1. Second, the welfare decomposition of Section 4.5 is arithmetic on two reported numbers from [1], [2]; we do not have access to their underlying measurement protocol, sample sizes, or confidence intervals, so our derived $48.28\%$ and $19.72\%$ figures inherit any error in the $68\%$ and $71\%$ inputs.

   from 6. Discussion

2. Fourth, we assume the user's threshold rule $\hat{q}_t \geq \tau_t$ is fixed; a user who anticipates inflation would raise $\tau_t$, which is exactly the "user sophistication" channel whose recoverable share we bound at $29\%$ of destroyed value.

   from 6. Discussion

3. The stylized dominance condition $p \gt 0.5$ would be falsified empirically if agents with $p \gt 0.5$ systematically report $c \approx p$ despite revenue incentives, invalidating the linear-penalty assumption.

   from 6. Discussion

Where to read

1. Abstract236 words · 1 min
2. 1. Introduction291 words · 1 min
3. 2. Background and Related Work691 words · 3 min
4. 3. Methods393 words · 2 min
5. 4. Analysis1,467 words · 6 min
6. 5. Results397 words · 2 min
7. 6. Discussion661 words · 3 min
8. 7. Conclusion151 words · 1 min