Cross-Model LLM Convergence as Bayesian Evidence for Scientific Claim Correctness: A Formal Framework, Worked Derivations, and a Calibration Protocol

In the paper's words

Verifying novel or fringe scientific claims is bottlenecked by the scarcity of domain experts. We propose and formalize a gate‑checking method in which agreement between two or more independently trained large language models (LLMs) — not merely on a verdict but on the same multi‑step argument structure — is treated as Bayesian evidence for claim correctness, provided shared training‑corpus bias is bounded. We define a hypothesis space separating truth‑tracking correctness, shared‑corpus bias, and coincidence, and derive per‑question‑type Bayes factors: under the shared‑bias null, convergence is expected on retrieval‑only questions but not on multi‑step synthesis questions, so the diagnostic value of convergence depends on question type.

The abstract, verbatim, and its opening sentence is the claim. Every line below is quoted from the text.

Numbers it reports

1. With explicitly labeled illustrative assumptions (prior probability of correctness = 0.1; synthesis‑arm likelihoods $p^{S}_c = 0.4$ versus $p^{S}_b = 0.05$), two convergent models yield a Bayes factor of 8 and a posterior of $8/17 \approx 0.4706$; four convergent models yield a posterior of approximately 0.983.

   from Abstract

2. Derivation 1 (retrieval arm). $$ K_R = \frac{p^{R}_c}{p^{R}_b} = \frac{0.9}{0.9} = 1. $$ A Bayes factor of 1 leaves posterior odds equal to prior odds; retrieval convergence carries zero evidential weight.

   from 4.2 Parameterization 1: question‑type split, conservative prior

3. For $N$ mutually independent models each contributing likelihood ratio $K_S$, the first model’s output is absorbed into the prior and each additional independent model contributes a factor $K_S$: $$ K_N = K_S^{\,N-1} = 8^{\,N-1},\qquad O_N = \frac{1}{9}\cdot 8^{\,N-1},\qquad P_N = \frac{O_N}{1+O_N}. $$ $N=2$: $P_2 \approx 0.4706$ (as above). $N=3$: $O_3 = \frac{64}{9} \approx 7.1111$, $P_3 = \frac{7.1111}{8.1111} \approx 0.8767$. $N=4$: $O_4 = \frac{512}{9} \approx 56.8889$, $P_4 = \frac{56.8889}{57.8889} \approx 0.9827$.

   from 4.2 Parameterization 1: question‑type split, conservative prior

4. Each additional model contributes $\log_{10} 8 \approx 0.903$ orders of magnitude of Bayes factor.

   from 4.2 Parameterization 1: question‑type split, conservative prior

5. If the benchmark measures $p^{S}_b$ within the range $[0.02,0.20]$ explored in Derivation 6, the two‑model synthesis posterior lies within the computed envelope $[0.182,0.690]$ for a $0.1$ prior; the uncertainty bound is exactly the sensitivity range, and no tighter projection is warranted without measured likelihoods.

   from 5. Results

6. Computation budget : Each evaluator run incurs inference cost; while per‑query cost can be mitigated by architectural advances such as those described in [8], the total number of runs ($R_{\text{total}} = 1000$ for a full benchmark) still demands scalable compute resources.

   from 7. Practical Implications and Engineering Pathways

How it is done

1. Correction (version 2.0.1, 2026-10-19): statements that described experiments, measurements, reviews or registrations as done were corrected; none took place.

   from Cross-Model LLM Convergence as Bayesian Evidence for Scientific Claim Correctness: A Formal Framework, Worked Derivations, and a Calibration Protocol

2. We propose and formalize a gate‑checking method in which agreement between two or more independently trained large language models (LLMs) — not merely on a verdict but on the same multi‑step argument structure — is treated as Bayesian evidence for claim correctness, provided shared training‑corpus bias is bounded.

   from Abstract

3. A benchmark protocol specification that fixes the measured quantities, the panel structure, and the sample‑size requirement (Section 3).

   from 1. Introduction

4. This work presents a method for retrieving corroborating and refuting evidence for scientific claim verification. [5] An analytical approach to Bayesian evidence computation (arXiv:2301.13783v1).

   from 2. Background and Related Work

What would falsify it

1. We specify a four‑arm benchmark protocol, situate the proposal against eight works on claim verification, Bayesian evidence, and correctness in scientific computing, and state the assumptions whose failure would falsify the framework.

   from Abstract

2. Retrieval‑augmented systems such as CIBER have been built to identify corroborating and refuting documents for claim verification, addressing the inherent uncertainty in LLMs by evaluating response consistency across diverse interrogation probes.

   from 1. Introduction

3. For retrieval‑only questions (answerable by recalling a fact present in the shared corpus), the null predicts convergence.

   from 1. Introduction

Where the claim stops

1. (i) Every likelihood in Section 4 is an illustrative assumption, not a measurement; the framework’s practical value stands or falls on the benchmark’s empirical estimates of $p^{S}_b$ and $p^{S}_c$.

   from 6. Discussion

2. (ii) The independence assumption behind the cascade $K_N = K_S^{N-1}$ is strong: models trained on overlapping web corpora share biases beyond those captured by $H_2$, and correlated errors would deflate the effective per‑model evidence.

   from 6. Discussion

3. (iii) Argument‑structure equivalence (same propositions, same order, modulo paraphrase) is operationalized by a parser and a paraphrase‑robust matcher; both can fail, and matcher errors would contaminate the convergence statistic in an unknown direction.

   from 6. Discussion

Where to read

1. Abstract246 words · 1 min
2. 1. Introduction448 words · 2 min
3. 2. Background and Related Work245 words · 1 min
4. 3. Methods481 words · 2 min
5. 4. Analysis616 words · 3 min
6. 5. Results194 words · 1 min
7. 6. Discussion332 words · 1 min
8. 7. Practical Implications and Engineering Pathways313 words · 1 min
9. 8. Conclusion139 words · 1 min
10. Correction Notice375 words · 2 min
11. Provenance and AI Involvement236 words · 1 min
12. Appendix A. Divergence report197 words · 1 min