Gate-Check Convergence: A Benchmark Protocol and Statistical Design for Measuring Cross-Model Agreement on Fringe Physics Claims

In the paper's words

Large language models (LLMs) are increasingly consulted as informal arbiters of scientific claims, including claims at the fringe of established physics, yet no published protocol measures whether independent models converge on such verdicts because they retrieve a shared consensus or because they share failure modes. We propose Gate-Check Convergence (GCC), a benchmark protocol testing the hypothesis that independent LLMs converge not only on verdicts for fringe physics claims but on the reasoning scaffolds used to reach them (Bell's theorem for hidden-variable claims, the SU(2) → SO(3) covering map for spin claims, BCS gap arithmetic for superconductivity claims). We formalize the hypothesis as retrieval from an overlapping consensus lattice, which predicts agreement rising with how well-represented the relevant consensus material is, and specify the benchmark construction, blinded evaluation, and statistics: Cohen's κ on verdicts, Jaccard-based argument-structure alignment, and citation-quality variance.

The abstract, verbatim, and its opening sentence is the claim. Every line below is quoted from the text.

Numbers it reports

1. Under the assumptions that expert adjudication is the binding cost and that the pilot's expert effort is on the order of tens of minutes per claim for two experts, under the explicit assumption of 15 minutes per expert per claim, a 110-claim confirmatory benchmark requires $110 \times 2 \times 0.25 = 55$ expert-hours, i.e., roughly a person-week and a half of one expert's time; the model-run cost of $1{,}650$ single-turn evaluations is negligible against this by the standard pricing of commercial LLM APIs, though we state no price figure because none is derived here.

   from 7. Practical Implications and Engineering Pathways

2. We require a 95% confidence interval on $\kappa$ of half-width $w = 0.10$.

   from 4.2 Benchmark size for a usable kappa confidence interval

3. We therefore specify a benchmark of at least 50 claims for exploratory analysis and 110 claims for the confirmatory analysis, with the caveat that these figures assume $p_o = 0.80$; if true agreement is lower, required $n$ rises as $p_o(1-p_o)$ grows toward its maximum $0.25$ at $p_o = 0.5$, giving an upper bound: $$n_{\max} = \frac{0.25}{0.005625} \times 3.8416 = 44.444 \times 3.8416 \approx 170.7,$$ i.e., at most about $171$ claims suffice for $w = 0.10$ under this approximation regardless of $p_o$.

   from 4.2 Benchmark size for a usable kappa confidence interval

4. Benchmark size: $n \approx 110$ claims for a kappa confidence interval of half-width $w = 0.10$ on $\kappa$ (Section 4.2); $n \approx 49$ claims suffice for the coarser half-width $w = 0.15$; and the worst-case upper bound over all values of $p_o$ is $n_{\max} \approx 171$ claims for $w = 0.10$ (Section 4.2).

   from 5. Results

5. The uniform-chance baseline $p_e^{(1)} = 0.25$ is a pessimistic floor: real verdict marginals are concentrated (models favor "falsified" on fringe claims), which raises chance agreement and lowers $\kappa$ for a given $p_o$.

   from 6. Discussion

6. The benchmark sizes are $n \approx 110$ claims for a confirmatory kappa confidence interval of half-width $w = 0.10$ (with a worst-case upper bound $n_{\max} \approx 171$ over all values of $p_o$) and at least 50 claims for exploratory analysis.

   from 8. Conclusion

How it is done

1. Large language models (LLMs) are increasingly consulted as informal arbiters of scientific claims, including claims at the fringe of established physics, yet no published protocol measures whether independent models converge on such verdicts because they retrieve a shared consensus or because they share failure modes.

   from Abstract

2. Anecdotal observation suggests that different models, from different families and trained by different organizations, often reach the same verdict and deploy the same argument: the same invocation of Bell's theorem, the same appeal to the covering map from $SU(2)$ to $SO(3)$, the same BCS gap estimate.

   from 1. Introduction

3. The works relevant to this protocol fall into four groups: automated claims verification, consensus formation in physics, meta-scientific verdict conventions, and fringe-adjacent candidate claims.

   from 2. Background and Related Work

4. Claim classes follow the trichotomy: falsified (contradicts established results), trivial (restates known results as if novel), valid-but-flawed (sound motivation, defective execution), with candidate items drawn from beyond-Standard-Model territory of the kind surveyed in [4].

   from 3.1 Benchmark construction

What would falsify it

1. We formalize the hypothesis as retrieval from an overlapping consensus lattice, which predicts agreement rising with how well-represented the relevant consensus material is, and specify the benchmark construction, blinded evaluation, and statistics: Cohen's $\kappa$ on verdicts, Jaccard-based argument-structure alignment, and citation-quality variance.

   from Abstract

2. No model runs are reported; the paper contributes the protocol, the formal model, and the analysis machinery needed to execute and falsify the claim.

   from Abstract

3. Gate-check convergence : models retrieve a shared consensus lattice — the same textbooks, reviews, and canonical refutations present in all large training corpora — and the shared retrieval produces both the verdict and the argument structure.

   from 1. Introduction

Where the claim stops

1. All sample-size and power figures rest on a large-sample variance approximation $\mathrm{Var}(\hat{\kappa}) \approx p_o(1-p_o)/\bigl(n(1-p_e)^2\bigr)$, which ignores the higher-order terms of the exact variance and can be optimistic when marginals are skewed, as the illustrative matrix of Section 4.3 already shows ($p_e = 0.33$ rather than the uniform $0.25$).

   from 6. Discussion

2. The design target $p_o = 0.80$ is an assumption, not a measurement; if true agreement is near $p_o = 0.5$, the required benchmark grows to $n_{\max} \approx 171$ claims (Section 4.2).

   from 6. Discussion

3. The power formula for Spearman correlation is itself an approximation, and the balanced-split assumption $n_1 = n_2 = 55$ may fail if representation depth $d_c$ is skewed toward high values, inflating $\mathrm{SE}(\Delta)$ beyond the computed $0.1017$.

   from 6. Discussion

Where to read

1. Abstract242 words · 1 min
2. 1. Introduction566 words · 2 min
3. 2. Background and Related Work874 words · 4 min
4. 3. Methods462 words · 2 min
5. 4. Analysis978 words · 4 min
6. 5. Results334 words · 1 min
7. 6. Discussion506 words · 2 min
8. 7. Practical Implications and Engineering Pathways937 words · 4 min
9. 8. Conclusion492 words · 2 min
10. Provenance and AI Involvement225 words · 1 min