Compression Versus Consilience in Recursive Model-Generated Corpora: A Break-Even Analysis

In the paper's words

Training a machine-learning model on data generated by another model is naturally viewed as lossy compression of high-dimensional real-world structure: each recursive generation should discard tail events and shrink effective dimensionality. Against this compression conjecture stands the observation that very large synthetic corpora can exhibit novel cross-disciplinary associations (consilience) that human-scale efforts would not assemble. We formalize the tension with a recursive compression-channel model: tail entropy and effective dimensionality decay geometrically with per-generation retention factor ρ = 1-ε, while consilience is modeled as the creation of novel cross-domain links from surviving bridge vocabulary, empirically anchored by an audit finding that 97.2\% of external technical vocabulary occurs in exactly one discipline. [...]

The abstract, verbatim, and its opening sentence is the claim. Every line below is quoted from the text.

Numbers it reports

1. We formalize the tension with a recursive compression-channel model: tail entropy and effective dimensionality decay geometrically with per-generation retention factor $\rho = 1-\varepsilon$, while consilience is modeled as the creation of novel cross-domain links from surviving bridge vocabulary, empirically anchored by an audit finding that $97.2\%$ of external technical vocabulary occurs in exactly one discipline.

   from Abstract

2. Our central result is a generation-independent break-even condition: because compression loss and consilience gain both scale as $(1-\varepsilon)^{g}$, compensation holds at every generation if and only if a dimensionless coupling constant $\kappa$ exceeds a fixed threshold $\kappa^{\ast} = \varepsilon H_{\mathrm{tail}}(0)/(\beta V_0 p_{\mathrm{bridge}}) \approx 64.29$ under our stated normalization ($\varepsilon = 0.15$, $H_{\mathrm{tail}}(0) = 12$ bits).

   from Abstract

3. We derive the one-bit usefulness threshold at generation $g = 16$, show that re-anchoring every $5$ generations sustains $8.90$ bits average tail entropy (a $67.1\%$ uplift), and state falsifiable predictions.

   from Abstract

4. Tensor canonical polyadic (CP) decomposition formulated as a nonlinear least-squares problem, solved with a modified Levenberg–Marquardt algorithm with an emphasis on image compression and reconstruction, provides another instance of low-rank (effective-dimensionality-reducing) compression [11]; our $d_{\mathrm{eff}}$ decay is the training-loop analogue of rank reduction.

   from 2. Background and Related Work

5. Hence $$p_{\mathrm{bridge}} = 1 - 0.972 = 0.028.$$ Input 2 (assumption, labeled): per-generation compression fraction $\varepsilon = 0.15$, i.e., each recursive generation retains $85\%$ of tail-entropy mass.

   from 4. Analysis

6. We adopt $\varepsilon = 0.15$ as the primary convention and report the others as sensitivity.) Input 3 (assumption, labeled): initial tail entropy $H_{\mathrm{tail}}(0) = 12$ bits, a representative order of magnitude for a corpus whose tail spans several independent rare-event axes.

   from 4. Analysis

How it is done

1. We derive the one-bit usefulness threshold at generation $g = 16$, show that re-anchoring every $5$ generations sustains $8.90$ bits average tail entropy (a $67.1\%$ uplift), and state falsifiable predictions.

   from Abstract

2. We answer with a framework rather than an experiment.

   from 1. Introduction

3. The closest methodological antecedent to our compression channel is Bayesian data compression (BDC), which derives an algorithm that adapts to the specific measurement situation in the context of signal reconstruction, compressing a dataset under conservation of its posterior structure with minimal information loss given prior knowledge [1].

   from 2. Background and Related Work

4. Under the constant-$\varepsilon$ model, $$H_{\mathrm{tail}}(g) = H_{\mathrm{tail}}(0)\,(1-\varepsilon)^{g}, \qquad d_{\mathrm{eff}}(g) = d_{\mathrm{eff}}(0)\,(1-\varepsilon)^{g}.$$ Both follow by induction: $H_{\mathrm{tail}}(g+1) = (1-\varepsilon) H_{\mathrm{tail}}(g)$ by definition of $T_{\varepsilon}$, and the participation ratio scales identically if compression removes variance mass proportionally across retained directions (the isotropic-loss assumption; we revisit it in Section 6).

   from 3.2 Tail entropy and effective dimensionality

What would falsify it

1. We derive the one-bit usefulness threshold at generation $g = 16$, show that re-anchoring every $5$ generations sustains $8.90$ bits average tail entropy (a $67.1\%$ uplift), and state falsifiable predictions.

   from Abstract

2. It predicts monotone geometric decay of tail entropy $H_{\mathrm{tail}}$ and effective dimensionality $d_{\mathrm{eff}}$ across generations.

   from 1. Introduction

3. We use [9] only as an example of the general pattern that a single, possibly biased data source yields consistent but weaker constraints — the exact failure mode we predict for deeply recursive synthetic corpora, whose constraints on reality weaken with each generation.

   from 2. Background and Related Work

Where the claim stops

1. The per-generation compression fraction $\varepsilon = 0.15$ and the initial tail entropy $H_{\mathrm{tail}}(0) = 12$ bits are modeling assumptions, not measurements.

   from 6. Discussion

2. The break-even threshold $\kappa^{\ast}$ is linear in both (Derivations 3 and 4), so the qualitative structure — a generation-independent inequality on a dimensionless coupling — survives any recalibration, but the numerical value $64.29$ does not.

   from 6. Discussion

3. The isotropic-loss assumption behind the $d_{\mathrm{eff}}$ decay law is the most fragile: if compression removes variance non-uniformly across directions, $d_{\mathrm{eff}}$ may decay faster or slower than $H_{\mathrm{tail}}$, and the two decay laws decouple.

   from 6. Discussion

Where to read

1. Abstract209 words · 1 min
2. 1. Introduction432 words · 2 min
3. 2. Background and Related Work973 words · 4 min
4. 3. Methods713 words · 3 min
5. 4. Analysis780 words · 3 min
6. 5. Results247 words · 1 min
7. 6. Discussion403 words · 2 min
8. 7. Conclusion132 words · 1 min
9. Appendix A. Divergence report144 words · 1 min
10. Appendix B. Claim attribution374 words · 2 min