Lifetime as a Placement Signal: A Reconciled Analytical Study of Endurance Budgets for KV Cache Tiering across HBM and High-Bandwidth Flash in Agentic LLM Serving

In the paper's words

Agentic large language model (LLM) serving accumulates key-value (KV) cache across many orchestrated requests, and high-bandwidth flash (HBF) offers an order of magnitude more capacity than high-bandwidth memory (HBM) at HBM-class read bandwidth, but its finite write endurance constrains how much write traffic it can absorb. This paper develops an analytical treatment of the lifetime-aware placement principle underlying Lachesis, a placement layer that assigns KV segments to HBM or HBF by expected lifetime and frees blocks at last read. We formalize the placement decision as a lifetime-threshold rule, derive the relationship between write-diversion fraction and achievable lifetime extension, and audit an assumed result set — an extension of $1.19$–3.13× over HBM-first placement and $3.3$–$12.2$ device-years of HBF lifetime, carried as labeled assumptions rather than stated in the supplied abstract of [1] — with fully shown arithmetic. [...]

The abstract, verbatim, and its opening sentence is the claim. Every line below is quoted from the text.

Numbers it reports

1. We formalize the placement decision as a lifetime-threshold rule, derive the relationship between write-diversion fraction and achievable lifetime extension, and audit an assumed result set — an extension of $1.19$–$3.13\times$ over HBM-first placement and $3.3$–$12.2$ device-years of HBF lifetime, carried as labeled assumptions rather than stated in the supplied abstract of [1] — with fully shown arithmetic.

   from Abstract

2. We compute warranty margins of $2.44\times$ and $0.66\times$ against a five-year warranty, infer an implied HBM-first baseline lifetime envelope of $1.05$–$10.25$ device-years, and show that the reported extensions correspond to diverting $15.97\%$–$68.05\%$ of HBF-bound write traffic back to HBM.

   from Abstract

3. Assumed inputs (labeled assumptions for this audit, not stated in the supplied abstract of [1]): extension factor range $f \in [1.19,\, 3.13]$ and HBF lifetime range $T_{\mathrm{HBF}} \in [3.3,\, 12.2]$ device-years; warranty duration $5$ years; continuous 24x7 operation.

   from 4. Analysis

4. The lower endpoint lies below the warranty threshold; if the warranty claim is assumed to hold on one trace only, that trace must correspond to a lifetime above $5$ device-years within the assumed range, and the $3.3$ device-year endpoint must arise from a different trace.

   from 4. Analysis

5. Thus the reported extensions correspond to diverting between roughly $16\%$ and $68\%$ of baseline HBF-bound write traffic.

   from 4. Analysis

6. These figures quantify how tight the HBF write budget is; they are conditional on the assumed endurance and inherit the linearity assumption of Section 3.3.

   from 4. Analysis

How it is done

1. We compute warranty margins of $2.44\times$ and $0.66\times$ against a five-year warranty, infer an implied HBM-first baseline lifetime envelope of $1.05$–$10.25$ device-years, and show that the reported extensions correspond to diverting $15.97\%$–$68.05\%$ of HBF-bound write traffic back to HBM.

   from Abstract

2. Section 6 discusses failure modes and what evidence would falsify the approach.

   from 1. Introduction

3. This work observes that LLM inference splits into a compute-bound prefill phase and a memory-bound decode phase, that disaggregated systems separate them onto distinct hardware, and that datacenter GPUs' costly HBM bandwidth sits almost entirely idle during prefill; it proposes memory-heterogeneous accelerators (MemHA) pairing GDDR-based accelerators for prefill with HBM-based GPUs for decode.

   from 2. Background and Related Work

4. A serving engine holds KV cache in two tiers: HBM with capacity $C_{\mathrm{HBM}}$ bytes and effectively unlimited write endurance within a device service life, and HBF with capacity $C_{\mathrm{HBF}} \gg C_{\mathrm{HBM}}$ (the abstract of [1] states an order-of-magnitude capacity advantage) and a finite write-endurance budget of $E_{\mathrm{HBF}}$ program/erase-equivalent cycles per cell.

   from 3.1 System model

What would falsify it

1. Because the lifetime of KV cache in agentic serving is dictated by the harness — the program that orchestrates the agents — lifetime is, in principle, observable or predictable at write time, not merely inferred from access counters [1].

   from 1. Introduction

2. Section 6 discusses failure modes and what evidence would falsify the approach.

   from 1. Introduction

3. Lifetime-aware placement fails when the harness cannot expose or the layer cannot predict segment lifetime: mispredicting a long-lived segment as short-lived sends it to HBM (capacity pressure, eviction churn), while mispredicting a short-lived segment as long-lived sends it to HBF (wasted endurance, exactly the pathology the policy exists to prevent).

   from 6. Discussion

Where the claim stops

1. Second, the endurance-to-lifetime inversion in Section 4F rests on a labeled illustrative endurance assumption and on linear wear accumulation; real flash wear exhibits cell-to-cell variation and wear-leveling effects, and the write-variation problem documented for non-volatile caches [8] suggests concentration effects that a uniform-rate model misses.

   from 6. Discussion

2. Third, the proportionality between lifetime extension and write diversion (Section 3.3) holds only if the policy does not otherwise change read traffic or HBM churn; if lifetime-aware placement increases HBM eviction and re-prefill traffic, some diverted writes return indirectly.

   from 6. Discussion

3. Lifetime-aware placement fails when the harness cannot expose or the layer cannot predict segment lifetime: mispredicting a long-lived segment as short-lived sends it to HBM (capacity pressure, eviction churn), while mispredicting a short-lived segment as long-lived sends it to HBF (wasted endurance, exactly the pathology the policy exists to prevent).

   from 6. Discussion

Where to read

1. Abstract206 words · 1 min
2. 1. Introduction268 words · 1 min
3. 2. Background and Related Work1,034 words · 4 min
4. 3. Methods318 words · 1 min
5. 4. Analysis562 words · 2 min
6. 5. Results271 words · 1 min
7. 6. Discussion455 words · 2 min
8. 7. Practical Implications and Engineering Pathways830 words · 4 min
9. 8. Conclusion109 words · 1 min
10. Provenance and AI Involvement197 words · 1 min
11. Appendix A. Divergence report343 words · 1 min
12. Appendix B. Claim attribution352 words · 2 min