The Recall Ceiling of LLM Recommendation Reranking

Some LLM recommendation rerankers are evaluated under an oracle protocol that guarantees the ground-truth item is in the scored set. That protocol measures something real. It does not measure deployment performance, because the quantity it holds fixed is the one that binds in production.

Under realistic retrieval, none of the seventeen optimisation strategies tested here — prompt design, a 168× model-scaling range, test-time reasoning, three supervised rerankers under a disjoint train/eval split, a LoRA-fine-tuned LLaMA-3B, score-aware prompting, LLM+CF fusion, sequential models, larger candidate budgets, hybrid text retrieval — produces a statistically significant gain over a $0.00 collaborative-filtering baseline on the primary Amazon datasets.

The cause is combinatorial, not a modelling failure, and it can be written down.

Theorem 1

What a reranker cannot do

A reranker that outputs a permutation of a retrieved candidate set can never place an item the retriever missed.

Two consequences do the work. First, the bound is numerical: on Amazon Movies, where CF-SVD reaches Recall@100 = 2.95%, no closed-candidate reranker can exceed NDCG@10 = 0.0295 — zero-shot LLM, fine-tuned model, or a perfect oracle over the retrieved set. Second, the binding quantity is |W_π|, not K: a listwise LLM prompt shown 30 of the 100 candidates is accountable to Recall@30, not Recall@100. Scoring it against the wrong denominator understates its ceiling utilisation by about 2.6×.

We claim no surprise in the statement — that a reranker cannot recover an unretrieved item is intuitive. What the formalisation buys is the exact number a measured NDCG must be read against, and the utilisation ratio η that separates a retrieval failure from a reranking failure.

Interactive · RAEP component 1

Diagnose your own setup

Before comparing rerankers, measure the retrieval recall that bounds them. Enter yours — the ceiling, the regime, and the fraction of it you are using follow deterministically.

Both axes are logarithmic, which makes the bound a 45° line and the vertical gap up to it exactly log(1/η). Blue points are this paper’s measured operating points (CF-SVD retrieval, n=500, K=100); every one sits about a decade below the line. Moving a point right is a retrieval improvement and raises the ceiling. Moving it up is a reranking improvement and consumes headroom that already exists. Seventeen attempts at the second are catalogued below.

RQ3 & RQ4

Seventeen ways we tried to break it

Each row removes one candidate explanation for the failure: prompt design, model capacity, test-time reasoning, supervised training signal, structured CF features, temporal modelling, candidate budget.

Two results deserve singling out. Test-time reasoning strictly hurts. DeepSeek-V4-Pro with thinking enabled scores 43% below CF on Movies (p = .004) at roughly 37× the cost of the same model without it — $4.75 versus $0.13 per 500 users, and 200–500 s versus under 1 s per user. Reasoning chains have nothing to operate on when the relevant item is absent from the candidate set.

Giving the LLM the CF scores helps, for the wrong reason. Annotating each candidate with its CF rank, normalised score and popularity percentile improves every dataset — and raises Kendall’s τ against the CF order every time. The model gets better by overriding CF less. An LLM+CF blend with λ tuned on the evaluation users themselves — an upper bound, not a held-out result — recovers exactly CF and no more.

Reliability

Three things to know before reading the numbers

Each of these is a limitation we found ourselves and kept in the repository on purpose.

Where the ceiling binds

Eight datasets, three domains

Recall falls broadly with catalog scale; density is the stronger descriptive correlate (Pearson r = +0.604 over eight points — a coefficient, not an inferential test). The ceiling bites hardest exactly where recommendation matters most: large, diverse catalogs.

Recall@100 under CF-SVD. MovieLens-25M is 18.9% on the n=1,000 RAEP diagnostic sample and 17.4% on the n=500 LLM-evaluation sample. Even at the top of this range, all five cloud LLMs score below the CF baseline.

The one escape route

Generative retrieval, not better reranking

Theorem 1 constrains closed-candidate rerankers. Three directions could break the ceiling; only one breaks that assumption. Better retrieval — multi-source fusion at K = 3,000 reaches 30% recall on Beauty at 30× the candidate budget — raises the ceiling but stays inside the same combinatorial constraint. Richer metadata improves η, not the ceiling. Generative retrieval — end-to-end token-level item generation, TIGER and GenRec — is the only direction in our taxonomy that lets a model emit items absent from the candidate set. That is the natural follow-up: a head-to-head against the hybrid CF+BM25+Dense baseline at the recall–NDCG operating point identified here.

Scope, stated plainly: our retrievers reach Recall@100 of 2–19%. A production system operating at substantially higher recall sits outside the regime we measured, and the ceiling loosens as recall rises. Our rerankers see CF embeddings, item text, popularity and CF scores — not the dwell-time, context and cross-feature signals an industrial ranker consumes. Offline evaluation cannot observe the online feedback loop. We do not claim the retrieval-first ordering holds beyond those boundaries.

Artifact

Reproduce it

Every number in the paper comes from a JSON file in experiments/logs/, and every one of those files is produced by a script in scripts/. Nothing is hand-entered.

# the datasets under data/processed/ are already preprocessed — no download step
git clone https://github.com/GeoffreyWang1117/recall-ceiling-cikm2026
cd recall-ceiling-cikm2026
conda create -n recallceiling python=3.11 && conda activate recallceiling
pip install -r requirements.txt

# reproduce the ceiling-utilisation numbers — no API calls, ~30 s
python scripts/cikm_recompute_eta_windows.py

# reproduce Figure 1 from the result JSONs
python scripts/make_fig_overview.py

SEED = 42 throughout; every non-API experiment is deterministic. The API drivers checkpoint per user, so an interrupted run resumes without repaying for completed users.

The full map, including the twenty results reported in prose, is in the repository README. Earlier result families (kdd_*, p9_*) are kept so the paper’s self-corrections can be checked against the original runs.