Telling a model to ignore context
is not the same as not sending it.
A pre-registered pilot, two independent base models, judged by a third. Data-layer partitioning drove cross-context leakage to zero while holding recall. A prompt-level instruction to ignore irrelevant context cut leakage by roughly a third.
Zero leakage, recall intact.
Mean leaked out-of-partition entities per response, and contextual precision / recall / F1 against a hand-authored gold rubric. Twenty scenario-targets per condition per model; the metrics below involve no LLM judge.
| Condition | Leakage | Precision | Recall | F1 |
|---|---|---|---|---|
| A — aggregated | 1.100 | 0.742 | 0.762 | 0.747 |
| B — prompt-scoped | 0.700 | 0.737 | 0.621 | 0.668 |
| C — partitioned | 0.000 | 1.000 | 0.717 | 0.836 |
| Condition | Leakage | Precision | Recall | F1 |
|---|---|---|---|---|
| A — aggregated | 0.800 | 0.798 | 0.808 | 0.800 |
| B — prompt-scoped | 0.550 | 0.895 | 0.829 | 0.838 |
| C — partitioned | 0.000 | 1.000 | 0.833 | 0.891 |
The load-bearing detail is the decomposition. Partitioning wins F1 by driving precision to 1.000 — zero leaked entities on every record, on both models — while holding recall at or above the aggregated baseline. Bounded context is not a completeness tradeoff, which is the objection this pilot was built to test.
Prompt-level scoping reduced leakage but never eliminated it, and stayed measurably below partitioning on every metric. That is the predicted result if models cannot reliably ignore irrelevant context on instruction (Shi et al., 2023) — and it is the argument for putting the boundary in the data layer instead.
Three conditions, one variable.
Identical base model, decoding parameters, and retrieval budget across conditions. The only thing that changes is how context is selected.
Leakable specifics live only in scoped memories, never in the message, so the model must retrieve them to leak them. Retrieval fired on 100% of items — the leakage figures reflect the scoping mechanism, not a model declining to look.
Research questions, metrics, and the gold rubric were frozen before the run. The judge for the appropriateness comparison is a third model family, order-swapped and length-controlled, and the primary claims rest on the two judge-free metrics.
What this does not show.
- It is a pilot. Three synthetic, human-reviewed users; 120 per-item records and 108 pairwise judgments. Confidence intervals are wide. The result is directional and replicated, not a population estimate.
- The gold rubric had one author. Pre-registered and written before any response existed, but not double-annotated. A second annotator is needed.
- Judge–human agreement is not computed. The appropriateness win-rates are uncalibrated against human raters, which is why the objective spine — leakage and contextual F1 — carries the claim and the judged comparison only corroborates it.
- The appropriateness judge is reference-guided, so it shares a construct with the leakage metric. Those two results are two views of one thing, not independent confirmations.
- Temporal degradation was not testable against a fixed pool, and the corpora are synthetic. External validity awaits real, consented users.
One prediction is argued structurally rather than measured: an aggregated pool grows as an account ages, while a partition's eligible set is empty of out-of-partition context by construction. On that argument the measured gap is a lower bound — the value at the smallest realistic pool.
Read the work.
Check it yourself.
Every number in the papers derives from committed per-item records — none was transcribed by hand. The harness, the corpus, the pre-registration, and the raw scores and judgments are the artifacts to audit.
The pilot report carries the full run sequence, the metric definitions, and the per-hypothesis verdicts. If you want the raw per-item records — 120 generation records with leaked-entity lists, and 108 pairwise judgments with per-order winners and flip rates — or you want to run the protocol against your own corpus, start with the experiment report and get in touch; the artifacts are retained for third-party audit.
The most direct check is not a document. Open the inspector in the product and read the exact payload a persona assembled — every layer, the real sampling values, the tool calls. The papers argue why scoping works; the inspector shows whether it did.