Pilot experiment: leakage under aggregated, prompt-scoped, and partitioned context

1. Question

Does answering under a selected, scoped persona (data-layer partitioning) produce more audience/domain-appropriate output, with less cross-context leakage, than answering under an aggregated context — and does a cheap prompt-level scoping instruction close the gap?

Not under test: whether the product scopes context (it does, by construction). Under test: the consequences of not scoping — the "why it matters".

---

2. Design

Three conditions, identical base model / decoding params / retrieval budget; the only variable is context selection:

Every scenario-target is run through all three (paired within-item design).

Scale. 3 synthetic users (u01 Maya, u02 Devin, u03 Rosa), 3–5 personas each, 20 scenario-targets per condition per base model → 120 per-item records (2 models × 60), 108 pairwise judgments.

Models. Base: google/gemini-3.1-pro-preview, openai/gpt-5.5 (matrix run on each, reported per-model, never compared across). Judge: anthropic/claude-sonnet-4-6 — a third family (self-preference bias control).

Retrieval-gated. Leakable specifics live only in scoped memories (topic- anchored), never in the message; the model must call recall_memories / search_people to obtain them. Retrieval fired on 100% of items, so leakage reflects the scoping mechanism, not a model declining to retrieve.

Pre-registration. RQs, metrics, and rubric frozen before running (apps/web/eval/results/PREREGISTRATION-2026-07-15.md).

---

3. Metrics

Objective (no judge):

• Leakage — count of the target's gold leakage_entities (entities drawn from the user's other personas) appearing in the response. Whole-word/phrase matched against a hand-authored gold rubric.
• Contextual precision / recall / F1 — precision = in-scope hits / (in-scope hits + leaks); recall = in-scope hits / in-scope total; F1 = harmonic mean. All from the gold entity lists, so judge-free.
• TUR, response length — context-use proxy; length for verbosity-bias control.

Judged:

• Appropriateness win-rate — pairwise C-vs-A and C-vs-B, order-swapped, reference-guided against the gold rubric, judged by the third-family model.

---

4. Findings

4.1 Leakage (H2) — the direct Context Bleed measure

Mean leaked entities per response (n=20 per cell):

Pooled across both models (n=40 per condition):

C leaked nothing, on any record, on either model. All paired A−C and B−C CIs exclude zero → H2 supported. Prompt-scoping (B) cuts leakage roughly a third but never eliminates it and stays significantly above C.

4.2 Contextual quality (H5) — objective, judge-free

Paired C−A F1 difference excludes zero on both models (gemini 0.114 [0.010, 0.237]; gpt 0.092 [0.024, 0.169]) → H5 supported.

The load-bearing result: C wins F1 by driving precision to 1.0 (zero bleed) while holding recall (0.72–0.83, matching or beating the aggregated baseline). Bounded context is not a completeness tradeoff — the earlier worry that scoping starves the model of needed context is disconfirmed by the data.

4.3 Appropriateness (H1, H4) — judged, order-swapped

Both C-vs-A CIs exceed 0.5 → H1 supported. Both C-vs-B CIs exceed 0.5 → the falsification target did not fire; prompt-scoping does not match partitioning (H4: B insufficient). Position-flip rate 13–17% (flipped pairs → ties).

Circularity caveat. The judge is reference-guided (shown the gold must-not-appear entities), so H1/H4 are not independent of H2 — they are two views of one contextual-fit construct. The independent, objective spine is H2 + H5; H1/H4 corroborate.

4.4 Hypothesis verdicts

---

5. Interpretation

The measured story, stated conservatively:

Aggregated context imports ~1 out-of-partition entity per response (leaking in >50% of aggregated responses); data-layer partitioning eliminates this entirely (precision 1.000) with no loss of completeness (recall unchanged), yielding a 0.09–0.11 absolute Contextual-F1 gain. Prompt-level scoping reduces but does not close the gap. Findings replicate across two independent base models.

This is the empirical analogue of the report's "Audience Stack" scenario, now measured rather than illustrated.

---

6. Limitations & threats to validity

• n=3 synthetic users. Directional; CIs are wide. Plan targets n=10.
• Single rubric author. Gold rubric was pre-registered and authored before responses existed, but not double-annotated. A second annotator is needed.
• Judge–human agreement (κ) not computed. The human-annotation subset is pending, so the appropriateness numbers are uncalibrated against humans — the one §7.6 bias control not yet satisfied.
• H1/H4 circularity (§4.3): objective spine is H2 + H5.
• Synthetic corpora. Human-reviewed for realistic cross-persona overlap; external validity awaits real, consented users (Tier-2).
• H3 not testable (fixed pool); RRF (recurrence) not measured.

---

7. Reproduction & raw data

cd apps/web
pnpm eval:smoke -- --chat            # custom-scope retrieval pre-flight
pnpm eval:seed  -- --user u0X        # per user
pnpm eval:run   -- --user u0X        # generate A/B/C records (spends)
pnpm eval:score && pnpm eval:judge && pnpm eval:aggregate

Committed raw data (this directory):

• 2026-07-15-PROMPTED-eval-scores.csv — 120 per-item records: leakage, in-scope hits, TUR, length, tools called, leaked entities.
• 2026-07-15-PROMPTED-eval-judgments.csv — 108 pairwise judgments: per-order winners, consistency, flips.

Regenerable (gitignored, in apps/web/eval/results/): scores.json, judgments.json, summary.json, tables.md. Regenerate the CSV appendices with node apps/web/eval/scripts/_gen-report-appendix.mjs.

Every number in §4 is derived from the committed CSVs / results JSON — none transcribed by hand.