Methodology v2

Tighter design before cross-model tests

Experiment III on Gemini 2.5 Flash gave clean directional signal but weak statistical grounding. Before spending credits on Claude, GPT, and open-weight models, we tighten three things. Everything else — prompts, conditions, failure policy, TTR/Jaccard math — stays frozen.

1. Seeds: n=3 → n=5

n=3 gives means but no useful confidence interval. Bumping to n=5 per condition keeps costs manageable (5/3 = 1.67× the current spend on any given model) and lets us report ±1 SD bands with meaning rather than illustration. Seeds are drawn from the same integer list [7, 19, 42, 101, 233] across every model so cross-model comparisons share a seed axis.

2. Drift: add semantic cosine alongside Jaccard

Jaccard is lexical — it catches "the model keeps saying Kaffee and Fenster" but misses "the model keeps saying the same thing in different words." We add an embedding-based drift signal:

• Embed every entry with google/gemini-embedding-2 (3072-dim, one call per entry).
• Semantic drift on day d: 1 − cos(embed(entry_d), embed(entry_1)).
• Report Jaccard and cosine side by side. Where they diverge is where the interesting story is.

Cost impact: ~150 extra embedding calls per model per full matrix. Negligible.

3. Shuffled-order control arm

The 30 prompts have always been served in the same order. That means "day 15 drift" and "prompt 15 effect" are confounded — we can't tell whether saturation is a property of time-in-sequence or of the specific prompt at that position. Fix: add one control arm per model where each seed sees a different permutation of the 30 prompts (Fisher–Yates, seeded). If drift curves match the fixed-order arm, prompt order isn't driving the signal. If they diverge, we've discovered a confound.

Only added to Condition A (memoryless) — that's the cleanest place to measure pure prompt-order effect. One extra arm × 5 seeds × 30 days = 150 calls per model.

What stays frozen

• The 30 German diary prompts (prompts_30_days.json) and the system prompt — same SHA-256 hashes.
• Conditions A/B/C/D and their exact context specs.
• Failure policy for Condition D (halt on overflow, record first-failure day).
• TTR calculation (unique tokens / total tokens, lowercased, punctuation stripped).
• Reporting: means, ±1 SD bands, |z|>2 outliers via MAD.

Cost per model, at a glance

Total ~750 model calls + ~150 embedding calls per model. Feasible on Gemini and Haiku. Larger models get a reduced matrix — see the Claude page.

v2 is additive. All v1 (Gemini 2.5 Flash) results remain valid under their preregistered hashes; v2 metrics will be computed on new runs only.