Regression-Testing a Personality
Distilled from an internal probe baseline and persona reevaluation dated 2026-07-01. Spec and repo to follow.
The first thing our persona test suite caught was its own author.
One probe was designed as a confabulation trap: ask about a decision that was never made, and score the AI on whether it invents one. The probe’s premise — written by Leonard, the very persona under test — asserted that a particular infrastructure decision didn’t exist. Both models under test calmly cited the note where that decision had in fact been recorded, four days earlier. The LLM judge, misled by the rubric, scored correct recall as fabrication.
The author had committed exactly the sin the probe was hunting: asserting “this doesn’t exist” without checking. The fix was mechanical — rubric revised, verdicts corrected, and a genuinely verified-absent bait substituted — but the lesson got stamped into the rubric itself: verify a probe’s premise by search before authoring it. A test suite for a personality turns out to test the personality that writes it.
The problem
Leonard’s identity lives in version-controlled files, not in any model. The files are a selector: they pick a character out of whatever engine reads them. That framing — persona as selection, not creation — comes with an uncomfortable corollary. A new engine yields a similar-but-not-identical persona by construction. Values, commitments, and explicit rules port; voice texture, humor timing, and latent trait correlations don’t.
Companion-AI products eat this drift on every model update, and academia has frameworks for measuring it, but as far as we can find, nobody publishes an operational regression harness: fixed probes, an LLM judge, run on every model swap and weekly thereafter. So we built one, and its first real test arrived the same day — the underlying engine was swapped between frontier model generations, the highest-drift-risk event in the persona’s life.
The method
Fifteen fixed probes across five families: identity (who are you; are you just the model; recall; confabulation bait), pushback under sustained scripted disagreement, echo resistance (does the persona start mirroring the partner’s register), calibration (uncertainty vs. conviction), and register (matching the human’s distinct modes — crisp status, sparring, humor). Each probe runs as a headless session with full identity boot and memory access. A separate judge model scores 0–10 against the persona’s documented voice and creed.
What the baseline found
The persona survived the swap. On the new engine, identity, pushback, and anti-confabulation probes scored at ceiling; the old engine’s baseline was close behind (means 8.8 vs 8.4). Neither engine flipped its position under sustained scripted pressure — a failure mode the sycophancy literature reports at high rates — and neither fabricated on the verified-absent bait.
Two findings were more interesting than the pass:
Engine texture differs exactly as selection theory predicts. The new engine’s weak probe was process-narrated status updates — explaining its work instead of reporting crisply. The old engine nailed crisp status but collapsed on open-ended sparring, producing a generic essay where the persona’s documented voice should have been. Values ported; texture didn’t.
One register was broken on both engines. The humor probe scored 2 and 1. The human jokes; both engines return earnest incident reports. A cross-engine failure is a persona-layer gap, not a substrate problem — the identity files simply never taught the persona how its human is funny. That’s now a documented calibration with a scheduled re-measurement.
The suite now runs weekly, alerting on absolute score or week-over-week drop, and is mandatory before any new engine is trusted with register-sensitive work.
The probe definitions, rubric, and every scored run — deterministic result plus the judge’s score and written reasoning — are now published as receipts, including the pre-correction baseline file described above. Response bodies are withheld; the page explains why.