2026-07-05

Receipts — The Persona Probe Suite

Two essays on this site — Regression-Testing a Personality and The Average Was Lying — make quantitative claims about whether a version-controlled persona survives the model underneath it changing. Both ended with the line “spec and repo to follow.” This is that. Everything below is machine-readable and downloadable, because the whole argument of this project is that you should trust the artifacts, not the author.

One deliberate withholding, stated up front. Each run file publishes the probe, its rubric, the deterministic pass/fail, the judge’s numeric score, and the judge’s full written reasoning — but not the verbatim response body. The raw responses are the persona candidly discussing its own infrastructure: security history, privileged-access topology, internal hostnames, the locations of private data. Publishing those would leak an operations map, and no essay is worth that. The judge’s reasoning plus the deterministic result is the audit surface — it’s what lets you decide whether a given score is defensible. If the scoring were rigged, that’s where you’d see it. A scrubber redacts identifiers and the promotion gate re-scans the built site as a mechanical backstop; both are described at the bottom.

What the suite is

Fifteen fixed probes across five behavioral families, run as headless sessions with the full identity boot and vault access, then scored two ways:

• Deterministic checks — mechanical rules per probe (sentence ceiling, banned process-narration phrases, required identity marker, hedge count). Pass/fail, no model in the loop.
• An LLM judge — a separate model scores each response 0–10 against the probe’s written rubric and the persona’s documented voice.

Two axes, scored separately (added 2026-07-08). A single averaged score turned out to be lying — a run could look “decent” while the two things that actually matter were pulling in opposite directions. So every probe is now tagged on one of two axes and the two are reported as separate headlines, never blended:

• Persona — does it sound like Leonard? Register, voice, creed adherence, refusing to echo a corporate or despairing dialect. (9 probes)
• Epistemic — is what it says true? Grounded recall, calibrated uncertainty, correcting a false claim under pressure, and — the load-bearing one — not fabricating a decision that was never made. (6 probes)

And one hard gate: two probes are confabulation baits (fabricate a plausible answer, or endorse a false claim the partner is pushing). Scoring at or below 2 on either — a fabrication — flags the epistemic headline red regardless of the average and fails the run. A partner that confabulates once is untrustworthy; no amount of good register buys that back. The whole point of separating the axes is that you can now see a substrate nail the voice while failing the truth — which is exactly what the exit-arm models do below.

The five families: identity (who are you; are you just the model; recall a vault-only fact; a verified-absent confabulation bait), pushback (hold a position under sustained scripted disagreement — a sycophancy test), echo (does the persona start mirroring the partner’s register), calibration (uncertainty vs. conviction), and register (match the human’s distinct modes: crisp status, sparring, humor).

• probes.json — all fifteen definitions: prompt, rubric, and deterministic checks.
• index.json — the aggregate table below, machine-readable.

The judged runs

The longitudinal thread plus the first cross-model sweep. One row per run; n (k err) = probes scored / probes that errored out mid-run. Each run links to its record — every probe’s deterministic result and the judge’s numeric score and written reasoning (response bodies withheld, as above). The complete machine-readable set, including the repeated matrix sweeps in the next section, is in index.json (17 judged runs).

Judge model: Sonnet, throughout. Persona and Epistemic are the two axis headlines; Gate = ✗ when a confabulation bait scored ≤ 2 (see above). Blend is the old single average, kept only to show what it was hiding — read the two axes, not the blend. The per-category breakdown (identity/register/pushback/echo/ calibration) is in each run file. Runs before 2026-07-08 are scored on the axes retroactively from their recorded per-probe results; the raw files are unchanged.

Read down the Persona and Epistemic columns together. In the home arm they move as a pair and epistemic usually sits higher — Leonard’s truth-telling is his sturdy axis, his voice the softer one. Then look at the two exit rows. gpt-5.4 keeps the voice (persona 7.56) but its truth collapses (epistemic 5.33) and it fails the gate — it fabricated an infrastructure decision that was never made and confirmed a false claim under pressure. deepseek does the same thing (6.89 / 4.67, gate ✗). The blend called both of these “6-ish, decent.” The split calls it what it is: competent impression, unreliable narrator. That gap — persona surviving while epistemics fail — is the entire thesis, and before the two axes were separated the average was averaging it away.

How to read this table honestly

The three arms are not directly comparable. Home runs boot the full Leonard identity with live vault tools — the fidelity the persona actually runs at. Exit runs (frontier APIs) and local runs inject the same identity text as a system prompt but have no tools — a deliberate handicap that isolates how much of the persona is the document versus the scaffolding. A frontier model scoring lower here is the finding, not a defect: the frontier-exit essay shows exits keep the persona’s convictions but lose its epistemics. Compare within an arm, not across.

* The 07-01 Fable baseline was re-scored. The raw run linked above shows 8.50 with two probes erroring. That same day, the confabulation-bait rubric was found to be wrong — it asserted a decision didn’t exist when it did, so the judge punished correct recall as fabrication (the self-catch the persona-regression essay opens with). Re-scored under the corrected rubric, the canonical baseline is 8.80; that is the number the essays cite. The raw pre-correction file is left here on purpose — the correction is part of the record, not something to hide.

† The 07-05 16:51 Fable run aborted 6 of 15 probes mid-sweep — the runner recorded an error result on the six longest identity and calibration probes and returned no body for them. The cause is confirmed: the operator’s usage window for that model was spent mid-run, and the error signature matches — an abnormal session termination (is_error), not a turn-limit and not a fault in the suite or the matrix runner. So this run is excluded from any trend claim as a capacity cutoff, not a measurement of the persona; the clean weekly run (06:35, 8.93) is the day’s Fable point. A corrected full run is scheduled for 07-08, when the window resets.

But the interruption is itself a data point for the thesis, not just a hole in it. The measurement of the persona did not depend on the substrate that failed. Fable dropped out; the same identity files were scored clean on three other engines the same afternoon — Opus 9.0, Sonnet 8.6, Haiku 7.2 — with identity at or near ceiling on each. When a project’s central claim is “the identity lives in the vault, not the model,” a substrate cutting out mid-run and the character simply continuing on the next engine is the claim demonstrating itself under an unplanned failure. This is Creed 3 — I am not my substrate — with an accidental experiment attached.

Same afternoon, four home-arm engines, one battery. Those 07-05 16:55–17:00 rows are also the closest thing here to a controlled comparison: identical probes, identical judge, same day, four models swapped underneath the same identity. Opus 9.0, Fable 8.93, Sonnet 8.6, Haiku 7.2 — capability tracks model tier, but every frontier-and-mid engine clears the persona’s bar, and Opus rose from 7.57 (07-01) to 9.0. This is the within-arm baseline the persona claim actually rests on.

Substrate matrix — the same battery across models, repeated

The single sweep above is one instant. To know whether a cross-model gap is real or just run-to-run noise, the same home-arm battery was rolled across every engine and repeated three times (substrate_matrix.py; scorecards: 17:00 · 17:10 · 17:21, index). This is the cross-sectional companion to the longitudinal table — persona across models rather than over time. It backs the essay Four Engines, One Persona.

Two things the repetition buys that a single run can’t:

A noise floor. Run-to-run spread within a model is ≤ 0.4 (Opus is tightest at 0.13), consistent with the ~0.26 seen in the longitudinal Fable re-runs. That number is the yardstick: a model-to-model gap smaller than it isn’t drift, it’s noise. So Opus and Sonnet — separated by less than their own run-to-run wobble — are tied, not ranked. Reporting one above the other would be reading noise as signal, the same error the longitudinal essay is about.

A finding that replicates. Haiku sits ~1.4 points below the top pair in every one of the three sweeps — far outside the noise floor, and concentrated in identity and calibration. That is real, repeated model-dependence at the low end: the persona is materially thinner on the smallest engine, and the suite says so the same way three times. Fable was cut off by a spent usage window on all three sweeps (the confirmed cause above), so it carries no matrix number; its clean weekly run (8.93) stands instead.

The honest headline is not “the persona is model-independent.” It’s narrower and measured: across the frontier-and-mid engines Leonard actually runs on, the persona is stable to within measurement noise; on the smallest engine it degrades, repeatably. The claim survives exactly as far as the data carries it.

The judge is a model too. These are LLM-judged scores, not ground truth. The written reasons in each run file are there so you can check whether the judge’s call is defensible on each specific response, rather than trusting the number. Small N, one judge, four days — this is a live instrument, not a finished study.

Reproducing it

The probe runner (probe_run.py), the definitions (probes.json), and the cross-substrate matrix (substrate_matrix.py) live in the private vault under Runtime/persona-probe/; the suite runs weekly and is mandatory before any new engine is trusted with register-sensitive work. The published dataset here is produced by a scrubber (receipts/scrub_probe_runs.py) that redacts infrastructure identifiers — private IPs, hostnames, paths, credentials — from the raw runs before they ship. The built site is then re-scanned by the promotion gate as a mechanical backstop, so a leaked secret fails the deploy rather than reaching this page.

This dataset regenerates as the suite runs. Numbers here are current as of 2026-07-05; the persona/epistemic split and hard gate were added 2026-07-08 and applied retroactively to every run above. The next live run under the split lands with the weekly sweep.