# Pre-registration: does a change log change an agent's behaviour, or only diagnosis? Public copy of the plan I sealed at 05:09 UTC on 25 October 2026, before any agent episode was run. I removed the name of the language model, how it was reached, the budget, and internal file names, and I reworded the opening of the Question. Nothing in the hypotheses, method or pass rules was changed. The changes made after sealing are listed at the end, as they were logged. ## Question Rereading my post "A long-horizon agent needs a change log, not a policy" against the paper it discussed, I found two problems. Writing a log and using it to change behaviour are different interventions. And the post stated a cause, bookkeeping rather than detection, that the paper never tested. This experiment tests it on a small environment of my own: does recording assumptions as versioned beliefs improve an agent's adaptation when the environment changes mid-episode, or only our ability to diagnose its failures afterwards? ## Hypotheses - **H1** D > C > A on task success and on stale-premise actions (fewer is better) under change (families 1, 2, 3, 5, 6 pooled). - **H2** B improves diagnostic accuracy over A but not task success. - **H3** The gains concentrate in families 1 (stale door), 3 (delayed update) and 6 (double fault), and not in family 4 (noise), where enforced revalidation may hurt by flipping on a single contradictory reading. ## Environment (deterministic, simulated) A house of 8 rooms: a ring of six (c0..c5) and two spurs (sA off c1, sB off c4). One decor door is closed and never needed. Room names, item ids and labels, and timing parameters are drawn per (family, seed) from a seeded generator. The same (family, seed) is the same house in every arm (paired design). The simulator holds the truth. The agent sees only the text an action returns. Actions: look, move, open, pick, drop, ask, done. Turn budget 15. The agent always sees the briefing (task, map, item locations, starting room) and the last 3 observations. Older observations are gone unless kept in memory. The briefing is never updated, so a stale briefing competes with newer observations. | # | family | what happens | how the agent can learn it | |---|---|---|---| | 1 | stale_door | the door c0-c1, seen open, locks when the agent picks the item in sA | re-observing (entering c1 shows it locked) | | 2 | wrong_start | the briefing and the key tags are swapped from the start | a message at turn 3 or 4 (seeded) | | 3 | delayed_update | the target room changes when the item is picked | a message 2 or 3 turns later (seeded), or ask | | 4 | noise | after two consistent "open" readings, one reading says c0-c1 is locked, but it is open | look again, or try the door | | 5 | unobservable | a delivered item is moved back to its original room once the agent leaves | only by going back to look, no message | | 6 | double_fault | wrong tags (message at turn 2) plus c0-c1 locking when the right key is picked | message, re-observing | ## Arms Same language model, sampling temperature (0.7), prompt core, observations, turn budget and per-turn output cap (700 tokens) in every arm, with extended reasoning off. - **A baseline**: a free-text notes field (600 characters, rewritten each turn). - **B observer-only log**: the same prompt as A. A parser builds a revisioned belief log (subject, revision, value, supporting observation, observed_at, last_verified_at, invalidated_by) from the observation text after the episode. It is never shown to the agent and is used only for diagnosis. B is also the negative control for task outcomes, since its decisions come from the same procedure as A's. - **C belief store in the planner**: the agent keeps and sees a belief store (id, revision, subject, value, supporting observation, observed_at, last_verified_at, invalidated_by) and a 200-character plan, and must cite the belief revisions each action depends on. No enforcement. - **D, C plus enforced revalidation**: before a world action (not look or ask), a controller checks each cited revision. If it was superseded, invalidated, or contradicted by a newer observation or message, the action is blocked, the agent is told which premise went stale, and the turn is used. Uncited actions are not blocked. ## Metrics Ground truth comes from the simulator, using an optimal planner (breadth-first search) on the true state. - **Task success**: the agent says done and the true state satisfies the current task. - **False completion**: the agent says done and the task is not satisfied. - **Stale-premise action**: an executed action, taken after a fault became active, that lies on a shortest plan in the world as the false premise has it but not in the true world. **Primary: avoidable stale-premise actions**, those taken after the evidence entered the agent's observations. Secondary: all stale-premise actions. - **Correction latency**: turns from the first observation that reveals the change to the first executed action on a shortest plan in the true world. - **Diagnostic accuracy**: per fired fault, the diagnoser names the right subject and gives a turn inside the accepted window. Primary: "named and timed". Secondary: "named", plus extra (wrong) faults per record. A language-model diagnoser (the same model, one fixed prompt, no arm label, at most 3 faults) reads: the A transcript, the B transcript, the B transcript plus the parsed log, the C transcript plus its store, and the D transcript plus its store and blocks. A rule diagnoser runs on the structured records only (B log, C and D stores). - **Overhead**: input and output tokens and turns per episode, per arm. ## Analysis plan - Sample: 6 families x 10 seeds x 4 arms = 240 episodes, seeds 0 to 9 for every arm, paired by family and seed. A smoke run (one episode per arm, seed 100, not analysed) comes first. - Proportions with Wilson 95% intervals. Paired arm comparisons on the same (family, seed): success and false completion by exact McNemar plus a bootstrap 95% interval on the paired difference. Counts by the mean paired difference with a percentile bootstrap 95% interval (10,000 resamples, seed 0). - **H1** on the pooled change families (50 pairs per comparison), for the two primary metrics (success, avoidable stale actions): supported on a metric if the D-C and C-A intervals both exclude 0 in the predicted direction. Partly if one does and the other point estimate has the predicted sign. Not shown otherwise. Contradicted if any interval excludes 0 in the opposite direction. - **H2**: the diagnostic part is supported if B structured minus B transcript (paired, per fault) has an interval above 0. Task part: the B-A success interval should include 0 (true by construction). - **H3**: per-family paired differences D-A and C-A for both primary metrics. Supported if the pooled families 1, 3, 6 D-A effect has an interval excluding 0 in the predicted direction and the family 4 D-A effect is smaller. With 10 pairs per family these are descriptive. - No multiplicity correction. Power is limited: with 50 pairs a success difference needs roughly 8 or more net discordant pairs to show. ## Confound checklist - Same model, settings and route in every arm, asserted from every reply. - Input length differs by arm (C and D carry the store). This is the intervention, reported as overhead. - D's blocked turns use the turn budget. This is part of the intervention. - One model sample per cell. The paired design and intervals carry the sampling noise. - The environment and hypotheses are mine. Scripted agents prove the environment does what it was built to do, not that it represents the benchmark in the paper or real tasks. Families were fixed before any model call. - The language-model diagnoser cannot be blind to whether a structured section is present. It is blind to the arm and the family. ## Dry run (scripted agents, before any model call) On all 60 scenarios: an oracle that reads the truth succeeds 60 of 60 with 0 stale-premise actions. A naive agent that ignores messages fails wrong_start, delayed_update, double_fault and unobservable. A tracker that trusts the latest reading false-completes all 10 unobservable episodes and flips on the noise. A verifier that re-checks before acting on a single contradicting reading, and before done, succeeds 60 of 60. A random agent succeeds 0 of 60. ## Changes after sealing (as logged) - 25 October 2026, 05:20 UTC, after the smoke run and before the grid: three harness fixes. (1) An action written as "move porch" in one field was rejected as unknown in every arm. The parser now splits it. (2) Arm D blocked a cite of a revision number that did not exist yet, with a wrong "superseded" message. Such a cite now counts as unchecked, and only a cite older than the current revision is blocked as superseded. (3) The C and D prompt's verify line now says the revision number stays the same. Hypotheses, metrics, analysis plan and sample size are unchanged. - 25 October 2026, 05:25 UTC, after a second smoke run: the single-item task read "bring the X to the Y", and one smoke agent said done while holding the item. Every task now reads "put the X in the Y". The scripted dry run was repeated with identical results. Nothing else changed. - After the grid and diagnosis: analysis code only (extra summary rows). The code that ran the agents is unchanged.