# Offline retrospective progress per turn Untested development tooling for [protocol §1](../03-experimental-protocol.md#1-research-questions-and-primary-comparison) and [§7](../03-experimental-protocol.md#7-independent-assessment); outputs are not empirical evidence and **all defaults remain unfrozen**. This tool validates existing human assessments and calculates retrospective scores. It creates no judgments, does not choose a historical winner, and never changes the prescribed final endpoint or feeds scores into adaptation. ## Coordinator inputs Copy [manifest.template.json](manifest.template.json) to a coordinator workspace. It is a blank schema sketch, not a completed study input. Supply: - `turn_count`: explicit positive integer `T`, defining expected turns 1…T. The protocol proposes ten turns; this is an unfrozen starting choice, not a silent default. Turn 1 is the first delivered worker artifact, not an imputed zero-score baseline. Do not compress numbering to hide missing turns. - Two analysis-level independent rater IDs and explicit condition/repetition declarations. Expected coverage at **every turn** is all twelve registry tasks × each declared repetition for that condition. Conditions/repetitions must be declared by supplied archives; an explicit subset panel is allowed. Unselected archived run IDs are reported. Dataset size is a development default, not a registered sample-size decision. - Explicit sealed archive paths and selected run identities. Each run's `snapshots` list names actual delivered records by `turn`, optional UTC `delivered_at_utc`, sealed artifact `record_id`/SHA-256, and an explicit assessment binding or `assessment: null`. Omit unavailable snapshots without renumbering later turns. Early stops/failures create missing later coverage, not carried-forward quality. At least one sealed run must be selected; its snapshots may all be unavailable. - `assessment_sets`: existing primary-assessment coordinator manifests and rating/adjudication form copies, each with an explicit path and raw-file hash. `rater_mapping` maps both analysis rater IDs bijectively to the set's source rater aliases. A set can cover many snapshots/cells, but the primary manifest permits only one artifact per condition/task/repetition cell. One set per assessed turn is a natural layout. Different blinded aliases across sets must be mapped explicitly; the tool cannot establish real-person identity from aliases. Snapshot `assessment` identifies `assessment_set_id` and `blinded_artifact_id`; its source assignment must match the run's task, task/rubric versions, family, condition, repetition and artifact hash. This explicit link is mandatory—scores are never found by searching for a matching or favorable artifact. An assessment assignment cannot be reused across turns/runs, even for identical bytes. Duplicate set IDs and repeated source batch/seal identities are rejected; combine split rating files within one set instead. Unused source assignments are reported and do not fill missing snapshot bindings. Missing assessment forms can be omitted from their arrays; missing assignments use `assessment: null`. Blank untouched rater templates are not completed primary forms and must not be substituted for missing records. Source paths resolve against the analysis manifest's parent; raw hashes cover exact bytes. Standard SHA-256 tools can compute them. No assessment files are created or edited. ## Validation and local usage ```sh python3 -B 'Japa two/progress-per-turn/analyze_progress.py' \ --manifest /coordinator/progress-selection.json \ --output /coordinator/new-retrospective-progress.json ``` `--registry PATH` optionally selects a compatible version-1.0 registry, including an archive's relocated registry; the default is the existing development assessment registry. The output parent must exist, and the destination must be new and outside all input archives. Existing paths, including broken symlinks, are refused. A complete report is staged, byte-checked and atomically linked without overwrite; a failed write/link publishes no report. Stdout contains only a success record/path/hash. Errors go to stderr with exit code 2 and no success output. Keep the output in a **coordinator-only workspace outside rater materials**; labels do not configure filesystem permissions. The tool reuses [sealed-archive validation](../run-sealing/RUN_OUTCOMES.md) and the [chronological trajectory validator](../repeated-work/README.md) through a private temporary field adapter. Only explicitly supplied snapshots are adapted; no instructions, extra history, candidate search, or assessment packet is generated. Turn order must be strictly increasing, duplicates/out-of-range turns and reused delivered record identities are rejected, and supplied timestamps must be chronological and precede archive sealing/run end. Unknown clocks remain null; ordinal chronology is caller-declared and cannot be proved from hashes alone. Source records must be sealed artifact members of the same run. Existing archive-contract limitations remain unchanged. Hash-checked private copies of source forms are passed through the unchanged [primary scorer](../development-tasks/assessor-only/manual-assessment/SCORING.md). It validates criterion/public-requirement mappings, binary values, evidence fields, stored scores, form/record identities and adjudication links. This tool reuses the scorer's normalized scores and eligibility decisions; it does not regrade content or trust unverified scalar values. Future rating/lock times are rejected, and every bound assessment must occur after the corresponding archive's batch was sealed. This does not prove that an externally registered batch contains no additional undeclared runs. Unsealed inputs, incompatible versions/rubrics, malformed fields, duplicate JSON keys and identity/hash mismatches abort analysis rather than dropping records. ## Scores, changes and missing coverage The reversible metric default is **normalized public-requirement satisfaction at each declared turn**. The current kit has twelve binary criteria mapped Cn→Rn. A usable snapshot score requires all twelve judgments and primary status `complete`; null criteria or `needs_resolution` yield no eligible score, even if a provisional arithmetic fraction exists. A zero score is an observed score, not missing data. Evidence locators remain human claims, not machine-verified semantic entailment. For each run, signed change at turn `t ≥ 2` is `score(t) − score(t−1)` only when both adjacent delivered turns have eligible assessments in the **same series**. Negative changes are retained. Missing snapshots, missing records, unresolved judgments, or absent bindings break adjacency: the tool never bridges 1→3 over missing turn 2, carries scores forward, or divides a larger jump across skipped turns. Turn 1 has no change baseline; its change mean is null with zero expected pairs. A one-turn analysis has no applicable change curve (null), not a zero-change finding. Each independent rater and adjudication is reported separately under `by_condition`. No independent scores are pooled with each other or filled from adjudication. Within a condition/turn, average all declared repetitions within each task, give tasks equal weight within each family, then give families equal weight. Changes use paired repetition-level differences before the same weighting; they are not differences of unmatched available-case means. There is no cross-condition pooled headline score. For every condition and turn, the report provides expected/assessed/missing score cells, missing runs, deliveries, assignments and records, ineligible records, and expected/observed/missing criterion judgments. `missing_snapshots` includes missing-run cells; it is not added again to `missing_run_cells`. Score-missing categories (snapshot, assignment, record, ineligible) otherwise distinguish the reason a score is unavailable. Partial criterion coverage is diagnostic only; it never reduces the requirement denominator. Adjacent-change points report their own complete-pair coverage. Task means require all expected repetitions; family means require all expected task means; overall means require every family. There is no renormalization over surviving observations. An incomplete point's aggregate mean is null, while a complete earlier/later point remains available with its own coverage. `curve_complete` is false and `fully_observed_score_curve` is null if any expected turn is incomplete; the analogous fields govern the full change curve. Complete family-specific points may remain visible when another family is missing, explicitly scoped to that family. Zero denominators return null. Per-run rows retain all declared turn positions and the archive's **prescribed endpoint identity/hash**, with its listed turn if present. A higher earlier score cannot move that endpoint. If the endpoint is outside the supplied curve, its listed turn stays null; no final assessment is substituted, synthesized or overwritten. An early-stopped run's endpoint is not copied to later expected turns. This analysis never replaces primary final-budget scoring. Observed score changes can reflect rater uncertainty, task differences, stochastic output, or other factors; they do **not** establish causal improvement by the prompts or planner. Turns/repetitions within tasks are dependent, and these descriptive curves supply no independent-sample intervals or power claims. Correctness of judgments, actual chronology, completeness of the declared history, and successful assessment blinding remain human/protocol responsibilities. Reports contain method-identifying metadata, remain coordinator-only, and must never enter planner or worker context. No actual study assessments or findings were generated for these assets. ## Temporary artificial-fixture tests ```sh cd 'Japa two/progress-per-turn' python3 -B -m unittest -v test_analyze_progress ``` Tests use temporary artificial sealed text/metadata and numeric binary-rating fixtures, not task answers, actual human study ratings or experimental results.