# Offline assessed requirement regression Version 1.0. **Development tooling; defaults are unfrozen.** Tests use temporary artificial numeric fixtures, not task answers or empirical observations. The utility adds no requirements, changes no forms/primary scores, and runs no models or experiments. This implements [protocol ยง7's](../../../03-experimental-protocol.md) secondary metric: a public requirement assessed as satisfied earlier and unsatisfied in the final artifact. The reversible default is **one explicitly designated earlier snapshot versus one designated final snapshot per run**. There is no best-history search, inferred intermediate assessment, trajectory maximum, or inferred causal explanation. Select the earlier comparison and endpoint policy before inspecting scores. ## Usage and existing validation Python 3.9+ and the standard library are sufficient. Copy [requirement-regression.template.json](requirement-regression.template.json), fill the coordinator manifest, and run from this folder: ```sh python3 -B analyze_requirement_regression.py --manifest /coordinator/paired-assessments.json ``` Optional `--registry` selects the matching task registry; the default is the current twelve-task registry. The tool reads locked **forms**, because version 1.0 scoring reports omit criterion judgments. It reuses `score_forms.py`'s unchanged full validation path, criterion mappings, binary checks, adjudication links and evidence conventions. Primary aggregates computed internally during that validation are not used as substitutes for requirement-level transitions. Source bytes are hash-checked and copied into temporary validation files, preventing a second read of changed assessment forms during the calculation. Temporary copies are deleted. The originals remain unchanged. JSON output on stdout is coordinator-only; it includes separate independent-rater series, adjudication, per-pair transitions, by-condition descriptive counts/rates, and coverage. Redirect only to a new report path. Exit 2 emits a JSON error on stderr and no analysis report. No live state, rater packet, or adaptation input is written. ## Pairing manifest Root fields are exactly `manifest_version` (`"1.0"`), `analysis_id`, `run_batch_seal`, `independent_rater_ids`, `earlier_assessments`, `final_assessments`, and `pairs`. The seal object has a nonempty `seal_record_id`, boolean `sealed: true`, and ISO 8601 UTC `sealed_at_utc` no later than now. Source assessment completion times must follow that declared run-batch seal. Forms must already have valid lock timestamps. Declarations/hashes cannot prove historical sealing, independence, or prespecification. Two named assessment sets are used because the existing primary coordinator contract permits only one artifact per condition/task/repetition cell. This adapter does not relax that rule or add endpoint metadata to rater forms. Each set has: - `coordinator_manifest`: `{ "path": string, "sha256": lowercase SHA-256 }` for an existing version 1.0 scoring manifest. - `rating_forms` and `adjudication_forms`: arrays of the same file descriptors. Paths resolve relative to the regression manifest, or may be absolute. Empty arrays represent absent assessments, not invented ratings. Nonexistent files or bad hashes are errors. - `rater_mapping`: a bijection from the two top-level analysis-rater identities to that set's two source rater aliases. Different blinded batches may use different aliases. The coordinator must map the **same actual rater identity** across endpoints; the tool cannot verify human identity. Merging two aliases into one series is rejected. No mapping combines raters with adjudication. Every pair has `pair_id`, `run_id`, `condition_id`, `task_id`, `task_version`, `rubric_version`, `repetition_id`, `earlier`, and `final`. Both endpoint objects contain exactly `blinded_artifact_id`, `blinded_artifact_sha256`, and positive integer `turn`. Earlier turn must be strictly less than final turn, with distinct snapshot identities. Run IDs, pair IDs, condition/task/repetition cells, and endpoint assignments cannot be duplicated. Conditions/tasks/repetitions, task/rubric versions, family, and hashes must match the selected source assignments and registry. A same-task comparison under a revised rubric is rejected rather than called regression. An unknown endpoint identity is an integrity error. A declared endpoint with no rating record is instead missing assessment coverage. Only the listed pairs are analyzed; other source assessments are validated but cannot become alternative earlier snapshots. No unlisted run/history is inferred. Pair chronology is caller-declared turn ordering, not a fact reconstructed from prose or file modification times. Keep this manifest and report outside every rater and model-facing packet. ## Criterion transitions, denominator, and missingness For each series/run, Cn is paired with Cn and its existing public Rn, regardless of criterion row order. For observed binary judgments, `1->0` is a regression, `1->1` retains satisfaction, and earlier-zero pairs do not enter the previously-satisfied denominator. If either judgment is null or its record is absent, transition and `regressed` are null; no pass/fail is imputed. The reported `eligible_previously_satisfied_denominator` counts **earlier = 1 with an observed binary final judgment**. `observed_regressions` counts observed `1->0` transitions. `assessed_regression_rate` divides those counts, or is **null when the denominator is zero**. This is explicitly an observed-pair conditional rate, not a claim about missing outcomes. The report also gives total expected/observed/missing criterion pairs, missing endpoint-record pairs, all known earlier positives, and the number of known earlier positives whose final judgment is missing. A zero observed numerator with incomplete coverage does not establish no regression. Known binary judgments in locked `needs_resolution` records can contribute individually even when another criterion remains null; status/issue fields stay visible. This differs from the primary score's whole-record aggregation eligibility and does not change that behavior. Adjudicated corrections never fill independent-rater nulls. If adjudication is absent, its series retains missing pairs and a null rate instead of silently copying agreement or a rater's score. Per-condition and total summaries pool descriptive **counts within each separate series**, so their ratios weight eligible earlier-positive requirements, not equal tasks/families. They are not the primary equal-family requirement-satisfaction endpoint, a new composite score, or a registered inferential contrast. Prefer per-run counts and disclose coverage; freeze any further task/family aggregation or statistical procedure before study analysis. No cross-rater pooling or confidence interval is supplied. This measures **assessed public-requirement regression**, not all semantic deterioration, factual drift, stylistic loss, harm, or causal effects of prompting. Rater disagreement/error can create apparent transitions; unassessed snapshots and requirements remain unknown. Comparing only one earlier snapshot can miss temporary gains/losses elsewhere. The software performs no regrading or hidden semantic evaluation. ## Focused checks ```sh python3 -B -m unittest -v test_requirement_regression.py test_score_forms.py ``` Artificial numeric fixtures cover all binary transitions, denominators/nulls, missing judgments/records, series separation, adjudication isolation, criterion-ID alignment, rater aliases, explicit endpoint scope, duplicates, reversed turns, task/rubric/hash errors, sealing declarations, unchanged source files and CLI behavior. Fixtures are deleted. No task answers, actual study inputs, purported empirical findings, model calls, experiments, credits, production changes, or submissions are generated.