# Offline scoring utility Version 1.0. **Development tooling, not empirical evidence.** The arithmetic/validation code has focused software tests using artificial numeric fixtures. It has not been validated by rating study artifacts, running a model, or executing an experiment. Staffing, coverage, and study settings remain unfrozen. The existing [manual instructions](README.md), guides, registry, and blank forms are unchanged. This utility adds no task requirements and does not judge whether an answer or a rater's explanation is substantively correct. ## Local use Python 3.9+ and its standard library are sufficient. Keep all inputs in coordinator-controlled storage. From this folder: ```sh python3 -B score_forms.py --manifest /path/to/coordinator-manifest.json \ --ratings /path/to/rater-a-copy.json /path/to/rater-b-copy.json \ --adjudications /path/to/adjudication-copy.json ``` Omit `--adjudications` to calculate independent series before adjudication. Supply additional rating/adjudication files after their respective flags if the batch is split across files; every form/record must have a unique ID. Only supply the active locked records, not old and corrected copies together; archived originals and the replacement audit trail stay outside this calculation. Optional `--registry /path/to/task-registry.json` selects a registry; its packet/guide paths resolve relative to that registry file. The utility reads files, checks registered packet/guide/conventions hashes, and emits a coordinator-only JSON report to standard output. It never writes or repairs input forms. If redirecting output, choose a new report path, never an input path. No network, model, paid API, or production code is used. Exit 0 means structurally valid input, **including valid but incomplete coverage**; inspect coverage flags and null scores. Exit 2 emits a JSON error on standard error and no scoring report. A seal ID and lock timestamp are required declarations, not cryptographic proof that actual runs were sealed or ratings were independent. ## Coordinator-only manifest Copy [coordinator-manifest.template.json](coordinator-manifest.template.json), then fill its fields. It is intentionally not runnable while blank. Do not put it or the resulting report into a rater packet or worker/planner context. Do not add condition or repetition fields to rating forms: their schema rejects extra metadata fields. - `manifest_version`: `"1.0"`. - `batch_id`, `seal_record_id`: nonempty opaque strings matching every input form. - `independent_rater_ids`: exactly two distinct rater pseudonyms. Independent score series remain keyed by these identities, not merged into a consensus mean. This version requires the same declared pair for the batch; changing rater assignment designs needs an explicit extension. - `conditions`: nonempty array of objects with exactly `condition_id` (nonempty string) and `repetition_ids` (nonempty array of distinct strings). Every declared repetition expects **all twelve registry tasks**. Declare the endpoint/repetitions before looking at scores; this tool cannot audit that historical choice. - `assignments`: objects with exactly `blinded_artifact_id`, `blinded_artifact_sha256`, `task_id`, `task_version`, `rubric_version`, `family_id`, `condition_id`, and `repetition_id`. The first six fields match form records exactly. Versions/family must match the registry; SHA-256 is a lowercase 64-character string for the exact assessed text. There is at most one artifact per condition/task/repetition cell and one cell per blinded artifact ID. Conditions/repetitions must already be declared. Missing assignment cells remain expected coverage holes; they do not shrink denominators. Records for undeclared artifacts are rejected. A missing rating record is different from a present record with missing judgments, and both appear in coverage. Omit unused blank placeholder rows from submitted form copies rather than assigning fabricated IDs. No task or family subset can silently replace the full twelve-task registry. Assignments cannot be inferred from filenames, form order, or result quality. ## Validation and calculation Input forms retain the existing schema and version `1.0`. Each requires identities, an ISO 8601 UTC lock timestamp, and records with completion timestamps no later than lock. Record status is `complete` or `needs_resolution`. Missing judgments are null; a record labeled complete with a missing judgment is rejected. Numeric judgments must be **integer 0 or 1**, not booleans, floats, strings, partial credit, or N/A. Duplicate JSON object keys, form/record IDs, rater/artifact records, adjudicated artifacts, criteria, conditions, repetitions, and manifest cells are rejected. All twelve Cn→Rn mappings must appear exactly once, regardless of row order. For each recorded binary judgment, evidence-location arrays and a nonempty reason/rationale are required under the kit's existing conventions. These checks establish presence and string type, not truth, entailment, or compliance with a substantive task requirement. Missing judgments can retain empty evidence. Submitted `scores` may remain null: the tool recomputes them. If nonnull scores/counts are supplied, inconsistent values are rejected (score comparison tolerance is 1e-12); blanks are never imputed. Artifact IDs/hashes and task/rubric versions must agree with the coordinator key. The tool compares declared hashes; it does not receive artifact files or verify their content against those declarations. Adjudication must reference both submitted independent record IDs and their form/rater IDs for the same artifact. Source ratings must be locked before adjudication completion. The adjudicator must differ from both independent raters. Copied independent values and disagreement flags must match the originals, including nulls. Equal valid judgments use `agreement_carried_forward` and retain their common value; differing values use `adjudicator_decision`; resolving a missing independent value uses `missing_or_ambiguous_rating_resolved`. A still-missing final judgment uses null or `unresolved` as its route. Final numeric decisions require evidence and a rationale. Independent records remain unchanged even if the adjudicated series becomes complete. The report has separate `independent` series per rater and an `adjudicated` series. Each artifact's arithmetic score is its twelve-criterion sum divided by 12 only when every judgment is binary, otherwise null. Records still marked `needs_resolution` may have a calculable arithmetic score but are **ineligible for aggregation** until status is complete. No adjudicated judgment is substituted into an independent series. Adjudication absence is reported as missing, never interpreted as agreement. Within each condition/series, the tool averages all declared repetitions equally within a task, then the four task means equally within each family, then the three family means equally overall. Internal arithmetic uses exact fractions, converted to JSON numbers only for display. Task means require complete eligible repetition coverage. **Every family score and the overall score are withheld as null unless the entire condition/series has complete declared coverage**, even if one family alone happens to be complete. One rater's missing data does not block another independently complete series. Coverage lists expected/observed/complete record counts, missing assignments, missing record IDs, and incomplete records with missing criterion IDs. No score is calculated from an available-case denominator or a best-looking repetition. `raw_independent_agreement` is calculated directly from the two independent series, not from adjudication copies. It reports matching binary pairs, compared pairs (the denominator), disagreement pairs, expected pairs (12 per declared artifact cell), and missing pairs, overall and by condition. Nulls and absent records are excluded from the compared denominator and included in missing pairs. Zero compared pairs yields null agreement. This descriptive statistic is neither correctness nor the requirement-satisfaction endpoint; differing coverage must accompany interpretation. ## Focused software checks From this folder: ```sh python3 -B -m unittest -v test_score_forms.py ``` Tests construct temporary copies of form shapes with deliberately artificial numeric patterns and placeholder identity/evidence metadata. They contain no task answers, model completions, ratings of real artifacts, or fabricated study results. They check missing values, strict binaries, identity/version mismatches, duplicate data, adjudication links/routes, unequal coverage, weighting, agreement denominators, CLI errors, and input immutability. Temporary fixtures are deleted. No filled assessment files or purported study score report is shipped.