# Offline human critical-requirement assessment Untested development tooling for [protocol §7](../03-experimental-protocol.md#7-independent-assessment), with **unfrozen defaults**, not empirical evidence. Criticality and violation judgments are human inputs. The tool does not read primary scores, promote failed criteria into critical requirements, add requirements, or generate judgments. It reuses sealed-archive validation, blinded-packet helpers, the public identifier parser, and the existing strict binary-rating validator. Primary scoring remains separate and unchanged. ## Declare the public critical set Copy [selection-manifest.template.json](selection-manifest.template.json). Explicitly select sealed designated final artifacts by archive/seal/run/condition/task/repetition identities, versions, artifact ID and SHA-256. A prespecified subset is allowed; coverage means the **selected artifact list**, with unselected archived run IDs reported separately. Failed or incomplete runs with designated final artifacts can be assessed; an artifact does not establish completion. Provide exactly one [designation file](designation.template.json) for each selected task, shared across its conditions and repetitions. This reversible default prevents condition-specific critical sets within a batch. Bind the file to the task/rubric versions and exact fixed public-packet hash, and put its raw-file hash in the selection descriptor. File paths resolve against the selection manifest's parent. Standard SHA-256 tools can compute those hashes; no generated content is needed. Humans explicitly select existing `R` IDs. Each selection must quote the **whole matching public requirement**, not a new custom criterion or stricter paraphrase. `requirement_location` supplies zero-based Unicode code-point `[start,end)` positions and exact `text` in the unchanged UTF-8-decoded public packet. Include the full requirement bullet/block, excluding trailing CR/LF separators. Any `source_locations` must name existing `S` IDs and quote exact spans inside the named source section. CR, LF and any BOM each count; byte/UTF-16 offsets and normalized editor text are not interchangeable. Source references may be empty where the public requirement itself is the authority, such as an explicit format constraint. The requirement quote is always mandatory. Duplicate requirement IDs, duplicate source references, unknown IDs, partial/mismatched requirement quotes, cross-section source spans, hidden fields, wrong hashes or incompatible versions are rejected. Existing public IDs and Cn→Rn mappings are validated structurally; citation validity does not prove semantic support, criticality, or a violation. Designator notes remain private and cannot add public obligations. Designation states are explicit: - `review_status: "reviewed"` requires a named human, past UTC completion time, and scope/limitations note. `requirements: []` declares a reviewed **empty critical set**, not that all requirements are satisfied. - `review_status: "unfinished"` requires a null completion time and explanatory note. Its list may be partial or empty; either prevents complete coverage. - For an unavailable designation, use selection `designation: null` and a nonempty `unavailability_reason`. Never substitute an empty reviewed file; the unknown critical-set size stays null. Prefer to freeze criticality from public tasks before viewing outputs, primary scores or condition identities. Designation review may therefore predate sealing. Development changes require new hashes and new packets/forms, preserving the old attempt. Hashes and timestamps do not prove preregistration, unbiased designation, or semantic appropriateness. All templates are blank schema sketches, not completed inputs or actual study designations. ## Build blank blinded materials ```sh python3 -B 'Japa two/critical-violations/assess_critical.py' build \ --manifest /coordinator/critical-selection.json \ --rater-output /coordinator/new-critical-rater-packets \ --coordinator-output /coordinator/new-private-critical-key \ --seed 0 ``` Two independent raters and a distinct adjudicator are the reversible default. Both output directories must be new, disjoint, have existing parents, and stay outside source archives. Existing paths, including broken symlinks, are refused. Validation reuses the [sealed-archive contract](../run-sealing/RUN_OUTCOMES.md), including commits, payload/reference hashes, run cells and endpoint identities; existing archive limitations remain unchanged. `--registry PATH` selects a compatible registry, including an archive's relocated registry; the default is the existing twelve-task development registry. Each opaque rater directory contains unchanged final artifact bytes, matching public packets, a sanitized critical designation, an index/instructions, and a judgment-blank form. Source/critical-set hashes and public requirement IDs remain visible for verification. Condition/run/repetition mappings, human designator identities, private notes, archive metadata, prompts, memory, resource records, primary scores and requirement-scoring assessor guides are excluded from distributed files and filenames. Public critical sets contain exact existing requirements and source quotes only. The seed shuffles artifact presentation order from canonical run order; opaque aliases use fresh UUIDs and are not seeded. Coordinator output retains mappings, exact source selection and designation snapshots; `critical-key.json` is written last. A failed write can leave partial directories but emits no success record; inspect and choose new destinations rather than merge. Directory separation does not enforce confidentiality. Intrinsic method disclosures in artifact/public text are preserved; opaque IDs cannot remove them or guarantee blinding. Distribute only the assigned opaque rater directory. ## Human violation judgments Forms follow the blank [independent](independent-form.template.json) and [adjudication](adjudication-form.template.json) schemas. For each designated artifact/requirement instance, supply integer `violation: 1` if violated, or `0` for no violation. Separately set `invented_restriction: 1` only when that violation includes an unauthorized restriction against **the same public requirement**; otherwise explicitly use `0`. A flag of 1 requires violation=1. Both remain null when unresolved. Booleans, fractional values and incomplete binary pairs are rejected. Invented eligibility rules, fees or approvals are assessable only insofar as they violate a designated public boundary; the tool creates no catch-all hidden prohibition. Multiple failures/restrictions within the same artifact/requirement count once. The flag is a descriptive annotation, not another denominator item or a new requirement. A human ordinary criterion failure is not imported as a critical judgment. Record a reason and nonempty `artifact_scope` explaining what was reviewed. Where possible, add exact artifact-evidence spans (`start`, `end`, `text`) under the same character-offset convention; these are checked against the artifact. For omissions, an empty evidence list is allowed, with scope and rationale explaining the absence. Exact quotes still do not establish semantic violation. Do not revise the critical set, task or artifact while rating; report ambiguity and leave the judgment null. Rate independently and lock each form after its judgments. Completion times must follow sealing and any later reviewed designation. The third human supplies a separate decision, evidence and resolution note after both independent locks; `independent_judgments` maps both opaque rater aliases to their exact `{violation, invented_restriction}` pairs. No automatic agreement copying or majority vote occurs. A disagreements-only adjudicated form leaves other items missing. Never feed any assessment back to the planner or worker. ## Summaries and limits ```sh python3 -B 'Japa two/critical-violations/assess_critical.py' summarize \ --key /coordinator/new-private-critical-key/critical-key.json \ --forms /coordinator/returned/independent-a.json \ /coordinator/returned/independent-b.json /coordinator/returned/adjudication.json ``` Omit unavailable forms, including the entire `--forms` option, to inspect coverage. Archives, source hashes, source references, identities, versions, binary fields, evidence spans, times and adjudication bases are revalidated. Original designation/selection working files can be removed: exact coordinator copies are used. Source archives and a matching registry must remain accessible. Inputs are read-only. Reports go to stdout; errors go to stderr with exit code 2 and no report. Save reports to new coordinator-only destinations outside inputs; shell redirection is not managed. Each independent rater and, separately, adjudication has `overall` and `by_condition` summaries: - Violated critical requirements over **nonnull assessed critical requirement instances**, plus provided-instance and missing-judgment counts. Instances are artifact/public-requirement pairs, repeated across distinct runs as declared; raters are never pooled. This descriptive ratio is not the primary equal-task/equal-family score. - Counts of artifacts with at least one assessed critical violation, and optional invented-restriction instance/artifact counts. Under partial coverage these positive counts are lower bounds. Per-artifact absence remains null until the designated set and judgments are complete. - Reviewed-empty, unfinished and unavailable designation artifact counts. Unavailable set sizes remain null and cannot enter the known-instance denominator. Unfinished sets may contribute provisional counts but prevent complete rates. - Complete requirement and artifact rates are withheld whenever any selected designation is unavailable/unfinished or any judgment is missing. Zero requirement denominators yield null. The artifact rate's explicit denominator is all selected artifacts; reviewed-empty sets contribute zero positives, not proof of general compliance. A reviewed-empty-only batch has a null requirement rate and a scoped zero artifact rate when its artifact denominator is nonzero. `independent_rater_agreement` is separate from adjudication: exact binary agreement for violation and invented-restriction labels, with a shared jointly assessed denominator, missing-joint-judgment count and unavailable/unfinished context. A zero joint denominator yields null. Agreement is conditional on the common human designation; it is not independent criticality agreement or adjudicator reliability. No inferential intervals are supplied. “Complete” means declared assessment coverage, not that the critical set is objectively correct or exhaustive. The tool cannot verify semantic source entailment, truthful artifact review, preselection, successful blinding, all invented restrictions, or every serious failure. Human selection/judgment errors and source defects remain possible. This metric neither replaces primary scoring nor establishes superiority. No actual study packets, designations or judgments were generated to create these assets. ## Artificial-fixture tests ```sh cd 'Japa two/critical-violations' python3 -B -m unittest -v test_assess_critical ``` Tests use temporary artificial text, metadata and binary fixtures, not task answers, actual designations or empirical findings.