# Offline blinded pairwise preference Development tooling for [protocol §§7–8](../03-experimental-protocol.md#7-independent-assessment); no empirical evidence, automatic judgments, model calls, or adaptation feedback. Defaults remain unfrozen. Usefulness and clarity are separate secondary dimensions, never a combined headline score or substitute for requirement satisfaction. ## Declare pairs and build packets Copy [pair-manifest.template.json](pair-manifest.template.json) into a coordinator workspace and fill its identities. One batch declares one ordered contrast (`target_condition_id` versus `comparator_condition_id`), two independent raters, a distinct adjudicator, and a nonempty list of expected repetition IDs. Expected coverage is **every task in the existing twelve-task registry × every declared repetition**, not just the pairs supplied. These development choices are not a registered sample size or held-out design. List explicit archives and artifact pairs. Each pair supplies its task/task version, rubric version, repetition, and both endpoints' archive alias, seal ID, run ID, condition ID, designated final artifact ID, and raw SHA-256. Both artifacts must have the same task, versions, and repetition. Relative archive paths resolve against the original selection manifest's parent. No endpoint selection, best-artifact search, missing-artifact tie, or semantic completion inference is performed. Incomplete/failed runs with designated final artifacts may be compared; endpoint availability does not establish successful completion. Missing pair cells stay coverage gaps. ```sh python3 -B 'Japa two/pairwise-preference/pairwise_preferences.py' build \ --manifest /coordinator/pair-selection.json \ --rater-output /coordinator/new-rater-packets \ --coordinator-output /coordinator/new-private-pair-key \ --seed 0 ``` Both output directories must be new, have existing parents, be disjoint, and stay outside all source archives. Existing paths (including broken symlinks) are refused. Preflight validates every supplied archive through the unchanged [sealed-archive reader](../run-sealing/RUN_OUTCOMES.md), including commit/sealing status, versions, complete inventories, identities, and payload/reference hashes. Endpoint bytes are checked again with the existing assessment-packet helper. Duplicate pairs, task/repetition cells, archive/seal/run identities, and incompatible references are rejected. An invalid input aborts the batch rather than silently dropping it. `--registry PATH` can select a compatible registry, including an archive's relocated registry; the default is the existing twelve-task registry. Original source files need not remain available. Existing archive-contract limitations remain unchanged. The reversible randomization default is Python's seeded PRNG: canonical source-pair-ID order, one left/right draw per pair, **the same orientation for both independent raters and the adjudicator**. Each form also gets shuffled comparison order. Opaque batch/rater/comparison/artifact/form IDs use unseeded UUIDs; the seed reproduces side assignments, not file identities. A new build is a new assessment batch, not interchangeable forms. The coordinator key retains seed, orientation, source-run/condition mappings, and assessor aliases outside every rater directory. Each of the three opaque rater directories contains an index, unchanged UTF-8 artifact bytes, the matching unchanged public task/source packets, instructions, and an identity-filled but judgment-blank form. Prompts, memory, usage logs, archive metadata, condition/run/repetition IDs, the coordinator manifest, and requirement-scoring assessor guides are not copied into rater files or filenames. Usefulness/clarity instructions replace the unrelated binary requirement-scoring rubric. Original method text **inside deliverables is preserved**: opaque IDs cannot remove intrinsic method disclosures or guarantee successful blinding. Directory separation does not enforce confidentiality. Distribute only the assigned rater directory, not its parent or coordinator output. Coordinator output contains `comparison-key.json` (written last), the exact source-manifest bytes, and their original path base in `source-location.json`. A failed write can leave partial directories; no success JSON is emitted. Inspect them and use new destinations rather than merging or overwriting. Keep source archives and coordinator files available for summary revalidation; relocation requires an explicit, carefully checked path-base update, not discovery. ## Human ratings and adjudication The [blank form schema](rating-form.template.json) documents the fields; its null identities are placeholders. Use generated forms to avoid identity transcription. `usefulness` means fitness for the intended task and practical value; `clarity` means readability, organization, and precision. These are proposed operational definitions to refine and freeze before a study. Raters use only the final artifacts and authorized public packet, accept supported alternatives, and add no hidden requirements. Better prose must not be interpreted as proof of factual correctness. Rate each dimension as `left_win`, `tie`, or `right_win`. Tie means equivalent preference; uncertainty, missing context, and unassessed comparisons remain null. Nonnull judgments require a reason, precise evidence locations, and UTC completion time after both source archives were sealed. Lock each form with `locked_at_utc` after its judgments. Missing rows, missing dimensions, or whole missing forms stay missing and never become losses or ties. Forms may be locked with unresolved entries, which still prevent complete coverage. Two raters work independently and lock their own forms before sharing. The adjudicator then reviews both locked forms and records a separate judgment, reason, evidence, timestamp, a resolution note, and `independent_judgments` mapping the two **opaque rater aliases** to their exact original left/tie/right judgments. All copies share orientation, so no side conversion is needed for this basis. Adjudication must follow both independent locks. No automatic resolution, majority vote, or copying of independent agreement is performed. This default adjudicated series requires explicit human review of every declared pair/dimension to be complete; adjudicating disagreements alone leaves its other entries missing. Preserve independent forms alongside adjudication. Never return any assessment to the planner or worker. ## Summaries and intervals ```sh python3 -B 'Japa two/pairwise-preference/pairwise_preferences.py' summarize \ --key /coordinator/new-private-pair-key/comparison-key.json \ --forms /coordinator/returned/independent-a.json \ /coordinator/returned/independent-b.json /coordinator/returned/adjudication.json \ --replicates 10000 --seed 0 ``` Omit unavailable forms (or `--forms`) to inspect coverage. Summarization revalidates the source-manifest hash, source archives, pair mapping/randomization, registry, form versions, opaque identities, artifact hashes, human judgment enums, lock/assessment times, and adjudication bases. Duplicate records/series, extra fields, duplicate JSON keys, stale bases, or unsealed/corrupt sources are rejected. Inputs are read-only. JSON goes to stdout; errors go to stderr with exit code 2 and no report. If saving stdout, choose a new coordinator-only destination outside all inputs; shell redirection is not managed by the tool. Each independent rater has its own series; adjudication has a separate series. **Raters are never pooled as independent task draws.** Each dimension separately reports: - Target-oriented win/tie/loss counts and the mirrored comparator counts, with the assessed-pair count as the raw-count denominator. Counts describe observed judgments, not task-weighted estimates. - Expected and supplied pair counts, absent task/repetition cells, and missing judgment comparison IDs. A null judgment cannot masquerade as a tie. - Per-task repetition means, and, only with complete declared coverage, equally weighted family preferences and an aggregate preference estimate with a 95% interval. Any absent pair or missing judgment withholds that series/dimension's family/overall estimates and interval; complete task means can still be inspected. Completeness of one rater/dimension does not establish another's completeness. The clearly defined preference statistic is **target win = 1, tie = ½, target loss = 0**, averaged over repetitions within task, equally over tasks within family, then equally over families. Thus ½ represents balanced preference with half credit for ties; it is not a requirement score or a combined usefulness/clarity index. The comparator's statistic, if needed, is its complement, not another independent observation. Intervals reuse the existing primary-contrast bootstrap helper: 10,000 seeded percentile replicates as an unfrozen default (`--replicates` accepts integers ≥2), linear interpolation at `q × (B−1)`, 2.5th/97.5th percentiles. Resample tasks with replacement **within each family**, retaining the original task count. Each sampled task carries both artifacts and all paired repetitions for the selected fixed rater series together; pre-averaging complete task repetitions is algebraically equivalent. No repetition-level, artifact-level, or rater-level resampling inflates the effective task sample. Intervals are conditional on the selected rater series, not estimates of rater-population uncertainty. The same draws are used for both dimensions with a common seed, but no joint interval or multiplicity correction is claimed. Seeds, replicate counts, and method are reported. Separate batches/contrasts require a prespecified multiplicity plan before confirmatory claims. Agreement is reported per dimension as exact categorical agreement divided by the number of jointly nonnull ratings, with that denominator and missing-joint-judgment count explicit; a zero denominator yields null. `independent_raters` compares the two independent forms. `adjudicated_vs_independent` reports descriptive agreement with each independent form separately, **not independent adjudicator reliability**. Its denominator is supplied pairs with both relevant judgments; absent expected pair cells are separately visible in coverage. Raw agreement counts are descriptive, not independent-sample confidence intervals. These tools cannot establish semantic correctness of ratings, representative task sampling, truthful endpoint selection, completeness of undeclared archive collections, successful blinding, or superiority. Rating hash fields bind claimed identities, not proof of what bytes a human actually read. Small development-task intervals are not power guarantees. No actual packets, ratings, or study estimates were generated to create these assets. ## Temporary artificial-fixture tests ```sh cd 'Japa two/pairwise-preference' python3 -B -m unittest -v test_pairwise_preferences ``` Tests use temporary artificial text/metadata and categorical numeric fixtures, never task answers or purported findings. They cover byte preservation, metadata separation, randomization, orientation, dimension separation, task/repetition/family weighting, clustered-bootstrap reproducibility, agreement denominators, missing coverage, adjudication, duplicate/hash/version/seal rejection, read-only inputs, and overwrite refusal.