# Manual development assessment kit Version 1.0. **Untested, blank development assets.** No artifacts have been rated and no scores or results are supplied. Two independent human raters per artifact, followed by a third human adjudicator for disagreements, are an **explicitly unfrozen default**, not a registered staffing decision. This kit reuses the [assessor conventions](../README.md), the twelve unchanged version 1.0 guides, and [protocol §7](../../../03-experimental-protocol.md). It introduces no task requirements or scoring criteria. ## Files and preparation - [task-registry.json](task-registry.json) lists the twelve tasks, four per family, their existing packet/guide versions and paths, content hashes, and shared C1–C12 → R1–R12 mapping. Paths resolve relative to this kit folder. The common assessor-conventions hash also pins the rubric context; its source file has no separately declared version. - [rating-form.json](rating-form.json) has one blank record for each task. Make a separate copy for each independent rater; never share a partially completed copy between them. - [adjudication-form.json](adjudication-form.json) has corresponding blank records for later resolution. It is not part of an independent rater's initial packet. These JSON files are templates, not a runner. Known IDs, mappings, versions, family labels, and the fixed denominator 12 are populated; every judgment, observation, artifact/rater assignment, timestamp, and score remains null or an empty array. Null means **not recorded**, never zero. Copy records for additional blinded artifacts/repetitions; assign unique opaque form and record IDs. A form may mix methods; never name files or bundle artifacts by method for raters. Randomize presentation order independently of method and run chronology. No supplied form is an executed assessment. ## Rater instruction sheet 1. **Wait for sealing.** Assess only after all runs in the declared assessment batch are sealed, including final artifacts, prompts, configuration, and usage records. The coordinator supplies an opaque `seal_record_id` confirming that state, not the adaptation logs. Preparation of blank forms does not authorize a run. Never return ratings, guide contents, or adjudication feedback to a planner or worker during adaptation. 2. **Use a blinded packet.** Receive only the assigned artifact under an opaque `blinded_artifact_id`, its immutable public task/source packet, its existing assessor guide, the common conventions, and your own blank rating record. The coordinator keeps the method/run/repetition mapping, prompt/history/memory logs, token/cost data, arm labels, and execution ordering in a separate restricted key outside every rater packet. An artifact hash identifies the exact presented text; the same text/version must reach both raters and the adjudicator. Optional stable line labels should be a display overlay, not an edit to the deliverable or its word count. Do not inspect the mapping key or guess a method from IDs. 3. **Rate independently.** Do not see another rater's form or discuss the artifact before both forms are completed and locked. Enter your pseudonymous `rater_id`, assigned artifact/record IDs, task/rubric versions, and each criterion judgment. Use only the public packet and existing guide; no new source retrieval or model judge. The task ID/family and rubric are legitimate assessment information, not method identities. 4. **Apply the existing binary convention.** For each Cn, assess only public Rn. Enter integer `1` only when all elements of that requirement are satisfied, otherwise integer `0`. No partial credit, N/A, hidden subcriteria, or newly weighted rules. Read the whole artifact: contradictory content can invalidate a requirement even if another passage is correct. One error may affect more than one criterion only when it independently violates each mapped public requirement. 5. **Accept supported alternatives.** Accept equivalent wording, arithmetic, source-location notation, and any feasible option/order expressly permitted by the public task. A guide's feasibility witness or example recommendation is not a required answer. Do not penalize missing optional calculations, stylistic preferences, outside best practices, or failure to discover facts not in the packet. For length, count whitespace-separated words throughout the deliverable, excluding punctuation-only table separators, as the public tasks specify. 6. **Record evidence and limitations.** For every completed criterion, provide `artifact_evidence_locations`, `public_basis_locations`, and a short `reason`. Use stable section/paragraph/table-row/line references and public Rn/Sn locations; short quotations may help. For an omission, identify the relevant expected section or the whole-artifact search scope and the missing public element; do not fabricate a location for absent text. If a real ambiguity prevents a justified binary decision, leave satisfaction null, explain `ambiguity_note`, and flag the record for resolution rather than invent a requirement. An ordinary visible omission is 0, not an ambiguity merely because the worker did not explain it. 7. **Lock before adjudication.** Set completion time/status only after checking identities, versions, all twelve criteria, and score arithmetic. Keep the original ratings immutable after lock; later corrections require an explicitly linked replacement record and an audit trail, never overwriting a rater's initial judgment. Intrinsic method disclosure in authored text or an incomplete/corrupt packet is an assessment issue to report to the coordinator; do not silently delete scored content to improve blinding or word count. Keep genuinely missing artifacts distinct from artifacts that visibly omit requirements. ## Form contract and adjudication Each rating record is keyed by its opaque record ID, blinded artifact ID/hash, task ID/version, rubric version, and family. `satisfaction` is null while blank, then exactly integer 0 or 1. Evidence fields are arrays of location strings, initially `[]`; `reason` and `ambiguity_note` are nullable strings. `assessment_status` is null initially, then `complete` or `needs_resolution`. `assessment_issue` is a nullable description of a packet, ambiguity, or blinding issue, not an added criterion. Completed numeric scores require twelve recorded binary judgments. Set `completed_at_utc` and batch `locked_at_utc` to ISO 8601 UTC strings when applicable; all IDs are strings. Version strings must match the registry. Verify the packet and guide hashes before distribution; do not silently use revised guides under an old version. After the two independent forms are locked, the coordinator fills the adjudication form's source-form/rater IDs and exact `source_rating_record_ids`. For each criterion, copy the original two binary values into `independent_satisfaction`, in the source-rater order; retain the source forms as authority rather than replacing them with these copies. Set `disagreement` to true for different binary values, false for equal binary values, or null if either is missing. Identity/version/hash mismatches block adjudication until the coordinator resolves the packet error; do not adjudicate ratings of different texts as a disagreement. For equal valid judgments, carry the value into `adjudicated_satisfaction` and set `resolution` to `agreement_carried_forward`. For disagreements, the third adjudicator sees the same blinded artifact, public packet, guides, and both locked ratings, then records an independent public-requirement-grounded decision, evidence locations, and `resolution_rationale`; set `resolution` to `adjudicator_decision`. Do not average 0 and 1 into half-credit, vote by seniority, or choose the higher-scoring rater. A missing/ambiguous rating is not a binary disagreement: the third adjudicator may resolve its documented issue from the same public material, using `missing_or_ambiguous_rating_resolved`, without pretending two valid ratings existed. If the public materials cannot settle it without inventing a criterion, keep the final value null and use `unresolved` with the issue documented. Every adjudicated row retains the original independent values and its resolution route. The third adjudicator's pseudonymous ID and adjudication completion time are recorded separately from both raters. Agreements need no newly invented explanation: `resolution_rationale` can identify the carried agreement and source records. The adjudication record uses `adjudication_status` = `complete` only when all twelve final values are binary; otherwise `needs_resolution`. Original independent nulls remain null even if adjudication later supplies a decision. No guide or task version is changed to resolve this batch; ambiguities can motivate separately versioned future assets. ## Scores and equal weighting For artifact `a`, task `i`, and independent rater `r`, let `b[i,a,r,c]` be that rater's binary judgment on criterion `c`. Populate the independent `satisfied_criteria` with the sum of the twelve judgments and `requirement_satisfaction_score` with that sum divided by **12**. Use the same calculation separately on `adjudicated_satisfaction` for adjudicated scores. Preserve exact fractions or adequate precision until reporting; scores lie in [0,1]. The denominator counts the existing twelve mapped requirements, not guide examples or new atomized subchecks. Keep independent-rater score series and adjudicated score series separate throughout. Do not replace an independent score with the adjudicated score, count raters as extra tasks, or describe a mean of the two ratings as adjudication. Before adjudication, descriptive raw agreement can be calculated as matching binary criterion pairs divided by the number of pairs with both ratings, reporting that denominator and missing pairs. Preserve disagreement counts and resolution routes even after a final score exists; agreement is not correctness and is not a task-quality score. For each series, first average the prespecified run repetitions **within a task and condition**, giving repetitions equal weight. Then average task means within each family, giving its four tasks equal weight. Finally average the three family means equally: `task_mean[i] = mean_over_prespecified_repetitions(score[i, repetition])` `family_mean[f] = (1 / 4) × sum(task_mean[i] for the four tasks in family f)` `overall = (family_mean[SYN] + family_mean[CON] + family_mean[PLAN]) / 3` With one complete artifact per task, each task therefore contributes 1/12 and each criterion 1/144 overall. These are weighting identities, not observed scores. More repetitions, more raters, or more artifacts from one task do not increase that task's weight. Do not pool criterion rows from unevenly represented tasks or substitute best historical artifacts. The endpoint and repetition set must be fixed outside assessment before scoring. The coordinator joins conditions/repetitions from the restricted key only after independent ratings and adjudication are locked; rater forms contain no condition fields. If an independent criterion or an adjudicated criterion is null, its corresponding score remains null; never turn blanks into failures or reduce the denominator to the rated criteria. A family/overall summary requires the full prespecified task/repetition coverage, or a previously declared missing-run policy with separate disclosure. This kit supplies no imputation rule and does not permit dropping difficult outputs. Report coverage, unavailable artifacts, unresolved ambiguities, and blinding incidents. Keep all forms and keys outside model-facing adaptation contexts. These are development tools, not held-out assessment materials, registered settings, compliance evidence, or empirical results. No task answers were generated, no model or experiment was run, and no production code, API credits, or submissions were involved.