# Offline human factual-claim support assessment Development tooling for [protocol §7](../03-experimental-protocol.md#7-independent-assessment), with **unfrozen defaults**, not empirical evidence. Humans identify factual claims, declare inventory review complete, and judge source support. The tool checks bytes, identities and spans, prepares blank blinded materials, and summarizes supplied judgments. It does not identify claims, assess truth, infer support from citations, or generate ratings. Never return inventories, ratings, or summaries to adaptation. ## Human enumeration and declared coverage Copy [selection-manifest.template.json](selection-manifest.template.json) into a coordinator workspace. Explicitly select each sealed **designated final** artifact by archive alias, seal/run/condition/task/repetition identities, task/rubric versions, artifact ID, and raw-file SHA-256. One artifact per selected run is permitted. Specify two independent raters and a distinct adjudicator as the reversible default. A prespecified sample is allowed: declared coverage is the selected artifact list, not all registry tasks or all archived runs. Unselected archived run IDs are reported, not silently treated as assessed. Run failures with designated endpoints may be selected; artifact availability does not establish completion or quality. Author one separate [claim inventory](claim-inventory.template.json) per artifact, referencing its exact artifact hash and task/version identities. Choose factual assertions and boundaries manually. The proposed counting unit is a minimally scoped, contiguous factual-claim **occurrence**. Distinct occurrences of a repeated proposition can count separately; there is no automatic semantic deduplication. Freeze segmentation and repetition policy before substantive analysis. Claim IDs must be unique within their artifact (their identity is artifact plus claim ID); identical IDs in different artifacts are distinct. Duplicate IDs or identical span boundaries within one inventory are rejected. Distinct overlapping spans are structurally allowed, without asserting they represent distinct facts. Compound or ambiguous claims require human segmentation decisions. Each span is zero-based **Unicode code-point** `[start,end)` into `artifact_bytes.decode('utf-8')`. End is exclusive. CR and LF each count, as does any BOM; no newline conversion, normalization, trimming, or boilerplate stripping occurs. UTF-8 byte offsets, UTF-16 editor offsets and line numbers are not interchangeable with these positions. Supply the exact selected substring as `text`; bounds must be integers (not booleans), nonempty and in range. Matching text establishes location only, not factuality, atomicity, support, or completeness. An inventory has three operational states: - **Reviewed:** `enumeration_status: "reviewed"`, named human `enumerator_id`, a UTC `completed_at_utc` after sealing, and a scope/limitations `note`. A reviewed `claims: []` explicitly declares that a human found no factual claims. This remains an unverifiable human completeness declaration. - **Unfinished:** `enumeration_status: "unfinished"`, `completed_at_utc: null`, an enumerator and explanatory note. It may contain partial claims or an empty list. Neither implies a reviewed claim-free artifact. - **Unavailable:** selection `inventory: null` with a nonempty `unavailability_reason`. Do not manufacture an empty inventory. Its unknown claim count stays null. For available inventories, selection `inventory` supplies an explicit `path` and SHA-256 of the **raw inventory-file bytes**; paths resolve against the selection manifest's parent. Compute hashes after finishing the file, for example: ```sh python3 -c 'import hashlib,pathlib,sys; print(hashlib.sha256(pathlib.Path(sys.argv[1]).read_bytes()).hexdigest())' /coordinator/inventory.json ``` All templates contain blank identities/observations and are schema sketches, not valid completed study inputs. No actual study inventories or claims have been generated for these assets. Enumeration may itself be biased by knowledge of conditions; this utility does not establish enumerator blinding or independent claim-identification reliability. ## Build blinded packets ```sh python3 -B 'Japa two/claim-support/assess_claims.py' build \ --manifest /coordinator/claim-selection.json \ --rater-output /coordinator/new-claim-rater-packets \ --coordinator-output /coordinator/new-private-claim-key \ --seed 0 ``` Output directories must be new, disjoint, have existing parents, and remain outside all source archives. Existing paths, including broken symlinks, are refused. The tool reuses [sealed-archive validation](../run-sealing/RUN_OUTCOMES.md) and the existing blinded-packet helpers for hashes, destinations and byte-preserving copies. Missing commits, unsealed inputs, corrupt payloads, duplicate run cells/selections, incompatible versions, wrong endpoints, malformed spans, inventory hash/identity mismatches, unknown fields and duplicate JSON keys abort preflight. `--registry PATH` can select a compatible registry (including an archive's relocated registry); otherwise the existing twelve-task development registry is used. Existing archive-contract limitations remain unchanged. Each opaque rater directory contains unchanged final artifact bytes, the exact matching public task/source packets, an index, instructions, a sanitized claim inventory with opaque claim IDs, and an identity-filled **judgment-blank** form. Claim text is copied only from the explicitly enumerated spans. Fixed source-packet and public-inventory hashes are included in rating identities. Source claim IDs, enumerator IDs, enumeration notes, condition/run/repetition mappings, archive metadata, prompts, memory, resource logs, and requirement-scoring assessor guides are excluded from distributed files and filenames. Inventory state is shown for coverage context. Only the assigned opaque directory should be distributed. Opaque IDs use fresh UUIDs. Seeded presentation shuffling (default 0) randomizes artifact order per rater from canonical run-ID order; it does not seed aliases, discover artifacts, or choose claims. Condition and identity mappings, exact source selection and original inventory snapshots stay in the separate coordinator output. `claim-key.json` is written last as the completed build record. Failed writes may leave partial directories but emit no success JSON; inspect them and use new destinations. Directory separation does not enforce confidentiality. Opaque IDs cannot remove intrinsic method disclosures in artifact or claim text; these are preserved, not heuristically redacted. ## Human support ratings Generated forms follow the blank [independent](independent-form.template.json) and [adjudication](adjudication-form.template.json) schemas. Assess each enumerated claim against **only the fixed authorized public source packet**. Proposed operational labels are `supported` (warranted there, including a justified derivation) and `unsupported` (not warranted there, including contradiction). This is packet-grounded support, not universal truth. A valid citation, reference identifier, or plausible assertion is never automatic support. Explain both contradiction and absence-of-support judgments with the relevant scope searched and locations. For each nonnull judgment, supply a reason, precise evidence-location strings, and UTC completion time. Locations are human descriptions, not machine-verified entailment. Ambiguity or inability to judge stays null with an optional reason; never turn it into supported/unsupported by default. If an enumerated item seems nonfactual or poorly split, report it and leave it unresolved; do not silently remove or alter it to change the denominator. A revised inventory requires a new hash-bound build and new forms, preserving the old attempt. Independent raters work separately and lock their forms after judgments using `locked_at_utc`. Times must follow sealing and, for reviewed inventories, inventory completion. A third human then supplies a separate adjudicated decision, rationale, evidence, resolution note and `independent_judgments` mapping both opaque rater aliases to their exact locked labels. Adjudication requires both independent judgments for that claim and must follow both locks. No agreement is automatically copied and no majority vote is generated. A complete adjudicated series requires explicit decisions on every enumerated claim; disagreements-only adjudication leaves other entries missing. Retain all independent ratings separately. ## Summarize after assessment ```sh python3 -B 'Japa two/claim-support/assess_claims.py' summarize \ --key /coordinator/new-private-claim-key/claim-key.json \ --forms /coordinator/returned/independent-a.json \ /coordinator/returned/independent-b.json /coordinator/returned/adjudication.json ``` Missing forms can be omitted, including the whole `--forms` option. Whole missing forms, omitted rows, and null judgments are reported as missing coverage. All supplied forms are validated; duplicate records/series, identity/hash/version mismatches, altered spans, extra fields, invalid labels, invalid times, and stale adjudication bases are rejected. Source archives and coordinator outputs are revalidated and read-only. Original selection/inventory files need not remain available: their exact copies and original path base are retained privately; archives and a matching registry must remain accessible. Ratings are bound to their declared identities, not proof of what a human actually viewed. Reports are coordinator-only development outputs. JSON goes to stdout; errors go to stderr with exit code 2 and no report. Save stdout only to a new destination outside inputs; shell redirection is not controlled by the tool. No production files or live planner memory are written. For each independent rater and, separately, adjudication, `overall` and `by_condition` give: - Unsupported-claim counts and **all nonnull support judgments** as the assessed-claim denominator. Supported and unsupported both count as assessed; null never counts as either. Counts are claim-weighted descriptive totals, not the primary equal-task/equal-family quality score; raters are never pooled. - Provided claim counts, missing judgments and judgment coverage (assessed/provided). Unavailable inventories have unknown claim counts and are excluded from that known denominator, with their artifact counts separately explicit. Unfinished inventories may contribute provisional counts but always prevent complete coverage. - Reviewed, reviewed-no-claim, unfinished and unavailable inventory counts, plus per-artifact counts/coverage. `declared_coverage_complete` requires every selected inventory to be reviewed and every listed claim to have a judgment. `unsupported_claim_rate` is withheld (null) if this coverage is missing, and is also null when the assessed denominator is zero. A fully reviewed claim-free set has a zero denominator and null rate, not a demonstrated zero factual-error rate. `independent_rater_agreement` is separate from adjudicated results. It reports exact supported/unsupported agreement over jointly nonnull ratings, its explicit denominator, provided claims missing joint judgments, and unfinished/unavailable inventory counts, overall and by condition. A zero joint denominator yields null, never perfect agreement. This agreement is conditional on the common human claim inventory; it measures neither enumeration agreement nor independent adjudicator reliability. No independent-sample intervals or significance claims are produced. “Complete” always refers to declared assessment coverage, not verified exhaustive enumeration. Omitted facts, incorrect segmentation, unsupported but unenumerated statements, source defects, semantic errors outside factual assertions, and biased or mistaken human judgments remain possible. This metric cannot certify factual accuracy or exhaustive factual-error detection, and does not replace requirement scoring or establish method superiority. ## Temporary artificial-fixture tests ```sh cd 'Japa two/claim-support' python3 -B -m unittest -v test_assess_claims ``` Tests create and remove temporary artificial text/span, metadata and categorical fixtures. They exercise no actual study claims, ratings, packets, task answers, model calls, or findings.