# Offline retrospective quality at common spend caps Untested development tooling for [protocol §§6–7](../03-experimental-protocol.md#6-budget-accounting), not empirical evidence. **All defaults remain unfrozen.** This distinct metric asks what quality was observed in the latest available deliverable at a common adaptation-spend cap. It is not first attainment of a quality threshold, a best-answer search, or a prospective stopping experiment. No judgments are generated and no scores enter adaptation. ## Coordinator manifest and usage Copy [manifest.template.json](manifest.template.json) into a coordinator workspace. It is a blank schema sketch, not an actual study input. Supply: - A raw-file-hash-bound [progress manifest](../progress-per-turn/README.md), with its explicit sealed delivered snapshots, human assessment bindings, condition/repetition panel and turn horizon. Its existing validation and normalized requirement scores are reused unchanged. - Resource-ledger descriptors and one accounting entry per supplied progress run, using the completed [request-boundary contract](../cost-to-quality/README.md#inputs-and-request-boundaries). These fields can be reused from a cost-to-threshold manifest; replace its threshold field with `common_caps`, and use this tool's version/analysis fields. - `common_caps`: one panel per explicit uppercase three-letter currency code, each with a nonempty, strictly increasing list of finite nonnegative **numeric** `amounts`. Caps are caller-declared common values, not derived from observed scores/spending. Duplicate currencies/caps, booleans, nulls, strings and nonfinite/negative values are rejected. Equal numeric amounts with different formatting still count as duplicates. All declared conditions and expected task/repetition cells participate in **each** currency panel. There is no automatic filtering by observed billing currency or method success. If currency incompatibility prevents a matched comparison, report it; use an explicitly scoped, prespecified progress panel for a different analysis rather than silently dropping runs. No exchange rates are inferred. ```sh python3 -B 'Japa two/quality-at-equal-cost/analyze_equal_cost.py' \ --manifest /coordinator/equal-cost-selection.json \ --output /coordinator/new-equal-cost-report.json ``` `--registry PATH` optionally selects the compatible registry used by progress analysis; otherwise the existing twelve-task development registry is used. Relative source paths resolve against this manifest's parent. Outputs must be new, have existing parents, and stay outside source archives. Existing paths, including broken symlinks, are refused. Publication is staged, byte-checked and atomic without overwrite. Stdout contains only success path/hash and development labels; errors go to stderr with exit code 2 and no success report. Keep outputs coordinator-only and outside rater packets; labels do not configure permissions. The existing progress, resource and request-boundary validators are reused through a private temporary schema adapter; no threshold event is evaluated. They check committed seals, hashes, task/rubric versions, chronological snapshots, primary ratings/adjudication, run/request identities, retry ordering, explicit complete/incomplete ledger declarations, and snapshot boundaries. Completed utilities and inputs are unchanged. No additional history is discovered, and no assessment packet is produced. ## Selection default For a run and cap, select the **latest chronologically delivered snapshot** with a known cumulative actual billed adaptation cost that does not exceed the cap. Selection uses chronology and costs only, before consulting rating values. Equal-spend ties select the later turn. A later affordable but unassessed snapshot is selected with missing quality; an earlier rated or higher-scoring answer cannot replace it. Each selected record reports its artifact/turn, observed normalized requirement satisfaction, **actual spend**, and **unused cap** separately. Spend and slack remain exact decimal strings in the panel currency; quality alone is averaged. Slack means `cap − spend at this snapshot`, not a refund or money actually saved by the whole run. Full-run adaptation resources and independent final-assessment expenses remain separate. Every cumulative prefix includes all declared adaptation requests through that boundary, across worker/planner/optimizer/comparator roles and failed, rejected, discarded, cancelled and retry attempts. Independent assessment charges are excluded from prefixes. Known billing is required; missing costs are never zero and no prices are inferred from token usage. The inherited conservative default withholds prefix totals when the adaptation log is explicitly incomplete. Known whole-prefix costs must be in the **single matching currency**. A foreign/mixed vector is not eligible under a one-currency cap by ignoring its other components. A genuinely known empty request prefix can fit any nonnegative cap with spend zero in the caller's panel unit; its original empty billed vector is preserved, without inventing a billed currency. Unknown currency or absent billing is not an empty prefix. ## Indeterminate eligibility and budget bounds An unknown later boundary, unavailable later snapshot, incomplete accounting, or incomparable currency vector can prevent establishing which snapshot is latest eligible. In that case `selection_state` is `indeterminate`, selected quality stays null, and the latest known eligible candidate is diagnostic only—it is not substituted into the comparison. Unknown earlier eligibility does not block a later fully known eligible snapshot, since an earlier snapshot cannot outrank it chronologically. The ledger contract permits only nonnegative charges, and validated cumulative request boundaries do not decrease. Consequently a known subtotal in the panel currency already **above** the cap proves a snapshot ineligible even if additional costs remain unknown. Once such an over-cap prefix is established, later snapshots cannot become affordable under that cap; missing later data cannot invalidate that exclusion. This is a cost lower-bound argument, **not monotonic quality**, currency conversion, interpolation, or treating unknown costs as zero. Diagnostics retain unknown totals and the exclusion reason. Unknown snapshots between an eligible candidate and a later over-cap snapshot can still block selection: an over-cap later point does not prove earlier points were over cap. If every possible snapshot is determinately over cap, report `no_snapshot_within_cap`, not zero quality. If selection is known but its human rating is missing/ineligible, report `missing_selected_assessment`. `observed_quality` requires a selected snapshot and an eligible primary score; an actual score of zero is valid observed quality. ## Cohorts, series and limits Reports separate every independent rater from adjudication. Selection is identical across series for the same run/cap; only observed quality and rating coverage can differ. Raw primary requirements, complete-status rules, hashes and adjudication links are preserved; unresolved criteria never reduce the denominator or get filled from another series. For each currency/cap/condition, report the full expected-run denominator, observed runs, determined selections, selected snapshots, comparison-state counts, and assessed/missing quality cells. Preserve equal repetition weighting within tasks, equal task weighting within families, and equal family weighting using the progress aggregator. A task mean requires every expected repetition, a family mean every task, and the overall mean every family. There is no surviving-case renormalization or observed-currency filtering. Incomplete condition means remain null. Complete condition-specific means can remain visible with their own coverage, but `complete_condition_quality_means` for the common-cap comparison is withheld unless **all declared condition cohorts** are complete. No-snapshot cases and indeterminate runs remain in coverage; neither is silently assigned zero or dropped. Zero denominators yield null. Common caps do not imply equal actual spend, which is why per-run actual spend and slack are shown separately; monetary amounts are not pooled across currencies. Coverage is **specific to this cap comparison**: identifying the latest eligible snapshot and assessing its quality. Missing nonselected ratings or eligibility already ruled out by a sound budget bound need not block that comparison. `all_snapshot_accounting_known` separately discloses broader accounting gaps; this tool does not claim that a whole progress curve is complete. The archived prescribed final endpoint/hash stays unchanged, even when the selected budget-view snapshot is earlier. This retrospective view does **not** reproduce a run prospectively stopped at the cap. A live stopping policy could change request scheduling, optimization, cancellations and outputs; whole runs may actually spend more after the selected snapshot. These are descriptive snapshot observations, not causal improvement, persistent quality, confidence intervals or superiority. Chronology, completeness/semantic provenance of inventories, billing truth and human judgments remain protocol responsibilities. No actual study inputs, assessments or findings were generated for these assets, and outputs must never feed back into a planner or worker. ## Temporary artificial-fixture tests ```sh cd 'Japa two/quality-at-equal-cost' python3 -B -m unittest -v test_analyze_equal_cost ``` Fixtures use temporary artificial text, metadata, numeric ratings and billing cases, not real task answers, measured costs, study assessments or empirical results.