# B3: reference-free SPO-style optimizer assets Version 1.0. **Hand-authored, untested study assets; configuration defaults are unfrozen.** No task answers, model calls, experiments, or results are supplied. This is a proposed text-packet adaptation of SPO, not a verified reproduction. It makes B3 concrete for development; it does not establish baseline strength or settle the protocol's final configuration. ## Evidence and fidelity Read alongside the existing [related-work evidence](../../06-evidence-and-related-work.md) and [experimental protocol](../../03-experimental-protocol.md). The primary specification is Xiang et al., [*Self-Supervised Prompt Optimization*, Findings of EMNLP 2025](https://aclanthology.org/2025.findings-emnlp.479/), especially [§3, Algorithm 1, §4.1, and Appendix A.1–A.2](https://aclanthology.org/2025.findings-emnlp.479.pdf). SPO proposes one revised prompt from the incumbent prompt and outputs, executes it on a fixed question set, and uses pairwise output judgments to update the incumbent. Algorithm 1 starts with three questions. The paper reports four randomized evaluation rounds and eight calls per iteration. Appendix A.1 supplies optimizer and comparator XML formats. Selection returns the retained prompt; it is not a separate generative selector in Algorithm 1. The paper does not fully specify vote aggregation or malformed-response handling. These are not grounds to invent a paper-defined rule. The authors' linked repository was also inspected read-only on 2026-11-12: - [Evaluator](https://github.com/FoundationAgents/SPO/blob/main/components/evaluator.py): randomly swaps the two output bundles and maps A/B choices back to prompt identities. - [Evaluation utilities](https://github.com/FoundationAgents/SPO/blob/main/utils/evaluation_utils.py): repeats four times, normally accepts when candidate votes exceed incumbent votes, but has a special neutral-dominant success branch. - [State utilities](https://github.com/FoundationAgents/SPO/blob/main/utils/data_utils.py): retrieves the latest successful round. Thus the neutral-success branch can advance the retained round; its comment alone does not establish incumbent preservation. - [Comparison prompt](https://github.com/FoundationAgents/SPO/blob/main/prompts/evaluate_prompt.py) and [generation prompt](https://github.com/FoundationAgents/SPO/blob/main/prompts/optimize_prompt.py): expose reference-answer fields absent from the paper's Appendix A.1 templates. This package deliberately has no such fields. These are observations of mutable `main`, not a pinned reproduction or proof of the historical experiment's implementation. The exact paper-era commit and intended neutral-case semantics remain unresolved. A reproduction claim requires a pinned code/configuration audit and resolution of these discrepancies. These assets follow the paper's reference-free mechanism, with the explicit conservative adapter below. ## Files | File | Function | Output | |---|---|---| | [candidate-generation.txt](candidate-generation.txt) | Propose one changed worker instruction from the incumbent instruction and its artifact | ``, ``, `` | | [candidate-worker.txt](candidate-worker.txt) | Execute a prompt against the same immutable packet in fresh context | One task deliverable | | [comparison.txt](comparison.txt) | Compare anonymously labeled output bundles against public requirements | ``, `A or B` | | [selection.txt](selection.txt) | Serialize a fixed vote rule; no new quality judgment | Selection JSON | The selection template is an **adapter**, not an additional SPO mechanism. Default selection is deterministic application of its rule by the prospective operator/harness, without a model call. If a future implementation sends this template to a model, charge that extra call, validate its result against the rule, and report the variant separately. Do not treat a selector's new opinion as another vote. ## Input contract Use single-pass substitution; all placeholders are JSON values, never paths. JSON-encode strings once, without extra surrounding quotes. Preserve the decoded packet and artifacts exactly. Empty/missing required inputs are configuration failures, not permission to insert an answer key. The existing [baseline conventions](../README.md) apply to public packet encoding and artifact export; this native-shaped optimizer uses incumbent state rather than the continuation arms' history window. | Placeholder | Value | |---|---| | `{{TASK_PACKET_JSON}}` | JSON string containing one entire unchanged `development-tasks/public/` packet, including assignment, requirement IDs, provenance, and all sources | | `{{INCUMBENT_PROMPT_JSON}}` | JSON string containing the retained worker directive, without appended packet/input blocks | | `{{INCUMBENT_ARTIFACT_JSON}}` | Object `{ "artifact_id": string, "text": string }`, exported final deliverable associated with that retained prompt | | `{{CANDIDATE_PROMPT_JSON}}` | JSON string containing only the generated `` content after XML decoding | | `{{PUBLIC_TASKS_JSON}}` | Array of `{ "task_id": string, "task_packet": string }`; the default has exactly one entry containing the same complete public packet | | `{{OUTPUTS_A_JSON}}`, `{{OUTPUTS_B_JSON}}` | Arrays of `{ "task_id": string, "text": string }`, identical task IDs/order on both sides; no prompt, method name, vote, or success label | | `{{INCUMBENT_ID_JSON}}`, `{{CANDIDATE_ID_JSON}}` | Distinct JSON strings identifying the two prompt–artifact pairs; metadata, not answers | | `{{COMPARISON_RECORDS_JSON}}` | Four records in round order: `{ "round": integer, "a_id": string, "b_id": string, "choice": "A" or "B" or null }`; IDs identify the randomized assignment, null denotes missing/invalid choice | No placeholder accepts assessor guides, hidden criteria, assessor scores, supported-answer sheets, reference solutions, worker completion reports, tool traces, private reasoning, repository content, or treatment memory. Never load `development-tasks/assessor-only/` into any request. Public numeric source facts remain available; they are not hidden performance scores. Comparison judgments are B3's **own fallible qualitative evaluations**, not independent assessor feedback. Generated analyses are brief inspectable justifications, not requests to expose private reasoning. ## Proposed usage sequence 1. **Initialize.** Select one completed public development packet without opening its assessor guide for prompt assembly. Set `P0` to the exact directive paragraphs in [common-initial-worker.txt](../common-initial-worker.txt), before `IMMUTABLE_TASK_PACKET_JSON`. Reuse this instruction rather than authoring a task-specific starting hint. Execute `candidate-worker.txt` with `P0` and the packet; retain its exported artifact as `A0`. Its fixed wrapper restates existing access constraints. For a future matched study, normalize this JSON/wrapper presentation across compatible arms before freezing; byte-identical initial requests to the existing B0–B2 templates are not claimed here. 2. **Generate.** Supply the public packet, incumbent directive `P*`, and associated artifact `A*` to `candidate-generation.txt`. Extract only the complete `` as `P'`. Archive the analysis and modification, but do not insert them, earlier critiques, votes, rejected prompts, or an accumulated lessons buffer into the next generator call. Retained state is the incumbent prompt–output pair, not absence of memory in all senses. 3. **Execute.** Execute `P'` using the same candidate-worker wrapper, packet, worker configuration, and fresh context used for `P0`. Do not give the candidate a different starting artifact or the incumbent deliverable to edit. The optimizer has seen that deliverable; the executor generates a fresh answer to the same task. Export only the task deliverable as `A'`. 4. **Compare.** Make four separate requests with `comparison.txt`, each containing the complete public packet and both outputs. Independently randomize A/B assignment with probability 1/2 each round and log the assignment/seed outside the model-facing prompt. Do not reveal prompt identities, which side is incumbent, earlier choices, or selector metadata to the comparator. The default compares the whole one-task bundle per request, not four extra worker samples. 5. **Select.** Map A/B choices back to pair IDs. If all four records are valid, promote only when candidate votes exceed incumbent votes; 2–2 retains the incumbent. Any invalid/missing record retains the incumbent and marks the comparison incomplete. Apply `selection.txt` deterministically by default. This conservative invalid-record rule is our adaptation, not the repository's neutral-success behavior. Do not repair an invalid response through an uncharged call. 6. **Repeat and seal.** The next generator receives only the newly retained `P*`, `A*`, and unchanged packet. At the fixed endpoint, seal the incumbent pair as the B3 output, plus every attempted prompt, artifact, judgment, failure, selection record, and usage entry in an external audit log. Independent assessment begins only after the registered batch is sealed; none of its information returns to adaptation. The retained artifact may come from an earlier iteration. This is native-style online selection using B3's own judgments, not retrospective selection with a hidden assessor. Disclose it alongside continuation arms' last-artifact endpoints. Do not silently replace it with the last rejected candidate or choose a historical winner after assessment. A final regeneration from `P*`, if separately studied, costs another worker call and may yield a different artifact. For development, require exactly the generator's three XML elements, a nonempty changed ``, and comparator `` text exactly `A` or `B`. Decode XML before JSON encoding the next input. A malformed proposal, missing artifact, or oversized required context ends that iteration without promotion; log the failure and consumed resources. Do not silently truncate the packet, substitute an assessor answer, or manually improve a candidate. These are mechanical assembly checks, not a separate human quality gate. Other parsing/retry policies require disclosure before freezing. ## Adaptations, budget, and open configuration The **unfrozen default is task-local optimization with one packet**, one candidate per iteration, and four randomized bundle comparisons. This retains prompt search and pairwise acceptance but changes the paper's three-question setup and cannot demonstrate prompt transfer. The generation template is consequently task-local. A future three-task/family study needs a separately frozen multi-task generator, disjoint optimization/evaluation packets, and transfer accounting; do not put three unrelated packets into this single-task generator or present task-local gains as transfer. Workers receive fresh task contexts, while the optimizer sees the incumbent artifact. Earlier artifacts and instruction histories are archived rather than supplied as continuation context. This is a disclosed native-state adaptation within the permitted public task/answer channels, not literal equality with B1/B2/C's history contents. If strict identical history is required for a comparison panel, design and name that altered B3 variant before freezing; these assets do not establish its equivalence to SPO. Worker/harness code, system boundary, tools, source packet, and configuration stay fixed; only the worker directive changes. Private worker reasoning is excluded, even where another SPO configuration might expose it. Count initialization plus **every** generator, worker rollout, four comparisons, optional model-based selection serialization, failure, and retry. Rejected candidates and unused comparisons still cost resources. With `k` fully completed iterations, one task, no retries, and deterministic selection: worker calls = `1+k`, optimizer calls = `k`, comparator calls = `4k`, total = `1+6k`. A nine-update/ten-worker-call ceiling is therefore **55 calls**, not ten; optional model selection adds nine calls, totaling 64. For the paper's three-question structure the analogous iteration costs `1+3+4=8` calls, excluding initialization. These are accounting identities, not measured costs or a supplied multi-task implementation. Use the protocol's common total adaptation-cost cap as well as the proposed ten-artifact-producing-call ceiling; the latter includes rejected worker candidates. Record input/cached/output/reasoning usage when exposed, billed cost, request count, and latency by role. Re-reading long source packets and both candidate outputs costs tokens. No numerical cost cap, model snapshot, decoding configuration, output-token allowance, retry policy, context limit, or randomization seed is frozen here. Select them on development/validation data with matched tuning allowance, then freeze before test. Do not inherit obsolete model identities or reported dollar costs from the paper as current settings. Reserve enough allowance for a complete iteration under chosen per-call caps before starting it. If a cap or failure interrupts an iteration, charge all consumed work and retain the last completely selected incumbent; mark incompletion. Do not assess a partially compared candidate as if it won. Quality declarations cannot extend the fixed budget. Assessor expense is separate reported study expense, never adaptation input. No runner or models have been invoked. These templates neither authorize API spending nor resolve the historical reproduction questions above.