# Experimental protocol Status: proposed design. Dataset sizes and resource caps below are starting specifications to finalize on development data before registration. No study has been executed. ## 1. Research questions and primary comparison Primary question: at equal total model budget, does artifact-guided planning improve the final deliverable over repeated generic continuation with the same worker? The minimum meaningful scientific comparison also includes ordinary self-refinement and a strong reference-free optimizer. A win against a deliberately weak continuation baseline alone is insufficient. The primary endpoint is the artifact at the prespecified final budget. Never choose the best-looking historical snapshot using a hidden evaluator. Score intermediate snapshots only after the run is sealed, for a retrospective progress curve. ## 2. Task suite and splits Use three primary families, all producing text and requiring no code edits: | Family | Example assignment | Independent checks | |---|---|---| | Source-grounded synthesis | Produce a decision brief from a fixed packet of research and stakeholder notes | Claim support, source attribution, coverage, contradictions, and stated uncertainty | | Constraint-heavy documents | Draft an operational specification with roles, exceptions, handoffs, and explicit scope boundaries | Atomic requirements, internal consistency, missing dependencies, and invented restrictions | | Structured planning and analysis | Produce a resource plan from a supplied calendar, inventory, numeric facts, and preferences | Arithmetic, feasibility, dependency ordering, consistency between prose and tables | Make tasks substantive enough to offer multiple useful increments, but small enough to complete within the chosen cap. Use fixed source packets, not live web research, in the primary suite. This removes retrieval changes and transient facts from the first study. Starting split: 12 development tasks for the pilot, 12 separate validation tasks for configuration selection, and **60 test tasks**, 20 in each family. Keep topics, source packets, and scenario templates disjoint across splits. Near duplicates with changed names do not constitute topic separation. Record source provenance and run a manual overlap audit. Use fresh tasks so historical Cruise artifacts do not become an answer key. Launchboard belongs in the motivation/case-study section, not the primary test set. Public benchmarks can serve as secondary checks, with their contamination limits disclosed. All requirements being assessed must be either present in the public task or implied by a prespecified public task contract. Hidden test cases check those requirements; they must not secretly introduce new ones. ## 3. Two information regimes ### Regime A: matched artifact access Every adaptive method sees the same task packet and artifact interface. No gold answers, numeric benchmark scores, private worker reasoning, execution traces, source code, or worker completion claims enter the planner's view. | Input | Strict planner | Worker | Independent assessor | |---|---|---|---| | Public goal and requirements | Yes, immutable | Yes | Yes | | Authorized source packet | Yes, fixed packet | Yes | Yes | | Current artifact | Yes | Yes | Yes | | Prior artifacts | Fixed bounded policy | Same policy across arms | All sealed snapshots | | Previous submitted instructions | Yes, bounded | Same policy across arms | Yes | | Planner memory | Own bounded state | Delivered memory only if arm specifies it | Yes, for audit | | Worker self-report, progress prose | No | Naturally generated if worker produces it; not relayed | Available separately for audit | | Tool traces and private reasoning | No | None required in primary text track | Operational logs only if needed for auditing | | Ground truth or hidden rubrics | No | No | Yes | | Numeric or pairwise acceptance judgments | No for our method | No for our method | Yes, after run | | Agent/harness source and settings | No modification | No modification | Version/configuration audit | SPO or Feedback Descent may generate their own comparison judgments as part of their published methods. That extra selection mechanism is disclosed, token-counted, and is precisely a methodological difference being tested. They must not receive independent hidden-assessor judgments. ### Regime B: native supervision Run GEPA with its intended development tasks, metric, feedback, traces, and candidate selection. Keep evaluation tasks and labels hidden. This measures the quality/cost tradeoff of purchasing or engineering supervision, rather than pretending all methods had identical information. Do not remove GEPA's score gate and still call the result GEPA. Label such a construction “GEPA-inspired reflection with scores removed,” and report it only as an ablation. ## 4. Baselines Prespecify and tune baseline configurations on validation tasks with the same tuning allowance as our method. | ID | Baseline | Purpose | |---|---|---| | B0 | One well-written task prompt; worker receives a long generation budget | Tests whether the same worker can solve the task without an outer loop | | B1 | Generic continuation: review the current deliverable and improve it toward the original goal | Controls for repeated attempts and ongoing worker context | | B2 | Self-Refine-style critique and revision under the same artifact boundary | Tests whether our principles and next-action planning add value beyond standard self-feedback | | B3 | SPO, implemented from its published method; same task/answer interface | Strong reference-free prompt-optimization comparison | | B4 | Feedback Descent, with published comparison/rationale/acceptance logic and the same artifact interface | Tests the value of explicit pairwise selection versus a single continuing trajectory | | C | Proposed Cruise artifact-guided planner | Treatment | The main matched-access multi-turn table uses **B1, B2, B3, B4, and C**. B0 is a separate compute-allocation control because its call structure differs. Include ACE and ProRefine in an extended comparison if full reading confirms a faithful fit; prioritize them over a large collection of weak baselines. If either is a closer match than B3 or B4, revise the registered set before test execution. Use identical worker and planner model families across arms where the methods permit it. Keep the worker's built-in system instructions, context retention, tools, temperature, and task workspace behavior fixed. A separate-planner advantage must be distinguished from a stronger model advantage. ### Direct GEPA comparison: specify what is being optimized GEPA usually learns task-family instructions; our main loop chooses task-specific next work. Make the bridge explicit with two comparison panels: 1. **Frozen family policy:** GEPA optimizes worker instructions on development tasks, then freezes them for test. During test, the worker receives the same fixed continuation procedure and budget as the other arms. Report both training cost and test cost, and amortized cost at stated deployment volumes. Our fixed principles and meta-prompt receive the same disclosure of development/tuning cost. 2. **Test-time GEPA extension:** when comparing at inference time, use an observable development feedback function and GEPA's actual selection mechanism. Deny hidden test-assessor access to all methods. GEPA's feedback can be a public checker or a learned judge, but describe exactly which it is and count its cost. Do not compare against an oracle that reads hidden answers as if it had equal information. If the benchmark cannot support a faithful common setup, publish separate panels instead of a single misleading ranking. Be explicit if GEPA is out of scope for the initial pilot; a title claiming superiority to GEPA requires the completed direct study. ## 5. Run procedure 1. Freeze the task packet, public requirements, worker configuration, planner configuration, observation policy, and memory policy. 2. Create a fresh isolated task state. Reset all cross-task memory. 3. Submit the same initial task instruction to every main multi-turn arm. This is completed worker turn 1. 4. Capture the first artifact using a deterministic artifact exporter. Do not include final-response boilerplate, worker diagnostics, or self-assessed scores. 5. Generate the arm's follow-up using only its permitted inputs. For our method, record an evidence pointer, the gap, the proposed action, and the next prompt. 6. Deliver the prompt to the unchanged worker and capture the next artifact. 7. Repeat until the fixed turn limit or total resource cap is reached. Record all attempts, retries, and protocol violations. 8. Seal all artifacts, prompts, memory, usage, timestamps, and configuration hashes. 9. Only after all runs in the registered batch are sealed, apply the independent assessment. Starting iteration schedule: **10 worker turns**, with at most nine between-turn planning steps for C. The last output is measured; no unused tenth planning call is needed. Other methods may generate more proposals internally, but every proposal, comparison, failed candidate, and discarded rollout counts toward their budget. For primary efficacy, use the turn limit plus a common total resource cap. For economic comparisons, allow methods to spend the same total cost cap with their native call structure; plot quality at equal spend. Do not silently equate ten worker turns with equal compute. ## 6. Budget accounting Track worker, planner, critic, comparator, scorer, artifact-rendering, and tool costs separately. Independent final-scoring expense is excluded from adaptation cost but still reported as study cost. Log input tokens, cached tokens, output tokens, reasoning tokens when exposed, image input units, tool runtime, request count, wall time, and actual billed cost. If model usage is unavailable, state that; do not infer a dollar result from a subscription fee. Choose the primary token/cost cap from the pilot so a typical ten-turn C run can finish; freeze it before test. Publish low/medium/high caps for the cost curve. For a mixed-model system, match dollars as well as reporting raw token totals, since a planner token and a worker token may have different prices. Record pricing date, provider, and model snapshot; no current price assumptions are made in this plan. Illustrative scale, excluding retries: - C alone: 60 tasks × 3 repetitions = 180 runs; at ten turns, 1,800 worker calls and 1,620 planner calls. - Five main arms: 900 whole runs and up to 9,000 completed worker turns, before baseline-specific candidate overhead and budget truncation. - Four-arm development pilot: 12 tasks × 3 repetitions × 4 arms = 144 runs, up to 1,440 worker turns. Start with B1, B2, C, and the closest reference-free optimizer. The full plan is substantially larger than a demonstration. Start with the pilot and use measured costs to size the study. This dossier authorizes no API expenditure. ## 7. Independent assessment ### Primary metric Use a normalized proportion of prespecified atomic task requirements satisfied, on `[0,1]`, assessed from the final artifact and the fixed source packet. Give each task equal weight within a family and each family equal weight in the aggregate. Have task authors define requirements before seeing outputs. Use deterministic checks where possible, and blinded human judgments for semantic criteria. An example rubric for a source-grounded brief might contain 12 checks: four coverage items, four support/attribution items, two consistency items, and two explicit output constraints. The numbers and weights must be frozen per task before test outputs exist. ### Secondary metrics - Blinded pairwise preference for usefulness and clarity. - Unsupported factual claims per assessed claim, with both numerator and denominator reported. - Critical requirement-violation rate, including invented restrictions. - Regression rate: previously satisfied public requirements broken in the final artifact. - Task completion and incomplete-run rates. - Memory drift: unsupported normative entries among all memory entries, plus the count of affected runs. - Repeated-work rate and progress per turn, coded after execution. - Quality at equal cost; cost required to reach a prespecified quality threshold. Do not combine these into a newly invented post-hoc headline score. Report important dimensions separately so better prose cannot hide worse correctness. ### Blinding and reliability Remove method names, planner text, token counts, and run ordering from artifacts shown to quality raters. Use two independent raters on a prespecified sample or the full primary set, and a third for disagreements. Record agreement and adjudication. If an LLM judge is used, freeze its prompt, use a family different from the planner when feasible, randomize left/right ordering, and calibrate against human ratings. Treat automated judging as fallible. Assess final artifacts directly rather than asking a model whether its own plan succeeded. Do not reward the planner for writing words that mimic the scoring rubric. ## 8. Statistics Use at least three independent whole-run repetitions per method/task. These capture run stochasticity, not 180 independent task draws. If a provider lacks seed controls, record that and randomize execution order within time blocks. Prespecify C versus B1 as the primary contrast. Let `d_i` be C's mean score across repetitions minus B1's mean score for test task `i`. Estimate the mean difference with a 95% confidence interval from a task-clustered, family-stratified bootstrap. Retain all repetitions and comparator pairings inside each resampled task. For secondary contrasts against B2–B4, apply a prespecified multiplicity correction, such as Holm, and show effect sizes and intervals. Use pilot variance to choose final sample size for a smallest practically important effect. The proposed 60 tasks is a starting design, not a power guarantee. More repetitions of the same few tasks do not substitute for diverse independent tasks. For preference, treat multiple ratings of one artifact pair as dependent. Aggregate or model by task. Report win/tie/loss and task-clustered intervals. Avoid presenting a 51% estimated win rate as decisive without uncertainty. If claiming noninferiority to a scored optimizer, define a justified margin before test and require its appropriate confidence bound to clear that margin. Failing to find a significant difference does not demonstrate equivalence. A cost advantage alone does not imply quality superiority. ## 9. Causal ablations and stress tests Run these first on development tasks, then register the most informative subset for a held-out analysis: | Change | What it isolates | Necessary control | |---|---|---| | Remove fixed principles | Contribution of principled decision rules | Replace removed token space with neutral structure or report the context-size difference | | No memory | Benefit of persistent state | Equal artifact and history access | | Unstructured memory vs provenance-linked memory | Whether provenance reduces requirement drift | Same memory token cap and update frequency | | No current artifact | Whether results depend on seeing delivered work | Same goal, prior prompts, and planning budget | | Matched stale artifact | Sensitivity to current state | Same task and artifact format; no impossible cross-topic swap | | Controlled artifact defect | Whether the next prompt targets the changed evidence | Change one observable gap, preserve other content, randomize versions | | Worker self-report only | Susceptibility to optimistic reporting | Include a prespecified misleading-report condition, separate from ordinary runs | | Same extra budget spent by worker | Whether gains are just more compute | Equal dollars/tokens rather than equal number of prompts | | Add external scoring | Value and cost of a score gate | Same candidate generator and evidence boundary | For controlled artifact interventions, measure how often the follow-up addresses the edited defect and whether that repair changes independent quality. Merely observing prompt differences is insufficient causal evidence for better outcomes. Add cases where a visible artifact is polished but numerically incorrect, where requirements conflict, where the worker falsely reports completion, and where important defects are not observable through the interface. Include 20- and 30-turn drift studies as secondary runs; keep the primary ten-turn endpoint unchanged. ## 10. Failures, retries, and protocol violations Distinguish service outages from task failures. A transport retry can repeat an identical request under a fixed retry policy; it cannot become a free extra optimization attempt. Count consumed resources and preserve failed responses. Keep incomplete and over-budget runs in the primary analysis under a prespecified rule, such as scoring their last delivered artifact and separately marking noncompletion. Do not drop them for low quality. If no artifact exists, apply the preregistered minimum score. If restricted information leaks into the planner, mark the run noncompliant and report its frequency. Analyze strict compliant runs secondarily, but do not quietly replace leaked runs until a favorable set is obtained. A systematic filter failure invalidates the strict-information claim and requires a new, clearly labeled experiment after repair. ## 11. Optional software track Use newly authored local application tasks. Freeze the worker/harness implementation, dependencies, permissions, and initial state. Define an artifact capture script with fixed viewport, timestamps, fixture data, and scripted user actions. Capture visible responses without assertions or pass/fail labels. Keep source maps, DOM dumps that expose hidden implementation, developer consoles, and test reports out of the planner packet. Independent evaluation can inspect source and run interaction checks after sealing. Report functionality, accessibility, requirement preservation, and visible quality separately. A screenshot cannot prove persistence, correct arithmetic, or working interactions. This track permits application-code changes by the worker. It does not support a literal “no code changes anywhere” claim. Keep its results separate from the primary text study. ## 12. Reproduction bundle Release task/source packets where rights permit, split definitions, frozen prompts, exact model IDs/settings, package versions, adapter and harness commits, configuration hashes, all submitted prompts, generated memory, allowed artifact packets, final deliverables, resource records, scoring code, rubric versions, rater instructions, and analysis scripts. Keep private or restricted logs separate; redact with explicit markers and retain an audit manifest. Log every human intervention. Report human-assisted runs separately from autonomous runs. Preserve unsuccessful cases and publish enough source/configuration to recreate the claimed information boundary. The final paper must distinguish a replay fixture from a live desktop integration. A working research wrapper is sufficient to study the algorithm, but cannot establish the operational reliability of Cruise's watcher or queue.