# Research plan: principle-guided prompting from artifact evidence ## Recommended thesis **A fixed agent can potentially make better progress on long tasks when a planner repeatedly translates visible artifact gaps into specific follow-up prompts, while preserving explicit user requirements and receiving no external scores or execution traces.** Working method name: **Cruise Artifact-Guided Prompting**. The folder name “Japa two” is a project label, not a proposed expansion of GEPA or a claim of a second version of that method. The publishable contribution would be a method plus evidence in a clearly defined information regime. “We keep prompting until it gets better” is insufficient: it must beat ordinary continuation and closely related self-refinement under controlled budgets. ## Translate the idea into academic terms | Everyday description | Precise research wording | |---|---| | Look at what the agent actually made | Observe the delivered artifact under an explicit observation protocol | | Give a better next prompt | Generate a task-conditioned control instruction for the next worker turn | | Use principles | Apply a fixed, public set of decision rules whose causal effect is tested by ablation | | Do not improve the code | Freeze worker/harness implementations; in the primary text track, generate no code at all | | Do not use evals | Do not query an external metric, grader, gold reference, or pairwise acceptance oracle during adaptation | | Remember useful lessons | Maintain bounded, provenance-linked textual state with fixed update rules | | Show that it works | Evaluate sealed outputs independently after all runs are complete | An inspecting planner is still making qualitative judgments. Call this **external-evaluator-free adaptation** or **artifact-guided adaptation without scored selection**. Avoid the stronger, misleading phrase “evaluation-free intelligence.” ## What is plausibly new, and what is already known Prompt improvement with fixed weights is established by [GEPA](https://arxiv.org/abs/2507.19457). Output feedback is established by [Self-Refine](https://arxiv.org/abs/2303.17651); label-free prompt refinement by [ProRefine](https://arxiv.org/abs/2506.05305); reference-free output comparison by [SPO](https://aclanthology.org/2025.findings-emnlp.479/); evolving textual memory by [ACE](https://arxiv.org/abs/2510.04618); and text-artifact optimization through comparison and rationales by [Feedback Descent](https://arxiv.org/abs/2511.07919). The narrower combination to investigate is: 1. A planner outside a fixed worker, using only an audited artifact interface. 2. A sequence of useful next actions on one evolving task, rather than scoring a population of candidate prompts. 3. No external score, preference gate, reference answer, or diagnostic execution trace returned to that planner. 4. Fixed principles and memory provenance that prevent locally convenient assumptions from replacing the user's objective. 5. Independent evidence separating artifact grounding from generic reflection, extra turns, and extra compute. This combination remains a **candidate contribution**. If full implementation-level comparison shows it is simply an existing method with different packaging, write an empirical study of information restrictions and memory drift rather than claiming a new algorithm. ## Formal setting Let: - `G`: immutable user goal and requirements. - `P`: immutable planner principles, written before the evaluation tasks are inspected. - `W`: worker with fixed weights, system instructions, harness code, tools, and permissions. - `s_t`: worker's task state after turn `t`. - `A_t = O(s_t)`: artifact produced by a fixed observation procedure `O`. - `M_t`: bounded planner memory, distinct from `G`. - `u_t`: the next instruction sent to the worker. - `Q`: fixed planner model and meta-prompt. The loop is: ```text (u_t, proposed_memory) = Q(G, P, A_t, M_t, prior_instructions) M_(t+1) = provenance_and_size_filter(proposed_memory) s_(t+1) = W(s_t, u_t) A_(t+1) = O(s_(t+1)) ``` The filter checks schema, size, evidence references, and immutable-goal separation. It does not score task quality or choose a candidate by performance. Its rules are frozen before experiments. No `gold_answer`, `reward`, `benchmark_score`, `test_result`, `worker_trace`, or `source_tree` is an input to the strict planner. The worker may possess ordinary task context and public evidence supplied by the task; the observation restriction concerns what crosses into the planner. A separate, stricter whole-system track can also prohibit worker tool diagnostics. The experiment measures `J(A_T; hidden_assessment)` after `T` turns and after outputs are sealed. This measurement cannot influence prompts, memory, stopping, retries, or candidate choice. In the primary experiment, `G`, `P`, `W`, `Q`, `O`, tools, and adapters remain fixed. Only `u_t`, `M_t`, and the task deliverable evolve. This is inference-time adaptation; it is not a weight-training algorithm or a claim of learning a better general model. ## Observation model Use two clearly separated tracks. **Primary: text deliverables, no code changes.** The planner sees the user's request, authorized source material, the latest deliverable, limited prior deliverables, its previous instructions, and bounded memory. It sees no private reasoning, worker self-reports, tool transcripts, graders, or hidden references. Tasks include source-grounded reports, structured operating documents, and internally consistent plans. Numerical source data can be supplied as plain text. **Secondary: software deliverables.** The worker edits application code as the user's task requires. The planner sees a frozen set of rendered states or user-visible outputs captured by a fixed script, but no source or developer-console output. The script captures; it does not grade. The planner and harness cannot rewrite themselves. All worker tests and browser diagnostics, if allowed, are logged and disclosed. This supports a planner-level information claim, not the claim that the whole system never receives execution feedback. Keep the existing Cruise report-aware planner as a separate comparison. Its access is broader than the strict method proposed here. ## Six principles to freeze and test 1. **Preserve the destination.** User requirements remain immutable. Memory cannot manufacture new requirements or permissions. 2. **Ground a change in evidence.** Name an observable gap and its artifact location. Do not assert an implementation defect from appearance alone. 3. **Separate observation from hypothesis.** A missing citation is visible; its underlying cause is uncertain. Ask the worker to resolve uncertainty when needed. 4. **Choose one coherent next increment.** Repair the highest-priority supported gap, preserving satisfied requirements. Priority is a qualitative planning decision, not a new hidden numeric judge. 5. **Keep memory bounded and attributable.** Save supported observations and explicitly provisional tactics; remove superseded state. Every entry points to an artifact or user requirement. 6. **Do not equate activity with progress.** Avoid repeated work and cosmetic expansion. A fixed budget terminates the primary run regardless of confidence. For conflict handling, explicit requirements take priority over planner preferences. Within requirements, repair blocking correctness and usability gaps before optional polish. The written rules and their token budget are held equal across relevant comparison arms. ## Hypotheses and how they could fail | ID | Testable hypothesis | Evidence that would count against it | |---|---|---| | H1 | Artifact-guided follow-ups improve final hidden quality over generic continuation at equal total budget | Gains disappear or reverse when worker and planning cost are matched | | H2 | Inspecting the actual artifact matters | Matched stale or swapped artifacts produce equivalent follow-ups and outcomes | | H3 | The fixed principles add value beyond ordinary self-refinement | A same-budget critique-and-revise baseline matches the method | | H4 | Provenance-linked memory reduces unsupported requirements | It does not reduce drift or harms task completion at the same memory budget | | H5 | Removing external scoring yields a useful quality/cost tradeoff | Savings are absorbed by critique costs, or independent quality drops materially | | H6 | The procedure is not specific to one worker family | Gains fail to replicate with a second frozen worker | H1 is the primary efficacy question. H2–H4 are mechanism questions. H5 is a cost-frontier question, not an assumption that no scores must be better. H6 supports portability; it does not prove cross-task learning. ## Two distinct notions of improvement **Within a task:** the next prompt helps the current deliverable. This is the primary claim. **Across tasks:** a learned strategy improves new tasks after being frozen. This requires a separate experiment: learn a reusable instruction on development tasks, strip task-specific facts, freeze it, and evaluate on disjoint tasks without further adaptation. Do not call within-task progress generalization. Do not carry memory between primary evaluation tasks. ## Work packages and decision gates | Stage | Work | Deliverable and gate | |---|---|---| | 1: one week | Complete method-level reading of closest competitors; trace their inputs, update rules, and selection logic | Novelty/access matrix; choose method paper or empirical-study framing | | 2: one week | Build an isolated experimental wrapper, artifact filter, snapshots, accounting, and immutable configuration | Auditable no-score/no-source boundary; frozen prototype before pilot | | 3: one week | Run development-only pilot with 12 tasks and three repetitions | Feasibility, latency, cost, failure modes, variance; no headline result | | 4: one week | Finalize tasks, rubric, power analysis, baseline settings, and analysis | Time-stamped protocol and untouched test split | | 5: two weeks | Execute paired main runs and prespecified ablations; keep hidden scores sealed | Complete logs including failed and incomplete runs | | 6: one week | Independent scoring, blinded review, statistics, and error analysis | Tables, intervals, full cost curves, negative cases | | 7: one week | Draft, verify claims against data, package reproduction materials | Reviewable manuscript and release bundle | Eight weeks is a planning estimate, not a promised timeline. Work can overlap only when it does not leak test outcomes into method development. This request creates the plan; it does not launch paid experiments. ## Minimum evidence for the eventual claim - The observation boundary holds in every claimed strict run. - Quality improves over continuation and an adequately tuned self-refinement baseline under a matched budget. - At least one strong reference-free optimizer is included; GEPA alone is an insufficient comparison set. - An artifact intervention demonstrates sensitivity to real task evidence. - Memory drift and regressions are reported, not excluded as inconvenient cases. - Independent scoring, whole-run repetitions, and task-level uncertainty support the stated effect. If the method matches competitors at lower measured cost, write that result. If it only helps observable-quality tasks, explain that boundary. If it fails to outperform self-refinement, a well-executed study of what artifact-only control can and cannot do remains more defensible than an unsupported superiority paper.