# GEPA: reading notes and implications ## Source and reading scope Primary source: [ICLR 2026 conference paper](https://proceedings.iclr.cc/paper_files/paper/2026/file/0e9e708b6f48e14fd0ac29e167413f76-Paper-Conference.pdf), 87 pages, by Lakshya A. Agrawal and colleagues. Local copy: [PDF](sources/GEPA-ICLR-2026.pdf), [page-marked text](sources/GEPA-ICLR-2026.txt). Page references below use this PDF's printed page numbers, which match its PDF page numbers. Also checked: [arXiv record](https://arxiv.org/abs/2507.19457), [arXiv v2 HTML](https://arxiv.org/html/2507.19457v2), and the authors' [reproduction repository](https://github.com/gepa-ai/gepa-artifact). The conference PDF is the numerical reference here. The arXiv version has different appendix lettering and rounds some headline numbers differently. Do not mix them. I read the complete extracted text: main paper, references, algorithm details, setup, plots' labels and captions, search trees, intermediate prompts, final prompt listings, and reflection-call counts. Page 82 is principally an image, so I separately inspected its embedded image. This is a literature reading, not an independent reproduction or an audit of every experimental log. ## Abstract and central argument, in plain language The paper asks whether language models can learn useful task strategies more efficiently by reading detailed feedback and changing instructions than by using scalar rewards to update weights. GEPA combines reflection with evolutionary prompt search. It reports favorable comparisons with GRPO and other prompt optimizers, plus exploratory kernel-generation and adversarial-search applications. These results concern the tested systems and budgets; they do not establish that prompting always beats reinforcement learning. [Source: conference PDF, pp. 1–9.](https://proceedings.iclr.cc/paper_files/paper/2026/file/0e9e708b6f48e14fd0ac29e167413f76-Paper-Conference.pdf) The essential lesson for us is that **better prompts from richer evidence is already the starting point of this literature**. A Cruise paper needs an additional, measurable contribution. ## What the method actually does Sections 2–3 and Appendix D define a compound system with fixed control flow, language-model modules, module prompts, and model weights. GEPA changes prompts, not weights. It uses training tasks, numeric metrics, textual feedback, and execution traces. It proposes a prompt mutation, tests it against its parent on a minibatch, and admits improving candidates into a pool. Validation scores support candidate selection. A Pareto-inspired procedure retains candidates that excel on individual instances. An optional merge combines complementary module changes from separate lineages. [Source: conference PDF, pp. 3–6, 18–19.](https://proceedings.iclr.cc/paper_files/paper/2026/file/0e9e708b6f48e14fd0ac29e167413f76-Paper-Conference.pdf) Three distinctions must remain clear in our paper: - **Mutation:** how a new prompt is proposed. - **Selection:** how the system decides which prompt to retain or extend. - **Measurement:** how an independent study determines whether the method helped. Our proposed departure concerns the evidence available for mutation and the removal of scored selection. It cannot remove measurement from the scientific study. ## Reading register: every section and appendix The rightmost column records our interpretation and study design, rather than claims that GEPA failed. | Part of conference PDF | Pages | Questions and implications for our paper | |---|---:|---| | Abstract; introduction | 1–2 | State exactly which signal is being removed. Avoid presenting language feedback or fixed weights as new. | | Problem statement | 2–4 | Specify our observation boundary, mutable prompt state, fixed worker, and total budget just as explicitly. | | Reflective evolution; candidate selection | 4–6 | Separate the effect of critique from the effect of retaining candidates by score. Can one evidence-based next action suffice? | | Evaluation and principal results | 6–8 | Compare under the same worker and full accounting, not a different flagship model against an old table. | | Extended applications | 8–9 | A test-time Cruise loop overlaps their inference-time setting; offline-versus-online is insufficient novelty. | | Related work; conclusion | 9 | Position against self-refinement, memory, prompt optimization, and rich feedback, not only RL. | | Acknowledgments; bibliography | 10–16 | Use the cited literature as a search map. Listing a reference does not mean we have read that cited paper in full. | | A: appendix map | 17 | Use this as a completeness check against accidental omissions. | | B: LLM use disclosure | 17 | Our eventual manuscript should disclose AI assistance and the human authors' responsibility for methods and claims. | | C: reflection meta-prompt | 17–18 | Ask whether our planner is merely paraphrasing this recipe. Domain details can become examples or memorized answers; audit memory content. | | D / D.1: algorithms and merge | 18–19 | Preserve the actual GEPA algorithm in comparisons. A version stripped of its selection rule is an ablation, not native GEPA. | | E: inference-time search | 19–21 | Distinguish improving a task's evolving deliverable from learning a reusable task-family instruction. Measure both separately if claiming both. | | F: adversarial search | 20–22 | Include a formatting-versus-substance audit: a score drop can reflect output parsing rather than lost task competence. | | G.1: tasks and feedback | 22–23 | Show exactly which information each competitor receives. Rich diagnostic feedback is additional supervision even if it is not a gold final answer. | | G.2–G.4: models, cost, optimizers | 23–25 | Record dated model IDs, decoding, all reflection calls, tool costs, and tuning. Report resource asymmetries explicitly. | | H: extended result analysis | 25–26 | Inspect improvements and regressions per task and model; do not conceal a failure behind an average. | | I: budget curves | 26–27 | Save snapshots at fixed budgets; do not select a favorable stopping point after seeing hidden scores. | | J: generalization gap | 26, 28 | A small validation/test gap is useful evidence, not proof of broad transfer. Test task-family and topic separation. | | K: prompt cost | 28–29 | Count prompt length and repeated context reads in actual cost. A short final prompt need not imply a cheap search. | | L: search trees | 29–31 | Make all trajectories inspectable, including unsuccessful branches and regressions. | | M / M.1: intermediate PUPA prompts | 29, 32–38 | Inspect which instructions accumulate, conflict, or overgeneralize. Longer instructions should earn their keep. | | N.1–N.2: HotpotQA prompts | 38–52 | Distinguish reusable retrieval strategies from example-specific entity and answer content. | | N.3–N.4: instruction-following prompts | 52–58 | Check for instructions learned too broadly from a subset, such as unconditional output-format rules. | | N.5–N.6: HoVer prompts | 58–70 | Measure factual support and retrieval utility separately. Fluent summaries can still contain unsupported bridges. | | N.7–N.8: PUPA prompts | 70–81 | Test whether privacy principles preserve enough task information and whether memory invents new restrictions. | | O.1: NPU prompt | 81–82 | The embedded prompt is visually inspected. Check examples and stated API rules for internal consistency before treating learned text as authoritative. | | O.2: CUDA prompt | 83–87 | Inspect redundancy, code/example correctness, and the difference between a persuasive instruction and a working output. | | P: reflection-call counts | 87 | Planning calls are real compute. Ten worker turns plus nine planning turns are not ten model calls. | ## Numerical checks that affect our framing The following are reported results, not results reproduced here. Source: Tables 1–3 and Appendix G of the [conference paper](https://proceedings.iclr.cc/paper_files/paper/2026/file/0e9e708b6f48e14fd0ac29e167413f76-Paper-Conference.pdf). - Qwen3-8B: GEPA's six-task aggregate is 54.85 versus GRPO's 48.91: **5.94 percentage points**, rounded to six. GEPA does not beat GRPO on AIME-2025: 32.00 versus 38.00. - GPT-4.1 Mini: GEPA scores 65.22; the base system scores 53.03. The **12.19-point improvement is over the base system**, not over MIPROv2. MIPROv2 scores 58.67, making that gap 6.55 points. GEPA+Merge scores 66.36, or 7.69 points over MIPROv2. - On GPT-4.1 Mini AIME, GEPA versus MIPROv2 is 59.33 versus 51.33: eight points. The abstract's approximate 12-point example aligns with Qwen GEPA versus Qwen MIPROv2, 32 versus 20. Always identify the model and variant. - The “up to 35× fewer rollouts” comparison uses the rollouts needed to reach the selected solution, rather than asserting that every full GEPA search costs 1/35 of every GRPO run. The paper reports both full budgets and intermediate budget milestones. Use total-budget curves in our comparison. - Merge helps some settings and harms others. For example, Qwen IFBench drops from 38.61 with GEPA to 28.23 with GEPA+Merge. A simpler method could be useful, but this does not predict our outcome. **Proposed reporting rule:** reconstruct every headline from a named table row, denominator, model, and budget. If our result beats one configuration, name that configuration rather than claiming to beat GEPA universally. ## What is strong about GEPA Our assessment: its main strength is a concrete, inspectable mechanism connecting rich feedback to prompt changes, coupled to a candidate-selection method and evidence across multiple task families. The appendices expose learned prompts, inference settings, and implementation choices. Its test-time experiments anticipate an obvious extension of the main method. The appropriate response is to extend its research question. A paper framed as “GEPA uses RL and Cruise uses prompts” would misdescribe its core contribution. ## Questions that remain open for our study These are proposed experiments or limits on inference, not findings against the authors. 1. **Information access.** What happens when module traces, evaluator explanations, labels, and numeric acceptance signals are unavailable? Publish a channel-by-channel access matrix. 2. **External evaluator cost.** If scoring is expensive or brittle, can direct artifact inspection deliver a useful quality/cost tradeoff? Count the critique model's cost too. 3. **Long tasks.** Can a planner choose the next useful subtask in a persistent workspace, rather than merely refine one answer? Test ordering, revisiting, and completion behavior. 4. **Memory integrity.** Does accumulated advice preserve user intent or turn temporary tactics into new requirements? The local Cruise case makes this a concrete concern. 5. **Independent verification.** Can an artifact look better while becoming less correct? Include factual, numerical, and interaction checks outside the loop. 6. **Stochasticity.** Repeat whole runs, not just final generations. Multiple samples of one question are not multiple independent tasks. 7. **Portability.** Do prompts transfer across worker families and unseen tasks? GEPA already reports cross-model transfer, so portability is a comparison question, not an assumed novelty. 8. **Unobservable defects.** An artifact-only observer cannot reliably diagnose what the artifact does not reveal. Include tasks with both visible and hidden failure modes. 9. **Stopping.** Without score-based acceptance, improvements are not guaranteed to be monotonic. Record final-state regressions and use a fixed primary endpoint. ## Conclusions we can and cannot draw now We can formulate a precise restricted-information setting, build fair tests, and use Cruise's history to identify failure modes. We cannot currently claim a new general paradigm, a win over GEPA, lower real cost, stable autonomous improvement, or better generalization. The research plan makes each of those an empirical question.