Self-Improvements in Modern Agentic Systems
A survey of how foundation model-based agents improve themselves through model updates, intrinsic demonstrations, intrinsic evaluative feedback, extrinsic experience, prompt evolution, memory, tool use, and recursive scaffolding.
Abstract
Foundation model-based agents are moving from static prompt-following systems toward systems that can improve themselves over time. This survey organizes that transition into two major branches: foundation model improvement, which updates the model itself, and scaffolding improvement, which updates surrounding components such as prompts, memory, tools, and executable agent logic.
Across the survey, the central distinction is between improving the underlying foundation model and improving the scaffolding around it. This distinction provides a practical way to compare methods, interpret learning signals, and connect representative papers to broader research trends.
Overview
Core distinction. Self-improvement can be framed by asking what is being updated.
- Foundation model improvement changes the model itself and tends to be slower, more persistent, and more training-centric.
- Scaffolding improvement changes the operational shell around the model and tends to be faster, cheaper, and more reversible.
- The paper library below follows this logic so that taxonomy and literature map onto each other directly.
Taxonomy
The survey can be read as a structured map of how an FM-based agent improves itself. Each item below links directly into the corresponding literature section.
1. Foundation Model Improvement 73 papers
2. Scaffolding Improvement 166 papers
Quick start: representative papers
These are not the only important papers in the survey. They are included here as a first reading path for readers who want to understand the landscape quickly before diving into the full library.
A canonical example of self-synthetic instruction generation for model alignment.
An influential AI-feedback framework where model-based judgments are used for alignment and policy optimization.
A representative web-agent training setup built around reinforcement learning and evolving curricula.
A concrete example of using learned environment dynamics to improve web agents.
A classic self-feedback loop for iterative improvement at the prompt/output level.
An influential formulation of automatic optimization via textual gradients.
A representative long-term memory architecture for LLM-based agents.
An open-ended embodied agent that accumulates skills and uses tools in a growing scaffold.
A strong representative for recursive self-improving agents that modify their broader operating logic.
Curated paper library
The literature below is organized to match the survey taxonomy. Within each subsection, papers are presented in roughly chronological order so that readers can follow the development of each research direction.
1.1 Intrinsic Generative Demonstrations
The agent or model improves by synthesizing demonstrations, instruction sets, reasoning traces, or task distributions that can be used for imitation-style parameter updates.
1.2 Intrinsic Evaluative Feedback
The system derives its own reward, critique, verification signal, or intrinsic supervision to guide further updates.
1.3 Extrinsic Exploratory Experience
The agent improves through trajectories gathered from interaction with environments or learned simulators.
1.3.1 Grounded Executable Environments 17 papers
Learning from trajectories, rewards, or observations obtained by acting in executable environments such as code runtimes, web interfaces, games, GUI systems, or robotics simulators.
1.3.2 Generative World Models 15 papers
Using learned world models, simulators, or imagined rollouts to improve agent behavior.
2.1 Prompt Optimization
The agent improves the prompt layer through scoring, reflection, evolutionary search, or textual-gradient style updates.
2.1.1 Scalar-Feedback Optimization 10 papers
Optimizing prompts with scalar objectives, scores, or evaluation signals.
| Year | Title | Venue | Links |
|---|---|---|---|
| 2022 | Large Language Models Are Human-Level Prompt Engineers | arXiv | Paper · Code |
| 2024 | Large Language Models as Optimizers | ICLR | Paper · Code |
| 2024 | Prompt Refinement with Image Pivot for Text-to-Image Generation | ACL | Paper · Code |
| 2024 | Learning from Contrastive Prompts: Automated Optimization and Adaptation | arXiv | Paper · Code |
| 2024 | PRompt Optimization in Multi-Step Tasks (PROMST): Integrating Human Feedback and Heuristic-based Sampling | EMNLP | Paper · Code |
| 2025 | The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation | arXiv | Paper · Code |
| 2025 | DRO-InstructZero: Distributionally Robust Prompt Optimization for Large Language Models | arXiv | Paper · Code |
| 2025 | CoolPrompt: Automatic Prompt Optimization Framework for Large Language Models | FRUCT | Paper · Code |
| 2026 | SePO: Self-Evolving Prompt Agent for System Prompt Optimization | arXiv | Paper · Code |
| 2026 | SAGE: Stochastic Prompt Optimization via Agent-Guided Exploration | arXiv | Paper · Code |
2.1.2 Qualitative-Feedback Refinement 10 papers
Improving prompts through critiques, revisions, hindsight, or other natural-language feedback.
2.1.3 Population-Based Evolution 6 papers
Evolving prompt populations with selection, mutation, or other population-based search methods.
2.1.4 Textual Gradient Optimization 11 papers
Treating language feedback like gradients for automatic textual optimization.
2.2 Memory
The agent improves what it stores, how memory is structured, and how memory is processed across interactions.
2.2.1 Memory Object 16 papers
What is stored in memory, such as notes, summaries, trajectories, or latent states.
2.2.2 Memory Structure 23 papers
How memory is organized, indexed, retrieved, or represented over time.
2.2.3 Memory Processing 22 papers
How memory is created, updated, compressed, retrieved, and otherwise processed over time.
2.3 Tool
The agent improves which tools it uses, how tools are routed, and whether it can refine or create new tools itself.
2.3.1 Dynamic Tool Routing 24 papers
Choosing, orchestrating, and routing among available tools as tasks and contexts evolve.
2.3.2 Iterative Tool Refinement 11 papers
Improving tool use through iterative practice, feedback, debugging, or skill refinement.
2.3.3 Autonomous Tool Creation 15 papers
Generating or assembling new tools, services, or agent capabilities on demand.
2.4 Full Scaffolding
The most ambitious setting: the agent revises its own broader operational logic or executable scaffold.
How to read this survey
If you are new to the area
Start with the overview and taxonomy, then read one representative paper from each branch in the quick-start section.
After that, use the paper library to go deeper into whichever mechanism matches your interests: model updates, prompt evolution, memory, tools, or recursive scaffolds.
If you are using the page as a reference
Jump directly to the literature tables. The search bar works best for titles and venues, while the filter buttons quickly separate training-centric model updates from scaffold-centric agent adaptation.
For deeper reading, use the search bar and filters to move between broad categories and more specific mechanisms.
Citation
Please cite this survey as follows.
@misc{ren2026selfimprovementsmodernagenticsystems,
title={Self-Improvements in Modern Agentic Systems: A Survey},
author={Zhe Ren and Yimeng Chen and Dandan Guo and Guowei Rong and Tonghui Li and R. B. Xiong and Qingfeng Lan and Wenyi Wang and Li Nanbo and Yibo Yang and Mingchen Zhuge and Jürgen Schmidhuber},
year={2026},
eprint={2607.13104},
archivePrefix={arXiv},
primaryClass={cs.AI},
url={https://arxiv.org/abs/2607.13104},
}
