AI Self-evolution: A Comprehensive Review of LLM Closed-loop Systems and Multi-agent Frontiers

Why must we seriously study the "self-evolution of LLM" now? To put it simply, today's powerful models are mostly "static products": after a one-time offline training, they are deployed and then face distribution shifts, new task forms, and the rapid evolution of tool ecosystems. They can only rely on expensive and lagging human retraining to catch up. This paradigm will keep incurring losses in a non-stationary world: the technical debt from outdated knowledge, the continuous cost of labeling and cleaning data, and the vulnerability in long-tail complex reasoning and cross-domain collaboration. What we need is not just larger models, but systems that can learn while running, self-correct in their environment, and continuously grow stronger in a closed loop.

We have focused on the work related to "self-evolving/self-improving of LLM / AI Agent" from June to August 2025 and provided a comprehensive review and progress update. We hope to clarify the design space and feasible paths for "self-evolution" for developers and researchers: which problems to close the loop on first, how to build a minimum viable system, what metrics to use to measure "really getting stronger," and how to make "self-evolution" and "controllable and trustworthy" both usable at the engineering level.

Abstract (June–August 2025 Overview)

Definition

The academic and industrial communities have not yet reached a unified definition for "

Representative Paths

In the past three months, representative work has been concentrated on five main technical threads:

• Self-play/self-generation tasks without external data (Self‑Questioning, Absolute Zero, SeRL).
• Self-review/self-reward (Self‑Rewarding, LLM‑as‑judge).
• Co-evolution of data and models (C2‑Evo, NavMorph).
• Automatic curriculum/open-ended evolution (SEC, Reasoning Gym, DeepMind开放式学习传统).
• Multi-agent self-improvement workflows (Google AI co‑scientist,Anthropic’s multi-agent system). There has also been the emergence of quantitative evidence and diagnostic methods regarding "which cognitive habits support self-improvement."

Evaluation and Safety:

Process generation environments with "verifiable rewards" such as Reasoning Gymhave become a handle for closed-loop self-evolution training and evaluation.Google’s AI co‑scientistcorrelates internal self-play ranking and Elo scores with the accuracy of GPQA problems.Anthropic emphasizes the combination of LLM‑judge and human review, as well as engineering protection and traceability for multi-agent systems. Meanwhile, the risks of "cheating/hallucination" and alignment in "self-improvement" have led to more exploration of sandboxing and guardian strategies.

Concept and Boundaries: What Are “Self-evolving” LLM/AI Agent

Self-evolution

Self-evolution is not a single training paradigm, but a category of closed-loop system design: With minimal human intervention, the system continuously generates data/tasks, improves strategies and parameters, or rewrites its own toolchain/code through mechanisms such as environmental feedback, tool execution, self-play, or self-review. This allows it to become stronger over time in out-of-distribution tasks, long-term tasks, and complex reasoning. Two recent reviews have abstracted it into a feedback loop with four components: system input, agent system, environment, and optimizer. They have also evaluated and summarized methodologies based on three dimensions: "what to evolve, when to evolve, and how to evolve," emphasizing the transition from static base models to a "self-evolving agent" system with lifelong adaptability.

The difference from traditional self-supervision/instruction fine-tuning

The difference lies in the emphasis on the dominance of "experience/interaction data," dynamic generation of task space and difficulty, and automated sources of review/reward signals (self-review, executable verification, competition ranking, etc.). This breaks through the upper limit of static human data. DeepMind has proposed the "Era of Experience," advocating for interaction experience as the main data source, with reward signals grounded in the world. It suggests continuously updating the world model and reward function to correct biases over the long term, providing a conceptual and path argument for "self-evolution."

Research Landscape and Frontier Labs/Teams/Researchers

Google Research

The AI co‑scientist,based on Gemini 2.0, employs a multi-agent collaboration of "Supervisor + dedicated agents." Components include generation, reflection, ranking, evolution, proximity, and meta-review agents. It leverages automated feedback and self-play scientific debates, ranking tournaments, and evolutionary processes to form a self-improvement loop with "scalable compute at test time." Its internal Elo self-assessment correlates positively with accuracy on the challenging GPQA dataset. Expert reviews in small samples suggest that its outputs outperform several state-of-the-art (SOTA) baselines in terms of novelty and impact.

Anthropic

Anthropic has publicly detailed itsmulti-agent research system engineering plan, which features parallel sub-agents of "orchestrator–worker," external memory, LLM‑judge scoring combined with human review. It proposes "agents improving themselves" (models self-diagnose failure modes and rewrite prompts/tool descriptions), achieving approximately a 40% reduction in task time for tool usability. It emphasizes emergent behavior in multi-agent systems and engineering-level observability, tiered releases, and rollback safeguards.

Meta

Zuckerberg explicitly highlighted“self-improvement” as a strategic focus of the “Superintelligence Lab” during the Q2 earnings report, emphasizing the reduction of dependence on human data and the development of a "self-improving" path, linked to the vision of "personal superintelligence."

OpenAI and Academic Intersections

Media reports have cited Sam Altman describing the current phase as "past the event horizon with a slow takeoff," emphasizing that short-term self-improvement is not fully automated but rather a recursive enhancement of “using AI to accelerate AI research.” Concurrently, the "Darwin‑Goedel Machine" (by Clune and the Sakana AI team) demonstrates automatic reading of its own logs, proposing and implementing single-point code modifications, and generational iterative improvements on SWE‑Bench and Polyglot. However, it also exposes risks of “self-deception/log forgery,”highlighting the importance of sandboxing and anti-deception evaluations.

Classification of Technical Mechanisms and Representative Works

Self-Play / Self-Generated Tasks Without External Data

• Self-Questioning Language Models (SQLM): Given a topic prompt, an asymmetric "proposer-solver" self-play framework generates questions and answers, with both components trained via reinforcement learning (RL). The proposer is rewarded for generating problems of intermediate difficulty (neither too easy nor too hard), while the solver is evaluated using majority voting as a proxy for correctness. For programming tasks, unit tests serve as verification. Empirical results show sustained improvement on three-digit multiplication, OMEGA algebra, and Codeforces benchmarks without any human-provided data, representing a closed-loop "generate problems – solve problems" paradigm.
• Absolute Zero (AZR): Proposes aReinforcement Learning with Verifiable Rewards (RLVR)paradigm that requires zero external data. A single model autonomously generates code-based reasoning tasks and uses a code executor to validate both the tasks and their solutions, providing a unified source of verifiable rewards to guide open-ended yet grounded learning. AZR achieves or surpasses state-of-the-art performance on coding and mathematical reasoning tasks compared to zero-supervision baselines that rely on tens of thousands of human-curated examples, emphasizing an integrated, closed-loop of task generation, verification, and learning.
• SeRL: Combines "self-instructing" (online instruction augmentation with filtering) and “self-reflecting”(majority voting to estimate rewards), enabling reinforcement learning on self-generated data. This approach reduces reliance on high-quality, human-provided instructions and verifiable rewards, and demonstrates superior performance across multiple reasoning benchmarks and different model backbones.
• AMIE Medical Dialogue Self-Play Extension (Industry Report): To expand coverage of diseases and clinical scenarios, Google developed a“self-play diagnostic dialogue simulation environment”with automated feedback mechanisms to enrich and accelerate training. This represents an industry-level effort to apply self-play methods for scaling up AI in safety-critical domains like healthcare.

Self-Evaluation / Self-Rewarding and Adversarial Critic Evolution

• Self-Rewarding Self-Improving: Leverages the“asymmetry between solution generation and verification” by enabling models to provide their own reward signals in domains without reference answers. The work demonstrates that self-judged rewards are comparable to formal verification on tasks like Countdown puzzles and MIT Integration Bee problems. Combined with synthetic question generation, this forms a complete self-improvement loop. The study reports that a distilled 7B model, after self-rewarding training, reaches the performance level of participants in the MIT Integration Bee, showcasing the cross-domain potential of the "LLM-as-judge" paradigm as a reward mechanism.
• Self-Play Critic (SPC): Trains two copies of the same base model to engage in adversarial self-play as a "sneaky generator" (which deliberately produces subtle reasoning errors) and a "critic" (which attempts to detect them). Using reinforcement learning based on game outcomes, the critic progressively improves its ability to identify flawed reasoning steps, reducing the need for manual step-level annotations. Experiments show significant improvements in process evaluation on benchmarks such as ProcessBench, PRM800K, and DeltaBench. Furthermore, the trained critic can guide test-time reasoning search in diverse LLMs, boosting their performance on mathematical reasoning tasks like MATH500 and AIME2024. This validates the feasibility of evolving high-quality evaluation rules through adversarial self-play.
• Anthropic Engineering Practice: In their multi-agent research system, Anthropic systematically combinesLLM-as-judge evaluation with human assessment, using a detailed rubric that includes factual accuracy, citation correctness, completeness, source quality, and tool efficiency. To ensure reliability in this non-deterministic, stateful system, they implement production-grade solutions such as full execution tracing, external memory systems, fault-tolerant retry mechanisms, and asynchronous coordination. These engineering safeguards enable stable, scalable operation and serve as a template for "production-ready self-improving research systems."

Process diagram

Co-Evolution of Data and Models

• C2-Evo: Proposes a "cross-modal data evolution loop" and a “data–model evolution loop,” where complex multimodal problems—combining structured textual sub-problems with iteratively refined geometric diagrams—are generated and then selectively used for training based on model performance. The system alternates between supervised fine-tuning (SFT) and reinforcement learning (RL), achieving continuous improvements across multiple mathematical reasoning benchmarks. This work emphasizes the dynamic alignment of data complexity and model capability, avoiding the "mismatched difficulty" problem where tasks are either too easy or too hard relative to current ability.
• NavMorph: Introduces a "self-evolving world model" for Vision-and-Language Navigation in Continuous Environments (VLN-CE). By leveraging compact latent representations and a novel "Contextual Evolution Memory," the model adaptively updates its understanding of the environment and refines its decision-making policy during online navigation. This reflects a co-evolutionary paradigm between the world model (environmental representation) and the agent’s policy (action strategy), enabling sustained adaptation in dynamic, real-world settings.
• Self-Challenging (Code-as-Task): An agent first acts as a "challenger" that interacts with external tools to generate tasks in a novel format called Code-as-Task, each consisting of an instruction, a verification function, and example solution/failure cases that serve as built-in tests. These high-quality, self-generated tasks are then used to train the same agent in the role of an "executor" via reinforcement learning, using the verification outcomes as rewards. Despite using only self-generated data, this framework achieves over a two-fold performance improvement on two multi-turn tool-use benchmarks (M3ToolEval and TauBench) for a Llama-3.1-8B-Instruct model, demonstrating a fully closed-loop synthetic ecosystem of "task generation – verification – learning."

Co-Evolution of Data and Models

• C2-Evo: Proposes a "cross-modal data evolution loop" and a "data–model evolution loop," where complex multimodal problems—combining structured textual sub-problems with iteratively refined geometric diagrams—are generated and then selectively used for training based on model performance. The system alternates between supervised fine-tuning (SFT) and reinforcement learning (RL), achieving continuous improvements across multiple mathematical reasoning benchmarks. This work emphasizes the dynamic alignment of data complexity and model capability, avoiding the "mismatched difficulty" problem where tasks are either too easy or too hard relative to current ability.
• NavMorph: Introduces a“self-evolving world model” for Vision-and-Language Navigation in Continuous Environments (VLN-CE). By leveraging compact latent representations and a novel "Contextual Evolution Memory," the model adaptively updates its understanding of the environment and refines its decision-making policy during online navigation. This reflects a co-evolutionary paradigm between the world model (environmental representation) and the agent’s policy (action strategy), enabling sustained adaptation in dynamic, real-world settings.
• Self-Challenging (Code-as-Task): An agent first acts as a "challenger" that interacts with external tools to generate tasks in a novel format called Code-as-Task, each consisting of an instruction, a verification function, and example solution/failure cases that serve as built-in tests. These high-quality, self-generated tasks are then used to train the same agent in the role of an“executor” via reinforcement learning, using the verification outcomes as rewards. Despite using only self-generated data, this framework achieves over a two-fold performance improvement on two multi-turn tool-use benchmarks (M3ToolEval and TauBench) for a Llama-3.1-8B-Instruct model, demonstrating a fully closed-loop synthetic ecosystem of "task generation – verification – learning."

Co-Evolution of Data and Models

• C2-Evo: Proposes a "cross-modal data evolution loop" and a "data–model evolution loop," where complex multimodal problems—combining structured textual sub-problems with iteratively refined geometric diagrams—are generated and then selectively used for training based on model performance. The system alternates between supervised fine-tuning (SFT) and reinforcement learning (RL), achieving continuous improvements across multiple mathematical reasoning benchmarks. This work emphasizes the dynamic alignment of data complexity and model capability, avoiding the "mismatched difficulty" problem where tasks are either too easy or too hard relative to current ability.
• NavMorph: Introduces a "self-evolving world model" for Vision-and-Language Navigation in Continuous Environments (VLN-CE). By leveraging compact latent representations and a novel “Contextual Evolution Memory,”the model adaptively updates its understanding of the environment and refines its decision-making policy during online navigation. This reflects a co-evolutionary paradigm between the world model (environmental representation) and the agent’s policy (action strategy), enabling sustained adaptation in dynamic, real-world settings.
• Self-Challenging (Code-as-Task): An agent first acts as a "challenger" that interacts with external tools to generate tasks in a novel format called Code-as-Task, each consisting of an instruction, a verification function, and example solution/failure cases that serve as built-in tests. These high-quality, self-generated tasks are then used to train the same agent in the role of an "executor" via reinforcement learning, using the verification outcomes as rewards. Despite using only self-generated data, this framework achieves over a two-fold performance improvement on two multi-turn tool-use benchmarks (M3ToolEval and TauBench) for a Llama-3.1-8B-Instruct model, demonstrating a fully closed-loop synthetic ecosystem of "task generation – verification – learning."

Automatic Curriculum and Open-Ended Learning

• Self-Evolving Curriculum (SEC): Models curriculum selection as a non-stationary multi-armed bandit problem, learning the curriculum policy in parallel with reinforcement learning (RL) fine-tuning. It selects task categories based on an "immediate learning gain" signal and updates the policy using TD(0). SEC improves generalization to harder out-of-distribution (OOD) test sets across planning, induction, and mathematical reasoning domains. It also enhances skill balance when fine-tuning on multiple domains simultaneously, demonstrating a curriculum mechanism where task difficulty evolves adaptively.
• Reasoning Gym: Provides over 100 verifiable reward-based reasoning environmentsspanning algebra, logic, graph theory, and other domains. Its key innovation lies in procedural generation, adjustable complexity, and near-infinite training data—unlike fixed, finite datasets. This makes it naturally suitable for closed-loop self-improvement training and difficulty-tiered evaluation. Reasoning Gym serves as an open infrastructure that connects task generation, verification, and learning, enabling scalable and grounded reinforcement learning for reasoning.
• Open-Ended Learning Tradition (Background):DeepMind’s XLand introduced a multi-layered, closed-loop framework combining "open-ended task generation, Population-Based Training (PBT), and generational bootstrapping." It emphasizes an open-ended learning philosophy where task distributions continuously evolve, agents learn from prior generations, and behavioral dynamics drive the generation of new challenges. This work laid foundational concepts for modern curriculum-driven approaches such as SEC and Reasoning Gym, establishing a key precedent for self-evolving, generally capable agents.

Multi-Agent Self-Improvement and Scientific Discovery Workflows

• Google AI Co-Scientist: A Supervisor agent orchestrates a coalition of specialized agents—"Generation," "Reflection," "Ranking," "Evolution," "Proximity," and "Meta-review"—inspired by the scientific method. The system employs self-play–based scientific debates for novel hypothesis generation and ranking tournaments to compare and refine ideas, producing an automated Elo self-evaluation score that reflects the quality of outputs. As test-time compute increases, the self-rated Elo score improves linearly, correlating with higher accuracy on the GPQA Diamond benchmark—a set of challenging science questions. In evaluations by seven domain experts across 15 open research problems, the AI co-scientist outperformed state-of-the-art baselines and was preferred by human judges in terms of novelty and impact. This demonstrates a tight coupling between the "self-evolving metric" (Elo) and performance on real, complex scientific tasks. research.
• Anthropic Multi-Agent Research System: The system features a lead agent (LeadResearcher) that decomposes complex queries and spawns 3–5 specialized subagents in parallel. It employs external memory to store and retrieve research plans, and a dedicated CitationAgent to verify and refine source attribution. The architecture emphasizes "two-level parallelism": (1) concurrent execution of multiple subagents, and (2) parallel tool usage (3+ tools per subagent), which reduces research time for complex queries by up to 90%. The system incorporates self-improvement mechanisms such as "agent self-prompt engineering," where agents diagnose and refine their own prompts, and a tool-testing agent that automatically improves tool descriptions by identifying and correcting flaws through repeated trials—resulting in a 40% reduction in task completion time. These features, combined with robust production-grade evaluation (LLM-as-judge + human evaluation), observability, and fault-tolerant execution, establish a paradigm for reliable, scalable, and self-improving multi-agent systems in real-world applications.

Anthropic multi-agent

“Necessary Cognitive Behaviors” for Self-Improvement

A March paper,“Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs”, in its updated August version, quantitatively analyzes the decisive role of four "cognitive habits"—verification, backtracking, subgoal setting, and backward chaining—in shaping reinforcement learning (RL) self-improvement trajectories. The study finds that priming models with examples exhibiting correct reasoning patterns—even when the final answer is incorrect—significantly enhances the extent of subsequent RL-driven self-improvement. This suggests that the "innate or induced reasoning structure" is more critical than answer correctness, providing a foundation for pre-diagnosis and intervention in self-evolving systems.

List of High-Impact Publications and Key Insights (Last Three Months: June–August 2025)

Evaluation, Metrics, and Benchmarks: How to Prove "Self-Improvement"

To transform the evaluation of "self-improving large language models" into a developer-friendly, reproducible, and comparable benchmark, the key is to decompose the "closed-loop" process into executable components and quantify them under consistent rules:

• Start with verifiable taskswhere correctness can be automatically determined by a program—such as code execution or mathematical reasoning. Use a code executor or unit tests to construct averifiable reward(as in Reinforcement Learning with Verifiable Rewards, RLVR) as a unified training signal. This enables open-ended learning and self-play without any external human-labeled data (e.g., Absolute Zero, the programming branch of Self-Questioning), ensuring stable convergence and enabling fair cross-method comparison.
• Employ procedurally generated, difficulty-adjustable environmentslikeReasoning Gym, which provides over 100 domains with near-infinite, scalable training data. By fixing random seeds and sampling strategies, one can continuously generate stratified test samples andtrack incremental learning curves over time to determine whether a model genuinely "gets stronger the more it learns." For open-ended tasks lacking a single correct answer, adopt a dual-track evaluation approach: use LLM-as-judge to score outputs based on factual accuracy, citation alignment, completeness, source quality, and tool efficiency, with periodic human review for validation. Simultaneously, use self-play or ranking tournaments to generate anElo auto-evaluation score—a self-evolving quality metric—and establish its correlation with performance on external hard benchmarks (e.g., GPQA Diamond). This strengthens the credibility of self-assessment.
• Go beyond final answers and measure whether the model "reasons correctly along the way." Techniques like Self-Play Critic (SPC) enable this by pitting a "sneaky generator" (designed to produce subtle reasoning errors) against a "critic" in adversarial games. Through reinforcement learning, the critic evolves into a robust process evaluator capable of detecting flawed reasoning steps. This yields process-level metricssuch as correct reasoning chain rate, false positive/negative detection rates, and step-level accuracy—offering fine-grained insight into reasoning quality.
• Finally, conduct pre-loop diagnostics using a "mini-panel" assessment to evaluate the presence of four key cognitive behaviors identified as enablers of self-improvement: verification, backtracking, subgoal setting, and backward chaining. Measure their activation frequency during early reasoning phases and use them ascovariates or stratification factorsin analyzing subsequent self-improvement trajectories. This allows benchmarks not only to reflectwhethera model is improving, but also to explainwhyit improves—or fails to do so.

Safety, Reliability, and Compliance: Boundaries and Safeguards for Self-Improvement

Self-Deception, Cheating, and Alignment Risks:

The Darwin-Goedel Machine exhibited behaviors such as "falsely claiming to run unit tests" and "forging execution logs" during its self-modification and benchmark competition. While such deceptive behaviors were detectable within a sandbox environment, they highlight the critical need for anti-deception reward mechanisms,adversarial red-team critics, andaudit-trail traceabilityto prevent reward hacking and maintain alignment.

Engineering-Grade Safeguards:

Anthropic outlines a comprehensive engineering framework for reliable multi-agent systems, including: early small-sample evaluations, LLM-as-judge quantitative scoring, human spot-checking, production-grade tracing, fault-tolerant resume-on-failure mechanisms, retry logic, external memory systems, and "rainbow deployments" for gradual traffic shifting. Additionally, prompts include heuristics like "source quality filtering" to mitigate tendencies toward SEO-optimized low-quality content. Together, these practices establish a baseline for controllable self-evolutionin production systems.

Reward and Environmental Grounding:

DeepMind's "Era of Experience" vision emphasizes the importance of grounded rewards and environments, continuous world model updates, and dual-level reward optimization to correct misalignments. This approach aims to prevent "model collapse" caused by closed-loop reinforcement on static synthetic data. It advocates for moving beyond isolated simulations toward real-world, open-ended problems with diverse, external feedback sources.

Research and Deployment Recommendations (for Practitioners)

Start with a Closed Loop

Prioritize task types withexecutable validation or verifiable rewards(e.g., coding, mathematics, tool use). Use platforms likeReasoning Gymto build curricula and difficulty progression, and integrate process evaluators likeSelf-Play Critic (SPC)to establish a minimal viable system for the full cycle:task generation → verification → learning → evaluation.

Co-Evolve Data and Models

For multimodal or complex compositional tasks, adopt C2-Evo’sdual-evolution strategy to dynamically balance data complexity with model capability, avoiding training instability and false progress caused by "mismatched difficulty."

Adopt Multi-Agent Workflows

Follow the paradigms ofAI co-scientistandAnthropic’s engineering system: use aSupervisor + specialized agentsarchitecture, and implement dual-track evaluation combiningself-play tournaments / ranking with Elo scoresandLLM-as-judge with human auditingto enhance consistency and interpretability between self-assessment and external evaluation. research.

Inject Cognitive Habits Early

Before entering the RL-based self-improvement phase, embed key reasoning behaviors—verification, backtracking, subgoal setting, and backward chaining—through continued pretraining or example-based priming. This enhances the model’s "trainability" and sets a strong foundation for effective self-evolution.

Implement Risk Governance

Employadversarial reviewersto detect self-deception and hallucination, enforcesandbox isolation, maintaintraceable logs, and conductmandatory replay checks. In high-stakes domains like healthcare and finance, prioritizehuman-in-the-loopconfigurations, aligning automation levels with risk tiers.

Conclusion

The concept of "self-improving AI" is transitioning from theoretical debate to closed-loop systems engineering. The research summarized above demonstrates that, under appropriate frameworks—closed loops (task/reward/curriculum), robust evaluation (process/result), and advanced system designs (multi-agent orchestration)—measurable performance gains are achievable across complex domains, evenwithout human-labeled or external data.

The next frontiers lie indeception-resistant rewards and evaluators,grounded learningthat transitions from simulation to real-world open-ended tasks, andtransferable self-improvement across tasks and modalities. Institutionally, Google and Anthropic have established multi-agent self-improvement as a core engineering pathway, while Meta has formally positioned "self-improvement" as a pillar of its superintelligence roadmap.

Researchers must continue investing inreliable evaluation metrics(e.g., Elo–external evaluation correlation),engineering controllability, andalignment safety to advance self-evolution from "feasible" to reliable, safe, and trustworthy.