From VLM Planners to Robotic Agents:
Self-Evolving Harnesses for History-Dependent Manipulation

Tingwen Du¹ · Haoyu Guo^2,† · Hanxue Zhang³ · Xuanxu Wang³ · Junting Dong² · Chao Gao²
Mingfei Han⁴ · Tao Lu² · Mulin Yu² · Chunhua Shen⁵ · Xiaojun Chang^1,‡ · Zhihui Li¹

¹ University of Science and Technology of China · ² Shanghai Artificial Intelligence Laboratory
³ Shanghai Jiao Tong University · ⁴ Mohamed bin Zayed University of Artificial Intelligence · ⁵ Zhejiang University
† Project leader ‡ Corresponding author

Paper ↗arXiv ↗HF paper ↗Code ↗Explore the cases ↓

TL;DR: SERA explores how an editable, self-evolving harness turns a VLM planner into a robotic agent for context management and history-dependent manipulation.

From planner to agent for history-dependent manipulation. Similar observations can require different decisions depending on task progress. SERA uses an editable harness to organize the VLM as an agent that maintains task context from observation and interaction history. Execution-driven self-evolution refines the harness using feedback from simulation rollouts initialized with task configurations from the benchmark’s training split. This follows the same train–test separation used by fine-tuning methods to support comparison. The selected harness is fixed before held-out testing.

Abstract

Vision-language-action (VLA) models achieve strong performance across embodied manipulation tasks by unifying visual understanding, language conditioning, and action generation. However, long-horizon tasks can involve non-Markovian dependencies: successful decisions may require retaining task progress and past interaction outcomes, while partial observability and execution errors compound over time. Existing work has begun to address these challenges by augmenting foundation-model policies with memory and by introducing hierarchical architectures that separate high-level planning from low-level execution. We organize the high-level VLM as a robotic agent through an editable harness to maintain task context and improve planning through execution experience. We instantiate this approach as SERA (Self-Evolving Robotic Agent), with a fixed VLA executor providing low-level control. The harness maintains and updates task context using observations and execution feedback to guide subsequent planning. Self-evolution uses environment rollouts to propose and evaluate harness revisions, with the selected configuration fixed before held-out testing. On RoboMME, SERA achieves 51.38% overall success with autonomous selection, compared with 44.38% without self-evolution. Our exploration suggests that agentic planning is a promising direction for history-dependent robotic manipulation.

Method

SERA organizes a high-level VLM as an agent through an editable harness, with a fixed VLA executor for low-level control. More specifically, the agent predicts a subtask describing what to do and, when needed, points in the observation image specifying where to act. These grounding points identify the object or destination, giving the VLA an instruction with explicit target coordinates.

Experiments

RoboMME Main Results

Task-averaged success rates (%; higher is better). Bold and underlined values denote the best and second-best scores across all listed methods.

Method	Counting	Permanence	Reference	Imitation	Average ↑
VLA baselines
π_0.5	28.78	17.00	17.17	8.78	17.93
π_0.5 w/ action history	29.09	22.75	14.59	11.17	19.40
Memory-augmented policies
SAM2Act	35.33	26.00	16.83	7.33	21.37
FrameSamp+Modul	65.22	25.11	36.33	51.39	44.51
Hierarchical policies
SimpleSG+QwenVL	44.61	19.61	25.22	26.56	29.00
GroundSG+Gemini	12.75	15.75	11.25	6.50	11.56
GroundSG+QwenVL	38.00	39.34	31.56	21.89	32.70
MemER	48.83	53.17	38.00	29.50	42.38
SERA (Ours)	49.00	57.00	69.50	30.00	51.38

Execution-Driven Self-Evolution

An initial harness may contain incomplete planning guidance, imperfect assessment criteria, or unsuitable control rules. Feedback obtained from execution provides an opportunity to identify where these limitations affect task performance and to revise the corresponding parts of the harness. With execution-driven self-evolution, SERA improves overall success from 44.38% to 51.38%, a gain of 7.00 percentage points.

Paper results figure: success rates by task suite, a VideoPlaceOrder execution with a predicted placement point marked by a yellow cross, and a PatternLock harness-revision example. — **Self-evolution results and complementary execution cases.** **(a)** RoboMME success rates before and after harness revision. **(b)** VideoPlaceOrder: the base agent retains the target identity after a swap and places the cube. The yellow cross in the third image of panel (b) illustrates grounding: the predicted placement point. **(c)** PatternLock: self-evolution revises direction guidance and the action candidate set. VideoPlaceOrder is a test episode; PatternLock is a development repair example.

Examples

Five test examples

TASK

Watch the task demonstration Input to the agent

The second target changes position.

The demonstration visits multiple targets, then swaps two of them. Retain the identity of the second target, rather than its earlier screen position.

HARNESS DETAIL

Case scope & interpretation

Runs have independent timelines. Each method retains its own playback position when switching; round numbers are not matched physical states. Context excerpts are recorded model assessments.

Citation

@misc{du2026sera,x
  title = {From VLM Planners to Robotic Agents: Self-Evolving Harnesses for History-Dependent Manipulation},
  author = {Du, Tingwen and Guo, Haoyu and Zhang, Hanxue and Wang, Xuanxu and Dong, Junting and Gao, Chao and Han, Mingfei and Lu, Tao and Yu, Mulin and Shen, Chunhua and Chang, Xiaojun and Li, Zhihui},
  year = {2026},
  note = {Preprint}
}

From VLM Planners to Robotic Agents:Self-Evolving Harnesses for History-Dependent Manipulation