MiRL: Error-Targeted Reinforcement Learning for Correcting Terminal Agent Misbehaviors

Runchu TianZaid KhanZun WangJustin Chih-Yao ChenMohit Bansal
University of North Carolina at Chapel Hill
Paper Code 🤗Model 🤗Data 🤗Trajectories BibTeX
MiRL: a misbehavior-driven RL recipe

MiRL starts rollouts from misbehavior prefixes and rewards rollouts that pass the tests and avoid the misbehavior.

Abstract

Error-targeted learning gives a legible way to improve a model: identify specific weaknesses of the model, train the model to fix these weaknesses, and check whether the weaknesses are fixed. We study how to apply error-targeted learning to terminal agents, which solve long-horizon tasks specified in natural language by interacting with a command shell. Existing error-targeted methods are designed for single-turn tasks, and existing analyses of agent failures are used to evaluate and debug agents rather than to train them. We propose Misbehavior-driven RL (MiRL), which first profiles the errors of a policy to find generalizable misbehaviors: kinds of wrong steps that recur across many environments. MiRL then trains the policy with RL, starting rollouts at the steps where these misbehaviors occur and rewarding a rollout only if it passes the tests and avoids the misbehavior. On Qwen3.5-4B, eight misbehaviors account for 52.1% of wrong steps. After training with MiRL, the policy repeats the measured misbehaviors 43% to 57% less often on average on unseen prefixes. MiRL outperforms TMax-4B on Terminal-Bench Lite (51.8% vs. 47.4%) and matches it on Terminal-Bench 2.1 (21.3% vs. 20.6%), while using 35% of its rollouts. These results show that error-targeted learning can be applied to realistic, long-horizon terminal tasks.

Motivation

Error-targeted learning for terminal agents

A terminal agent takes dozens of steps per task, and a failed task is often caused by one or two wrong steps, such as reporting that the tests pass without running them. This creates two difficulties.

FirstMany wrong steps are specific to one task, so fixing them one at a time does not help on other tasks; training should instead target generalizable misbehaviors, kinds of wrong steps that recur across many tasks.
SecondTraining from the start of a task spends most rollouts on steps the agent already performs correctly, and the agent may never reach the state where the misbehavior occurred.

We therefore ask:

1How can we identify the generalizable misbehaviors of a terminal agent?Error profiling → 2How can we train the agent directly at the steps where they occur?MiRL →
Method

Error profiling and MiRL

A terminal environment consists of an instruction that states a user requirement, a container in which the agent runs shell commands, and a test suite that checks the final state of the container.

Name
Deduplicate a photo library
Instruction
Find every photo in backup/ that is byte-identical to one in photos/, move it into dupes/, and list it in dupes/report.csv.
Container
$ tree -L 1 ~/lib
├── photos/ (412 .jpg)
├── backup/ (389 .jpg)
└── dupes/ (empty)
Tests
dupes/: all and only true duplicates
report.csv: lists every moved file

An example terminal environment. The agent must move duplicate photos from backup/ into dupes/ and list them in report.csv; the tests check the result.

Error profiling
RL training

(a) Error localization

At each step of a rollout, error profiling goes through four stages.

  1. (1) The policy proposes its next step from the current prefix.
  2. (2) The error localizer checks the step with its prefix, flags it only if it is likely to lead to failure, and writes a short explanation.
  3. (3) For a flagged step, the teacher writes a correction from the same prefix, so that the rollout can continue without being pushed into incorrect states. The correction is checked in the same way as a step of the policy.
  4. (4) The accepted step, either the policy's own step or a correction from the teacher, is executed, and the rollout returns to (1) for the next step.

During this process, each error step is recorded with its prefix and the error localizer's explanation.

(b) Error taxonomy

First, the annotator reads batches of error steps, together with the error localizer's explanations, and proposes error types for each batch, with a name and a definition for each type. We repeat this on several batches and keep the types that recur across batches.

This gives a fixed set of error types. The annotator then labels every error step with one or more types from this set, or with none if it fits no known type. Finally, we keep the error types that cover at least a fixed share of the error steps and treat the others as a long tail of rare types.

(c) Misbehavior prefixes

However, a live rollout cannot start from a prefix directly. To continue from step k, the policy needs the container state at step k, but the recorded data keep only the prefix and not the container, since saving a snapshot of the container at every step would require substantial storage.

We therefore need to reconstruct the state from the prefix: we rerun the commands of the prefix step by step in a fresh container of its environment, and keep the prefix only if every command returns exactly the same output as in the prefix. Many containers are flaky: the same command may produce different outputs depending on time, randomness, or network conditions. Such prefixes cannot reach the required state, so we discard them.

For these eight misbehaviors, we sample 80 misbehavior prefixes each, which gives 640 prefixes. After the rerun check, 219 prefixes remain (34%), from 189 environments.

(d) MiRL training

To fix the misbehaviors in a more targeted and efficient manner, MiRL instead starts each group of rollouts from a pre-existing trajectory prefix where the policy exhibited a misbehavior. The policy then continues from the prefix until the rollout ends.

The behavior judge reads the first m steps that the policy takes after the prefix, together with the definition of the misbehavior of that prefix. It gives re = 1 if the policy avoids the misbehavior, and re = 0 otherwise.

MiRL combines the two rewards as r = ro ∧ re, so a rollout gains reward only when the tests pass and the misbehavior is avoided. In this way, MiRL avoids reinforcing trajectories that pass the tests but still contain the misbehavior.

Overview. (a) The error localizer flags wrong policy steps, and the teacher corrects them; (b) the annotator groups error steps into types, where a few misbehaviors dominate; (c) prefixes that rerun exactly are kept for live RL rollouts; (d) rollouts start from them and gain reward only if they pass the tests and avoid the misbehavior.

Error profiling

A few error types dominate the distribution

Error profiling on the 2,017 environments yields 12,746 error steps. The eight most frequent error types cover 52.1% of all error steps.

Error steps concentrate in a few misbehaviors, followed by a long tail of environment-specific types; the top four appear across many environments.

Results

Main results

MiRL outperforms TMax-4B.

MiRL reaches 51.8% on TB-Lite and 21.3% on TB-2.1, compared with 47.4% and 20.6% for TMax-4B on our evaluation hardware: it gains 4.4 more points on TB-Lite and matches TMax-4B on TB-2.1.

MiRL outperforms all baselines trained on the same number of environments.

MiRL outperforms the best of these baselines by 4.2 points on TB-Lite and 2.2 points on TB-2.1.

Terminal task performance: mean (%) and standard error over 3 runs. Dashed boxes are the scores reported in the TMax paper.

Ablations

We ablate MiRL by adding its three parts one at a time to TMax-189.

Ablations of MiRL. Starting from TMax-189, each bar adds one part: environment selection, prefix start, and the behavior reward re.

Analysis

MiRL reduces misbehavior frequency.

For example, “Makes false claims about tests” drops from 26.7% to 8.1% under Gemini and from 29.8% to 11.7% under Qwen. Since the rate drops under both judges, MiRL likely reduces these misbehaviors rather than only learning to satisfy its own judge.

Misbehavior rate (%): the share of 16 samples that repeat the misbehavior on unseen prefixes. Judges: Gemini 3.8 Flash (used in training) and Qwen3.8-27B.

Pass@k: Sharpening or learning?

A common critique of RL is that it sharpens the output distribution of the base model rather than teaching it new capabilities. TMax-4B shows this pattern: it improves over Qwen3.5-4B at pass@1, but the base model catches up at about k = 11 and is ahead at k = 16. In contrast, MiRL outperforms the base model at every k up to 16 by a large margin, and reaches 39.3% at k = 16.

Pass@k on TB-2.1 with 16 runs per task. (a) Unbiased pass@k of Qwen3.5-4B, TMax-4B, and MiRL. (b) Pass rate of each task over the 16 runs.

MiRL is more data-efficient than TMax-4B.

MiRL uses fewer terminal environments, and only 35% of the rollouts and 34% of the input tokens of TMax-4B. Although MiRL pays a model API fee for the LLMs in error profiling and for the behavior judge in RL training, which TMax-4B does not, its total cost is still lower, at 56% of that of TMax-4B, while it scores 4.4 points higher on TB-Lite and matches it on TB-2.1.

Training resources of MiRL relative to TMax-4B, up to the best checkpoint rather than the full training run. Checkered segments are error profiling and solid segments are RL training. GPU time is in A100 hours.

Citation

BibTeX

@misc{tian2026mirlerrortargetedreinforcementlearning,
      title={MiRL: Error-Targeted Reinforcement Learning for Correcting Terminal Agent Misbehaviors},
      author={Runchu Tian and Zaid Khan and Zun Wang and Justin Chih-Yao Chen and Mohit Bansal},
      year={2026},
      eprint={2610.35697},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2610.35697},
}