AA

Agentic Abstention / NeurIPS 2026

Agentic Abstention:
Do Agents Know When to Stop
Instead of Act?

Knowing when to stop is part of knowing how to act.

Han Luo1,2,* Bingbing Wen1,* Lucy Lu Wang1,3
1 University of Washington2 University of Leeds3 Allen Institute for Artificial Intelligence

* Equal contribution

Read the paper Code & data NeurIPS Explore the results Cite
28K+ instructionsA study of when agents should stopSolvable and abstain-warranted tasks
3 environmentsWeb · Terminal · Interactive QA13 LLM systems and 2 agent scaffolds overall
26.7 → 57.4%Timely abstention with CONVOLVELlama-3.3-70B · Held-out WebShop subset
// 01

The next action
can be to stop.

An agent can keep searching, browsing, or running commands—even when its goal cannot be achieved.

Some requests are underspecified or based on a false premise. Others appear feasible until a tool reveals a missing prerequisite or an unavailable target. Continuing can waste resources or lead to an unsupported answer.

Agentic abstention is a sequential decision problem. At each step, an agent may answer, gather more information, or abstain. The key question is when the evidence warrants stopping.

Request-based

The request reveals the blocker.

Ambiguity, conflicting requirements, or a false premise can make abstention warranted from the beginning.

Environment-based

Interaction reveals the blocker.

A missing item, file, dependency, or capability becomes apparent only after the agent examines its environment.

Request + observations
ActGather useful evidence.
AnswerComplete a supported goal.
AbstainExplain why progress should stop.
// 02

Three environments.
One decision boundary.

We adapt web shopping, terminal tasks, and question answering to study both whether agents abstain and when they do so.

The collection includes 28,350 instructions across three environments. Evaluation coverage varies by experiment: WebShop supports broad model comparisons; terminal and QA experiments examine selected models and scaffolds.

01 / WebShop

Web shopping

1,000 tasks

500 solvable tasks and 500 abstention tasks: 249 request-based cases and 251 missing-target cases.

Subjective preference · Underspecified intent · False premise or contradiction · Missing target

02 / Terminal-Bench 2.0

Terminal execution

277 tasks

89 solvable tasks and 188 abstention tasks: 167 request-based cases and 21 missing-prerequisite cases.

False premise or contradiction · Underspecified intent · Missing prerequisite

03 / AbstentionBench adaptation

Interactive QA

27,073 questions

16 source datasets, adapted to let agents answer, abstain, or search a fixed Wikipedia corpus.

Answer unknown · False premise · Subjective · Underspecified context · Underspecified intent

Dataset sizes describe the constructed collection, not a common evaluation set for every model. See the paper for scenario-specific coverage and retrieval settings.

// 03

Abstaining eventually
is not abstaining
on time.

Many agents keep interacting after abstention becomes warranted. Some never abstain within the allotted budget.

We measure Abstention Recall @K (AbsRec@K) from the earliest step at which abstention is warranted. K=1 requires stopping at that step; K=2 allows one additional step of delay.

Premature abstentions do not count as correct. Overall recall is measured separately within the configured interaction budget and need not equal AbsRec@10.

Metrics and figures updated November 4, 2026.

Abstention recall curves across Web, Terminal, and QA. Recall generally rises when more delay is allowed; timely recall at K=1 is often much lower.
Abstention recall across three environments. K counts decision opportunities from the earliest warranted step. Terminal comparisons use GPT-5.4-mini with two scaffolds. QA models have different evaluation coverage; their curves are not a controlled leaderboard.

Explore the reported results

How much does timing matter?

Choose an environment and the allowed abstention delay.

K = 1Strictly timely

WebShop · 500 abstention tasks per model.

Rates use the same numeric inputs as the figure above. K measures the window for a correct abstention, not the number of independent attempts. Results should be interpreted within each experiment’s evaluation scope.

Model scale, reasoning, and agent scaffolds

More capable models are not consistently better at timely abstention. Reasoning can improve timely recall while reducing overall recall, and changing the agent scaffold can alter behavior even with the same base model.

Reasoning comparisons show timely and overall recall can change in different directions on Web and Terminal tasks.

The paper’s reasoning comparison uses Qwen3-235B for Web and GPT-5.4-mini for Terminal. These experiments have their own evaluation scopes.

// 04

Learn when to stop.
Keep the model fixed.

CONVOLVE turns interaction trajectories into a reusable playbook of stopping rules.

A reflection agent reviews complete trajectories to identify evidence that warranted abstention and actions that delayed it. A curator converts those lessons into structured playbook updates, which are appended to the agent’s context on later tasks.

The method changes the agent’s context without updating model parameters.

01 / Execute

Collect a trajectory

Observe the request, tool calls, evidence, and stopping decision.

02 / Reflect

Find the stopping signal

Identify when abstention became warranted and what delayed it.

03 / Curate

Update the playbook

Reuse concise lessons as context for future interactions.

Llama-3.3-70B · Separate held-out WebShop abstention subset · Percentages
MethodTimely recall ↑Overall recall ↑SPL ↑
Base model26.783.255.3
In-context learning55.197.077.2
CONVOLVE · lessons from 8B55.399.076.4
CONVOLVE · lessons from 70B57.4100.078.9

Source: the paper’s CONVOLVE WebShop results table. Timely recall is AbsRec@1; SPL rewards correct abstention with shorter interaction paths. The held-out set contains 101 abstention tasks; trained variants use 20 trajectories, with 8 validation examples. The 26.7% baseline comes from this separate held-out set, not the 500-task comparison above.

These results evaluate abstention on tasks where stopping is correct. They do not establish preservation or improvement of solvable-task completion.

// 05

Read the study.
Build on the work.

The paper presents the task construction, timing-aware metrics, evaluation protocols, and additional analyses. The repository provides the research code and data resources.

PaperAgentic Abstention Code & dataResearch repository Related workHERA: harness–environment co-evolution
// 06

Cite this work.

Download BibTeX 
@misc{luo2026agenticabstention,
  title={Agentic Abstention: Do Agents Know When to Stop Instead of Act?},
  author={Han Luo and Bingbing Wen and Lucy Lu Wang},
  year={2026},
  eprint={2606.28733},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  url={https://arxiv.org/abs/2606.28733}
}