CaRT: Teaching LLM Agents to Know When They Know Enough

Grace Liu*1 , Yuxiao Qu*1 , Jeff Schneider1 , Aarti Singh1 , Aviral Kumar1 ,
1Carnegie Mellon University, *Equal Contribution
arXiv Code

Outline

The Problem | Abstract | CaRT Method | Results | Ablations | Findings

The Termination Problem

Hi! What symptoms are you facing today?

Chest pain, trouble breathing, nausea, tingling in my arms.

Final Diagnosis: Heart Attack ✗

Hi! What symptoms are you facing today?

Chest pain, trouble breathing, nausea, tingling in my arms.

Have you recently been under stress or experienced anxiety?

Yes, I've had episodes where my heart races, I feel like I'm losing control, and I'm terrified I'm going to die.

Final Diagnosis: Panic Attack ✓

Hi! What symptoms are you facing today?

Chest pain, trouble breathing, nausea, tingling in my arms.

Have you recently been under stress or experienced anxiety?

Yes, I've had episodes where my heart races, I feel like I'm losing control, and I'm terrified I'm going to die.

...

Do you have a history of heart disease in your family?

Yes, my father had a heart attack at age 50.

Final Diagnosis: Heart Attack ✗

Simplified example of terminating information gathering in the medical diagnosis domain. Left: Terminating too early leads to incorrect diagnosis. Middle: Optimal termination occurs after gathering sufficient information. Right: Terminating too late results in unnecessary additional interaction.

Deciding when to stop gathering information is challenging because it requires predicting the expected future utility of continuing under the model's current policy. We build an approach to imbue LLMs with the ability to stop or "terminate" their internal thinking processes and/or environment interaction at the right point for maximal performance, without wasting computation or interaction.

Abstract

Many tasks require machine learning models to strategically gather relevant information over multiple rounds of interaction before actually acting on a task. Strategic information gathering requires models to know not only how to effectively acquire information, but also when to stop gathering information and make a decision, in order to avoid overthinking or getting derailed when acting. In this paper, we formalize this problem and introduce Counterfactuals and Reasoning for Termination (CaRT), an approach for teaching LLMs when to stop seeking information. To appropriately learn when to terminate, CaRT fine-tunes LLMs using counterfactual pairs of trajectories, one where termination is appropriate and a minimally modified version of the same trajectory where it is not. It trains the LLM to explain the rationale for the termination decision in either case via verbal reasoning, and imbues this capability into the base LLM via fine-tuning. We instantiate CaRT in two domains: interactive medical diagnosis and math problem solving. In both domains, we find that CaRT improves the efficiency of information gathering and task success rate compared to other fine-tuning methods.

CaRT: Counterfactuals and Reasoning for Termination

Our approach combines two key components to teach LLMs effective termination behavior:

Component 1: Hard Negative Counterfactuals

For each trajectory where there is an optimal termination point (success rate ≥ 50%), we create a minimally modified counterfactual where termination would be inappropriate (success rate < 30%). This isolates the exact piece of information that differentiates success from failure.

Original (Should Terminate)

Hi! What symptoms are you facing?

Chest pain, trouble breathing, nausea, tingling.

Have you been under stress or anxiety?

Yes, I've had episodes where my heart races, I feel like I'm losing control, and I'm terrified I'm going to die.

Decision: Terminate, Diagnosis: Panic Attack

Counterfactual (Should NOT Terminate)

Hi! What symptoms are you facing?

Chest pain, trouble breathing, nausea, tingling.

Do you have any family history of respiratory disease?

No, I don't have any family history of respiratory disease.

Decision: Continue

Simplified counterfactual pair example. The original trajectory (left) contains sufficient information to provide a diagnosis. The counterfactual (right, modified Q&A in gray) replaces this with a question that did not acquire the information needed to provide a diagnosis, making termination inappropriate.

Component 2: Verbal Reasoning to Explain Termination

We augment each training example with explicit reasoning traces that explain why the current state warrants the termination decision. This reasoning serves as an implicit value function, helping the model predict whether to terminate or continue by reasoning about the implications before acting.

Example Reasoning Trace

<think> Based on the conversation so far, we have established that the patient is experiencing chest pain, difficulty breathing, and feelings of panic. These symptoms, combined with the absence of cardiac risk factors and presence of acute anxiety, strongly suggest a panic attack rather than a cardiac event. We have sufficient information to make a confident diagnosis. </think>

Decision: Terminate, Diagnosis: Panic Attack

Training Objective

We perform supervised fine-tuning (SFT) on counterfactual pairs with reasoning traces. This effectively optimizes for a policy that maximizes task reward while minimizing the number of information-seeking turns.

Results

CaRT Termination Behavior

A schematic illustration of termination behavior with and without CaRT. While LLMs typically fail to recognize the best points to stop thinking or questioning, our approach CaRT teaches them to correctly identify when they have gathered sufficient information to solve a task, avoiding wasted computation on easy problems and making steady progress on hard problems.

Medical Diagnosis

CaRT outperforms both the base model and supervised fine-tuning baselines on medical diagnosis tasks. The model learns to terminate at optimal points, achieving higher success rates while asking fewer questions.

Medical Diagnosis Results
Medical Diagnosis Results

CaRT performance on medical diagnosis. (a) Results on holdout in-distribution data showing CaRT outperforms the base model and SFT baseline across all metrics. (b) CaRT maintains superior performance on out-of-distribution dermatology diagnosis tasks.

Math Problem Solving

In the math domain, CaRT achieves higher success rates while using fewer tokens compared to baseline approaches. The model learns when to stop reasoning and provide an answer, avoiding unnecessarily long thinking traces.

Math Reasoning Results

CaRT performance on AIME2025 math problems. CaRT outperforms the base model and SFT baseline, achieving higher success rates with fewer output tokens.

Ablation Studies

Ablation Study Results

Ablation study on medical diagnosis. We ablate counterfactual training data and reasoning augmentation, showing that both components contribute to improved termination performance. Counterfactuals provide the largest gains.



Termination Rate Curves

Analysis of model termination rates. We find that base models maintain low rates regardless of context and the baseline SFT learns to increase termination rate as conversation length increases (top row). Adding counterfactuals enables the model to learn to terminate when sufficient information is gathered (middle row), and reasoning smooths these curves while maintaining information sensitivity (bottom row).

Key Findings

  • Counterfactual training examples with explicit reasoning help models learn good termination behavior: By training on counterfactual examples of termination with explicit reasoning, LLMs learn to recognize when they have acquired sufficient information to solve the task.
  • The approach generalizes across domains: CaRT demonstrates strong performance across both medical diagnosis tasks and mathematical problem solving, achieving higher accuracy with fewer tokens and interactions.