ReAct Paper Explained: Reasoning and Acting in LLMs
Introduction
By 2022, large language models could reason through problems step by step with chain-of-thought promptingClick to read tutorial: Chain-of-Thought Prompting Paper Explained, but only using what they already knew - and when their knowledge was wrong or missing, the reasoning confidently built on hallucinatedClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations facts. Separately, researchers had used language models to choose actions in interactive environments, but without explicit reasoning. The 2022 paper ReAct: Synergizing Reasoning and Acting in Language Models, from Princeton and Google, combined the two: the model alternates between thinking about what to do, acting - for example by searching Wikipedia - and reading the observation that comes back. This thought-action-observation loop is the pattern behind today's AI agents. This post explains the ReAct paper with diagrams. We will compare it with reasoning-only and acting-only approaches, walk through a ReAct trace, go through the results and failure modes, and build a ReAct agent loop in Java.The paper: ReAct: Synergizing Reasoning and Acting in Language Models
Authors: Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao (Princeton University, Google Research)
Published: October 2022 - ICLR 2023
Authors: Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao (Princeton University, Google Research)
Published: October 2022 - ICLR 2023
Reason only, act only, or both
Three ways of prompting a model, as compared in the paper.
- break a task into steps and track its progress ("I need to find X, then compare it with Y"),
- pull out the important part of an observation ("The paragraph says it was released in 2018"),
- use common sense or arithmetic on what it has found, and
- notice when something went wrong and change its plan ("The search found nothing; I should search for a broader term").
The ReAct loop and the paper's setup
The ReAct loop. Every thought, action and observation is added to the prompt for the next step.
- HotpotQA - questions which need facts from two or more Wikipedia articles, and
- FEVER - checking whether a claim is supported or refuted by Wikipedia,
search[entity], which returns the first sentences of the entity's page, lookup[string], which finds the next sentence on the page containing the string, and finish[answer]; and -
- ALFWorld - a text-based game of household tasks, such as "put a clean mug on the desk", and
- WebShop - buying a product matching a description on a simulated shopping website.
Question: How many years after Java was first released did Java 11 come out? Thought 1: I need the year Java was first released, then the year of Java 11. Action 1: search[Java (programming language)] Observation 1: Java is a programming language ... It was first released by Sun Microsystems in 1995. Thought 2: Java was first released in 1995. Now I need Java 11's release year. Action 2: search[Java 11] Observation 2: Java 11 was released in September 2018 and was a long-term support release. Thought 3: 2018 - 1995 = 23, so Java 11 came out 23 years later. Action 3: finish[23]
Results
- Interactive tasks: on ALFWorld and WebShop, ReAct prompted with only one or two examples beat imitation learning and reinforcement learning methods trained on thousands of examples, by an absolute 34% and 10% in success rate. Act-only prompting did much worse, showing that the thoughts mattered.
- Fact verification: on FEVER, ReAct (60.9% accuracy) beat chain-of-thought prompting (56.3%).
- Question answering: on HotpotQA, ReAct alone was slightly behind chain-of-thought in accuracy, because a bad search result could send it off track. But its answers were far more grounded: in the authors' analysis, hallucination caused over half of chain-of-thought's failures, and only a small fraction of ReAct's. The best results came from combining the two - falling back from ReAct to chain-of-thought with self-consistency when ReAct did not finish within a set number of steps, and the other way round when the chain-of-thought answers disagreed.
- Interpretability and control: because the reasoning is visible, a person can read why the model acted, and even edit a thought mid-trajectory to correct its course.
- Fine-tuning: smaller PaLM models fine-tuned on a few thousand ReAct trajectories outperformed much larger models that were only prompted.
Failure modes
The paper's error analysis is still a good checklist for anyone building agents -- Repetitive loops: the model sometimes repeats the same thought and action over and over, unable to break out.
- Uninformative observations: a search which returns nothing useful can derail the reasoning, and the model may fail to recover.
- Reasoning errors: the model can still misread an observation or draw a wrong conclusion from correct facts.
A ReAct agent loop in Java
The program below implements the ReAct loop for the Java trace above. The tools are plain Java functions, registered by name in a map ofFunction<String, String>. To keep the example runnable without an API key, the language model is replaced by a scripted stand-in which returns the steps of the trace above, with one change - it uses a calculate tool for the subtraction instead of doing it in a thought; in a real agent, callModel() would send the prompt to an LLM API, for example with Java 11's HttpClient. The loop itself - parse the action, run the tool, append the observation, repeat until finish or a step limit - is the same either way.
A ReAct agent loop with tools
//ReActAgent.java
import java.util.*;
import java.util.function.Function;
import java.util.regex.*;
public class ReActAgent {
// Tools: name -> function from argument to observation
static final Map<String, String> WIKI = Map.of(
"java (programming language)", "Java is a programming language. It was first released by Sun Microsystems in 1995.",
"java 11", "Java 11 was released in September 2018 and was a long-term support release.");
static final Map<String, Function<String, String>> TOOLS = Map.of(
"search", q -> WIKI.getOrDefault(q.toLowerCase(), "Could not find [" + q + "]. Try a different search."),
"calculate", expr -> {
String[] p = expr.split("-");
return String.valueOf(Integer.parseInt(p[0].trim()) - Integer.parseInt(p[1].trim()));
});
// Stand-in for the LLM: returns the next "Thought ... Action ..." for the current prompt
static final List<String> SCRIPT = List.of(
"Thought: I need the year Java was first released, then the year of Java 11.\nAction: search[Java (programming language)]",
"Thought: Java was first released in 1995. Now I need Java 11's release year.\nAction: search[Java 11]",
"Thought: Java 11 was released in 2018. I will compute the difference.\nAction: calculate[2018 - 1995]",
"Thought: The difference is 23 years.\nAction: finish[23]");
static int calls = 0;
static String callModel(String prompt) { return SCRIPT.get(calls++); }
static final Pattern ACTION = Pattern.compile("Action: (\\w+)\\[(.*)\\]");
public static void main(String[] args) {
String prompt = "Question: How many years after Java was first released did Java 11 come out?\n";
final int maxSteps = 6;
for (int step = 1; step <= maxSteps; step++) {
String output = callModel(prompt);
System.out.println(output);
Matcher m = ACTION.matcher(output);
if (!m.find()) { prompt += output + "\nObservation: Invalid action format.\n"; continue; }
String tool = m.group(1), argument = m.group(2);
if (tool.equals("finish")) {
System.out.println("=> Final answer: " + argument + " (after " + step + " steps)");
return;
}
String observation = TOOLS.containsKey(tool) ? TOOLS.get(tool).apply(argument) : "Unknown tool " + tool;
System.out.println("Observation: " + observation);
prompt += output + "\nObservation: " + observation + "\n"; // the trajectory grows each step
}
System.out.println("=> Stopped: no answer within " + maxSteps + " steps");
}
}
Thought: I need the year Java was first released, then the year of Java 11. Action: search[Java (programming language)] Observation: Java is a programming language. It was first released by Sun Microsystems in 1995. Thought: Java was first released in 1995. Now I need Java 11's release year. Action: search[Java 11] Observation: Java 11 was released in September 2018 and was a long-term support release. Thought: Java 11 was released in 2018. I will compute the difference. Action: calculate[2018 - 1995] Observation: 23 Thought: The difference is 23 years. Action: finish[23] => Final answer: 23 (after 4 steps)
Explanation of the code
TOOLSmaps each action name to aFunction<String, String>. Adding a tool, such as a database query or a REST call, means adding one entry - this is essentially what agent frameworks such as LangChain4j and Spring AI do with annotated Java methods.- The regular expression extracts the tool name and argument from the model's text. Modern LLM APIs offer tool calling, where the model returns a structured tool call in JSON instead, which is more reliable than parsing text.
- After each step, the model's output and the observation are appended to
prompt, so the next call sees the whole trajectory - the model has no memory of its own between calls. Long trajectories therefore fill the context windowClick to read tutorial: LLM Tokens, Context Windows, Embeddings and Hallucinations, which is why agents also limit or summarize old steps. maxStepsguards against the repetitive loops described above. Thesearchtool also returns a helpful message when nothing is found, so a model can recover by trying a different search.
ReAct's legacy
ReAct's loop is the core of most AI agents today: coding agents that read files, run tests and fix errors; research agents that search and read the web; and assistants that call company APIs. What changed since 2022 is mostly the plumbing - native tool calling in model APIs, standard protocols such as the Model Context Protocol (MCP) for connecting tools, and models trained specifically to plan and use tools - while the idea of interleaving reasoning with actions and observations remains the same. It also pairs naturally with RAGClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation: an agent can decide for itself when, and what, to retrieve, using the same retrieval steps described in RAG architecture explainedClick to read tutorial: RAG Architecture Explained: Chunking, Embeddings, Retrieval. And the step-by-step chains that the loop is built from are the subject of prompt engineering techniquesClick to read tutorial: Prompt Engineering Techniques Explained with Examples.Summary
In this post we went through the ReAct paper. We saw how ReAct interleaves thoughts, which plan and track progress, with actions, which fetch real information from tools or an environment, and observations, which feed the results back into the prompt. We compared it with reasoning-only chain-of-thought and acting-only approaches, followed a ReAct trace for a multi-hop question, and went through the results on HotpotQA, FEVER, ALFWorld and WebShop, along with the failure modes found in the paper. Lastly, we built the ReAct loop in Java, with tools as functions, a step limit, and a growing trajectory. In the next post, we will see how a model can be trained to reason, in the DeepSeek-R1 paperClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL.Continue Reading: AI and LLM Papers Explained
DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained
DeepSeek-R1 Paper Explained: Reasoning via RLClick to read tutorial: DeepSeek-R1 Paper Explained: Reasoning via RL InstructGPT Paper Explained: Training LLMs with RLHFClick to read tutorial: InstructGPT Paper Explained: Training LLMs with RLHF Attention Is All You Need: The Transformer Paper ExplainedClick to read tutorial: Attention Is All You Need: The Transformer Paper Explained BERT Paper Explained: Bidirectional TransformersClick to read tutorial: BERT Paper Explained: Bidirectional Transformers Scaling Laws for Language Models (and Chinchilla) ExplainedClick to read tutorial: Scaling Laws for Language Models (and Chinchilla) Explained GPT-3 Paper Explained: Language Models Are Few-Shot LearnersClick to read tutorial: GPT-3 Paper Explained: Language Models Are Few-Shot Learners RAG Paper Explained: Retrieval-Augmented GenerationClick to read tutorial: RAG Paper Explained: Retrieval-Augmented Generation LoRA Paper Explained: Low-Rank Adaptation of LLMsClick to read tutorial: LoRA Paper Explained: Low-Rank Adaptation of LLMs Chain-of-Thought Prompting Paper ExplainedClick to read tutorial: Chain-of-Thought Prompting Paper Explained