KARL: Knowledge Agents via Reinforcement Learning

🍞 Hook: Imagine you’re doing a school project with a giant box of mixed papers—notes, articles, and charts. You can’t just guess your answers; you must find the right pages, pull out the facts, do some math, and then write a clear report with citations. That’s hard, right?

🥬 The situation before this work: What it is: For years, AI models got good at quick facts, math puzzles, or code, but they struggled with grounded reasoning—tasks where answers must be discovered outside the model’s memory by searching, reading, calculating, and citing evidence. How it worked: Typical systems used one-size-fits-all prompts or relied on web tools that change over time, making results hard to compare and not always reliable on private data. Why it mattered: Companies in finance, health, law, and engineering hold huge private document collections. If an AI can’t reliably search and reason over these, it can’t handle their most valuable work.

🍞 Anchor: Think of trying to write a science report using only your brain and no textbook. You might remember some facts, but to be accurate, you must look things up and prove them. That’s what grounded reasoning asks AI to do.

Now, the problem researchers faced:

• Deep research demos looked impressive on public web tasks but didn’t prove the skills would transfer to enterprise settings with fixed corpora and privacy needs.
• Benchmarks were narrow: some measured multi-hop questions, others finance math, but few tested a wide mix at once. Training on one style didn’t make models good at the others.
• Data for training was hard: You want challenging, diverse, and truly grounded examples, not easy or vague ones. Simple prompting or static data generation often produced shallow or repetitive cases.
• Reinforcement learning (RL) was promising but tricky at scale: online RL can be unstable and expensive, especially with modern large models and tool-using agents.

So what was missing?

• A single, honest testbed covering many grounded reasoning styles, run on closed corpora with consistent tools.
• A way to generate hard, high-quality, evidence-grounded training data at scale without leaking test answers.
• A training method that’s efficient, stable, and can blend multiple tasks so the model truly generalizes.
• A serving approach that lets you turn up quality with more compute (parallel tries) while keeping latency practical.

🍞 Anchor: It’s like building a decathlon for AI research skills, teaching the athlete with a smart practice plan, and timing their sprints with a stopwatch that can start several lanes at once and pick the best run.

Let’s introduce the key concepts in the order we’ll use them.

🍞 Reinforcement Learning (RL) — Hook: You know how a dog learns tricks faster when it gets a treat for doing the right thing? 🥬 The concept: RL is a way for AI to learn by trying actions and getting feedback (rewards) for good outcomes.

• What it is: A learning style where the model takes steps, receives a score, and adjusts to do better next time.
• How it works: 1) Try a strategy; 2) Get a reward; 3) Prefer strategies with higher rewards; 4) Repeat to improve.
• Why it matters: Without feedback tied to real success, the model can’t learn behaviors like when to search, when to stop, and how to justify answers. 🍞 Anchor: Like practicing free throws and keeping the ones that score.

🍞 Multi-task Training — Hook: In school you don’t just learn math—you also learn reading, science, and art at the same time. 🥬 The concept: Multi-task training teaches an AI to handle different kinds of jobs together.

• What it is: Training on several task types so the model picks up general skills that transfer.
• How it works: 1) Mix tasks; 2) Balance the training so none dominates; 3) Learn shared patterns; 4) Test across all tasks.
• Why it matters: If you only study one subject, you might ace it but struggle with others. Multi-task practice builds flexible reasoning. 🍞 Anchor: A student who does math word problems, lab reports, and book summaries usually becomes a stronger all-around thinker.

🍞 Grounded Reasoning — Hook: Detectives don’t just guess—they collect clues, compare stories, and then explain how it all fits. 🥬 The concept: Grounded reasoning means the AI must fetch knowledge from outside its memory, reason over it, and support answers with evidence.

• What it is: Multi-step searching, cross-checking, and explaining using the actual documents.
• How it works: 1) Plan what to look for; 2) Search; 3) Read and connect information; 4) Calculate or aggregate; 5) Answer with sources.
• Why it matters: Without grounding, answers can be confident but wrong (hallucinations). 🍞 Anchor: Writing a report with quotes and page numbers rather than guessing from memory.

Before this paper, attempts often failed because:

• Narrow benchmarks encouraged overfitting to one style (e.g., just entity linking or just finance math), hurting generalization.
• Synthetic data was often shallow or poorly grounded, especially when created from static contexts.
• Online RL setups were finicky, expensive, and sometimes brittle with large models and tool calls.

The gap this paper fills:

• A broad, controlled benchmark (KARLBench) with six search regimes on fixed corpora.
• An agentic synthesis pipeline that explores, writes tough questions, and verifies answers with evidence.
• An iterative, large-batch off-policy RL recipe that’s stable, sample-efficient, and naturally multi-task.
• Test-time compute strategies (parallel thinking and value-guided search) to squeeze more accuracy with modest latency.

Real stakes in daily life:

• Faster, cheaper, and more accurate answers from your company’s private documents: less time searching, more time deciding.
• Safer decisions in finance or medicine because answers are tied to cited sources.
• Lower costs by teaching the model to search efficiently instead of brute-forcing long, wasteful explorations.

🍞 Anchor: Picture an assistant who can skim thousands of pages, pick the right paragraphs, do the math, and then explain exactly where every number came from—reliably, quickly, and at a reasonable cost. That’s the world KarL pushes toward.

🍞 Hook: You know how a great librarian doesn’t just point you to one shelf but can also help you compare books, summarize key parts, and double-check your math table?

🥬 The “Aha!” in one sentence: Teach one agent to search, read, and reason across many document types by training it with a steady stream of grounded, challenging examples and reward it for finishing the job well across multiple task styles.

Multiple analogies:

1. Swiss Army Knife Learner: Instead of sharpening just the knife, you train the whole toolkit—scissors (entity search), corkscrew (report writing), screwdriver (tables and math)—so the agent can handle any campsite (dataset).
2. Sports Cross-Training: Runners lift weights, do sprints, and stretch; here, the model practices different search drills so it’s stronger on everything—not just one event.
3. Cooking School: The agent doesn’t just memorize recipes; it shops for ingredients (documents), cooks (reasoning), and plates the dish (final answer with citations), then gets judged on taste and presentation (rewards), improving over time.

Before vs. After:

• Before: Models tuned on a single benchmark often looked good there but stumbled elsewhere; training was fragile and costly; synthetic data lacked diversity or solid grounding; test-time boosts were ad hoc.
• After: With KARL, multi-task RL over carefully synthesized, well-grounded data grows general search habits. Off-policy RL makes training stable and efficient. At test-time, parallel thinking and value-guided search reliably add extra points with modest latency.

Why it works (intuition, no equations):

• Rewarding outcomes shapes behavior: When the model gets higher scores for retrieving the right documents, compressing context wisely, and answering accurately with evidence, it naturally prefers those patterns next time.
• Multi-task signals create common skills: Different tasks share hidden muscles—planning, query writing, evidence merging, knowing when to stop. Practicing all of them makes those muscles stronger.
• Off-policy learning is like reviewing past games: Instead of only learning from today’s plays, the model learns from a big library of previous tries, which is cheaper and safer.
• Test-time compute is like taking multiple photos and keeping the sharpest one: Parallel tries reduce unlucky errors; a value model guides search toward promising paths.

Building blocks (each with the sandwich pattern):

🍞 KARLBench — Hook: Imagine a school decathlon with six different events so no one can win by being good at just one. 🥬 The concept: KARLBench is a six-task test suite for grounded reasoning.

• What: It covers constraint-driven entity search, biomedical report synthesis, finance tables, exhaustive entity lists, procedural tech help, and enterprise note aggregation—on fixed corpora.
• How: Each task isolates a different skill; agents use vector search only, and answers are graded via nugget-style evaluation to count correct pieces.
• Why: Without a broad, controlled test, you can’t tell if the model truly generalizes beyond a single trick. 🍞 Anchor: Like testing reading, math, science, and writing together so students can’t just cram one subject.

🍞 Agentic Synthesis Pipeline — Hook: You know how a curious student makes their own practice quizzes from the textbook and then checks the answers? 🥬 The concept: An agent generates hard, diverse questions and reference answers by exploring the corpus; then multiple solvers try them; low-signal items are filtered out.

• What: Two stages—(1) create grounded Q&A from retrieved docs, (2) have several solution attempts to estimate difficulty, remove too-easy/too-hard, and quality-check.
• How: Search the corpus, propose Q&A with citations, deduplicate against eval sets, run multiple solve attempts, filter extremes, and judge for ambiguity/errors.
• Why: Without tough, grounded, well-checked data, the model can’t learn robust search-and-reason behavior. 🍞 Anchor: Making flashcards, trying them a few times, and tossing out the ones that are trivial or broken.

🍞 Iterative Large-Batch Off-Policy RL (OAPL) — Hook: Think of a coach who studies recordings of many past games and then runs focused drills. 🥬 The concept: A stable, efficient RL recipe that learns from batches of prior attempts (off-policy) and iterates to improve.

• What: Train the policy to prefer higher-reward trajectories compared to a reference policy, using large offline batches.
• How: 1) Collect many rollouts; 2) Score them; 3) Update the model to favor better ones while staying close to a reference; 4) Repeat with the improved model.
• Why: Online RL can be unstable and costly; off-policy with large batches is steadier, cheaper, and works well across multiple tasks. 🍞 Anchor: A basketball team watches last week’s plays, learns what worked, then practices those moves in bulk.

🍞 Test-Time Compute (TTC) — Hook: When you take a hard photo, you snap a burst of shots and pick the best one. 🥬 The concept: Spend a bit more compute at inference to try multiple reasoning paths and combine them.

• What: Parallel Thinking runs several full attempts; a value model can guide which steps look promising.
• How: 1) Generate N solutions in parallel; 2) Aggregate to a final answer; 3) Optionally use a value model to steer choices.
• Why: Without TTC, you might trust the first try even if it’s unlucky; with TTC, quality rises while keeping latency low by running in parallel. 🍞 Anchor: Taking multiple drafts of an answer and then writing a polished final version.

🍞 Value-Guided Search (VGS) — Hook: Like a treasure hunter who brings a map showing which caves are promising. 🥬 The concept: Train a small value model to predict which partial paths are likely to end well and use it to steer search.

• What: A model scores partial rollouts with a “chance of success” and helps pick the next branch.
• How: 1) Train on past successes/failures; 2) At each step, sample candidates; 3) Choose the one with the highest predicted value; 4) Aggregate multiple runs.
• Why: Random exploration wastes steps; a value model focuses effort where it counts. 🍞 Anchor: Using metal detectors to choose where to dig on the beach.

Put together, the core idea is: use a broad test (KARLBench), generate grounded training data with an agent (agentic synthesis), learn from batches of past attempts (OAPL), and add parallel tries and guidance at inference (TTC + VGS). This ensemble produces a cost-effective, general-purpose knowledge agent that actually cites its work.

At a high level: Documents + Questions → [Agent uses Vector Search] → [Reason over retrieved evidence with compression] → [Answer + Citations] → [Reward and training updates with OAPL] → A better knowledge agent. At test time, we can run several parallel attempts and combine them.

We’ll explain each step like a recipe, and use the sandwich pattern for key tools.

1. The Agent Harness and the Single Tool

🍞 Vector Search — Hook: You know how you press Ctrl+F to find words in a long file? Vector search is like a super-smart Ctrl+F that finds related ideas, not just exact words. 🥬 The concept: The only external tool the agent can use is vector search, which retrieves relevant chunks from a fixed corpus based on meaning.

• What: An environment function that takes a query and returns top-k semantically similar chunks.
• How: 1) The model writes a search query; 2) The tool returns chunks; 3) The model reads them and decides next steps.
• Why: Limiting to one tool isolates core retrieval and reasoning skills instead of tool juggling. 🍞 Anchor: If you ask, “find passages about ‘operating income growth’,” vector search returns the right report pages even if they don’t use the exact phrase.

1. Managing Long Conversations

🍞 Context Compression — Hook: When your notebook gets full, you keep a short summary of important pages so you can keep working. 🥬 The concept: The agent compresses its own history when it gets too long, trained end-to-end so it learns what to keep.

• What: When tokens exceed a threshold, the agent writes a shorter summary to free space.
• How: 1) Detect limit; 2) Ask the model to summarize key facts; 3) Continue searching/ reasoning with the compact history.
• Why: Without compression, long searches overflow the context window and lose critical clues. 🍞 Anchor: In a 200-step search, the model might summarize past evidence (“we found X meets conditions A and B; still missing C”) and press on efficiently.

1. Data Synthesis for Training

Stage I: Question-Answer Synthesis

• Inputs: A small set of seed examples and the target corpus.
• Process: The synthesis agent explores the corpus with vector search, proposes a grounded question with a nuggetized answer and citations, and removes near-duplicates of eval items.
• Why it exists: Ensures training data is diverse, tough, and truly grounded in the corpus.
• Example: The agent searches biomedical abstracts for vaccine efficacy across variants, then writes: “What evidence supports mRNA vaccine effectiveness against emerging variants?” with an answer organized by study type and citations.

Stage II: Solution Synthesis and Filtering

• Process: Multiple solver attempts per question measure difficulty. Filter out tasks solved every time (too easy) or never (too hard or flawed). A quality filter removes ambiguous or incorrect-reference items.
• Why it exists: Keeps only high-signal examples where learning is richest.
• Example: If 8 attempts score mixed on a finance question, it’s kept; if all 8 are perfect or all 8 fail, it’s removed.

1. Post-Training via Iterative Large-Batch Off-Policy RL (OAPL)

• What happens: We collect big batches of trajectories from a reference policy, score them, and update the model to prefer higher-reward behavior while staying near the reference. We iterate this process a few times.
• Why this step exists: Online RL can be unstable and expensive; off-policy large-batch learning is steadier and lets you amortize data costs.
• Example with actual data: For TREC-Biogen, we compile many rollouts, each scored by nugget completeness. Trajectories that retrieve evidence, combine findings coherently, and cite correctly earn higher rewards, so the model shifts behavior toward those patterns.

1. Multi-Task RL Training

🍞 Multi-task Training — Hook: If you practice piano and basketball, your hands get faster and your footwork gets smarter. Training on several skills builds shared strength. 🥬 The concept: Mix different grounded reasoning tasks (e.g., BrowseComp-Plus and TREC-Biogen) so the agent learns generally useful search and reasoning habits.

• What: Combine losses from both tasks, roughly balancing total training tokens.
• How: 1) Prepare batches from both datasets; 2) Apply the same OAPL logic; 3) Iterate; 4) Evaluate on held-out tasks.
• Why: Without multi-task practice, the model risks specializing to one pattern and failing elsewhere. 🍞 Anchor: After this, KARL improved on both in-distribution tasks and generalized to others like finance tables and tech procedures.

1. Evaluation with Nugget-Based Scoring

🍞 Nugget-Based Evaluation — Hook: When grading an essay, teachers often check for key points—did you include each important idea? 🥬 The concept: Answers are broken into “nuggets” of information; the score counts how many nuggets the model covered.

• What: A fair way to grade multi-paragraph or list-style answers across tasks.
• How: 1) Define gold nuggets (e.g., entities or facts); 2) Compare the model’s answer; 3) Count coverage.
• Why: Without nugget scoring, long answers might look good but miss crucial facts. 🍞 Anchor: For “Which countries won the World Cup?”, each country is a nugget; the model must list them all.

1. Test-Time Compute (TTC) to Boost Results

🍞 Parallel Thinking — Hook: When solving a riddle, you and your friends try separately and then share answers to pick the best one. 🥬 The concept: Run N rollouts in parallel and then aggregate to a final answer, sometimes synthesizing a better answer than any individual try.

• What: A general-purpose way to improve quality while keeping latency low via parallelism.
• How: 1) Launch N independent solutions; 2) Feed their final answers to an aggregator; 3) Produce a combined best answer.
• Why: Without it, one unlucky try might miss key evidence. With it, odds improve across tasks. 🍞 Anchor: On PMBench, the aggregator often combined pieces from different rollouts into a superior final.

🍞 Value-Guided Search (VGS) — Hook: It’s like having a coach whisper, “This path looks promising—follow it.” 🥬 The concept: A small value model predicts the chance of success from any partial solution and steers which branch to take next.

• What: For short, factual answers (like entity names), VGS plus weighted voting can beat simple parallel thinking.
• How: 1) Train value model on past rollouts; 2) At each step, sample 2 candidates; 3) Pick the higher-value one; 4) Repeat for N trees; 5) Aggregate with weighted voting.
• Why: It wastes fewer steps and lifts recall without being trained to optimize recall directly. 🍞 Anchor: On BrowseComp-Plus, VGS with weighted votes reached higher accuracy and recall than basic methods.

The Secret Sauce:

• End-to-end training with compression included: The agent learns not just to search, but to manage memory wisely.
• Off-policy, large-batch RL: Stable, sample-efficient progress without complicated online tricks.
• Multi-tasking: Shared skills like planning, diverse exploration, and stopping wisely transfer across domains.
• TTC: Parallel tries and guided branching give consistent boosts while keeping latency practical.

Example walkthrough (small, concrete):

• Input: “Find the company that meets constraints A, B, C.”
• Step A (Search): The agent writes a vector query targeting constraint A; reads results.
• Step B (Refine): It queries for B using terms learned from A’s results; compresses history when long.
• Step C (Aggregate): With A and B satisfied, it searches for C; once found, it composes a short answer citing sources.
• Output: The final answer plus source snippets covering A, B, and C.
• What breaks without each step: No search → guesses; no compression → lose earlier clues; no aggregation → incomplete answer; no reward training → repeats bad habits; no TTC → one-off mistakes go uncorrected.

The Test: The team measured grounded reasoning across six distinct regimes in KARLBench: constraint-driven entity search (BrowseComp-Plus), biomedical report synthesis (TREC-Biogen), finance tables (FinanceBench), exhaustive entity lists (QAMPARI), procedural tech help (FreshStack), and enterprise notes (PMBench). They used nugget-based scoring to count correct pieces and fixed corpora with only vector search to keep comparisons fair.

The Competition: KARL was compared to strong closed and open models (e.g., Claude family, GPT-5.x, Qwen, GLM 4.5 Air, Minimax). Single-task RL experts (KARL-TREC, KARL-BCP) and multi-expert SFT distillation were also evaluated.

Scoreboard with context (plain-language):

• In-distribution strength: Single-task experts shined on their home turf (e.g., KARL-TREC scored very high on TREC-Biogen), but they didn’t transfer well to the other in-distribution task, showing how different the skills are.
• Multi-task gains: The multi-task KARL matched or beat models of similar size and pushed close to the strongest closed models even without extra compute at test time. With parallel thinking (N=10), KARL matched top-tier models’ quality while being cheaper and faster.
• Cost and latency trade-offs: KARL sat on the Pareto frontier—meaning you couldn’t beat it in quality without paying more or waiting longer. It even undercut the cost of its own base model while scoring higher, thanks to more efficient search.
• TTC scaling: As N increased for parallel thinking, scores rose across all tasks, with diminishing returns past ~15. On BrowseComp-Plus, value-guided search plus weighted voting outperformed simple parallel thinking, highlighting the power of reward-guided branching on short factual answers.

Surprising findings and what they mean:

1. RL teaches new capabilities, not just “sharpening” probabilities: When checking best-of-k behavior, improvements appeared across all k, not just at k=1. This means the model solved problems it never solved before, even with many tries—evidence of true capability growth.
2. Efficiency improved alongside accuracy: RL reduced wasted post-retrieval searches (continuing to search after all necessary documents were already in context). It also shortened many trajectories without damaging quality, by learning better stopping rules.
3. Diversity of exploration rose: The trained model retrieved more unique documents over time, suggesting broader and less repetitive search patterns.
4. Compression got smarter: Swapping in KARL as the compression-only model boosted another model’s performance, showing compression learned what to keep without separate pretraining.

Concrete numbers in everyday terms:

• Think of an A as a top score and a B as a good score. KARL’s multi-task model, with moderate parallel thinking, performed like the class valedictorian while spending less time and allowance money per question than other top students.
• On the biomedical report task, the single-task expert rose from a solid “B” to a strong “A” across iterations, showing iterative off-policy RL keeps adding value.
• On the tough BrowseComp-Plus entity task, guiding test-time search with a small value model bumped both accuracy and recall beyond voting-only methods.

Behavior case studies (what changed in how the agent thinks):

• Persistence where needed: KARL sometimes continued beyond where other models quit and reached the correct answer, showing better long-horizon stamina.
• Smarter stopping: KARL often stopped earlier when extra searches weren’t improving evidence—trading exhaustive verification for timely, well-supported answers.
• Reasoning focus: In some numerical cases, KARL still avoided hard arithmetic, preferring more searches over calculation—pointing to a future area to strengthen.

Bottom line: Across budgets, KARL delivers high-quality grounded reasoning at lower cost and latency than peers. With parallel thinking or value-guided search, it reaches or beats top closed models on the benchmark suite.

Limitations (clear-eyed view):

• Single tool constraint: The agent only used vector search. Many real tasks benefit from more tools (e.g., browsing, code execution, table parsing). Extending the action set should help.
• Arithmetic and table-heavy reasoning: While improved, some cases showed avoidance of deeper numerical computation once the evidence was already in hand.
• Synthetic data dependence: Quality hinges on the synthesis pipeline. Although carefully filtered and de-duplicated, synthetic tasks may still miss rare real-world edge cases.
• Aggregation scaling: Parallel thinking’s aggregator reads many answers; very large N can bloat context, soft-limiting gains without more advanced summarization.

Required resources to reproduce/use:

• A capable base model (e.g., mixture-of-experts like GLM 4.5 Air) and GPUs suitable for large-batch offline rollout collection and RL training.
• A high-throughput vector search stack (embedded database, prebuilt indexes) to sustain hundreds of queries per second per host during synthesis and evaluation.
• The aroll agent harness (or equivalent) to keep training, evaluation, and serving consistent and avoid environment drift.

When not to use:

• If your task needs internet browsing, structured database queries, or code execution right now, a vector-search-only agent may underperform until those tools are added.
• If the domain demands precise arithmetic on complex tables without retrieval (e.g., pure math competitions), a specialized math model might be better.
• If data privacy forbids any form of synthetic data generation or offline logging for rewards, the approach needs careful adaptation.

Open questions and next steps:

• Tool expansion: How much does performance jump when adding structured retrieval, code tools, or spreadsheet engines under the same RL recipe?
• Memory and compression: Can hierarchical or learned memory modules further cut latency and push quality at larger N without context blowup?
• Robustness and auditing: How to best verify that synthetic data never overlaps with eval or production items—beyond current de-dup pipelines?
• Reward shaping: Can we incorporate fine-grained signals (like partial arithmetic correctness) to lift weaknesses without harming generalization?
• Theory of generalization: Why does multi-task off-policy RL lead to such consistent out-of-distribution gains versus distillation, and under what conditions might that reverse?

Overall, KARL shows that carefully designed synthetic data, multi-task off-policy RL, and practical test-time compute form a powerful trio—but it also maps out clear frontiers to push next.

Three-sentence summary: This paper introduces KARL, a knowledge agent trained with iterative large-batch off-policy reinforcement learning on carefully synthesized, grounded data across multiple task types. Using a broad benchmark (KARLBench), a two-stage agentic synthesis pipeline, and test-time compute (parallel thinking and value-guided search), KARL reaches frontier quality while reducing cost and latency. Results show genuine generalization beyond single-task training and beyond simple “probability sharpening.”

Main achievement: Demonstrating that tailored synthetic data plus multi-task off-policy RL can produce a Pareto-optimal, grounded reasoning agent that matches or exceeds top closed models when allowed modest parallel thinking—at lower cost and latency.

Future directions: Expand the toolset (structured queries, code execution, spreadsheet reasoning), strengthen numerical/table reasoning rewards, and develop smarter, hierarchical memory for compression and aggregation at large N. Explore richer reward shaping and theoretical analysis of why off-policy multi-task RL generalizes so well.

Why remember this: KARL turns grounded reasoning from a narrow, benchmark-specific trick into a general, trainable skillset—and it does so efficiently. For anyone building enterprise search assistants, it’s a blueprint for creating agents that cite sources, manage context, and improve with parallel thinking. It charts a practical path toward trustworthy, cost-effective AI that can navigate real document ecosystems.