RelayGen: Intra-Generation Model Switching for Efficient Reasoning

🍞 Hook: Imagine you’re solving a long math problem. Some parts are tough puzzles, and some parts are just writing out the final answer neatly. You wouldn’t call your genius friend for the neat-writing part—you’d ask them for help only on the hardest steps.

🥬 The Concept (Large Reasoning Models, or LRMs): LRMs are big AI models that solve complex, multi-step problems by writing long chains of thought before giving the final answer. How it works:

1. The model thinks out loud in many steps (the reasoning stage).
2. Then it writes a short final answer (the answer stage).
3. Longer, smarter models tend to do better on tricky puzzles, but take more time and money to run. Why it matters: If we use the biggest model for every single word, it gets slow and expensive, even when the text is easy. 🍞 Anchor: Like asking a math whiz to invent the strategy, then having a classmate copy it neatly.

The world before: As reasoning got better, models grew bigger and wrote longer chains of thought. That increased accuracy, but also made inference expensive and slow. People noticed that not every part of the chain is equally hard, yet most systems still treated it all the same.

The problem: How do we speed up long reasoning without losing the accuracy boost from a big model? We want to save time on the easy stretches while keeping the big model on the hard parts.

🥪 New Concept Sandwich — Input-level routing 🍞 Hook: You know how a coach picks one player to play the whole game, no matter what? 🥬 The Concept: Input-level routing chooses one model per question and sticks with it. How it works: You score the difficulty of the whole input, pick a model, and let it generate everything. Why it matters: It misses the fact that difficulty changes inside one answer, so it can overuse the big model or underperform on hard parts. 🍞 Anchor: Using one backpack for a whole trip—even when sometimes you only need a light daypack.

🥪 New Concept Sandwich — Token-level routing 🍞 Hook: Imagine switching drivers every single second on a road trip. 🥬 The Concept: Token-level routing chooses which model to use at every word. How it works: A trained router predicts, for each token, whether the big or small model should produce it. Why it matters: It’s very fine-grained, but needs extra training, adds system complexity, and breaks other speedup tools like speculative decoding. 🍞 Anchor: It’s like changing drivers so often that you lose time instead of saving it.

Failed attempts: Step-level or heuristic approaches split the answer into chunks but often pick chunks using hand-made rules that don’t always match where the hard thinking really is. Token-level routers are precise but heavy and hard to deploy, often clashing with speculative decoding.

The gap: We need a simple, training-free way to switch models inside one answer, at just the right times, using signals that truly reflect difficulty—and that still works with speculative decoding.

🥪 New Concept Sandwich — Difficulty varies within one output 🍞 Hook: When you write an essay, brainstorming is hard, but writing the final summary is easier. 🥬 The Concept: Difficulty rises and falls during a single generated answer. How it works: Early parts might explore and revise; later parts may summarize or conclude. The model’s uncertainty changes accordingly. Why it matters: If we can detect easy stretches, we can switch to a smaller model to save time. 🍞 Anchor: You do deep thinking first, then just clean up the final paragraph.

Real stakes: Faster answers mean lower costs, less waiting, and greener computing. It lets more people use powerful reasoning on everyday devices, and it makes big systems more practical in classrooms, coding assistants, and research tools.

🥪 Aha! Moment In one sentence: Watch for moments when the generation becomes easier and hand off those stretches to a smaller model—keep the hard thinking on the big model.

Three analogies:

1. Relay race: The sprinter (big model) handles steep hills; on flat ground, they pass the baton to a steady jogger (small model).
2. Chef and prep cook: The head chef designs the dish (reasoning), then the prep cook plates and serves (answering) quickly.
3. Off-road to highway: Use a 4x4 for the rocky trail, then switch to a scooter for smooth city streets.

Before vs. After:

• Before: Either one model did everything, or complex routers decided token-by-token, adding overhead and hurting compatibility with speculative decoding.
• After: Training-free, segment-level handoffs triggered by empirically chosen switch cues. Big model for hard parts; small model for easy continuations and final answers.

🥪 New Concept Sandwich — Discourse-level cues 🍞 Hook: In stories, words like “Therefore,” or “Now,” tell you what kind of sentence is coming next. 🥬 The Concept: Discourse-level cues are words that signal shifts in the flow of reasoning, like moving from exploring to summarizing. How it works: The paper profiles many cue words in advance and keeps only those that, on average, are followed by easier, more predictable text for a given model pair. Why it matters: Good cues let us switch models at the right times without training a router. 🍞 Anchor: Seeing “Thus,” often means a neat wrap-up is coming—perfect time to hand off.

🥪 New Concept Sandwich — Token probability margin 🍞 Hook: When you’re sure the next word in a sentence is “the,” you barely hesitate; when unsure, you pause. 🥬 The Concept: The token probability margin is how much more confident the model is in its top guess than its second-best guess for the next word. How it works: Larger margins mean higher certainty. The paper measures average margins after certain cues and picks cues that reliably lead to higher certainty. Why it matters: Certainty is a useful thermometer for difficulty—higher certainty often means the small model can handle it. 🍞 Anchor: If you’re 95% sure the next word is “Therefore,” that’s a low-difficulty moment.

Why it works (intuition):

• During the reasoning stage, uncertainty jumps around; after certain cues, the text often becomes summarizing or reflective, which is easier.
• The answer stage depends on the entire previous reasoning, so it’s more like formatting the final thought than inventing it—small models do well there.
• Switching at sentence-sized chunks avoids constant overhead and stays compatible with speculative decoding.

Building blocks:

• Offline cue selection using probability margins from a small calibration set (no training).
• Runtime switching using stop tokens at selected cues and at the boundary between thinking and answering.
• Prefix caching so switching doesn’t mean redoing all the work.
• Full handoff of the final answer to the small model because it’s stable and cheap.

High-level recipe: Prompt → Big model starts reasoning → On certain cue words, delegate a sentence to small model → Return to big model for next hard bit → After </think>, small model writes the final answer.

Step 1: Offline pick the right cues (once per big–small pair) What happens: The team collects a small set of reasoning traces (e.g., 40 AMC problems, 4 traces each ≈ 160 samples). For each candidate cue (like “Thus,” “Now,” “Therefore,”), they measure the average token probability margin from the cue to the end of the sentence and compare it to the global average. Why this step exists: Not all cues mean “easy.” Some cues begin new explorations. Picking only cues that consistently raise certainty avoids bad handoffs. Example: If “Thus,” is usually followed by higher confidence sentences, it becomes a switch cue; if “So,” is inconsistent, it’s rejected.

Step 2: Start reasoning on the big model What happens: At runtime, the big model begins generating the chain-of-thought. Its stop tokens include: the approved cue words and the special boundary token that ends reasoning (like </think>). Why this step exists: We want the big model to lead the hard thinking but pause right when an easy segment begins. Example: The model writes, “We solved for x. Therefore,” and stops on “Therefore,” because it’s a chosen switch cue.

Step 3: Temporary handoff to the small model for an easy sentence What happens: After a cue triggers a stop, the small model continues just until the sentence ends (e.g., at a period). Then control returns to the big model if reasoning continues. Why this step exists: Sentences after good cues are often simpler and more predictable. The small model handles them fast and cheaply. Example: Small model writes, “Therefore, x = 7 and y = 3.” On hitting the period, it stops; control returns to the big model.

Step 4: Repeat as needed during reasoning What happens: This back-and-forth can happen multiple times. Thanks to prefix caching (reusing past context), the system doesn’t re-run the entire prompt each time—only the new bits. Why this step exists: Keep hard chunks on the big model and easy chunks on the small model, many times, without paying switching penalties. Example: Later, “Now,” appears; small model takes one sentence; return to big model again.

Step 5: Final handoff for the answer stage What happens: When the special token indicates the end of thinking (e.g., </think>), the rest—the answer formatting—is fully delegated to the small model. Why this step exists: The paper shows answer writing is highly stable under handoff (about 99.86% matching the big model), while being the most expensive per token due to long attention spans. Perfect moment to save time. Example: Small model formats: “Final answer: 7.”

Secret sauce:

• Coarse, sentence-sized switching triggered by empirically validated cues, not training.
• Answer-stage stability: small model finishes reliably while saving lots of compute.
• Compatibility: Because switching happens in chunks, speculative decoding can still form long draft spans for extra speed.

Concrete data example:

• Calibration set: 40 AMC problems × 4 traces = 160 samples; took ~100 minutes offline on two A100s (about 80 minutes to generate traces, 20 minutes to compute margins and pick cues). Smaller calibration sets (10–40 samples) still worked well in ablations.
• Example cues that worked (for certain model pairs): “Thus,” “Therefore,” “Now,” “Similarly,” “Specifically,” and variants.

What breaks without each step:

• Without cue selection: You might switch at the wrong times, harming accuracy.
• Without segment boundaries: Switching too often adds overhead and blocks speculative decoding.
• Without full answer handoff: You’d miss the biggest cheap-win zone where small models excel.
• Without prefix caching: Switching would force costly re-prefills, erasing the gains.

The tests and why: The paper checks two main things—accuracy and speed. Accuracy is measured by pass@1 on reasoning benchmarks (how often the top answer is correct). Speed is measured as end-to-end latency and reported as a speedup compared to using only the large model.

Competitors:

• Small model only: Fast but much weaker.
• Large model only: Strong but slow and expensive.
• Speculative Thinking: Step-level heuristic switching using hand-picked cues.
• R2R: Token-level routing with a trained router (more complex and often incompatible with speculative decoding).

Scoreboard with context:

• Accuracy: With Qwen3-32B/Qwen3-1.7B on AIME 2025, large model alone gets about 70% pass@1; small model alone about 32%. RelayGen reaches about 68.3%, keeping close to the big model while using the small model for easy parts. On GPQA-Diamond, RelayGen is also close to the big model and clearly above the small model. Compared to Speculative Thinking, RelayGen keeps more accuracy (because cue selection is evidence-based). Against R2R, RelayGen is competitive or better on several benchmarks without training a router.
• Speed: RelayGen by itself achieves speedups similar to R2R but with higher large-model utilization (it keeps the hard parts on the big model). Crucially, RelayGen composes with Eagle-3 speculative decoding to reach up to about 2.20× speedup, while staying training-free. That’s like finishing two homework sets in the time it used to take to finish one—without your grades really dropping.

Surprising findings:

• Answer-stage stability: When the small model finishes the answer after the big model’s reasoning, the answer matches the big model’s answer in about 99.86% of cases (728 samples, only one mismatch). This is a free win for speed and cost.
• Small calibration works: Even with as few as 10–40 calibration samples, performance remained strong. You don’t need lots of data to pick good cues.
• Not all cues are equal: Using every candidate cue hurts accuracy. Selecting only cues that reliably raise certainty matters a lot.

Big picture: RelayGen lives at a sweet spot—near-big-model accuracy with meaningful speedups, and it stacks with speculative decoding instead of fighting it.

Limitations:

• Works best when the model actually writes out long, structured reasoning. If tasks are short or don’t have a clear reasoning/answer split, there’s less to gain.
• Needs a reasonably capable small model. If the small model can’t even handle the easy sentences, switching hurts.
• Cues are model-pair specific. A cue that’s easy for one pair might not be for another; a brief calibration is required.
• Most results shown are on English. Multilingual behavior likely transfers but needs testing.

Required resources:

• One large and one small model available at inference (often two GPUs or careful scheduling on one).
• vLLM or an engine with prefix caching and simple stop-token control.
• A tiny calibration pass (10–160 samples) done once per model pair; no training.

When not to use:

• Very short answers or tasks without visible “thinking” segments.
• Highly creative writing where cues don’t map to difficulty.
• Ultra safety-critical settings where any offloading risk is unacceptable (until further validation).

Open questions:

• Can cue discovery be automated further or adapted on-the-fly per user/task?
• What’s the best way to pick among multiple small models (e.g., math-specialist vs. generalist)?
• How does this scale with extremely long contexts and memory-optimized hardware?
• Can we extend beyond cue words to structural markers (lists, equations, verified steps) while staying training-free?
• How does this work across many languages and domains with different discourse patterns?

Three-sentence summary: RelayGen is a simple, training-free way to speed up long reasoning by switching between a big and a small model at the right moments. It uses evidence-based cue words and sentence-sized segments to keep hard thinking on the big model and easy parts on the small one, then hands the final answer to the small model. It preserves most of the big model’s accuracy, composes with speculative decoding, and delivers up to about 2.2× speedups.

Main achievement: Showing that coarse-grained, empirically guided switching inside a single answer is enough to capture real difficulty shifts—no trained router needed—while staying compatible with speculative decoding.

Future directions: Broaden to multilingual setups; refine automatic cue selection; consider multiple small models specialized by domain; integrate smarter runtime signals (still training-free); and explore tighter hardware–software co-design to cut latency further.

Why remember this: RelayGen proves you don’t need complicated token-by-token routing to get big wins. A little bit of smart, sentence-level timing—grounded in actual model certainty—can make large reasoning models faster, cheaper, and easier to deploy at scale.