LaSER: Internalizing Explicit Reasoning into Latent Space for Dense Retrieval

You know how a librarian doesn’t read every book to help you, but still knows where to find the right one quickly? That’s how modern search engines work with something called dense retrieval.

🍞 Hook: Imagine you ask for “books that explain how rainbows form.” A great librarian understands your idea and grabs the right shelf fast. 🥬 The Concept (Dense Retrieval): It’s a way to turn text into compact vectors (numbers) so we can quickly find the most similar items in a huge collection.

• How it works:
  1. Turn your question and every document into vector points in the same space.
  2. Measure closeness (similarity) between your question vector and document vectors.
  3. Return the nearest documents.
• Why it matters: Without it, searching would be slow and miss matches that use different words but the same idea. 🍞 Anchor: Even if you ask “rainbow creation,” dense retrieval can still find “light refraction and reflection in water droplets.”

The world before: Dense retrievers used smaller encoders (like BERT). They were great at matching words and meanings but struggled with queries that needed extra thinking—like multi-step logic or hidden intent. Then Large Language Models (LLMs) arrived. People started using LLMs as stronger encoders for retrieval. These LLM-based retrievers understood language better, but here’s the catch: they were trained mostly just to separate relevant and irrelevant stuff (contrastive learning). That means we used them like super-librarians who never got to explain their reasoning—they matched, but didn’t think out loud.

The problem: Real questions can be tricky. Users might ask something vague (“What’s the process called when chickens make eggs?”), multi-hop (“Which scientist’s idea explains both A and B?”), or ambiguous (“Apple earnings in spring”). For these, plain similarity isn’t enough; we need step-by-step reasoning.

🍞 Hook: You know how solving a math word problem is easier if you write out the steps? 🥬 The Concept (Chain-of-Thought, CoT): Chain-of-Thought is a step-by-step explanation that breaks a hard problem into smaller steps.

• How it works:
  1. Restate the problem.
  2. List what’s needed to solve it.
  3. Follow the steps to reach the answer.
• Why it matters: Without steps, models guess. With steps, models reason. 🍞 Anchor: “Eggs come from hens via ovulation → shell forms in the oviduct → so the process is egg production.”

Failed attempts: A popular fix is “rewrite-then-retrieve.” Another LLM first writes extra text—expansions or a CoT—then the retriever uses that richer query to search. It works, but it’s slow because generating text step by step takes time and money. On the other hand, some tried “implicit reasoning” with hidden, made-up tokens inside the model’s vector space (no text). That’s faster, but those silent thoughts can become meaningless without clear guidance, because they’re only nudged by the final loss.

The gap: We want the brains of explicit CoT without the slowness, and the speed of latent tokens without the “empty thoughts.” What’s missing is a way to teach the silent thoughts using the explicit steps—so the model learns to think inside, quietly, but correctly.

The stakes: Faster, smarter search improves everyday tools—web search, assistants, customer help, and Retrieval-Augmented Generation (RAG). It reduces waiting, cloud costs, and energy use. It also makes small models more helpful on devices with limited power. For students, doctors, researchers, or shoppers, that means better answers sooner, even for messy questions.

Enter LaSER: it internalizes explicit reasoning (the written steps) into a model’s latent space (its hidden vectors). During training, it reads explicit CoTs as a teacher. During inference, it thinks silently with only a few latent tokens—no text generation. The result: near the quality of rewrite-then-retrieve, at the speed of a single-pass retriever.

The “Aha!” in one sentence: Teach a retriever to do silent, step-by-step reasoning by aligning its hidden thinking tokens with a teacher’s explicit Chain-of-Thought, so it keeps speed while gaining smarts.

Three analogies:

• Map vs. Guided Tour: The teacher gives a full guided tour (explicit CoT). The student learns to navigate later with just a compact map (latent tokens), reaching the same spots without the long narration.
• Training Wheels: The teacher’s steps act like training wheels, keeping the student’s silent path stable until it can balance by itself.
• Movie Trailers: The teacher shows the whole movie (full reasoning). The student learns to make a short trailer (latent keyframes) that captures the story arc.

Before vs. After:

• Before: LLM retrievers acted like quiet matchers. If you needed reasoning, you had to bolt on a slow rewriter.
• After: The retriever “thinks” internally using a few soft tokens, guided by a teacher’s steps during training. No extra rewriter needed at inference.

Why it works (intuition, no equations):

• Explicit steps are rich but long; latent tokens are short but can become empty. If you align the student’s hidden states with the teacher’s reasoning path at multiple points (not just the final output), the student’s silent thoughts carry real meaning. This keeps the process compact and meaningful.
• Matching score distributions (not just vectors) teaches preferences—what should rank above what—so the student learns fine judgment rather than copying coordinates.
• Using the same backbone for both teacher and student views ensures the knowledge lives in the model itself, not in a separate component.

Building blocks (introduced with sandwiches):

🍞 Hook: You know how a coach can teach a player to self-correct by showing perfect form first? 🥬 The Concept (Self-Distillation): The model uses its own explicit-reasoning view as a teacher to guide its silent-reasoning view.

• How it works:
  1. Run the explicit view with a Chain-of-Thought to get a rich “teacher” signal.
  2. Run the latent view with silent tokens to get a “student” signal.
  3. Align their rankings and intermediate thinking steps.
• Why it matters: Without self-distillation, the student’s silent thoughts may drift and become useless. 🍞 Anchor: Like practicing basketball: watch the coach’s form (teacher), then mirror it quietly until your body remembers.

🍞 Hook: Imagine jotting tiny symbols instead of full sentences to remember your plan. 🥬 The Concept (Latent Thinking Tokens): These are continuous, soft “thought vectors” the model appends internally to reason without generating text.

• How it works:
  1. Start with the query’s hidden state.
  2. Predict a soft token (a weighted mix of word embeddings).
  3. Append it and repeat a few steps to refine understanding.
• Why it matters: They’re fast to compute and can store reasoning if trained well. 🍞 Anchor: Instead of writing “find causes, then timeline,” you draw two icons; you still remember the plan.

🍞 Hook: You know how photographers shoot from two angles to understand a scene better? 🥬 The Concept (Dual-View Training): Train the same model in two ways: an explicit view (with CoT) and a latent view (silent tokens), sharing parameters.

• How it works:
  1. Explicit view reads the query + CoT and produces a strong embedding.
  2. Latent view reads only the query, thinks with K soft tokens, and produces an embedding.
  3. Align them so the latent view learns from the explicit view.
• Why it matters: Without two aligned views, the model can’t learn silent reasoning from explicit steps. 🍞 Anchor: One student reads the solution, another solves mentally; then they compare notes to improve.

🍞 Hook: Think of matching both the big picture and the small steps in a recipe. 🥬 The Concept (Multi-Grained Alignment): Align not only the final rankings but also the in-between thinking stages.

• How it works:
  1. Output alignment: match score distributions over a batch of documents.
  2. Process alignment: match intermediate hidden states to checkpoints along the explicit reasoning.
  3. Balance these with normal contrastive learning.
• Why it matters: Without process alignment, tokens may be hollow; without output alignment, the final ranking can wobble. 🍞 Anchor: It’s like learning to bake: taste at each step and also judge the final cake.

🍞 Hook: Picture lining up two train tracks so the stops match. 🥬 The Concept (Trajectory Alignment): Make each latent token step correspond to a key step in the explicit Chain-of-Thought.

• How it works:
  1. Split the explicit reasoning into M segments.
  2. Use K latent steps and map each step to one segment (downsample if needed).
  3. Align the student’s hidden state at that step with the teacher’s state for that segment.
• Why it matters: Without this, the silent steps drift and don’t follow the logic arc. 🍞 Anchor: Like syncing a highlight reel to the full match at the right timestamps.

Put together, LaSER compresses long reasoning into a few meaningful silent steps, keeping speed and gaining depth.

At a high level: Query → Two parallel passes during training (Explicit view with CoT, Latent view with soft tokens) → Align outputs and trajectories + contrastive loss → Final retriever that, at inference, uses only the latent view.

Step-by-step details with sandwiches for key parts:

1. Inputs and basic encoding

• You provide a query q. During training, an external reasoner (like GPT-4o-mini) also provides an explicit Chain-of-Thought (CoT) rationale r_q.
• The shared LLM backbone encodes text into hidden states.

🍞 Hook: Think of labeling boxes so you can stack them neatly. 🥬 The Concept (Contrastive Learning): A training method that pulls matching pairs together and pushes non-matching ones apart in vector space.

• How it works:
  1. For each query, mark one positive document and several negatives.
  2. Compute similarities (query vs. docs) and a temperature-scaled softmax.
  3. Increase probability of the positive; decrease for negatives.
• Why it matters: Without it, the model wouldn’t learn what “relevant” means. 🍞 Anchor: Like teaching a dog “sit” vs. “not sit” with treats and gentle corrections.

1. Explicit View (teacher)

• Input: [q ; r_q ; EOS]. The model does one forward pass and outputs an explicit-view embedding v*_q.
• It also saves intermediate hidden states tied to segments of r_q (think of them as checkpoints along the reasoning path).
• Contrastive learning is applied here too, so the teacher knows how to rank documents well.

1. Latent View (student)

🍞 Hook: Imagine solving a puzzle by adding a few quiet hints to yourself. 🥬 The Concept (Latent Thinking Tokens): Continuous soft tokens that the model appends to its hidden sequence to reason without text.

• How it works:
  1. From the last hidden state, predict a soft distribution over the vocabulary.
  2. Turn that distribution into a soft embedding (a weighted average of word embeddings).
  3. Append it and repeat K times (usually small, like 3) with causal attention.
  4. Pool the thinking states to form the final query vector v_q.
• Why it matters: Without these steps, the model has no room to refine its understanding. 🍞 Anchor: Instead of writing a paragraph, you add three sticky-note icons that remind you of your plan.

Example with tiny numbers: Suppose K=3 and your query is “How do bees make honey?” Step 1 latent token leans toward words like “nectar,” step 2 toward “enzymes,” step 3 toward “evaporation.” The pooled states produce an embedding that ranks documents describing those steps higher.

1. Output-level Distillation (score alignment)

🍞 Hook: You know how learning to grade homework like a teacher helps you understand what makes a good answer? 🥬 The Concept (Output-level Distillation): Make the student mimic the teacher’s ranking preferences over a batch of documents.

• How it works:
  1. Compute similarity scores (query vs. each doc) for both views.
  2. Turn scores into probability distributions with temperature.
  3. Minimize the difference (KL divergence) from teacher to student.
• Why it matters: Without matching preferences, the student may miss fine-grained ranking cues. 🍞 Anchor: If the teacher says Doc A > Doc B > Doc C, the student learns that order, not just the final pick.

Concrete example: For a 1×N batch, if the teacher gives A:0.6, B:0.3, C:0.1 and the student gives A:0.4, B:0.4, C:0.2, distillation nudges the student to shift probability from B and C toward A.

1. Process-level Trajectory Alignment (step alignment)

🍞 Hook: Think of matching the chapters of two books that tell the same story—one long, one short. 🥬 The Concept (Trajectory Alignment): Align each of K latent student steps with M teacher segments from the explicit CoT using temporal downsampling.

• How it works:
  1. Split the teacher’s rationale into M segments (e.g., by sentences or markers).
  2. Map each student step i to segment j_i = floor(i×M/K).
  3. Compare their induced ranking distributions over the doc batch and minimize the gap.
• Why it matters: Without this, silent steps can drift and fail to represent meaningful parts of reasoning. 🍞 Anchor: It’s like ensuring your 3 key photos correspond to the beginning, middle, and end of the full vacation album.

1. Overall loss

• Add up: latent-view contrastive + explicit-view contrastive + output distillation + trajectory alignment, each with a weight. This balances learning to rank with learning to think.

1. Inference

• Only the latent view runs. No text generation. Just K soft steps, then produce the query embedding and retrieve.
• Optional: If you already have an external rewrite, you can feed it too—the backbone learned to encode such text even better due to co-training.

The secret sauce:

• Multi-grained alignment (both outputs and steps) prevents the silent tokens from becoming hollow.
• Score-based distillation (not raw embedding matching) focuses on what really matters: ranking preferences.
• Shared-backbone co-learning lets the teacher and student improve together, so the “thinking skill” lives inside the model itself.

What breaks without each part:

• No latent tokens: the model has no room to refine thoughts; reasoning stalls.
• No explicit teacher: tokens lack guidance; thoughts drift.
• No output distillation: final rankings wobble.
• No trajectory alignment: intermediate steps lose meaning.
• No contrastive loss: relevance basics are missing.

The test: Can LaSER retrieve reasoning-heavy answers as well as slow pipelines, but with the speed of a normal retriever? Researchers measured ranking quality and latency.

Datasets and why they matter:

• ReasonEmb (train): Queries paired with LLM-generated reasoning paths. Teaches the model how explicit steps look.
• BRIGHT (test, in-domain): A challenging benchmark for reasoning-intensive retrieval (e.g., coding forums, math, theorems).
• FollowIR (test, out-of-domain): Evaluates instruction-following in retrieval with special pairwise metrics (p-MRR) and standard ones.
• BrowseComp-Plus (test, out-of-domain): Deep research agent benchmark; checks recall at different cutoffs.

Metrics with meaning:

• nDCG@10: Measures how well the top 10 results are ordered. Think of it like your “grade” for the front page; higher is better.
• Recall@K: Did the right document appear in the top K? Like checking if the treasure is somewhere in your top-K picks.
• MAP@5, p-MRR: Different ways to judge ordering and instruction-following quality.

Competition:

• Standard dense retrievers (e.g., BGE-M3, E5, Qwen3-Embedding).
• Pipeline methods (Rewrite-then-Retrieve): Use an external LLM to generate a rewrite/CoT before retrieval—high quality but high latency.
• Explicit reasoning retrievers: Generate CoT inside the retriever (still slow and harder to train).
• Implicit reasoning retriever (GIRCSE): Uses latent tokens but relies mainly on the final contrastive loss.

Scoreboard (with context):

• On BRIGHT, LaSER with Qwen3-8B averages nDCG@10 ≈ 29.3. That’s like getting an A- when the strong fine-tuned baseline is around 25.7 (a solid B). It also surpasses Qwen3-Embedding-8B (≈14.0), showing the importance of internalized reasoning.
• Against implicit reasoning (GIRCSE), LaSER wins in 8/9 settings across model families and sizes, highlighting the value of process + output alignment.
• Against pipelines: LaSER matches or even beats rewrite-then-retrieve in some reasoning-heavy cases but runs vastly faster at inference because it avoids text generation.

Latency: In a BRIGHT subset test, LaSER achieved roughly 0.3% of the pipeline’s latency while staying competitive in quality. Compared to a basic retriever, LaSER adds a small overhead (about 1.7×) due to a few latent steps, but this shrinks with larger models and KV caching.

Surprising findings:

• Few steps suffice: Training with K=3 latent tokens often matches or beats longer horizons. The explicit teacher helps compress reasoning into compact keyframes.
• More steps at inference can help a bit: When you allow extra steps at inference, quality can improve, suggesting the model truly iteratively refines its understanding.
• Co-learning matters: Training the explicit and latent views together (shared backbone) works better than a frozen teacher. The backbone grows better at both reading CoT and thinking silently.

Takeaways by audience:

• Practitioners: You can get pipeline-like quality with a single model and just a handful of latent steps. This simplifies deployment and reduces serving costs.
• Researchers: Process-level alignment is crucial for making silent thoughts meaningful. Score-distribution distillation focuses learning on ranking choices, not arbitrary vector matching.
• Product teams: The method is robust across backbones and sizes; even small models gain notable reasoning ability without external services.

Limitations:

• Sensitivity to settings: Performance depends on hyperparameters like temperatures, loss weights, and the number of latent steps K. Poor choices can add noise or under-train the tokens.
• Quality of teacher rationales: If the explicit CoT (from the external reasoner) is weak or off-topic, the student learns a distorted path.
• Fixed step budget: Mapping K latent steps to M explicit segments via uniform downsampling is simple but may not fit uneven reasoning structures.
• No direct control of interpretability at inference: The model thinks silently; you don’t get visible rationales unless you switch to an explicit mode.

Required resources:

• An LLM backbone (0.6B–8B in the paper) and GPU memory for fine-tuning (e.g., 4×A100 with LoRA in the experiments).
• Access to a reasoner during data creation (e.g., GPT-4o-mini) to produce training-time CoTs.
• Time to run joint training with multi-loss optimization.

When not to use:

• Ultra-low-latency on tiny hardware where even a few latent steps are too costly.
• Domains where training-time reasoning labels are unavailable or too noisy to trust.
• Tasks that require visible, auditable reasoning at inference time (compliance scenarios), unless you run the explicit view.

Open questions:

• Adaptive step scheduling: Can the model learn how many latent steps a query needs on the fly?
• Smarter alignment: Beyond uniform downsampling, can dynamic matching (e.g., attention-based alignment) improve step-to-step mapping?
• Reinforcement learning: Can we directly optimize latent trajectories for retrieval utility, as the authors suggest for future work?
• Safety and bias: How do we detect and mitigate biased or brittle reasoning compressed into the latent space?
• Interpretability: Can we probe or visualize latent tokens to make silent thoughts more understandable without full text generation?

Three-sentence summary: LaSER teaches a retriever to think silently by aligning its hidden, compact reasoning steps (latent tokens) with a teacher’s explicit Chain-of-Thought during training. This multi-grained self-distillation—matching both final rankings and intermediate steps—produces pipeline-level quality without pipeline-level latency. At inference, the model uses only a few latent steps, keeping it fast and practical.

Main achievement: Unifying explicit and implicit reasoning in a single backbone so that explicit CoT can be compressed into latent tokens that actually carry semantic weight, delivering strong reasoning-aware retrieval at near standard retriever speed.

Future directions: Let the model decide how many steps to think per query, align steps with smarter dynamic methods, and use reinforcement learning to optimize entire latent trajectories directly for retrieval outcomes. Explore tools to interpret or audit silent thoughts when needed.

Why remember this: LaSER shows you don’t have to choose between brains (explicit reasoning) and speed (dense retrieval). By teaching silent thoughts to follow explicit logic, it gives you both—making complex search practical in real systems.