Multi-Vector Index Compression in Any Modality

🍞 Hook: Imagine you have a giant scrapbook with pages of text, photos, and even QR codes that link to sounds. You want to find the exact page that answers your question fast. But if you keep every tiny detail from every page, your backpack gets too heavy to carry.

🥬 The Concept (Multi-vector representations): What it is: Many modern search systems break each document (text, image regions, video frames, audio chunks) into lots of little meaning-points called vectors, so the system can match very specific parts to your query. How it works: (1) Chop the document into tokens (words, image patches, frames). (2) Turn each token into a vector using a model. (3) During search, compare query vectors to all document vectors. (4) Add up the best matches to rank results. Why it matters: Without this fine-grained view, the system might miss the exact table cell, frame, or sentence that answers your question. 🍞 Anchor: When you ask, “Show me the video where a dog catches a red frisbee,” the system can match the “red frisbee” token to the exact video frame that shows it.

🍞 Hook: You know how packing every toy you own into one suitcase makes it impossible to lift? Search indexes are like suitcases—they can get too big.

🥬 The Concept (Index compression): What it is: A way to shrink stored vectors so the index is smaller and faster but still helpful for search. How it works: (1) Decide how many vectors you can keep (a budget). (2) Combine or select the most important vectors. (3) Save only those. (4) Use them for search later. Why it matters: Without compression, video, image, and audio-heavy collections become so big they’re slow and too costly to store. 🍞 Anchor: Instead of saving all 10,000 video frames, you keep 32 very informative ones that still let you find that red frisbee.

🍞 Hook: Think of a book report—you read everything first, then decide what to write. That’s different from deciding after every page.

🥬 The Concept (Late interaction): What it is: A search technique that compares query pieces with document pieces at the very end (after both are fully processed), using a “best match” rule. How it works: (1) Turn query into vectors. (2) Turn document into many vectors. (3) For each query vector, find the most similar document vector (MaxSim). (4) Add these best scores to rank the document. Why it matters: Without late interaction, you either miss fine details or do too much heavy computation at the wrong time. 🍞 Anchor: The word “capital” in your question matches best with “Paris” in the document, so the document with “Paris” floats to the top.

🍞 Hook: Libraries hold not just books—there are maps, films, audio archives. Real life is multimodal!

🥬 The Concept (Omni-modal retrieval): What it is: Searching across many types of media—text, images, visual PDFs, videos, and even audio. How it works: (1) Use a model that can encode different media into vectors. (2) Store them in one index. (3) Let a text query find matches in any modality. Why it matters: Without omni-modal retrieval, we can’t search the web as it actually is—rich in pictures, video, and sound. 🍞 Anchor: Typing “drum solo with confetti” retrieves a video clip—even if the query is text and the answer lives in audio+video.

The world before: Late interaction systems like ColBERT made search both accurate and efficient for text, and newer models extended this to images, visual documents, and videos. But they hit a wall: storage and compute grew linearly with document length. One video could need megabytes to index, and a platform like YouTube would require petabytes—unrealistic for many providers.

The problem: Multimodal documents are long and repetitive—silent audio stretches, static scenes, repeated frames—and late interaction stores vectors for everything. Worse, during a full evaluation pass, only about 1% of those vectors actually get used.

Failed attempts: Researchers tried three main families: (1) Sequence resizing (project the whole list into a shorter list). It often underuses its budget, leaving many stored vectors wasted. (2) Memory tokens (append learnable tokens and throw away the originals). It can “mush” information together (collapse), losing distinct details. (3) Hierarchical pooling (greedily merge similar tokens). It’s simple and reduces redundancy but can get tricked by noisy outliers.

The gap: We needed a compression method that is (a) query-agnostic at indexing time, (b) aware of what’s semantically important, (c) removes redundancy without erasing fine details, and (d) flexible across different index sizes.

Real stakes: Smaller, faster, and smarter indexes mean cheaper search, greener compute, and room to include more diverse content (videos with sound, scanned PDFs with complex layouts). For users, that means better answers sooner; for builders, it means running advanced multimodal search within real-world budgets.

🍞 Hook: Picture sorting a huge box of LEGO by first spotting the most special pieces (wheels, windows), then grouping nearby bricks around each special piece.

🥬 The Concept (Attention-Guided Clustering, AGC): What it is: A way to pick the most important tokens (centroids) using learned attention, group the rest by similarity, and average them with importance weights—so a tiny set of vectors still carries the document’s meaning. How it works: (1) Add a few universal query tokens that “look” over the document. (2) Use their attention to score how important each token is. (3) Select the top-m tokens as centroids. (4) Assign every other token to its nearest centroid (hard clustering). (5) Make each final vector by a weighted average using the importance scores. Why it matters: Without attention-guided picks and weighted merging, you either keep noise, collapse details, or merge the wrong things—hurting search quality. 🍞 Anchor: For a sports video, AGC spots key frames like the kickoff and touchdown, groups nearby frames to each, and saves a small, smart summary that still answers “Who scored?”

The Aha! moment (one sentence): If we let learned attention find the few most meaningful spots and then group and weight everything around them, we can compress any modality while keeping the details that matter for retrieval.

Three analogies:

1. Field trip chaperones: Pick a few responsible leaders (centroids) and assign students (tokens) to the nearest leader; louder or more on-topic students get a bit more say (weights).
2. Museum map: Mark top exhibits (centroids), draw sections around them (clusters), and print a small guide where the biggest font highlights what most visitors care about (weights).
3. Grocery store aisles: Choose aisle signs (centroids), shelve similar products nearby (clusters), and give end-caps to best-sellers (weights) so shoppers (queries) find what they need fast.

Before vs. After:

• Before: Compressors either squished sequences blindly (SeqResize), smoothed them into a few learned tokens (MemTok), or greedily merged by similarity (H-Pool), each with trade-offs in redundancy, detail, or robustness.
• After: AGC uses attention to anchor clusters on globally salient tokens and weights aggregation by importance, keeping discriminative details and reducing noise across text, visual documents, and video.

Why it works (intuition, no equations):

• Attention as a spotlight: Universal queries act like trained spotlights that consistently shine on semantically rich tokens—even without knowing the user’s future question.
• Hard clusters preserve edges: Assigning tokens to just one centroid prevents over-smoothing, so important differences don’t blend away.
• Weighted averaging respects density: Some tokens carry more meaning (like an action frame vs. a near-duplicate), so weighting by saliency builds sharper, more useful vectors.

Building Blocks (each as a sandwich):

• 🍞 Hook: You know how a teacher’s eyes naturally land on students raising their hands? 🥬 Attention-based centroid selection: What it is: Learned “universal query” tokens rank document tokens by importance and pick the top ones as cluster centers. How it works: (1) Append universal queries. (2) Collect their attention over tokens. (3) Average across heads/queries. (4) Take the top-m tokens. Why it matters: Without smart centers, clusters might form around boring or noisy spots. 🍞 Anchor: In a lecture PDF, the chosen centers land on section headers, figure captions, and bolded terms.
• 🍞 Hook: Imagine putting magnets down, then each paperclip jumps to the closest one. 🥬 Hard clustering: What it is: Each remaining token joins the nearest centroid based on similarity. How it works: (1) Compute similarity to each centroid. (2) Pick the closest. (3) Repeat for all tokens. Why it matters: Without hard assignment, distinct concepts can blur together. 🍞 Anchor: Frames of a goal celebration go to the “goal” centroid, not the “halftime” centroid.
• 🍞 Hook: Not every word in a paragraph is equally important—names and numbers often matter more. 🥬 Weighted aggregation: What it is: Combine tokens inside each cluster with weights from attention saliency. How it works: (1) Take each token’s importance score. (2) Weighted-average the vectors. (3) Normalize. Why it matters: Without weights, rare but crucial details get drowned out by repeats. 🍞 Anchor: In a table, the row with the exact year mentioned in your query pulls more weight into the final vector.

Put together, AGC keeps indexes tiny yet sharp, so ranking by late interaction stays accurate—even at aggressive compression levels.

At a high level: Multimodal Document → Encode with universal queries → Score token saliency → Select top-m centroids → Assign other tokens to nearest centroid → Weighted-average per cluster → Compressed multi-vector index → Late-interaction search (MaxSim) → Ranked results.

Step-by-step (with the sandwich pattern for each key step):

1. 🍞 Hook: Think of adding coaches to a sports practice who watch and point out the most skilled moves. 🥬 Add universal queries and encode: What it is: Append a small set of learnable “universal query” tokens to the document tokens and run a bidirectional transformer. How it works: (1) Tokenize text, split images into patches/regions, sample video frames, chunk audio. (2) Append K universal queries. (3) Encode everything with a transformer to get last-layer states and attention maps. Why it matters: Without these overseer tokens, we don’t get a reliable signal about which document parts are globally meaningful. 🍞 Anchor: A scanned contract page plus universal queries produces token states where headings and signatures stand out in attention.

2. 🍞 Hook: Like shining spotlights on a stage to see which actors the audience watches most. 🥬 Compute saliency from attention: What it is: Use attention from universal queries to each document token to get importance scores. How it works: (1) Take the last-layer attention from each universal query to all tokens. (2) Average across heads and across queries. (3) Get one saliency score per token. Why it matters: Without saliency, we might pick centers based on mere geometric closeness instead of meaning. 🍞 Anchor: In a news article, names in the headline and bold subheads receive high saliency.

3. 🍞 Hook: Choose team captains first, not at random. 🥬 Select top-m centroids: What it is: Pick the m tokens with highest saliency as cluster centers. How it works: (1) Sort tokens by saliency. (2) Take the top m as centroids. (3) Record their vectors as initial cluster representatives. Why it matters: Random or greedy-local choices may anchor clusters on noise; saliency picks semantically rich anchors. 🍞 Anchor: For a product page PDF, centroids land on the product name, price, key spec bullets, and the main figure caption.

4. 🍞 Hook: Every student stands by the chaperone whose sign matches their class. 🥬 Hard assign tokens to nearest centroid: What it is: Each non-centroid token joins the closest centroid by cosine similarity. How it works: (1) Compute similarity between a token and all centroids. (2) Pick the best one. (3) Repeat for all tokens. Why it matters: Without hard assignment, clusters can blur; with it, distinct ideas remain distinct. 🍞 Anchor: Video frames of a fireworks finale join the “finale” centroid, not the “intro” centroid.

5. 🍞 Hook: When mixing a fruit salad, you don’t add equal amounts of everything—you add more of the favorites. 🥬 Weighted aggregation per cluster: What it is: Make one final vector per cluster by a weighted average using saliency scores. How it works: (1) For each cluster, multiply token vectors by their saliency. (2) Sum them. (3) Divide by total saliency to normalize. Why it matters: Without weights, many duplicates can drown out rare but crucial details. 🍞 Anchor: In an academic PDF, the exact phrase matching the query (“p-value < 0.05”) gets more pull inside its cluster vector.

6. 🍞 Hook: Put your small, neat suitcase in the cargo hold so check-in is quick. 🥬 Build the compressed index: What it is: Store m vectors per document, not thousands. How it works: (1) Repeat steps 1–5 for every document. (2) Save the m vectors to a retrieval index (e.g., FastPlaid). Why it matters: Without a compact index, storage and search time explode, especially for video+audio. 🍞 Anchor: A million videos now fit in your server budget because each has 32 smart vectors instead of megabytes of frames.

7. 🍞 Hook: During a scavenger hunt, you match each clue to the best item you can find. 🥬 Late interaction ranking (MaxSim): What it is: For a query, compare each query vector to all document vectors and add the best matches to score the document. How it works: (1) Encode query into vectors. (2) For each query vector, find the highest dot-product among the document’s m vectors. (3) Sum those maxima to get the score. (4) Rank all documents by score. Why it matters: Without MaxSim, fine details like a number, name, or object can get washed out. 🍞 Anchor: The query “red frisbee catch” finds the document vector representing the catch frame and ranks that video high.

Comparative baselines (what, why they break):

• 🍞 Hook: Squishing a long loaf into a smaller pan doesn’t guarantee the slices you want are on top. 🥬 SeqResize: What it is: A small neural network maps the long sequence into m vectors directly. How it works: (1) Encode fully. (2) Pad/truncate to a fixed length. (3) Project along the sequence dimension to m. Why it matters: Without selection, it may underuse or over-focus on a few positions; performance can plateau across budgets. 🍞 Anchor: On MSR-VTT, many stored vectors end up barely used.
• 🍞 Hook: Sticky notes can help, but if you throw away the whole book and keep only the notes, you might miss details. 🥬 MemTok: What it is: Append m learnable tokens; after encoding, keep only their final states. How it works: (1) Add memory tokens. (2) Self-attend across document+mem. (3) Keep memory positions only. Why it matters: Can cause information collapse—tokens become too similar and lose discriminative power. 🍞 Anchor: Visual tokens all look alike in similarity maps, so nuances vanish.
• 🍞 Hook: Merging duplicate photos saves space, but if you merge too greedily, an outlier can mess up groups. 🥬 H-Pool: What it is: Iteratively merge most similar tokens (agglomerative), replacing pairs with their mean. How it works: (1) Compute pairwise distances. (2) Merge by Ward linkage until m remain. Why it matters: Removes redundancy, but can be sensitive to noise; no attention to meaning. 🍞 Anchor: It often does well non-parametrically, but learned methods beat it in many multimodal cases.

Secret sauce: AGC mixes learned attention (to pick meaningful centers) with hard clustering (to preserve distinctions) and weighted averaging (to highlight signal over repeats). This trio targets exactly what multimodal retrieval needs: keep discriminative details, trim redundancy, and fit any budget.

🍞 Hook: You know how report cards make more sense when you know the class, the grading scale, and how other students did? That’s how we’ll read the results here.

The tests (what and why):

• BEIR (text): Diverse text retrieval tasks—medical, finance, arguments—to see if compression still works well for plain documents.
• ViDoRe v2 (visual documents): Scanned/visual PDFs where meaning lives in both text and layout (figures, tables, headers).
• MSR-VTT (video, vision-only): Retrieve a single correct video per text query among 1k candidates—stress test for matching key frames.
• MultiVENT 2.0 (audiovisual video): Many queries, large video pool, and audio matters—tests if methods can handle both sight and sound.

How we score: Recall@k (did the right item appear in the top k?) and nDCG@k (are the right items ranked near the top?). We also track percent of baseline (compressed vs. full model) and compression budgets (e.g., 32 or 64 vectors per document).

The competition: SeqResize, MemTok, H-Pool, and the full, uncompressed multi-vector baseline where possible. Retrieval engine uses Flat or FastPlaid depending on feasibility; large uncompressed visual indexes often didn’t fit.

Scoreboard with context:

• Big picture (Table 1): AGC is the most reliable learned compressor across modalities, typically preserving about 97% of uncompressed performance at nDCG@10. That’s like getting an A when others drop to B or B+ under the same space limit.
• Standout result: On MSR-VTT, AGC at budgets 32 and 128 achieves higher R@1 than the full index—like cleaning your room and finding things faster because the clutter is gone. This suggests compression can reduce noise in multimodal inputs and even improve accuracy.
• H-Pool: Strong among non-parameteric methods, often beating SeqResize and sometimes MemTok on non-text tasks. But AGC tends to edge it out while offering better flexibility across index sizes.

Per-benchmark highlights:

• BEIR (text): With a tiny budget (e.g., 32 vectors), AGC and MemTok are close and stable; H-Pool varies more by dataset. Text has less redundancy than video, so gains from attention-guided selection are smaller but still present. Overall, learned methods keep performance surprisingly high despite ~80% sequence compression in many corpora.
• ViDoRe v2 (visual documents): AGC and H-Pool significantly outperform SeqResize and MemTok. AGC is more stable across domains (Biomedical, Economics, ESG-English). Some uncompressed visual indexes didn’t fit memory, reinforcing why compression is necessary.
• MSR-VTT (video): Every compressed method sets a new SOTA versus prior multi-vector and dense baselines, even at extreme budgets (e.g., 5 vectors/document). AGC notably surpasses the uncompressed baseline at R@1 for budgets 32 and 128—evidence that removing redundancy helps retrieval zoom in on meaningful frames.
• MultiVENT 2.0 (audiovisual): Full uncompressed index was infeasible; compression was required. AGC remains competitive, though audio sampling limits in the backbone (e.g., needing 4 KHz for batch size constraints) suggest future work to ingest audio more efficiently.

Surprising/insightful findings:

• Index underuse: In a full pass, only ~1% of tokens are actually used by late interaction. Carrying thousands of vectors per doc is overkill—compression matches how the system really behaves.
• SeqResize plateau: Changing the budget often didn’t change its performance much—analysis shows it underutilizes many of its stored vectors.
• MemTok collapse: Similarity heatmaps show over-smoothing; many vectors look alike, hurting discriminative power.
• H-Pool diversity: Produces the most diverse vectors (low inter-token similarity), which helps—but without attention to meaning, it can miss semantic anchors.
• Utilization predicts performance: Evenness of MaxSim matches across positions correlates with better retrieval metrics (strong Pearson r). While based on few samples, it hints you can estimate which compression will work well without building huge indices.

Takeaways: AGC balances diversity and meaningful overlap. It selects semantically strong centroids, preserves differences with hard clustering, and boosts signal with weights. That’s why it beats or matches others across text, visual PDFs, and videos, and sometimes even improves over the full index.

Limitations (be specific):

• Training cost and plumbing: AGC needs attention maps from a bidirectional encoder and learnable universal queries. This adds training complexity versus purely heuristic pooling.
• Query-agnostic by design: Since compression happens before seeing the user’s query, AGC preserves generally salient content, not tailor-made for a specific question. Good retrieval models help, but there’s still a gap from ideal query-aware selection.
• Audio bottlenecks: On large audiovisual sets (e.g., MultiVENT 2.0), practical limits like audio sampling rates and memory pressure can cap performance; better audio ingestion strategies are needed.
• Model dependence: Bigger/stronger backbones improve AGC (e.g., moving from 3B to 7B, or newer Qwen versions), so results hinge on the quality of the encoder.

Required resources:

• Compute/GPU memory to fine-tune multimodal encoders with universal queries and attention capture. Storage for building FastPlaid/flat indexes at chosen budgets.
• For videos: frame sampling pipelines; for PDFs: vision-language extraction (text + layout tokens). For audio: careful downsampling to fit training, with quality trade-offs.

When NOT to use:

• Tiny text-only apps where a single dense vector already works great and storage is trivial; the added complexity of multi-vector and late interaction may not pay off.
• Highly structured, short documents with little redundancy; simple indexing may suffice without compression.
• Real-time streaming scenarios where precomputing and storing multi-vectors per chunk isn’t feasible and latency is ultra-strict.

Open questions:

• Adaptive budgets: Can we allocate more vectors to complex documents and fewer to simple ones, guided by light saliency stats?
• Better universal queries: How many do we need, and can they be specialized by modality (text/image/video/audio) to boost centroid quality?
• Audio efficiency: How to pass richer audio (≥16 KHz) without blowing memory, so audiovisual retrieval shines?
• Utilization-aware training: If evenness of MaxSim matches predicts performance, can we directly regularize for balanced token use?
• Hybrid methods: Can we blend AGC with lightweight residual storage or product quantization to capture very fine details at almost no extra space?

Three-sentence summary: This paper introduces Attention-Guided Clustering (AGC), a query-agnostic way to compress multi-vector representations across text, visual documents, and videos. By selecting salient centroids with attention, hard-clustering nearby tokens, and weighting their aggregation, AGC keeps discriminative details while trimming redundancy. Experiments show AGC consistently matches or exceeds other compression methods and can even outperform a full, uncompressed index on MSR-VTT.

Main achievement: A single, modality-agnostic compression recipe that respects what late interaction needs—fine detail and smart coverage—while making indexes small, fast, and flexible.

Future directions: Build adaptive budgets per document based on quick saliency checks; refine universal queries per modality; improve audio ingestion efficiency; and train with utilization-aware objectives so every stored vector reliably “earns its keep.”

Why remember this: Most multimodal indexes carry far more tokens than they actually use; AGC shows how to keep only the few that matter most, without losing—and sometimes improving—search quality. That shift makes web-scale, truly multimodal retrieval more affordable and more accurate, opening the door to better search experiences across text, images, PDFs, videos, and sound.