MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning

Top Bread (Hook): Imagine trying to understand a busy day from your family's photo album: morning breakfast, a soccer game, grocery shopping, then dinner. You wouldn’t look at just one photo—you’d flip through many and connect the dots. Filling: What it is: Before this work, most AI reasoning tests focused on single images or on puzzles and expert trivia, not the kind of multi-photo, real-life sense-making we do daily. How it works (history): 1) Language models got good at math and science problems using step-by-step text thinking. 2) Multimodal models learned to look at images and text together. 3) But tests often used either single images or synthetic puzzles (like abstract grids) or expert-heavy questions. Why it matters: Everyday life is not a single snapshot or a classroom exam—it’s a stream of scenes with people moving, things changing, and events unfolding. Bottom Bread (Anchor): Picking the next frame in a driving scene is different from answering a textbook physics question; it demands tracking positions, motion, and timing across several pictures.

Top Bread (Hook): You know how a scavenger hunt makes you gather many separate clues to find the prize? You rarely solve it from a single hint. Filling: The Problem: Existing multimodal tests don’t reflect how we reason in the real world: they (1) overuse symbolic or expert tasks and (2) usually show just one image. That hides big gaps in AI’s ability to connect multiple moments, views, and causes. Failed Attempts: Two main routes came up short—knowledge-heavy exams (great for facts, not daily sense-making) and synthetic puzzles (great for pure logic, not lived reality). The Gap: We need a benchmark that (a) uses natural, real-life images, (b) demands integrating multiple pictures, and (c) measures many reasoning types at once without requiring specialized knowledge. Why it matters: Without this, we might think AIs are good reasoners when they’re only good at niche tasks or cherry-picked examples. Bottom Bread (Anchor): An AI might ace a chart-based puzzle but still fail to order four crowd photos by time—exactly the kind of skill you need for safety cameras or home robots.

Top Bread (Hook): Imagine building a fair obstacle course to test how well a bike rides in real streets—not just on a smooth gym floor. Filling: What it is: MMR-Life is a benchmark designed to evaluate everyday, multi-image reasoning across seven thinking styles. How it works (story): 1) Gather natural, related images from real sources (homes, traffic, sports, nature). 2) Create multi-image questions that require joining clues across pictures. 3) Cover a wide toolkit of reasoning types (abductive, analogical, causal, deductive, inductive, spatial, temporal). 4) Use multiple choice so answers are comparable. 5) Filter out too-easy or ambiguous items. Why it matters: It tests what truly breaks in the wild—like understanding motion, space, and time across scenes. Bottom Bread (Anchor): A temporal task asks: “Which frame comes next?” from a dashcam sequence; a spatial task asks: “How did the camera rotate between these two views?” That’s everyday perception, not a riddle.

Top Bread (Hook): Think of checking a team’s skills in passing, defense, speed, and stamina—not just scoring goals. Filling: What it is: Seven reasoning types model the different “mental muscles” we use in life. How it works: Each type gets several tasks—like step order for recipes (deductive) or route planning (spatial)—to probe depth, not just surface. Why it matters: If a model only excels at analogies but fails at time and space, it won’t be safe or reliable in real-world applications. Bottom Bread (Anchor): A robot that can name objects (easy) but can’t predict where a ball will go next (temporal/causal) is not ready for your living room.

Top Bread (Hook): You know how assembling a big LEGO set needs many kinds of skills—reading the guide (rules), spotting similar pieces (analogies), guessing what went wrong (abduction), and making sure pieces fit in space and time? A single skill won’t finish the model. Filling: The “Aha!” Moment in one sentence: To fairly judge multimodal AI reasoning, we must test how it pieces together multiple real images across seven core reasoning types, not just single-image or expert quizzes. How it works: 1) Use only natural, real-life images, often related in space or time. 2) Build tasks that require combining clues across images. 3) Ensure each problem is solvable with commonsense, not niche knowledge. 4) Make all questions multiple-choice with carefully curated wrong answers to avoid shortcuts. 5) Analyze model behavior across types and thinking styles. Why it matters: This shows what today’s models truly can’t do—especially spatial, temporal, and causal sense-making—so we can fix it. Bottom Bread (Anchor): A task might show four mall-camera images and ask for the correct order; solving it needs tracking recurring people and their movement across cameras, not memorizing trivia.

Multiple Analogies for the same idea:

• Toolbelt analogy: Different fixes need different tools; MMR-Life checks if the AI’s whole toolbox works, not just its favorite screwdriver.
• Detective analogy: Real cases use many photos, times, and angles; this benchmark checks if AI can connect those clues like a careful detective.
• Sports tryout analogy: It’s not enough to shoot three-pointers—you must also pass, defend, and read plays; MMR-Life runs drills for all seven skills.

Before vs After:

• Before: We judged AI with single pictures, symbolic puzzles, or expert-heavy questions and assumed good scores meant broad reasoning.
• After: We can now see uneven strengths across seven real-life reasoning types and pinpoint weak spots like space, time, and cause.

Why it works (intuition, not equations):

• Natural multi-image inputs force models to build internal maps of who/what/where/when.
• Diverse tasks reduce shortcut learning; models can’t coast on memorized patterns.
• Multiple-choice with carefully designed distractors avoids guessable templates and tests genuine reasoning.

Building Blocks (each with a Sandwich explanation):

1. Multimodal Multi-image Reasoning

• Top Bread (Hook): Imagine solving a mystery from several security photos, not just one.
• Filling: What it is: Using more than one natural image (plus text) to reach a conclusion. How it works: (1) Look across images for shared people/objects. (2) Track changes and relations (who moved, what changed). (3) Combine clues to answer. Why it matters: One photo can hide key steps; multiple photos reveal the story. Bottom Bread (Anchor): Ordering four store-aisle photos by time.

1. Reasoning Types

• Top Bread (Hook): Like a toolbox with seven different tools for seven kinds of jobs.
• Filling: What it is: A set of thinking styles we use in daily life. How it works: Each type stresses a different skill (e.g., spotting rules vs. predicting effects). Why it matters: Strong AI needs balance, not one-trick skills. Bottom Bread (Anchor): A recipe task (deductive) vs. a driving next-frame task (temporal) probe different skills.

1. Abductive Reasoning

• Top Bread (Hook): Like guessing why the cookie jar is open.
• Filling: What it is: Choosing the most plausible explanation for an observed event. How it works: (1) Notice the event. (2) List possible causes. (3) Pick the most likely cause. Why it matters: Life often gives effects first; we infer causes. Bottom Bread (Anchor): Girl opens fridge—most plausible reason among options.

1. Analogical Reasoning

• Top Bread (Hook): Spotting how two pairs match in the same way.
• Filling: What it is: Transferring a relation from a known pair to a new pair. How it works: (1) Find relation A↔B. (2) Apply same relation to C. (3) Choose D that fits C↔D. Why it matters: It’s key for generalization from examples. Bottom Bread (Anchor): Choosing an animal that relates to a third one like the first pair did.

1. Causal Reasoning

• Top Bread (Hook): Knowing pushing a ball makes it roll.
• Filling: What it is: Understanding what causes what. How it works: (1) Identify candidate causes. (2) Predict likely effects. (3) Check with physical intuition. Why it matters: Without it, models mix up what comes from what. Bottom Bread (Anchor): Predict which object moves after collisions.

1. Deductive Reasoning

• Top Bread (Hook): Using a recipe to cook in the right order.
• Filling: What it is: Applying general rules to specific cases. How it works: (1) State rules. (2) Match facts. (3) Conclude what must be true. Why it matters: Ensures logically certain steps. Bottom Bread (Anchor): Reordering cooking steps from images.

1. Inductive Reasoning

• Top Bread (Hook): Seeing many examples to spot a pattern.
• Filling: What it is: Forming a general rule from observations. How it works: (1) Compare examples. (2) Find shared features or trends. (3) Predict next case. Why it matters: Helps with learning from data. Bottom Bread (Anchor): Bird migration heatmaps over years to predict next season.

1. Spatial Reasoning

• Top Bread (Hook): Figuring how to rotate your phone to match a photo.
• Filling: What it is: Understanding positions and orientation. How it works: (1) Map objects in space. (2) Track rotations/moves. (3) Plan routes or compare views. Why it matters: Navigation and camera changes depend on it. Bottom Bread (Anchor): Estimating camera rotation between two pictures.

1. Temporal Reasoning

• Top Bread (Hook): Sorting a comic strip into the correct order.
• Filling: What it is: Understanding order and duration. How it works: (1) Identify recurring people/objects. (2) Track progress. (3) Decide what comes next or when something happens. Why it matters: Forecasting and surveillance rely on it. Bottom Bread (Anchor): Predicting the next sports scene or next driving frame.

Overview (like a recipe): Input (natural, related images and a question) → Step A: Curate and group real-life images → Step B: Design seven reasoning types and tasks → Step C: Generate questions and answers (some automatic, some manual) → Step D: Create high-quality wrong options → Step E: Filter for quality and difficulty → Step F: Evaluate many models and analyze their thinking → Output: Accuracy per reasoning type and insights about reasoning patterns.

Step A: Curate and group real-life images

• What happens: Collect natural images from public datasets, web screenshots, videos (by frame extraction), and carefully chosen existing resources; group them when they share a real relationship (same scene, time, or activity).
• Why this step exists: Multi-image reasoning needs meaningful links between images (e.g., time series), not random piles.
• Example: Extract six clear frames from a public driving video showing a car approaching an intersection; these frames become the input set for a “what comes next?” question.

Step B: Design seven reasoning types and 21 tasks

• What happens: Map real-life needs to seven core skills: abductive, analogical, causal, deductive, inductive, spatial, temporal. For each type, craft tasks that naturally require that skill, like recipe-step deduction (deductive) or camera rotation estimation (spatial).
• Why this step exists: A single skill can’t capture everyday reasoning; we need a balanced toolkit.
• Example: Temporal—crowd timeline reconstruction asks for ordering four surveillance photos by time using people positions and motion cues.

Step C: Generate question–answer pairs (automatic or manual)

• What happens: If image sets have explicit structure (e.g., a true time order), write code and rules to auto-build Q&A; if the reasoning requires implicit understanding (e.g., plausible motives), have humans author questions and gold answers to ensure commonsense quality.
• Why this step exists: Some tasks are structural (easy to synthesize); others hinge on subtle, human-level judgment.
• Example: For recipe images, auto-generate a “Which order?” multiple-choice from a known correct sequence; for abductive scenes, humans craft the most plausible explanation and four tempting but wrong alternatives.

Step D: Create strong negative options (distractors)

• What happens: Use rules for image distractors (e.g., pick too-early or too-late frames) and prompt multiple advanced models to propose wrong text options; then humans select and polish the four best distractors.
• Why this step exists: Weak distractors make tests easy to game; strong distractors force real reasoning.
• Example: A driving-next-frame question’s distractors include frames from before the car turned or far after the turn, all visually similar but logically wrong.

Step E: Quality control with three filters

• What happens: 1) Difficulty filter: If three small MLLMs all solve an item, remove it as too easy. 2) Format filter: Adjust option lengths and styles to avoid telltale patterns. 3) Quality filter: Remove ambiguous, multi-correct, or expert-only items.
• Why this step exists: Keeps the benchmark fair, realistic, and challenging without hidden shortcuts.
• Example: If a Gemma 4B-sized model, a Qwen 7B, and an InternVL 8B all get a question right, that question is dropped.

Step F: Evaluate and analyze

• What happens: Run 37 models (with and without “thinking” styles like long Chain-of-Thought) in a consistent zero-shot setup; average across runs for stability; measure accuracy per reasoning type; analyze how thinking length and methods influence performance; cluster reasoning types.
• Why this step exists: We want apples-to-apples comparisons and deeper insights than a single overall score.
• Example: Compare a no-CoT run and a CoT run of the same model to see which reasoning types benefit from longer thoughts.

The Secret Sauce (what makes it clever):

• Natural, multi-image, real-life focus means models must really track people, places, and changes—not just tag objects.
• Seven reasoning types expose a skill profile; we can see exactly where models stumble (space/time/cause) rather than only a single grade.
• Strong distractors and difficulty filtering prevent shortcut hacking and inflate true reasoning effort.
• Thinking-pattern analysis (length, methods like self-consistency/best-of-N/RL, and type clustering) turns scores into guidance for training better models.

Concrete running example across steps:

• A temporal sequence question uses four store-camera images (input) → task label: temporal → human or code builds the correct order and four distractor orders → small models try it; if all pass, drop it; else keep it → final evaluation shows many big models still struggle to track the same couple across cameras, revealing a true weak spot.

The Test: The benchmark includes 2,646 multiple-choice questions from 19,108 images, spread across seven reasoning types. Models answer by selecting one of five options per question. Accuracy is reported both overall and by reasoning type to expose strengths and weaknesses.

The Competition: 37 models were evaluated, including non-thinking and thinking versions of leading systems. Humans also took a mini version to provide a reality check for what skilled people can do without web lookup.

The Scoreboard with Context:

• Overall: The top model reached about 58.69%—that’s like a solid C when the human comparison is about 72.28% (a comfortable B). Many open-source models scored below 40%, and some dipped near or below random-guess territory on tough categories.
• By type: Models often did well on analogical, deductive, and inductive reasoning—these are like pattern-matching and rule-using strengths. But they stumbled badly on spatial, temporal, and causal reasoning—the very skills needed to follow movement, predict what comes next, or connect causes to effects in real scenes. For example, the best spatial accuracy hovered around the mid-20% range, far behind human performance.

Surprising Findings:

• Longer thoughts aren’t magic. Adding Chain-of-Thought helped some types (like analogies) but not others (like induction), and sometimes even hurt. More words doesn’t always mean better reasoning.
• Reinforcement learning (RL) trained models didn’t generalize as well as simple best-of-N selection on small models, suggesting overfitting to training-style problems. In some cases, RL models trailed the base models using plain CoT.
• Reasoning types cluster: Analogical and inductive go hand in hand (both depend on generalizing from features), while spatial sits far away from the rest—suggesting it needs targeted training signals that other tasks don’t provide.

What this means in plain terms: Today’s models may recite patterns and follow rules, but they aren’t yet reliable at building internal maps of the world—where things are, how they move, and what will happen next. MMR-Life turns this from a hunch into measurable evidence.

Limitations:

• Even though images are natural, they are still curated; real field deployments can be noisier or messier (e.g., extreme lighting, occlusions). Multiple-choice simplifies evaluation but doesn’t capture open-ended reasoning or explanations. Some tasks use frames from videos or web screenshots; despite checks, subtle distribution biases can remain. Spatial and temporal demands are high; some models may underperform due to current vision backbones rather than pure reasoning.

Required Resources:

• Running all tasks at scale benefits from GPUs and batch processing for visual encoders. To reproduce results, users need access to the image sets (released) and consistent prompts. For analysis beyond accuracy (like thinking-length sweeps), APIs or code that control decoding length and sampling are helpful.

When NOT to Use:

• If your goal is expert knowledge recall (e.g., chemistry formulas) or single-image captioning, specialized benchmarks may be better. If you need free-form long-answer grading or generation quality (not multiple-choice accuracy), consider complementary datasets.

Open Questions:

• How can we best train spatial and temporal representations that generalize—video pretraining, 3D understanding, or synthetic-to-real curricula? Why does RL underperform best-of-N on small models here—overfitting, reward misspecification, or exploration limits? Can we design thinking controllers that adapt length per task type, instead of using a one-size-fits-all Chain-of-Thought? What new negative-option generation methods further reduce shortcuts while maintaining fairness? How can we extend from recognition to world modeling (predictive, counterfactual, and uncertainty-aware reasoning) using this benchmark family?

Three-sentence summary: MMR-Life is a new benchmark that tests how AI pieces together multiple real-life images across seven essential reasoning types, without relying on expert trivia. Across 37 models, it reveals strong skills in analogies and rules but persistent weaknesses in space, time, and cause—leaving a clear gap from human performance. Its analyses of thinking length, training methods, and type clusters turn scores into a roadmap for building truly general multimodal reasoners. Main Achievement: Creating a realistic, multi-image, multi-skill benchmark—plus a comprehensive evaluation and analysis suite—that exposes where current multimodal models actually break in the wild. Future Directions: Targeted training for spatial/temporal/causal skills; adaptive thinking strategies that tailor Chain-of-Thought length by task; improved reward models and search strategies that generalize; extensions from multiple-choice toward structured explanations and free-form predictions. Why Remember This: MMR-Life moves evaluation from puzzles and trivia to the everyday visual stories we live in—showing what today’s AI can and cannot do, and lighting the path to make it truly world-ready.