HiFi-Inpaint: Towards High-Fidelity Reference-Based Inpainting for Generating Detail-Preserving Human-Product Images

You know how when you see a cool poster online, you can immediately spot if a brand logo looks a little off or the pattern on a shoe is slightly wrong? Your eyes are really good at catching tiny mistakes. AI image generators used to be great at making overall nice pictures, but they often smudged or guessed the small stuff, like tiny letters, stitching, or exact label shapes. That was the world before this paper.

The World Before: Diffusion models made huge progress in turning text into images. They could place objects, choose colors, and make scenes that look realistic. But for human–product images—like a person holding a bottle, wearing a watch, or showing a snack—the smallest details really matter. A missed curve in a logo or a fuzzy barcode can make the whole image feel untrustworthy. People tried using general text-to-image prompts or broad editing tools, but these focus on big-picture changes and often lose fine patterns, textures, and branding.

The Problem: The hardest part is preserving high-fidelity product details. When you replace a masked area (say, the person’s empty hand) with a specific product from a reference photo, you need the exact shape, logo, label text, and texture to match. Diffusion models tend to average things during denoising. That smoothing makes tiny letters blur and sharp patterns fade.

Failed Attempts: People tried two main routes. First, free-form text editing: “Put a blue water bottle in the hand.” This can put something there, but the exact bottle from the reference often gets lost—logos bend, text melts, and textures don’t match. Second, reference-based methods tried to look at a product image while filling the hole, but they didn’t strongly enforce local, pixel-precise alignment, so details still drifted.

The Gap: We needed a method that (1) pays special attention to the product’s fine details, (2) actually guides the model with a detail map rather than hoping it figures it out, (3) uses a training signal that punishes losing those details at the pixel level, and (4) has enough diverse, high-quality data to learn this skill well.

Real Stakes: This matters in everyday life. When you shop online, you want to see the actual product, not a blurry guess. For ads and e-commerce, brand trust depends on precise visuals—logos, labels, and textures must be right. Accurate images help customers make better choices, reduce returns, and protect brand identity. They also speed up content creation for businesses, letting teams produce consistent, high-quality photos at scale.

🍞 Top Bread (Hook): Imagine finishing a jigsaw puzzle using the box cover to know exactly what the missing piece should look like.

🥬 The Concept (Reference-based inpainting): Reference-based inpainting is an AI technique that fills a missing part of an image using a separate reference picture as a guide.

• How it works:
  1. Take a photo with a hole (the masked region) and a reference photo of the object you want to place.
  2. The model studies the reference’s look (shape, colors, patterns).
  3. It paints inside the hole to match the reference while blending with the scene.
• Why it matters: Without using the reference, the model might invent a “similar” object that’s wrong—off-brand logos, smeared labels, or wrong shapes.

🍞 Bottom Bread (Anchor): Think of putting a specific soda can into a person’s hand: reference-based inpainting makes sure it’s that exact can, with the right logo and text, not just any red can.

The Aha! Moment in one sentence: If you want crisp, trustworthy human–product images, explicitly highlight and protect the sharp details during both generation and training.

Analogy 1 (Microscope): You know how a microscope makes tiny things clear? HiFi-Inpaint uses a special “sharp-detail map” to magnify the parts of the product that must stay crisp. Analogy 2 (Tracing Paper): Imagine laying tracing paper over a logo to copy every edge exactly. The model uses a high-frequency detail map to trace the logo’s fine edges while it paints. Analogy 3 (Two Chefs): One chef cooks the whole dish, while another chef focuses on seasoning only the most important spots. The main model paints the scene; a shared attention branch adds precise detail only where needed.

Before vs After:

• Before: Models blended and averaged details during denoising; tiny text and logos often went fuzzy.
• After: The model is guided by high-frequency maps and trained with a detail-focused loss, so letters, edges, and textures stay sharp and faithful to the reference.

Why It Works (intuition, not equations):

• High-frequency map = a spotlight on edges, small text, and fine textures. By feeding this map into the model, we tell it where the precious details live.
• Shared Enhancement Attention (SEA) = a shared-weights helper branch that injects those detail cues only inside the mask. It’s like whispering, “Pay extra attention here.”
• Detail-Aware Loss (DAL) = a teacher that checks the student’s work at the pixel level for sharp details. If edges or tiny letters get smudged, the loss nudges the model to fix them.
• Together, they solve the averaging problem: the model no longer treats all regions equally; it prioritizes the right details in the right place.

🍞 Top Bread (Hook): You know how a blacklight reveals hidden ink on a poster?

🥬 The Concept (High-frequency map): A high-frequency map is a special image that highlights the sharp, detailed parts—like edges, tiny text, and fine patterns.

• How it works:
  1. Convert the product image into the frequency world (where slow changes and quick changes live).
  2. Filter out the slow, smooth stuff and keep the quick, sharp changes.
  3. Turn it back into a picture that glows where details are.
• Why it matters: Without this map, the model doesn’t know which tiny parts must be preserved; it may blur them by accident.

🍞 Bottom Bread (Anchor): If the product has a thin gold border and small letters, the high-frequency map lights those up so the model keeps them crisp.

🍞 Top Bread (Hook): Picture two synchronized dancers mirroring each other so the routine stays tight.

🥬 The Concept (Shared Enhancement Attention, SEA): SEA is an attention module that runs a twin branch with shared weights to inject high-frequency detail cues into the main painting process, but only inside the mask.

• How it works:
  1. The main branch processes the usual visual tokens (scene + product).
  2. A twin branch processes tokens built from the product’s high-frequency map.
  3. A learnable weight mixes the twin branch’s detail hints back into the main branch, only where the mask says to paint.
• Why it matters: Without SEA, detail cues can get lost or spill into the wrong places; SEA keeps them strong and targeted.

🍞 Bottom Bread (Anchor): When adding a watch to a wrist, SEA boosts the tiny ticks and logo on the watch face only on the wrist area—not the whole photo.

🍞 Top Bread (Hook): Imagine your art teacher zooms in and checks your pencil strokes on the smallest parts of your drawing.

🥬 The Concept (Detail-Aware Loss, DAL): DAL is a training rule that compares only the sharp-detail parts of the predicted image to the ground truth, inside the mask.

• How it works:
  1. Build high-frequency maps for both the predicted image and the real image.
  2. Compare them pixel by pixel within the masked area.
  3. Penalize differences so the model learns to match fine details.
• Why it matters: Without DAL, training mostly rewards getting the big picture right and may ignore tiny errors that people still notice.

🍞 Bottom Bread (Anchor): If the product’s label has 6-point text, DAL makes sure those tiny letters stay readable instead of turning into gray smudges.

At a high level: Text prompt + Masked human image + Product reference image → Encode and merge tokens → Guide with high-frequency maps → Enhance with SEA inside the mask → Decode to a sharp, faithful human–product image, trained with global and detail-aware losses.

Step 1: Build the right training data (HP-Image-40K)

• What happens: The authors auto-generate paired data: a clean product image, a human–product image, a matching text description, and a mask showing where the product goes. They use a “diptych” trick (left: product, right: person-with-product), then split, check consistency with object detection, CLIP similarity, and text/logo matching, and keep only high-quality pairs.
• Why this exists: Without lots of clean, diverse pairs, the model won’t learn precise product placement and identity.
• Example: If the left side shows a “ZENLUX” bottle and the right side shows someone holding it, the system verifies the bottle regions match and the text “ZENLUX” appears on both.

🍞 Top Bread (Hook): Think of a giant sticker book where every sticker (product) has a matching scene page (human holding it).

🥬 The Concept (HP-Image-40K dataset): HP-Image-40K is a 40,000+ example collection of carefully filtered human–product pairs for training.

• How it works:
  1. Generate diptych images that keep product identity consistent.
  2. Split and detect the product region; check visual and text matches.
  3. Create masks and text captions; keep only reliable pairs.
• Why it matters: Without a strong dataset, the model can’t reliably learn exact shapes, logos, and tiny text under many conditions.

🍞 Bottom Bread (Anchor): The dataset teaches the model “this exact tube with THIS label goes in THIS hand,” over and over, so it becomes precise.

Step 2: Turn images into tokens and merge conditions

• What happens: The model encodes the ground-truth image (with noise), the masked human image, and the product image into tokens. It concatenates these visual tokens so the model can jointly reason about them.
• Why this exists: Without merging the conditions, the model might ignore either the human context or the product reference, causing mismatch.
• Example: Putting a tall bottle into a small hand must respect both the hand’s pose (from the masked human image) and the exact bottle shape (from the product reference).

🍞 Top Bread (Hook): Imagine stacking transparent sheets—one with the human pose, one with the product, and one with the target image outline—so you can see how they line up.

🥬 The Concept (Token merging mechanism): Token merging is the step where visual tokens from different inputs (masked human, product, noisy target) are concatenated so the model can fuse the information.

• How it works:
  1. Encode each image into tokens.
  2. Concatenate them into one sequence.
  3. Let the transformer attend across all tokens together.
• Why it matters: Without this, the model can’t connect the product to the right place in the human image.

🍞 Bottom Bread (Anchor): It’s like putting the product transparency on top of the hand transparency so the model sees they must match.

Step 3: Highlight sharp details with a high-frequency map

• What happens: The product image is converted into a high-frequency map (keeps edges and tiny textures, removes smooth color fields). The same idea is used for training supervision later.
• Why this exists: The model needs a clear pointer to what must stay crisp.
• Example: A bottle’s thin label border and small lettering pop out in the high-frequency map, telling the model “don’t blur this.”

Step 4: Enhance details inside the mask with SEA

• What happens: In each dual-stream visual transformer block, a twin branch (sharing weights) processes the high-frequency tokens. A learnable weight blends the twin’s output into the main branch, but only inside the masked region.
• Why this exists: Without SEA, detail cues may be weak or spread into the background; SEA focuses them exactly where inpainting happens.
• Example: When inserting a shoe, SEA boosts stitching and logo edges just on the shoe area, not on the floor.

Step 5: Decode to an image and train with two complementary losses

• What happens: After denoising, a decoder turns tokens back into an image. Training uses two losses: (1) a global latent-space MSE to keep overall composition and semantics correct, and (2) Detail-Aware Loss (DAL) that compares high-frequency maps of prediction vs. ground truth, inside the mask.
• Why this exists: Global loss keeps the big picture coherent; DAL protects the tiny details people care about.
• Example: The arm angle and lighting stay natural (global), while the product’s tiny serial number remains readable (detail).

🍞 Top Bread (Hook): Think of a librarian who catalogs all information, and a proofreader who checks punctuation.

🥬 The Concept (Diffusion Transformer, DiT): A Diffusion Transformer is a model that refines a noisy image step by step using attention, like a careful editor improving each draft.

• How it works:
  1. Start with a noisy version of the target.
  2. Use attention to pull the right info from text, masked human, and product tokens.
  3. Iteratively denoise until a clean image emerges.
• Why it matters: Without iterative refinement, you can’t carefully align the product with the scene while preserving details.

🍞 Bottom Bread (Anchor): It’s like restoring an old photo little by little, each pass bringing back clearer edges and correct shapes.

Secret Sauce:

• The trio of high-frequency maps, SEA, and DAL. High-frequency maps point out what’s precious; SEA injects those details exactly where needed; DAL enforces them during training. Together, they beat the usual blur-from-averaging that hurts logos, labels, and textures.

The Test: The team measured three things. (1) Text Alignment: Does the image match the prompt’s words? (2) Visual Consistency: Does the inpainted product really match the reference image in both overall identity and local details? (3) Image Quality: Is it sharp, natural, and aesthetically pleasing? They also introduced a special detail-aware score (SSIM-HF) that focuses on high-frequency structure—perfect for checking tiny text and logos.

The Competition: HiFi-Inpaint was compared with well-known methods: Paint-by-Example, ACE++, Insert Anything, and FLUX-Kontext. All models were tested with the same resolution and fair settings.

The Scoreboard (with context):

• Visual Similarity: HiFi-Inpaint reached top CLIP-I (about 0.95) and DINO (about 0.92). Think of this like getting an A+ in matching the reference product, when others often got B’s.
• Structural Similarity: Highest SSIM and SSIM-HF. That’s like saying the big shapes line up (SSIM) and so do the tiny edges and details (SSIM-HF). Many baselines lost points on crisp details; HiFi-Inpaint kept them.
• Text Alignment: Competitive CLIP-T—meaning the generated image didn’t wander away from the prompt.
• Aesthetics/Perceptual Quality: Best or near-best on LAION-Aes and Q-Align-IQ, showing images look good to both machines and people.

Surprising/Notable Findings:

• Instruction-based editors (like FLUX-Kontext) struggled to link the specific reference product to the masked region. Sometimes they produced a separate product image instead of a proper insertion.
• Insert Anything did fairly well on details but formed artifacts when the masked area was small. HiFi-Inpaint remained robust even in small, tricky masks.
• User Study: With participants choosing favorites for text alignment, visual consistency, and quality, HiFi-Inpaint won the largest share in all three categories—lining up human opinions with the metrics.

Real-World Stress Test:

• On a tougher, real-world dataset, HiFi-Inpaint still led on key visual-consistency metrics (best CLIP-I and DINO) and structure (best SSIM and SSIM-HF). Aesthetics remained competitive. This suggests the method generalizes beyond the cleaner synthetic data and keeps detail fidelity under messy lighting, different poses, and complex backgrounds.

What the numbers mean in plain words:

• If you need the exact logo, text, and texture of a product to survive the inpainting, HiFi-Inpaint is the safest bet among the tested methods. It’s not just making the area look okay—it’s making it look right.

Limitations:

• Synthetic-first training: While the HP-Image-40K dataset is large and filtered, synthetic data can miss some real-world messiness (odd lighting, motion blur, extreme occlusions). This can still lower performance compared to carefully curated real photos in some edge cases.
• Extreme small masks or ultra-tiny text: Although the method is robust, there are limits—think very tiny logos or 2–3 pixel text at low resolution.
• Domain shifts: Unusual materials (e.g., holographic foil) or heavy reflections can break assumptions in the high-frequency cues.

Required Resources:

• A decent GPU setup for training and inference (DiT-based diffusion is compute-heavy).
• Access to the curated training pairs (HP-Image-40K plus real-world data) and the model’s code to reproduce.
• Pretrained encoders/transformers (e.g., FLUX.1-Dev backbone, VAE) and the filtering tools (YOLO, CLIP, InternVL) if you want to expand the dataset.

When NOT to Use:

• If exact product identity is unimportant (e.g., just any blue bottle is fine), a simpler editor may suffice.
• If the reference is extremely low-resolution or heavily compressed so that high-frequency details vanish, the method has little to preserve.
• If the product is highly deformable or translucent in ways not represented in training, results may drift.

Open Questions:

• Can we further improve realism under extreme lighting, motion blur, or glassy/metallic reflections without losing detail fidelity?
• How can we scale to video, where every frame must keep fine details consistent over time?
• Could adaptive or learned high-frequency extraction (instead of a fixed filter) capture even more subtle details?
• How to handle multiple products with conflicting detail cues in the same mask region?
• Can we reduce compute costs while keeping fidelity, perhaps by selective high-frequency enhancement only where the model is uncertain?

Three-Sentence Summary:

• HiFi-Inpaint is a reference-based inpainting method that protects tiny, important product details by using high-frequency maps during generation and training.
• Its Shared Enhancement Attention (SEA) injects detail cues exactly inside the mask, and its Detail-Aware Loss (DAL) enforces pixel-level sharpness where it counts.
• With a strong, carefully filtered dataset (HP-Image-40K) and thorough experiments, it delivers state-of-the-art, detail-preserving human–product images.

Main Achievement:

• Proving that explicitly highlighting and supervising sharp details—via high-frequency maps, SEA, and DAL—solves the common blurring/averaging problem in human–product inpainting.

Future Directions:

• Extend to video so logos and textures remain stable across frames.
• Improve robustness to tough lighting, reflections, and unusual materials.
• Explore learned or adaptive detail maps and even better, localized training signals.
• Expand datasets with more real-world variety and rare edge cases.

Why Remember This:

• It shifts the mindset from “hope the model keeps details” to “teach the model exactly which details to protect and how.” For e-commerce, ads, and branding, that’s the difference between a guess and the genuine product. HiFi-Inpaint shows how to combine the right guidance (high-frequency maps), the right attention (SEA), and the right supervision (DAL) to get crisp, trusted results.