Netflix released VOID: Inpainting that understands cause-and-effect relationships 🍿

Regular video inpainting works like this: you select a person, the network fills their pixels with background. It becomes blurry, but that's okay. However, if the person was holding a guitar or leaning on a table, the guitar will remain hanging in the air, and the tablecloth will retain the shape of a non-existent elbow.

Researchers from Netflix, together with INSAIT, published the VOID (Video Object and Interaction Deletion) model. This thing doesn't just fill background pixels; it removes the object along with all its physical interactions in the scene.

Removed a person holding a guitar — the guitar realistically falls to the floor.
Removed a stand — the object rolls off the table.

How it works:
The base is CogVideoX from Zhipu AI. All the magic here lies not in overcomplicating the architecture, but in the data preparation pipeline and clever conditioning.

1️⃣ Quadmask (4-value mask)
Instead of a classic binary mask (delete/keep), the network is fed a mask with four values:
0 (black) — the object itself to be removed.
127 (grey) — affected region (the interaction zone where physical processes will change, e.g., fall trajectory).
63 (dark grey) — overlay.
255 (white) — static background, which we don't touch.
This mask is automatically assembled during preprocessing by a combination of SAM2 and Gemini (VLM-Mask-Reasoner).

2️⃣ Counterfactual dataset
The model does not simulate physics in real time. To teach the diffusion model physics, engineers generated a dataset of paired videos in Blender (based on HUMOTO mocap data) and Kubric. In one video, the object is present; in the second, the object is absent, and the engine calculates the physics of the remaining elements falling. The model simply learned this physical prior.

3️⃣ 2-Pass Inference
The first pass basically removes the object. But we know that video diffusion models love to flicker. So there is Pass 2: it uses optical flow to generate warped noise in latent space. This rigidly fixes temporal consistency.

To run this pipeline, you will need a GPU with at least 40GB VRAM (A100). Diffusion transformers on video consume memory mercilessly. Fortunately, the code comes with CPU-offload and FP8 quantization out of the box, so you can try to squeeze it into simpler hardware if you have a lot of time to wait for tensors from RAM.

Code, weights (in safetensors format) and demo notebook are already open.

#good_opensource