Overview
InstructX is a unified framework for image and video editing. By integrating MLLMs with diffusion models, it enables flexible and precise instruction-guided manipulation across image and video.
Method
InstructX framework.
Core questions
- Q1: Do MLLMs actually help with visual editing tasks, and by how much specifically?
- Q2: Among the many ways to combine MLLMs and diffusion models, which is better suited for editing tasks?
- Q3: Can a large amount of high-quality image-editing data be used to improve video editing tasks?
Our answers
- A1: Yes. As shown in the below Figure (a), the model using MLLM outperforms the diffusion-only model across all tasks.
- A2: As shown in Figure (b), among the four structures, the combination of metaquery + LoRA fine-tuned MLLM + small connector achieves the best performance. The key insight here is that the editing should be accomplished within the MLLM itself; LoRA is helpful for adapting the MLLM to the editing task; and a large connector is unnecessary.
- A3: Yes. Training on image data enables capability transfer to video editing, which unlocks zero-shot abilities on video tasks. As shown in the below Figure (c).
