Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement
A research paper proposes FISA, a framework for MLLM self-improvement that generates augmented images from the model's own failure cases. Experiments on visual question answering benchmarks show consistent performance improvements in both in-distribution and out-of-distribution settings.
Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks but rely on costly annotated data. Self-augmentation can expand training data without external supervision, yet existing methods are text-centric and image augmentation remains underexplored. The paper introduces Failure-informed Image Self-Augmentation (FISA), which constructs challenging but answer-preserving image complications from failure cases, verifies utility via self-examination, and applies dual fidelity filtering. Experiments demonstrate consistent gains on visual question answering benchmarks across in-distribution and out-of-distribution settings.
FISA leverages model failures to guide image augmentation, creating visually challenging variants that preserve semantic answers. It employs self-examination and dual fidelity filtering to avoid semantic distortion, enabling targeted self-improvement without external data.
This approach reduces dependency on expensive human-annotated multimodal data, potentially lowering the cost and accelerating the iteration cycle for MLLM development in commercial applications.
By enabling models to self-improve from their own mistakes, FISA can reduce data annotation costs and improve model robustness, benefiting enterprises deploying MLLMs in vision-language applications.
Next signals include integration of FISA into MLLM training pipelines, extension to other multimodal tasks, and evaluation on larger-scale models. Compatibility with other self-improvement methods may further amplify gains.