LLaVA: Visual Instruction Tuning: An Open-Source Milestone in Multimodal Instruction Tuning
In April 2023, LLaVA first proposed using pure language GPT-4 to generate multimodal language-image instruction data, and trained an end-to-end large-scale multimodal model (connecting visual encoder and LLM) based on this. On Science QA, LLaVA+GPT-4 achieved 92.53% accuracy, setting a new record; on synthetic multimodal instruction-following datasets, it reached 85.1% relative score of GPT-4.
LLaVA pioneered the introduction of the instruction tuning paradigm from NLP into the multimodal domain, proving that multimodal data generated solely by language models can effectively train vision-language models. It reduces the reliance on human-annotated instruction data for multimodal models, making it possible to build general-purpose visual assistants. LLaVA's code, data, and models are fully open-source, quickly becoming a benchmark framework for multimodal dialogue systems.
LLaVA architecture is simple: uses CLIP visual encoder to extract image features, and maps visual tokens to the input space of the LLM (Vicuna) via a linear projection layer. Training has two stages: first stage freezes the visual encoder and LLM, training only the projection layer to align visual and language features; second stage uses 158K multimodal instruction data generated by GPT-4 to fine-tune the entire model (or only the projection layer + LLM via LoRA). Key innovation lies in data generation: converting image descriptions and bounding box information into text, and letting GPT-4 generate human-like multi-turn dialogues based on this text. Evaluation shows LLaVA performs well on tasks like visual reasoning, OCR, and dialogue, but suffers from hallucination issues.
LLaVA sparked an open-source race for multimodal dialogue models, directly influencing subsequent works like MiniGPT-4 and mPLUG-Owl. For industries such as intelligent customer service, education, and content creation, LLaVA provides a low-cost solution for building multimodal interactive systems. Its data generation method can also be used to automatically construct instruction data for vertical domains (e.g., healthcare, e-commerce).
Recommend product teams to rapidly prototype multimodal dialogue features based on LLaVA, such as image captioning, chart question answering, and document understanding. It can serve as an open-source alternative to avoid reliance on closed-source APIs. Additionally, leverage its data generation method to build domain-specific instruction sets, improving model performance in vertical scenarios.
Focus on LLaVA's hallucination mitigation, extension to long video understanding, and integration with other modalities (audio, 3D). Whether its data generation pipeline will be replaced by higher-quality human-annotated data, and the model's performance in safety and bias.