Can General Foundation Models Surpass Specialized Fine-Tuning? A Medical Case Study: Decision Insights: Prompt Engineering Can Unlock Deep Professional Capabilities of General Models, Surpassing Specialized Models
This study uses systematic prompt engineering (Medprompt) to enable GPT-4 to surpass the previous best specialized model Med-PaLM 2 on all nine medical benchmarks in MultiMedQA, reducing error rate by 27% on MedQA and achieving over 90% accuracy for the first time, with an order of magnitude reduction in model calls. The method requires no domain expert involvement and generalizes to exams in electrical engineering, machine learning, philosophy, accounting, law, nursing, and clinical psychology.
The paper challenges the common assumption that general foundation models cannot match specialized fine-tuned models. By designing a general prompt strategy Medprompt, GPT-4 significantly outperforms the previous leading specialized model on medical benchmarks, and the method is transferable to other professional domains, indicating that innovations in prompt engineering can unlock deep professional capabilities of general models.
Medprompt combines multiple prompt strategies (e.g., dynamic few-shot selection, chain-of-thought, self-consistency) to systematically optimize GPT-4 output without domain expert involvement. The experimental design strictly controls for prompt engineering overfitting, achieving state-of-the-art results on all nine MultiMedQA datasets. Compared to Med-PaLM 2, Medprompt achieves higher accuracy with one-tenth the model calls, reducing error rate by 27% on MedQA. The method also performs well on exams in seven non-medical domains such as electrical engineering and machine learning.
This research is disruptive for the medical AI industry: prompt engineering can replace expensive domain fine-tuning, reducing model deployment costs. Moreover, the generality of the method suggests the potential of foundation models in professional domains (e.g., law, accounting), potentially changing the industry's reliance on specialized models.
It is recommended that medical AI companies prioritize prompt engineering over fine-tuning to improve model performance. They can purchase GPT-4 API and apply Medprompt strategies to quickly build diagnostic assistance systems, reducing R&D costs. At the same time, focus on developing automated tools for prompt strategies to form differentiated competitiveness.
Future work needs to verify Medprompt's generalization on larger-scale and more diverse professional tasks, as well as automated search and optimization of prompt strategies. Key signals include zero-shot performance on more medical subfields (e.g., imaging, pathology) and compatibility with other foundation models (e.g., Claude, Gemini).