LLaVA-CKD: Bottom-Up Cascaded Knowledge Distillation for Vision-Language Models

Researchers have introduced LLaVA-CKD, a novel framework designed to optimize Large Vision-Language Models (VLMs) for practical deployment by addressing their high memory and computational demands. While Knowledge Distillation typically transfers knowledge from a large Teacher network to a smaller Student network, significant capacity gaps often hinder effective transfer. To overcome this, the proposed Bottom-Up Cascaded Knowledge Distillation (CKD) method employs intermediate-capacity Teachers that gradually elevate the Student network's capabilities, mimicking human formal education systems. This approach ensures smoother knowledge transfer and improved generalization performance. The study includes a theoretical analysis of cascaded distillation effects and applies the framework to models based on the LLaVA methodology. Evaluations across seven standard, publicly available Visual Question Answering (VQA) benchmarks demonstrate that the derived models achieve State-of-the-Art (SotA) performance. This advancement offers a promising solution for making powerful VLMs more efficient and accessible for real-world applications without compromising accuracy.