Adaptive Greedy Frame Selection for Long Video Understanding
Researchers have proposed a new method called Adaptive Greedy Frame Selection to improve long-video question answering using Large Vision-Language Models (VLMs). Current VLM inference is often bottlenecked by the high number of input frames and visual tokens. Existing solutions like naive sparse sampling miss key moments, while relevance-driven selection often results in duplicate frames and poor temporal coverage. This new approach constructs a candidate pool at 1 FPS and embeds frames in two complementary spaces: SigLIP for question relevance and DINOv2 for semantic similarity. It selects frames by greedily maximizing a weighted sum of relevance and coverage terms, ensuring a standard approximation guarantee. To handle different query types, the system uses a lightweight classifier to route questions to one of four preset strategies. Experiments on the MLVU benchmark demonstrate consistent accuracy improvements over uniform sampling and recent baselines, particularly under tight frame budgets. This research addresses critical efficiency and accuracy challenges in computer vision and artificial intelligence, offering a robust solution for processing long-form video content effectively.
Wire timeline
Adaptive Greedy Frame Selection for Long Video Understanding
Researchers have proposed a new method called Adaptive Greedy Frame Selection to improve long-video question answering using Large Vision-Language Models (VLMs). Current VLM inference is often bottlenecked by the high number of input frames and visual tokens. Existing solutions like naive sparse sampling miss key moments, while relevance-driven selection often results in duplicate frames and poor temporal coverage. This new approach constructs a candidate pool at 1 FPS and embeds frames in two complementary spaces: SigLIP for question relevance and DINOv2 for semantic similarity. It selects frames by greedily maximizing a weighted sum of relevance and coverage terms, ensuring a standard approximation guarantee. To handle different query types, the system uses a lightweight classifier to route questions to one of four preset strategies. Experiments on the MLVU benchmark demonstrate consistent accuracy improvements over uniform sampling and recent baselines, particularly under tight frame budgets. This research addresses critical efficiency and accuracy challenges in computer vision and artificial intelligence, offering a robust solution for processing long-form video content effectively.
cs.AI updates on arXiv.org