GUI-SD: On-Policy Self-Distillation Framework for GUI Grounding
Researchers have introduced GUI-SD, the first on-policy self-distillation (OPSD) framework specifically designed for Graphical User Interface (GUI) grounding. This technology enables autonomous agents to accurately map natural language instructions to visual coordinates on a screen. While recent reinforcement learning methods like GRPO have shown promise, they suffer from high computational costs due to multiple rollouts and sparse feedback signals. GUI-SD addresses these limitations by providing dense token-level supervision through a single rollout. The framework utilizes a visually enriched privileged context for the teacher model, employing target bounding boxes and Gaussian soft masks to guide learning without revealing exact coordinates. Additionally, it implements entropy-guided distillation to adaptively weight tokens based on digit significance and teacher confidence. Extensive experiments across six benchmarks demonstrate that GUI-SD consistently outperforms existing GRPO-based methods and naive OPSD approaches in both accuracy and training efficiency. The study highlights a significant advancement in making autonomous GUI agents more efficient and effective. Code and training data for the framework have been made publicly available to support further research and development in this area of artificial intelligence.
Wire timeline
GUI-SD: On-Policy Self-Distillation Framework for GUI Grounding
Researchers have introduced GUI-SD, the first on-policy self-distillation (OPSD) framework specifically designed for Graphical User Interface (GUI) grounding. This technology enables autonomous agents to accurately map natural language instructions to visual coordinates on a screen. While recent reinforcement learning methods like GRPO have shown promise, they suffer from high computational costs due to multiple rollouts and sparse feedback signals. GUI-SD addresses these limitations by providing dense token-level supervision through a single rollout. The framework utilizes a visually enriched privileged context for the teacher model, employing target bounding boxes and Gaussian soft masks to guide learning without revealing exact coordinates. Additionally, it implements entropy-guided distillation to adaptively weight tokens based on digit significance and teacher confidence. Extensive experiments across six benchmarks demonstrate that GUI-SD consistently outperforms existing GRPO-based methods and naive OPSD approaches in both accuracy and training efficiency. The study highlights a significant advancement in making autonomous GUI agents more efficient and effective. Code and training data for the framework have been made publicly available to support further research and development in this area of artificial intelligence.
cs.AI updates on arXiv.org