bViT: Investigating Single-Block Recurrence in Vision Transformers for Image Recognition
Researchers have introduced bViT, a novel Vision Transformer architecture that utilizes single-block recurrence to process images. Unlike standard ViTs that stack independently parameterized blocks, bViT applies one transformer block repeatedly, preserving iterative structure while significantly reducing parameter count. Experimental results on the ImageNet-1K dataset demonstrate that a 12-step bViT-B model achieves accuracy comparable to standard ViT-B under identical training conditions and computational budgets, yet uses an order of magnitude fewer parameters. The study highlights that performance improves with representation width, a phenomenon termed implicit depth multiplexing, where the shared block adapts its computation across steps via evolving hidden states. Mechanistic analyses confirm that the block changes behavior rather than repeating identical operations. Furthermore, bViT shows competitive transfer capabilities to downstream tasks and supports parameter-efficient fine-tuning. These findings suggest that much of the depth in traditional ViTs can be effectively implemented through recurrent reuse if the representation space is sufficiently wide, offering a more efficient approach to vision model design.
Wire timeline
bViT: Investigating Single-Block Recurrence in Vision Transformers for Image Recognition
Researchers have introduced bViT, a novel Vision Transformer architecture that utilizes single-block recurrence to process images. Unlike standard ViTs that stack independently parameterized blocks, bViT applies one transformer block repeatedly, preserving iterative structure while significantly reducing parameter count. Experimental results on the ImageNet-1K dataset demonstrate that a 12-step bViT-B model achieves accuracy comparable to standard ViT-B under identical training conditions and computational budgets, yet uses an order of magnitude fewer parameters. The study highlights that performance improves with representation width, a phenomenon termed implicit depth multiplexing, where the shared block adapts its computation across steps via evolving hidden states. Mechanistic analyses confirm that the block changes behavior rather than repeating identical operations. Furthermore, bViT shows competitive transfer capabilities to downstream tasks and supports parameter-efficient fine-tuning. These findings suggest that much of the depth in traditional ViTs can be effectively implemented through recurrent reuse if the representation space is sufficiently wide, offering a more efficient approach to vision model design.
cs.AI updates on arXiv.org