Mixture of Layers with Hybrid Attention
Researchers Ivan Ternovtsii and Yurii Bilak have introduced a novel transformer architecture termed Mixture of Layers (MoL) in a new arXiv preprint. Unlike standard Mixture-of-Experts (MoE) models that route tokens to expert subnetworks within monolithic layers, MoL replaces full-width transformer blocks with multiple parallel thin blocks of reduced dimensionality. These blocks are connected via learned projections and composed using top-k block routing. The study addresses the attention coverage problem inherent in scaling sparse block routing, where individual blocks process fewer tokens. To solve this, the authors propose a hybrid attention mechanism that combines a shared softmax block for maintaining global context with Gated DeltaNet linear attention within the routed blocks. This approach aims to enhance efficiency and contextual understanding in large-scale machine learning models. The paper, categorized under Machine Learning and Artificial Intelligence, was submitted on May 10, 2026, representing a significant technical advancement in transformer model design and sparse routing strategies.
Wire timeline
Mixture of Layers with Hybrid Attention
Researchers Ivan Ternovtsii and Yurii Bilak have introduced a novel transformer architecture termed Mixture of Layers (MoL) in a new arXiv preprint. Unlike standard Mixture-of-Experts (MoE) models that route tokens to expert subnetworks within monolithic layers, MoL replaces full-width transformer blocks with multiple parallel thin blocks of reduced dimensionality. These blocks are connected via learned projections and composed using top-k block routing. The study addresses the attention coverage problem inherent in scaling sparse block routing, where individual blocks process fewer tokens. To solve this, the authors propose a hybrid attention mechanism that combines a shared softmax block for maintaining global context with Gated DeltaNet linear attention within the routed blocks. This approach aims to enhance efficiency and contextual understanding in large-scale machine learning models. The paper, categorized under Machine Learning and Artificial Intelligence, was submitted on May 10, 2026, representing a significant technical advancement in transformer model design and sparse routing strategies.
cs.AI updates on arXiv.org