Sparsity Moves Computation: How FFN Architecture Reshapes Attention in Small Transformers
A new research paper published on arXiv investigates how architectural choices within the Transformer feedforward network (FFN) block influence the broader computational dynamics of small Transformer models. The study, conducted by Gabriel Smithline and Chris Mascioli, examines one-layer Transformers trained on tasks such as digit addition with carry, modular arithmetic, and histogram counting. By comparing dense FFNs, gated linear units (GLUs), mixture-of-experts (MoE), and MoE-GLUs, the authors discover that sparse MoE routing shifts computational load from the FFN to the attention mechanism, particularly impacting carry-based addition. Crucially, the research indicates that this redistribution is driven primarily by architectural sparsity rather than learned router specialization, as frozen random routing yields similar results. Additionally, the study finds that GLU-style multiplicative gating rotates task-relevant Fourier structure into distributed subspaces, reducing neuron-level interpretability while maintaining structured computation. These findings highlight that local design decisions in FFN architecture have significant nonlocal consequences for overall model behavior, validated through various controls including parameter-matching and width-scaling analyses.
Wire timeline
Sparsity Moves Computation: How FFN Architecture Reshapes Attention in Small Transformers
A new research paper published on arXiv investigates how architectural choices within the Transformer feedforward network (FFN) block influence the broader computational dynamics of small Transformer models. The study, conducted by Gabriel Smithline and Chris Mascioli, examines one-layer Transformers trained on tasks such as digit addition with carry, modular arithmetic, and histogram counting. By comparing dense FFNs, gated linear units (GLUs), mixture-of-experts (MoE), and MoE-GLUs, the authors discover that sparse MoE routing shifts computational load from the FFN to the attention mechanism, particularly impacting carry-based addition. Crucially, the research indicates that this redistribution is driven primarily by architectural sparsity rather than learned router specialization, as frozen random routing yields similar results. Additionally, the study finds that GLU-style multiplicative gating rotates task-relevant Fourier structure into distributed subspaces, reducing neuron-level interpretability while maintaining structured computation. These findings highlight that local design decisions in FFN architecture have significant nonlocal consequences for overall model behavior, validated through various controls including parameter-matching and width-scaling analyses.
cs.AI updates on arXiv.org