MIDUS: Memory-Infused Depth Up-Scaling for Efficient LLM Expansion
Researchers have introduced MIDUS (Memory-Infused Depth Up-Scaling), a novel method for expanding pre-trained language models without the high computational costs associated with traditional Depth Up-Scaling (DUS). While standard DUS increases model capacity by duplicating Transformer blocks, it also replicates parameter-heavy Feed-Forward Network (FFN) branches, leading to significant resource overhead. MIDUS addresses this inefficiency by replacing duplicated FFN branches with lightweight memory layers, transforming added depth into retrieval-based residual capacity. The core innovation is the Head-wise Memory Layer (HML), which integrates multi-head product-key memory with Head-wise Implicit Value Expansion (HIVE). This architecture assigns distinct key spaces to each attention head while deriving head-specific values from a shared latent bank via compact projections. Empirical analyses demonstrate that MIDUS offers improved performance and efficiency compared to FFN-based expansion. Structural evaluations characterize HML with HIVE as a distinct, head-conditioned alternative that leverages the heterogeneous roles of attention heads. This development represents a significant advancement in optimizing large language model scalability, offering a practical solution for enhancing model capacity while minimizing parameter and compute costs.
Wire timeline
MIDUS: Memory-Infused Depth Up-Scaling for Efficient LLM Expansion
Researchers have introduced MIDUS (Memory-Infused Depth Up-Scaling), a novel method for expanding pre-trained language models without the high computational costs associated with traditional Depth Up-Scaling (DUS). While standard DUS increases model capacity by duplicating Transformer blocks, it also replicates parameter-heavy Feed-Forward Network (FFN) branches, leading to significant resource overhead. MIDUS addresses this inefficiency by replacing duplicated FFN branches with lightweight memory layers, transforming added depth into retrieval-based residual capacity. The core innovation is the Head-wise Memory Layer (HML), which integrates multi-head product-key memory with Head-wise Implicit Value Expansion (HIVE). This architecture assigns distinct key spaces to each attention head while deriving head-specific values from a shared latent bank via compact projections. Empirical analyses demonstrate that MIDUS offers improved performance and efficiency compared to FFN-based expansion. Structural evaluations characterize HML with HIVE as a distinct, head-conditioned alternative that leverages the heterogeneous roles of attention heads. This development represents a significant advancement in optimizing large language model scalability, offering a practical solution for enhancing model capacity while minimizing parameter and compute costs.
cs.AI updates on arXiv.org