mHC-SSM: Manifold-Constrained Hyper-Connections for State Space Language Models
Researchers Abdulvahap Mutlu, Şengül Doğan, and Türker Tuncer have introduced mHC-SSM, a novel architecture enhancing State Space Models (SSMs) for language modeling. Published on arXiv, this study adapts Manifold-Constrained Hyper-Connections (mHC) to SSMs by expanding residual streams into parallel paths constrained by doubly stochastic matrices via Sinkhorn-Knopp projection. The method includes stream-specialized adapters that add lightweight capacity through shared bottlenecks. Evaluated on the WikiText-2 dataset, the static mHC-SSM reduced validation loss from 6.3507 to 6.2448 and perplexity from 572.91 to 515.35 compared to baseline single-stream SSMs. Incorporating adapters further improved performance, lowering loss to 6.1353 and perplexity to 461.88. These quality gains come with modest efficiency trade-offs, including reduced throughput and increased GPU memory usage. The findings demonstrate that constrained multi-stream residual mixing significantly boosts SLM performance, offering a viable path for optimizing state space models in natural language processing tasks despite slight computational costs.
Wire timeline
mHC-SSM: Manifold-Constrained Hyper-Connections for State Space Language Models
Researchers Abdulvahap Mutlu, Şengül Doğan, and Türker Tuncer have introduced mHC-SSM, a novel architecture enhancing State Space Models (SSMs) for language modeling. Published on arXiv, this study adapts Manifold-Constrained Hyper-Connections (mHC) to SSMs by expanding residual streams into parallel paths constrained by doubly stochastic matrices via Sinkhorn-Knopp projection. The method includes stream-specialized adapters that add lightweight capacity through shared bottlenecks. Evaluated on the WikiText-2 dataset, the static mHC-SSM reduced validation loss from 6.3507 to 6.2448 and perplexity from 572.91 to 515.35 compared to baseline single-stream SSMs. Incorporating adapters further improved performance, lowering loss to 6.1353 and perplexity to 461.88. These quality gains come with modest efficiency trade-offs, including reduced throughput and increased GPU memory usage. The findings demonstrate that constrained multi-stream residual mixing significantly boosts SLM performance, offering a viable path for optimizing state space models in natural language processing tasks despite slight computational costs.
cs.AI updates on arXiv.org