Priming: Efficient Conversion of Pre-trained Transformers to Hybrid State Space Models
Researchers have introduced 'Priming,' a novel method that converts pre-trained Transformer models into Hybrid State-Space Models (SSMs) without requiring training from scratch. By initializing hybrid architectures with existing Transformer weights and employing short alignment phases, Priming recovers downstream performance using less than 0.5% of the original pre-training token budget. This approach overcomes previous barriers in exploring hybrid design spaces at scale. The study evaluates various SSM layers, establishing an expressiveness hierarchy where Gated KalmaNet outperforms Gated DeltaNet and Mamba-2 in long-context reasoning tasks. Scaling this method to 32B parameter models, the resulting Hybrid GKA model demonstrated a 3.8-point improvement in reasoning over its Qwen3-32B source and achieved up to 2.3x higher decode throughput. The team released open-source code, optimized kernels, and a model zoo under the Apache 2.0 license to facilitate further research into efficient, long-context AI architectures.
Wire timeline
Priming: Efficient Conversion of Pre-trained Transformers to Hybrid State Space Models
Researchers have introduced 'Priming,' a novel method that converts pre-trained Transformer models into Hybrid State-Space Models (SSMs) without requiring training from scratch. By initializing hybrid architectures with existing Transformer weights and employing short alignment phases, Priming recovers downstream performance using less than 0.5% of the original pre-training token budget. This approach overcomes previous barriers in exploring hybrid design spaces at scale. The study evaluates various SSM layers, establishing an expressiveness hierarchy where Gated KalmaNet outperforms Gated DeltaNet and Mamba-2 in long-context reasoning tasks. Scaling this method to 32B parameter models, the resulting Hybrid GKA model demonstrated a 3.8-point improvement in reasoning over its Qwen3-32B source and achieved up to 2.3x higher decode throughput. The team released open-source code, optimized kernels, and a model zoo under the Apache 2.0 license to facilitate further research into efficient, long-context AI architectures.
cs.AI updates on arXiv.org