Caracal: Causal Architecture via Spectral Mixing
Researchers have introduced Caracal, a novel architecture for Large Language Models designed to overcome the scalability limitations of traditional Transformers. Current models struggle with long sequences due to the quadratic cost of attention mechanisms and positional encoding constraints. Caracal addresses these issues by replacing standard attention with a parameter-efficient Multi-Head Fourier (MHF) module, achieving O(L log(L)) complexity through Fast Fourier Transform (FFT) based sequence mixing. A key innovation is the application of frequency-domain causal masking, which uses asymmetric padding and truncation to enforce autoregressive capabilities, solving a major hurdle for Fourier-based generative models. Unlike other efficient architectures such as Mamba that rely on hardware-specific implementations, Caracal utilizes standard library operators, ensuring robust portability and easier deployment. Evaluations indicate that Caracal performs competitively against Transformer and State Space Model (SSM) baselines. This development offers a scalable, simple pathway for efficient long-sequence modeling in AI, with code available for further research and implementation.
Wire timeline
Caracal: Causal Architecture via Spectral Mixing
Researchers have introduced Caracal, a novel architecture for Large Language Models designed to overcome the scalability limitations of traditional Transformers. Current models struggle with long sequences due to the quadratic cost of attention mechanisms and positional encoding constraints. Caracal addresses these issues by replacing standard attention with a parameter-efficient Multi-Head Fourier (MHF) module, achieving O(L log(L)) complexity through Fast Fourier Transform (FFT) based sequence mixing. A key innovation is the application of frequency-domain causal masking, which uses asymmetric padding and truncation to enforce autoregressive capabilities, solving a major hurdle for Fourier-based generative models. Unlike other efficient architectures such as Mamba that rely on hardware-specific implementations, Caracal utilizes standard library operators, ensuring robust portability and easier deployment. Evaluations indicate that Caracal performs competitively against Transformer and State Space Model (SSM) baselines. This development offers a scalable, simple pathway for efficient long-sequence modeling in AI, with code available for further research and implementation.
cs.AI updates on arXiv.org