Positional Encoding via Token-Aware Phase Attention

Researchers have introduced Token-Aware Phase Attention (TAPA), a novel positional encoding method designed to overcome limitations in Rotary Positional Embedding (RoPE) for large language models. The study demonstrates that RoPE introduces an intrinsic distance-dependent bias in attention scores, which restricts its effectiveness in modeling long-context sequences. While existing RoPE extensions attempt to mitigate this through post-hoc adjustments like rescaling, TAPA integrates a learnable phase function directly into the attention mechanism. This approach preserves token interactions over long ranges and allows for extension to longer contexts via direct and light continual pretraining. Experimental results indicate that TAPA extrapolates effectively to unseen sequence lengths, achieving substantially lower perplexity and superior retrieval performance in long-context regimes compared to RoPE-style baselines. This advancement addresses critical challenges in scaling transformer architectures for extensive data processing, offering a more robust solution for maintaining accuracy and coherence in long-form text generation and analysis without requiring complex hyperparameter retuning after initial training.