LoKA: Low-precision Kernel Applications for Recommendation Models At Scale
Researchers have introduced LoKA, a novel framework designed to enable the practical use of low-precision FP8 arithmetic in Large Recommendation Models (LRMs). While recent GPU generations support FP8 for higher performance, its direct application to LRMs has been limited due to numerical sensitivity, small matrix multiplications, and communication-intensive training environments, often leading to degraded model quality. LoKA addresses these challenges through a system-model co-design approach based on three principles: profiling under realistic distributions, co-designing model components with hardware, and orchestrating kernel libraries. The framework comprises three key components: LoKA Probe, an online benchmarking method that identifies safe zones for FP8 adoption by quantifying per-layer errors; LoKA Mods, reusable model adaptations enhancing numerical stability and efficiency; and LoKA Dispatch, a runtime system that selects optimal FP8 kernels based on statistical insights. This innovation aims to maximize computational gains while maintaining accuracy, marking a significant advancement in optimizing large-scale recommendation systems for modern hardware architectures.
Wire timeline
LoKA: Low-precision Kernel Applications for Recommendation Models At Scale
Researchers have introduced LoKA, a novel framework designed to enable the practical use of low-precision FP8 arithmetic in Large Recommendation Models (LRMs). While recent GPU generations support FP8 for higher performance, its direct application to LRMs has been limited due to numerical sensitivity, small matrix multiplications, and communication-intensive training environments, often leading to degraded model quality. LoKA addresses these challenges through a system-model co-design approach based on three principles: profiling under realistic distributions, co-designing model components with hardware, and orchestrating kernel libraries. The framework comprises three key components: LoKA Probe, an online benchmarking method that identifies safe zones for FP8 adoption by quantifying per-layer errors; LoKA Mods, reusable model adaptations enhancing numerical stability and efficiency; and LoKA Dispatch, a runtime system that selects optimal FP8 kernels based on statistical insights. This innovation aims to maximize computational gains while maintaining accuracy, marking a significant advancement in optimizing large-scale recommendation systems for modern hardware architectures.
cs.AI updates on arXiv.org