CoTAR: A Centralized MLP Module Improving Transformer Efficiency for Medical Time Series
Researchers have introduced CoTAR (Core Token Aggregation-Redistribution), a novel module designed to address limitations in Transformer-based models for analyzing medical time series (MedTS) data, such as EEG and ECG signals. While standard Transformers effectively capture temporal dependencies, their decentralized attention mechanism struggles with the centralized nature of MedTS channel dependencies. CoTAR replaces this decentralized attention with a centralized MLP-based approach, utilizing a global core token to facilitate inter-token interactions. This structural alignment with MedTS signal characteristics reduces computational complexity from quadratic to linear. Experimental results across five benchmarks demonstrate significant improvements, including an 11.6% accuracy increase on the APAVA dataset. Furthermore, the model exhibits superior efficiency, requiring only 33% of the memory and 20% of the inference time compared to previous state-of-the-art methods. The study highlights a critical rethinking of architecture design for healthcare applications, offering a more effective and resource-efficient solution for diagnosing brain and heart diseases through advanced deep learning techniques.
Wire timeline
CoTAR: A Centralized MLP Module Improving Transformer Efficiency for Medical Time Series
Researchers have introduced CoTAR (Core Token Aggregation-Redistribution), a novel module designed to address limitations in Transformer-based models for analyzing medical time series (MedTS) data, such as EEG and ECG signals. While standard Transformers effectively capture temporal dependencies, their decentralized attention mechanism struggles with the centralized nature of MedTS channel dependencies. CoTAR replaces this decentralized attention with a centralized MLP-based approach, utilizing a global core token to facilitate inter-token interactions. This structural alignment with MedTS signal characteristics reduces computational complexity from quadratic to linear. Experimental results across five benchmarks demonstrate significant improvements, including an 11.6% accuracy increase on the APAVA dataset. Furthermore, the model exhibits superior efficiency, requiring only 33% of the memory and 20% of the inference time compared to previous state-of-the-art methods. The study highlights a critical rethinking of architecture design for healthcare applications, offering a more effective and resource-efficient solution for diagnosing brain and heart diseases through advanced deep learning techniques.
cs.AI updates on arXiv.org