Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection
Researchers have introduced Echo-LoRA, a novel parameter-efficient fine-tuning (PEFT) method designed to enhance the adaptation of large language models (LLMs) for downstream tasks. Unlike traditional LoRA variants that primarily modify weight spaces within individual layers, Echo-LoRA leverages cross-layer representation injection. During training, it aggregates boundary hidden states from deeper layers into sample-level echo representations, which are then injected into shallow LoRA or DoRA modules using lightweight projection and gating networks. To ensure stability and minimize the discrepancy between training and inference, the method employs answer-only masking, masked distillation, and stochastic routing. Experimental results across eight commonsense reasoning benchmarks demonstrate that Echo-LoRA outperforms standard LoRA baselines by an average of 5.7 percentage points on LLaMA-7B, LLaMA2-7B, and LLaMA3-8B models. Crucially, the auxiliary Echo path is discarded post-training, ensuring that the deployed model retains the original low-rank structure without adding any inference-time parameters or computational overhead. This approach offers a significant performance boost while maintaining the deployment efficiency characteristic of PEFT methods.
Wire timeline
Echo-LoRA: Parameter-Efficient Fine-Tuning via Cross-Layer Representation Injection
Researchers have introduced Echo-LoRA, a novel parameter-efficient fine-tuning (PEFT) method designed to enhance the adaptation of large language models (LLMs) for downstream tasks. Unlike traditional LoRA variants that primarily modify weight spaces within individual layers, Echo-LoRA leverages cross-layer representation injection. During training, it aggregates boundary hidden states from deeper layers into sample-level echo representations, which are then injected into shallow LoRA or DoRA modules using lightweight projection and gating networks. To ensure stability and minimize the discrepancy between training and inference, the method employs answer-only masking, masked distillation, and stochastic routing. Experimental results across eight commonsense reasoning benchmarks demonstrate that Echo-LoRA outperforms standard LoRA baselines by an average of 5.7 percentage points on LLaMA-7B, LLaMA2-7B, and LLaMA3-8B models. Crucially, the auxiliary Echo path is discarded post-training, ensuring that the deployed model retains the original low-rank structure without adding any inference-time parameters or computational overhead. This approach offers a significant performance boost while maintaining the deployment efficiency characteristic of PEFT methods.
cs.AI updates on arXiv.org