Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation
Researchers from arXiv have introduced Parametric Skill Transfer (PaST), a novel framework designed to address the knowledge cutoff challenge in Large Language Models (LLMs). While Supervised Fine-Tuning (SFT) updates factual knowledge, it often fails to improve reasoning capabilities, and Reinforcement Learning (RL) is computationally prohibitive for online adaptation. The study observes that parameter updates from SFT and RL are nearly orthogonal. Leveraging this, PaST extracts a domain-agnostic Skill Vector from a source domain and linearly injects it into target models after lightweight SFT. This method enables efficient knowledge manipulation without the high costs of full RL training. Experimental results demonstrate significant performance improvements: PaST outperforms state-of-the-art self-editing SFT baselines by up to 9.9 points on SQuAD, achieves an 8.0-point accuracy gain on long-context QA benchmarks like LooGLE, and improves zero-shot success rates on ToolBench by 10.3 points. These findings highlight the framework's scalability and cross-domain transferability, offering a practical solution for continual adaptation in AI systems.
Wire timeline
Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation
Researchers from arXiv have introduced Parametric Skill Transfer (PaST), a novel framework designed to address the knowledge cutoff challenge in Large Language Models (LLMs). While Supervised Fine-Tuning (SFT) updates factual knowledge, it often fails to improve reasoning capabilities, and Reinforcement Learning (RL) is computationally prohibitive for online adaptation. The study observes that parameter updates from SFT and RL are nearly orthogonal. Leveraging this, PaST extracts a domain-agnostic Skill Vector from a source domain and linearly injects it into target models after lightweight SFT. This method enables efficient knowledge manipulation without the high costs of full RL training. Experimental results demonstrate significant performance improvements: PaST outperforms state-of-the-art self-editing SFT baselines by up to 9.9 points on SQuAD, achieves an 8.0-point accuracy gain on long-context QA benchmarks like LooGLE, and improves zero-shot success rates on ToolBench by 10.3 points. These findings highlight the framework's scalability and cross-domain transferability, offering a practical solution for continual adaptation in AI systems.
cs.AI updates on arXiv.org