MARLaaS: Multi-Tenant Asynchronous Reinforcement Learning as a Service
Researchers have introduced MARLaaS, a novel system designed to address the high computational costs and limited accessibility associated with fine-tuning Large Language Models (LLMs) using Reinforcement Learning from Verifiable Rewards (RLVR). This approach is particularly relevant for multi-turn agentic settings involving tool use. MARLaaS enables concurrent RL fine-tuning across multiple users and tasks by leveraging two core innovations: sharing a base model among tenants via lightweight LoRA adapters, and employing a disaggregated asynchronous architecture. This architecture decouples rollout generation, environment interaction, and policy training into independently scheduled, event-driven stages. By allowing tasks to progress at their own pace, the system significantly reduces cross-task interference, idle time, and end-to-end latency. In tests involving up to 32 concurrent tasks, MARLaaS achieved state-of-the-art performance comparable to single-task setups while improving accelerator utilization by up to 4.3 times and reducing total training time by 85 percent. This development marks a significant step toward more efficient and scalable AI model optimization.
Wire timeline
MARLaaS: Multi-Tenant Asynchronous Reinforcement Learning as a Service
Researchers have introduced MARLaaS, a novel system designed to address the high computational costs and limited accessibility associated with fine-tuning Large Language Models (LLMs) using Reinforcement Learning from Verifiable Rewards (RLVR). This approach is particularly relevant for multi-turn agentic settings involving tool use. MARLaaS enables concurrent RL fine-tuning across multiple users and tasks by leveraging two core innovations: sharing a base model among tenants via lightweight LoRA adapters, and employing a disaggregated asynchronous architecture. This architecture decouples rollout generation, environment interaction, and policy training into independently scheduled, event-driven stages. By allowing tasks to progress at their own pace, the system significantly reduces cross-task interference, idle time, and end-to-end latency. In tests involving up to 32 concurrent tasks, MARLaaS achieved state-of-the-art performance comparable to single-task setups while improving accelerator utilization by up to 4.3 times and reducing total training time by 85 percent. This development marks a significant step toward more efficient and scalable AI model optimization.
cs.AI updates on arXiv.org