Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoning

Research Summary

Contribution Summary

• Traditional on-policy Reinforcement Learning with Verifiable Rewards (RLVR) frameworks suffer from experience waste and reward homogeneity, which directly hinders learning efficiency on difficult samples during large language models post-tr
• In this paper, we introduce Batch Adaptation Policy Optimization (BAPO), an off-policy RLVR framework to improve the data efficiency in large language models post-training.
• It dynamically selects training batches by re-evaluating historically difficult samples and reusing high-quality ones, while holding a lower bound guarantee for policy improvement.

Related Papers

• TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Models Embedding Neighbor
• Explicit Grammar Semantic Feature Fusion for Robust Text Classification Embedding Neighbor
• STAR: Similarity-guided Teacher-Assisted Refinement for Super-Tiny Function Calling Models Embedding Neighbor
• Virtual Biopsy for Intracranial Tumors Diagnosis on MRI Embedding Neighbor
• CodePDE: An Inference Framework for LLM-driven PDE Solver Generation Embedding Neighbor
• Meenz bleibt Meenz, but Large Language Models Do Not Speak Its Dialect Embedding Neighbor
• Reinforced Fast Weights with Next-Sequence Prediction Embedding Neighbor
• The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera Footage Embedding Neighbor
• GATES: Self-Distillation under Privileged Context with Consensus Gating Tag Overlap
• Proof-RM: A Scalable and Generalizable Reward Model for Math Proof Tag Overlap
• Esoteric Language Models: Bridging Autoregressive and Masked Diffusion LLMs Tag Overlap
• Recursive Concept Evolution for Compositional Reasoning in Large Language Models Tag Overlap