General Exploratory Bonus for Optimistic Exploration in RLHF
Wendi Li, Changdae Oh, Sharon Li
Abstract
Optimistic exploration is central to improving sample efficiency in reinforcement learning with human feedback, yet existing exploratory bonus methods to incentivize exploration often fail to realize optimism. We provide a theoretical analysis showing that current formulations, under KL or $α$-divergence regularization, unintentionally bias exploration toward high-probability regions of the reference model, thereby r...
Results & Benchmarks
| Task | Dataset | Metric | Value |
|---|---|---|---|
| Reinforcement learning | MATH | pass@16 | 6 |
Hardware Requirements
- Expect multi-day setup/compute for meaningful reproduction based on current guidance.
Best Implementation
Maintained implementation evidence is not confirmed for this paper yet.
Use the Implementation Status and Reproduction Path sections below for the current action plan.
Reproduction Path
Follow this baseline workflow to decide if this paper is worth immediate prototyping.
- 1
No maintained paper-verified implementation was found; start with the closest related repositories below.
- 2
Compare repo methods against the paper equations/algorithm before trusting metrics.
- 3
Create a minimal baseline implementation from the paper and use adjacent repos as references.
- 4
Prioritize reproducing the core method first: Reinforcement learning.
Related Implementations
These are not paper-verified. Use them as reference points when no direct implementation is available.
Matches contextual method/domain keyword: reinforcement learning
Matches contextual method/domain keyword: reinforcement learning
Additional Implementations
No additional verified repositories beyond the primary recommendation.
Hugging Face Artifacts
No trustworthy direct or curated related Hugging Face artifacts were found yet.
Continue with targeted Hugging Face searches: