HF Daily Papers 2026-07-14
范围：优先 RLHF / preference optimization / alignment 在 Diffusion 或生成式 Diffusion 中的应用；其次为 RLHF 在 VLM / LLM 中的工作。排除 VLA、机器人、具身智能、robot manipulation、coding / software-engineering agents、long-horizon agents 与 agentic-RL。
筛选说明
- 本列表只基于 Hugging Face Daily Papers 可见元数据与摘要完成初筛和排序；不应将作者摘要中的主张视为已被独立验证的事实。
- 中文 AI Summary 是摘要级判断，不等同于全文结论。
- 相关论文数：2。
- 全文精读候选数：2。
排序后的相关论文
1. ⭐ Weak-to-Strong Generalization via Direct On-Policy Distillation
- 方向： RLHF-VLM/LLM
- 研究阅读价值： 5/5
- 潜在工程价值： 5/5
- 作者： Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
- Hugging Face： 论文页
- arXiv： 摘要页 · PDF
- 代码： 未提供
- 项目页： https://bytedtsinghua-sia.github.io/Direct-OPD/
中文 AI Summary
该论文提出了一种弱到强泛化新方法（Direct-OPD），将弱模型在强化学习（RL）前后产生的策略偏移量（对数比例）作为强学生模型的隐式奖励信号，并在学生自身的同策略状态上进行蒸馏。该方法避免了在强模型上运行高昂的稀疏奖励RL，实验表明仅用8张A100训练4小时即可将Qwen3-1.7B在AIME 2024上的表现提升至58.3%。
Abstract
Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher’s final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher’s RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student’s own on-policy states. This directly reuses the weak model’s RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.
排序依据与全文待核验点
- 为什么相关： Direct-OPD introduces a novel weak-to-strong reinforcement learning distillation paradigm by distilling the policy shift (log-ratio of post-RL to pre-RL checkpoints) as an implicit reward signal on target-model on-policy states, achieving 58.3% on AIME 2024 using a 1.7B teacher to boost a stronger student in 4 hours on 8 A100s.
- 全文待核验： Verify the exact mathematical formulation of the dense implicit reward signal transfer and how the sequential composition of multiple policy shifts is implemented.
2. ⭐ Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals
- 方向： RLHF-VLM/LLM
- 研究阅读价值： 5/5
- 潜在工程价值： 5/5
- 作者： Daocheng Fu, Rong Wu, Yu Yang, Xuemeng Yang, Jianbiao Mei, Licheng Wen, Pinlong Cai, Yong Liu, Botian Shi, Yu Qiao
- Hugging Face： 论文页
- arXiv： 摘要页 · PDF
- 代码： 未提供
- 项目页： 未提供
中文 AI Summary
该研究提出了PUST框架，将大语言模型后训练中的策略探索与分布对齐完全解耦。利用轻量级代理模型发现高奖励行为，并提取其状态提升的相对更新信号转移至目标强模型，从而使优化信号可以异步生成、缓存与复用。在数学和代码领域的Qwen3系列模型上验证了其能以极低成本稳健提升强模型性能。
Abstract
Post-training is essential for refining the domain-specific capabilities of large language models (LLMs), yet existing reward optimization and distribution matching methods tightly couple policy exploration with distribution alignment. This coupling forces expensive exploration directly on the policy model and severely hinders the asynchronous generation, reuse, and cross-model transfer of optimization signals. In this paper, we propose Proxy-guided Update Signal Transfer (PUST), a novel post-training framework that fundamentally decouples update-signal exploration from distribution alignment. Instead of utilizing the primary model for costly exploration, PUST employs a lightweight proxy model as an efficient testbed to discover high-reward behaviors. We extract the relative improvement signal between the proxy’s initial and optimized states, transferring this directional update to the primary model to guide its policy alignment. This decoupled pipeline, comprising proxy exploration, update-signal extraction, and signal transfer, significantly reduces computational overhead and enables optimization signals to be asynchronously generated, cached, and reused. Crucially, by transferring relative improvements rather than absolute policy distributions, PUST naturally supports weak-to-strong improvement and seamless cross-model transfer. Systematic evaluations on Qwen3-family models across math and code domains demonstrate that update signals extracted from substantially weaker proxies can robustly and adjustably enhance stronger primary models. Ultimately, PUST transforms post-training from a monolithic online optimization process into a highly modular, reusable, and cost-efficient paradigm.
排序依据与全文待核验点
- 为什么相关： PUST decouples RL policy exploration from distribution alignment by using a lightweight proxy model to discover high-reward behaviors and transferring the relative improvement signal as a directional update to the primary model, offering high modularity, reusability, and asynchronous signal caching.
- 全文待核验： Verify the structural design of the proxy-guided update signal and whether any performance regression occurs when the scale gap between the proxy and target model is exceptionally large.
筛选备注
No papers matching the highest priority RLHF-Diffusion scope were found in today’s set. Therefore, the selection focuses on two highly innovative post-training / RLHF-VLM/LLM papers (Direct-OPD and PUST) that introduce significant paradigm shifts in weak-to-strong distillation and decoupled proxy exploration.