My Obsidian Blog

HF Daily Papers 2026-07-21

10分钟阅读

HF Daily Papers 2026-07-21

范围：优先 RLHF / preference optimization / alignment 在 Diffusion 或生成式 Diffusion 中的应用；其次为 RLHF 在 VLM / LLM 中的工作。排除 VLA、机器人、具身智能、robot manipulation、coding / software-engineering agents、long-horizon agents 与 agentic-RL。

筛选说明

  • 本列表只基于 Hugging Face Daily Papers 可见元数据与摘要完成初筛和排序；不应将作者摘要中的主张视为已被独立验证的事实。
  • 中文 AI Summary 是摘要级判断，不等同于全文结论。
  • 相关论文数：4。
  • 全文精读候选数：2。

排序后的相关论文

1. ⭐ Group Entropy-Controlled Policy Optimization

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 4/5
  • 潜在工程价值： 4/5
  • 作者： Guangran Cheng, Chengqi Lyu, Songyang Gao, Wenwei Zhang, Kai Chen
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

针对大语言模型 RLHF 对异构任务混合训练时全局与 Token 级熵控制不足以及 GRPO 组内优势归一化引入偏置的问题，提出了组熵控制策略优化（GEPO）。该方法基于历史组熵统计进行非对称优势重塑，在降低低熵组过拟合的同时保护高熵组的探索，在数学、代码与指令遵循等多项基准上取得了优于 GRPO 的效果。需要注意，此评估仅基于论文摘要内容。

Abstract

Entropy control has become an effective tool in reinforcement learning (RL) of large language models (LLMs), helping balance exploration-exploitation trade-off during alignment process. Such RL paradigm is often conducted on mixtures of heterogeneous tasks, which induce distinct entropy regimes under the same policy, making global or token-level entropy regulation insufficient to corresponding heterogeneous needs of exploration. This heterogeneity further makes GRPO-style normalized advantages induce an entropy-dependent bias, making advantage signals across prompt groups statistically non-comparable. To address this issue, we propose Group Entropy-Controlled Policy Optimization (GEPO), a lightweight extension to GRPO that uses group entropy, estimated from existing grouped samples to perform entropy-conditioned asymmetric advantage shaping. GEPO attenuates positive advantages in low-entropy groups to reduce over-exploitation, and negative advantages in high-entropy groups to preserve exploration, with adaptive thresholds derived from historical entropy statistics. Extensive experiments on two base models across thirteen benchmarks spanning mathematics, physics, science, code generation, and instruction following show that GEPO consistently outperforms GRPO and recent entropy-controlled methods, delivering balanced cross-task improvements while preserving task-specific exploration levels throughout training.

排序依据与全文待核验点

  • 为什么相关： Directly addresses entropy regulation and advantage bias in GRPO-style RLHF on heterogeneous tasks, offering a practical optimization technique for LLM policy training.
  • 全文待核验： Verify the exact statistical formula for group-entropy estimation and how the dynamic thresholds are calculated across training iterations.

2. ⭐ Distilled Reinforcement Learning for LLM Post-training

中文 AI Summary

为解决大模型 Post-Training 中传统 RL 细粒度反馈缺失与在线蒸馏（OPD）强制无条件模仿教师分布的局限，本文提出了蒸馏强化学习（Distilled RL）。该方法通过带截断的反向重要性采样、负样本重置及序列级几何归一化，将教师模型的监督信号选择性地融入 RL 目标，实现了跨模型家族的有效知识迁移。需要注意，此评估仅基于论文摘要内容。

Abstract

Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.

排序依据与全文待核验点

  • 为什么相关： Proposes Distilled Reinforcement Learning to integrate teacher supervision into RL post-training for LLMs, tackling credit assignment and knowledge transfer limits of standard RL/OPD.
  • 全文待核验： Verify how the teacher model’s logits are sampled and incorporated into the student’s RL policy loss update without causing training instability.

3. Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 3/5
  • 潜在工程价值： 3/5
  • 作者： Zitong Huang, Gustavo Lucas Carvalho, Deqing Fu, Robin Jia
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

论文提出了 Token 级离策略标注（TOPL），将 Post-Training 转化为 Token 级别的正确性预测任务，用以识别生成结果中的优劣 Token，从而避免直接在 Off-Policy 样本上进行生成训练的缺陷。实验表明 TOPL 在文档摘要和机器翻译任务中具备较强的分布外泛化能力，且 LoRA 权重表现出可解释的线性分类特征。需要注意，此评估仅基于论文摘要内容。

Abstract

We propose Token-Level Off-Policy Labeling (TOPL), an off-policy training paradigm that reframes post-training as a token-level correctness prediction task. Our key intuition is that by training the model to distinguish good and bad tokens in a response, we naturally guide the model towards generating good tokens, while avoiding the pitfalls that come with directly training the model to generate off-policy tokens. Experiments on document summarization tasks show that TOPL achieves strong out-of-distribution generalization across 11 datasets against a diverse set of sequence-level and token-level baselines. We further demonstrate that TOPL transfers effectively to machine translation, suggesting that its benefits generalize across different faithful generation tasks. Through ablation studies, we confirm that our token-level learning signal is critical to good performance; sequence-level analogues do not confer similar benefits. Finally, we show that TOPL induces interpretable model updates: the LoRA adapters learned through TOPL function as linear classification heads and steering vectors.

排序依据与全文待核验点

  • 为什么相关： Explores token-level off-policy correctness prediction for post-training LLMs, improving faithful generation and out-of-distribution performance.
  • 全文待核验： Verify the exact architectural implementation of the token-level classification head and how off-policy tokens are assigned correctness ground-truth.

4. LLM-as-a-Coach: Experiential Learning for Non-Verifiable Tasks

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 3/5
  • 潜在工程价值： 3/5
  • 作者： Tianzhu Ye, Li Dong, Guanheng Chen, He Zhu, Xun Wu, Shaohan Huang, Furu Wei
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

针对不可验证开放式任务中标量奖励 RL 丢失文本反馈细节、易引发 Reward Hacking 的问题，本文提出经验学习（EL）框架，将裁判模型转化为“教练”，将评估转化为可迁移的经验知识并通过上下文蒸馏作用于策略模型。实验表明 EL 在非验证性任务上的泛化表现优于基于 Rule/Rubric 的标量 RL。需要注意，此评估仅基于论文摘要内容。

Abstract

Reinforcement learning (RL) on open-ended tasks compresses an LLM’s rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.

排序依据与全文待核验点

  • 为什么相关： Presents Experiential Learning (EL) as an alternative to scalar-reward RL for non-verifiable tasks by distilling LLM-as-a-Judge feedback into transferable context guidance.
  • 全文待核验： Verify the prompt structure and computational overhead of on-policy context distillation during the coaching feedback loop.

筛选备注

No directly matching RLHF-Diffusion papers found in today’s batch. Ranked scoped RLHF-VLM/LLM optimization methods, strictly excluding robotics/VLA, embodied agents, and domain-specific molecular generation.

Edit

继续完善这篇笔记

编辑本文AI 协助

关系图谱