My Obsidian Blog

HF Daily Papers 2026-07-24

8分钟阅读

HF Daily Papers 2026-07-24

范围：优先 RLHF / preference optimization / alignment 在 Diffusion 或生成式 Diffusion 中的应用；其次为 RLHF 在 VLM / LLM 中的工作。排除 VLA、机器人、具身智能、robot manipulation、coding / software-engineering agents、long-horizon agents 与 agentic-RL。

筛选说明

  • 本列表只基于 Hugging Face Daily Papers 可见元数据与摘要完成初筛和排序；不应将作者摘要中的主张视为已被独立验证的事实。
  • 中文 AI Summary 是摘要级判断，不等同于全文结论。
  • 相关论文数：3。
  • 全文精读候选数：3。

排序后的相关论文

1. ⭐ Visual Contrastive Self-Distillation

中文 AI Summary

提出 Visual Contrastive Self-Distillation (VCSD) 方法，通过对比图像内容抹除后的输出分布差异，为多模态大模型提供在线自蒸馏（OPSD）信号。在不需要外部 Teacher 或额外推理开销的前提下，显著提升了 Qwen3-VL 等模型的多模态基准表现。该结论基于论文摘要判断。

Abstract

On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student. Existing methods create this asymmetry either through privileged answers or visual evidence. We ask whether both can be removed, yielding a simpler form of OPSD driven purely by input conditioning. For this purpose, we propose Visual Contrastive Self-Distillation, namely VCSD, which converts image-content removal into an on-policy self-distillation signal. At each student-generated response prefix, the EMA teacher produces two next-token distributions under the same prompt and prefix — one conditioned on the original image and the other on a content-erased control. Their token-wise log-probability difference highlights candidates whose likelihood is specifically increased by the instance-level visual content. We use this contrast to sharpen the teacher’s original-image distribution within its plausible support, and distill the resulting full-distribution target into the student. Using ViRL39K dataset, VCSD consistently outperforms matched OPSD across Qwen3-VL and Qwen3.5 models. For example, on Qwen3-VL, it improves the seven-benchmark aggregate from 62.27% rightarrow 67.04% at 2B, 71.30% rightarrow 73.16% at 4B, and 72.51% rightarrow 76.26% at 8B. Furthermore, VCSD requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference-time cost.

排序依据与全文待核验点

  • 为什么相关： Introduces Visual Contrastive Self-Distillation (VCSD), an on-policy self-distillation approach using contrastive image-content removal signals for VLMs without requiring an external teacher.
  • 全文待核验： Verify the specific formulation of the EMA teacher contrastive loss and how content erasure is implemented during student rollout.

2. ⭐ Multi-Turn On-Policy Distillation with Prefix Replay

中文 AI Summary

针对多轮 Agent 任务中在线策略蒸馏（OPD）成本极高的问题，提出了 Replayed-Prefix On-Policy Distillation (ReOPD)。通过重用 Teacher 轨迹并结合衰减采样策略，在不进行新环境交互的情况下实现了高效的蒸馏与对齐。该结论基于论文摘要判断。

Abstract

We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4times faster per rollout than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.

排序依据与全文待核验点

  • 为什么相关： Proposes Replayed-Prefix On-Policy Distillation (ReOPD) to address off-environment multi-turn agentic distillation for LLMs while avoiding distribution shift / prefix trap.
  • 全文待核验： Verify the exact schedule of prefix step-decay sampling and its impact on teacher reliability metrics in multi-turn environments.

3. ⭐ Predictive Divergence Masks for LLM RL

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 4/5
  • 潜在工程价值： 3/5
  • 作者： Xiangxin Zhou, Jiarui Yao, Penghui Qi, Bowen Ping, Jiaqi Tang, Haonan Wang, Tianyu Pang
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

针对 LLM 强化学习中传统 PPO 比例掩码与概率散度方向不一致的问题，提出了预测性散度掩码（Predictive Divergence Mask）。通过闭式推导与轻量级 Top-K 估计器，实现了更稳定的 off-policy RL 更新。该结论基于论文摘要判断。

Abstract

Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates. The dominant PPO-style approach uses the sampled-token importance ratio for two criteria: a proximity criterion, which asks whether the policy has moved too far from the behavior policy, and a direction criterion, which asks whether the update pushes it farther away. Recent work DPPO improves the proximity criterion by replacing PPO’s ratio-based test with a probability divergence between the behavior and training policies. However, its direction criterion is still inherited from PPO. A token can be masked only when the sampled-token importance ratio moves away from one. We observe that this ratio-based direction criterion is a single-sample proxy that can disagree in sign with the change of the divergence that defines the proximity criterion. We therefore propose the predictive divergence mask, which asks whether the next policy-gradient step will increase or decrease the same divergence used by the trust region. For the discrete softmax policies used in LLM RL, we derive this prediction in closed form. Because production rollout engines expose only a truncated (top-K) view of the vocabulary, we develop two lightweight top-K estimators for this prediction. Detailed analysis shows the divergence-based direction is better aligned with the realized change of the divergence than the sampled ratio, and the resulting masks improve RL training across model scales and precision settings.

排序依据与全文待核验点

  • 为什么相关： Develops predictive divergence masks for LLM RL (PPO/DPPO style) to better align policy gradient updates with trust-region probability divergence constraints.
  • 全文待核验： Verify the mathematical derivation of the closed-form prediction for discrete softmax policies and the accuracy of top-K estimators under truncated sampling.

筛选备注

No Diffusion-focused RLHF/post-training papers were present in today’s dataset. Filtered out domain-specific robotics/embodied agents, general agent frameworks, and visual generation without RLHF/preference alignment.

Edit

继续完善这篇笔记

编辑本文AI 协助

关系图谱