My Obsidian Blog

HF Daily Papers 2026-07-20

11分钟阅读

HF Daily Papers 2026-07-20

范围：优先 RLHF / preference optimization / alignment 在 Diffusion 或生成式 Diffusion 中的应用；其次为 RLHF 在 VLM / LLM 中的工作。排除 VLA、机器人、具身智能、robot manipulation、coding / software-engineering agents、long-horizon agents 与 agentic-RL。

筛选说明

  • 本列表只基于 Hugging Face Daily Papers 可见元数据与摘要完成初筛和排序；不应将作者摘要中的主张视为已被独立验证的事实。
  • 中文 AI Summary 是摘要级判断，不等同于全文结论。
  • 相关论文数：5。
  • 全文精读候选数：3。

排序后的相关论文

1. ⭐ SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 4/5
  • 潜在工程价值： 4/5
  • 作者： Mingyuan Wu, Jingcheng Yang, Shengyi Qian, Xudong Wang, Jize Jiang, Qifan Wang, Aashu Singh, Khoi Pham, Fei Liu, Zhaolun Su, Zhuokai Zhao, Klara Nahrstedt, Jianyu Wang, Hanchao Yu
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

提出 SVR-R1 框架，将模型自身的自我验证结果作为 VLM 多模态推理的 RL 学习信号。基于 GRPO 和异步多轮 Rollout 架构，模型在没有外部监督或 Critic 的情况下实现了自我纠错与推理能力提升。注意：该结论仅基于摘要级信息。

Abstract

We introduce Self-Verified Reasoner (SVR-R1), a multi-turn RL framework that turns a model’s own verification into a learning signal for multimodal reasoning. For each query, the model proposes an answer using the same weights, and issues a binary self-verdict (Yes/No). A ‘No’ triggers a second-chance rethink; a ‘Yes,’ or a turn cap, finalizes the output for computing the outcome-based reward. SVR-R1 is implemented with GRPO and an asynchronous multi-turn rollout framework and needs no external supervision or auxiliary critics. We evaluate SVR-R1 on vision-language reasoning benchmarks and show that it improves accuracy by a large margin over strong standard GRPO baselines. Training dynamics show decreasing reliance on verification-fewer verification turns, yet higher test accuracy-indicating that the gap between verification and generation narrows as the policy internalizes self-correction and chooses the most confident answer via our framework. SVR-R1 bridges the less explored intersection of inference-time self-refinement and RL training for VLMs, offering a simple yet effective recipe for bootstrapping multimodal reasoning. We will open-source SVR-R1 to facilitate future research in VLMs.

排序依据与全文待核验点

  • 为什么相关： Applies multi-turn RL with GRPO to multimodal reasoning via self-verification and self-correction, directly aligning with post-training and RLHF/RLVR for VLMs.
  • 全文待核验： Verify whether the self-verification reward design avoids reward hacking (e.g., model learning to systematically output ‘No’ to get extra turns or artificially boost confidence).

2. ⭐ On-Policy Delta Distillation

中文 AI Summary

提出 On-Policy Delta Distillation (OPD^2)，利用教师模型与基础模型之间的 Output Delta 作为蒸馏奖励信号，转移推理能力。实验表明在数学、科学和代码推理任务上优于传统 On-policy 蒸馏。注意：该结论仅基于摘要级信息。

Abstract

On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher’s output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD^2). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD^2 consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2

排序依据与全文待核验点

  • 为什么相关： Explores on-policy distillation as an alternative post-training alignment/distillation signal using teacher-base delta signals for reasoning models.
  • 全文待核验： Verify how sensitive the delta reward signal is to vocabulary distribution differences and whether temperature scaling is needed during on-policy sampling.

3. ⭐ Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 4/5
  • 潜在工程价值： 3/5
  • 作者： Weiwen Xu, Jia Liu, Hou Pong Chan, Long Li, Deng Cai, Min Chen, Hao Zhang
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

针对可验证奖励 RL (RLVR) 中熵正则化无法区分有用不确定性与混乱的问题，提出对比策略优化 (CPO)。通过参考引导与 Vanilla 生成分布之间的 Token 级对比分歧实现精确的 Advantage Shaping。注意：该结论仅基于摘要级信息。

Abstract

Reinforcement learning with verifiable rewards (RLVR) commonly uses entropy for advantage shaping. However, entropy cannot distinguish useful uncertainty from detrimental confusion, limiting its effectiveness as a correctness signal. We propose Contrastive Policy Optimization (CPO), which uses token-level contrastive disagreement between reference-guided and vanilla generation distributions for correctness-aware advantage shaping. Both theoretical and empirical results show that this disagreement reliably indicates token-level correctness. We further show that On-policy Distillation is a special case of CPO, where the posterior distribution is instantiated by an external teacher model. CPO also resolves the zero-advantage problem. Experiments on in-domain and out-of-domain benchmarks demonstrate that CPO substantially outperforms entropy-based RLVR methods while maintaining strong generalization. Further analysis shows that correct and incorrect responses naturally support exploration and exploitation respectively, and balancing both leads to the best performance.

排序依据与全文待核验点

  • 为什么相关： Focuses on advantage shaping via contrastive policy optimization for reinforcement learning with verifiable rewards (RLVR) in LLMs.
  • 全文待核验： Verify the computational overhead of maintaining reference-guided distributions during parallel RL updates in scaling experiments.

4. Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

中文 AI Summary

针对 GRPO 等仅针对最终答案打分导致模型倾向于堆砌字数而非改进推理的问题，提出 Agon 交叉模型对抗 RL 机制。两个模型互为 Grader 和 Rival，通过隐式推理对决提升 Trace 质量。注意：该结论仅基于摘要级信息。

Abstract

Reinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today’s reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other’s graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO’s pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.

排序依据与全文待核验点

  • 为什么相关： Introduces a cross-model competitive RL approach (Agon) extending GRPO by implicitly grading reasoning traces without external reward models.
  • 全文待核验： Verify whether policy collapse or cyclic game-theoretic dominance occurs when training cross-model rivals over long iteration windows.

5. Understanding Reasoning from Pretraining to Post-Training

中文 AI Summary

通过受控实验（国际象棋与数学领域）系统研究预训练与 RL 后训练的接口机制。发现预训练 Loss 可预测 RL 收益，且 RL 能使 SFT 下原本几乎不出现的正确解被挖掘出来。注意：该结论仅基于摘要级信息。

Abstract

Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.

排序依据与全文待核验点

  • 为什么相关： Provides empirical analysis on how pretraining compute/loss directly determines and predicts the returns of RL compute during post-training.
  • 全文待核验： Verify if the linear relationship between pretraining tokens and RL reward slope holds on general open-domain multi-step reasoning LLMs beyond chess/math.

筛选备注

No RLHF-Diffusion papers were present in today’s dataset. Selected high-value RLHF/post-training/RLVR methods applied to VLMs and LLMs while excluding agentic, robotics, code-review, and hardware-specific papers according to exclusion guidelines.

Edit

继续完善这篇笔记

编辑本文AI 协助

关系图谱