HF Daily Papers 2026-07-23
范围：优先 RLHF / preference optimization / alignment 在 Diffusion 或生成式 Diffusion 中的应用；其次为 RLHF 在 VLM / LLM 中的工作。排除 VLA、机器人、具身智能、robot manipulation、coding / software-engineering agents、long-horizon agents 与 agentic-RL。
筛选说明
- 本列表只基于 Hugging Face Daily Papers 可见元数据与摘要完成初筛和排序；不应将作者摘要中的主张视为已被独立验证的事实。
- 中文 AI Summary 是摘要级判断，不等同于全文结论。
- 相关论文数：3。
- 全文精读候选数：3。
排序后的相关论文
1. ⭐ Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization
- 方向： RLHF-VLM/LLM
- 研究阅读价值： 4/5
- 潜在工程价值： 4/5
- 作者： Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Mingxuan Wang, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
- Hugging Face： 论文页
- arXiv： 摘要页 · PDF
- 代码： https://github.com/Aiolus-X/RIPO
- 项目页： 未提供
中文 AI Summary
本文分析了 PPO-Clip 在大语言模型强化学习中因假设欧氏度量而导致的探索崩溃问题。作者提出了基于黎曼流形保距更新的 RIPO 算法，旨在平衡探索与利用。实验表明，该方法在 AIME24 等竞赛级推理基准上比 GRPO 取得了显著提升，但该结论仅基于论文摘要及作者声称的结果。
Abstract
Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs’ reasoning capabilities. However, RL algorithms with PPO-Clip are inherently limited by exploration collapse. Subsequent works remain primarily heuristic and fail to identify the essential cause of PPO-Clip’s failure. This work reveals the fundamental flaw of PPO-Clip: it implicitly measures policy discrepancy using Euclidean metric, which is theoretically inconsistent with the intrinsic geometry on the policy Riemannian manifold. This geometric mismatch results in overly conservative updates in low-probability regions while aggressive in high-probability regions, ultimately collapsing exploration. To correct this geometric flaw, we propose Riemannian Isometric Policy Optimization (RIPO), which guarantees isometric policy updates on the Riemannian manifold, effectively balancing exploration and exploitation. We further show that RIPO achieves a favorable bias-variance trade-off, which stabilizes optimization. Extensive experiments demonstrate that RIPO significantly surpasses existing LLM RL algorithms across seven competition-level benchmarks (up to 60% improvement over GRPO on AIME24).
排序依据与全文待核验点
- 为什么相关： Directly targets fundamental exploration collapse in PPO-Clip for LLM RL by introducing Riemannian Isometric Policy Optimization (RIPO) to match non-Euclidean policy manifold geometry.
- 全文待核验： Verify mathematical formulation of the Riemannian isometric update rule and actual implementation stability compared to standard PPO/GRPO.
2. ⭐ SLPO: Scaling Latent Reasoning via a Surrogate Policy
- 方向： RLHF-VLM/LLM
- 研究阅读价值： 4/5
- 潜在工程价值： 3/5
- 作者： Runyang You, Zhiyuan Liu, Yongqi Li, Wenjie Li
- Hugging Face： 论文页
- arXiv： 摘要页 · PDF
- 代码： https://github.com/ModalityDance/SLPO
- 项目页： 未提供
中文 AI Summary
针对隐空间推理模型缺少显式 Step Likelihood 导致无法直接应用 Outcome-Reward RL 的问题，作者提出了 SLPO 方法。该方法利用经验替代策略密度与可变步长停止头，实现了连续隐空间推理轨迹的强化学习与测试期扩展，此结果仅基于摘要描述。
Abstract
Reinforcement learning with verifiable rewards has become the predominant recipe for eliciting test-time scaling in explicit Chain-of-Thought reasoners. Yet this scaling path remains computationally costly, since every intermediate step must be decoded as a language token. Latent reasoning instead carries intermediate computation as continuous vectors and already matches or surpasses explicit CoT at far shorter horizons. Despite this promise, latent reasoners remain largely imitation-bound, while explicit CoT has already moved past imitation via outcome-reward RL. Latent trajectories lack a tractable per-step likelihood and an adaptive stopping interface under fixed thinking budgets, so outcome rewards cannot elicit latent test-time scaling. We introduce Surrogate Latent Policy Optimization (SLPO) to bring outcome-reward RL to autoregressive latent reasoners: an empirical surrogate policy density over latent transitions for trajectory-level credit assignment, and a correctness-supervised stopping head that outcome-reward optimization refines into a variable-horizon policy. Across continuous and soft thinking settings, SLPO improves Pass@k under parallel sampling and allocates longer latent computation to harder instances with higher deterministic accuracy.
排序依据与全文待核验点
- 为什么相关： Addresses outcome-reward RL for continuous latent reasoners by introducing Surrogate Latent Policy Optimization (SLPO) to overcome missing per-step likelihoods and enable test-time scaling.
- 全文待核验： Verify how the empirical surrogate policy density is calculated without incurring intractable continuous likelihood computation errors.
3. ⭐ Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning
- 方向： RLHF-VLM/LLM
- 研究阅读价值： 3/5
- 潜在工程价值： 4/5
- 作者： Md Tanvirul Alam
- Hugging Face： 论文页
- arXiv： 摘要页 · PDF
- 代码： https://github.com/maveryn/trace
- 项目页： https://maveryn.github.io/trace/
中文 AI Summary
论文构建了具备可验证奖励的多领域视觉推理环境 Trace，通过场景语法与可执行程序生成 1,000 个任务。基于该环境对 Qwen2.5-VL 进行 RLVR 训练，在 24 个外部评测基准上获得了平均 3.5–4.0 个百分点的提升，结论仅基于摘要推断。
Abstract
Reinforcement learning with verifiable rewards (RLVR) has substantially improved language-model reasoning, yet its extension to vision-language models remains constrained by the lack of training data that are simultaneously broad, exactly verifiable, and reproducible. We introduce Trace, a taxonomy-guided environment for multidomain visual reasoning. Trace factorizes task construction into a scene grammar and an executable task program, separating visual realization from answer computation. A shared semantic state determines the rendered image, prompt, typed answer, verifier state, and replayable instance trace. The resulting environment comprises 1,000 tasks over 277 scene grammars and 11 visual domains, with controlled semantic and visual variation. RLVR on 64,000 Trace instances improves the macro-average across 24 external benchmarks by 3.51 percentage points for Qwen2.5-VL-3B and 4.06 points for Qwen2.5-VL-7B, providing evidence that broad procedural training can transfer beyond the generated task distributions. Project page: https://maveryn.github.io/trace/.
排序依据与全文待核验点
- 为什么相关： Explores RLVR (RL with verifiable rewards) scaling for multimodal vision-language models using a procedurally generated visual reasoning environment (Trace).
- 全文待核验： Verify whether the RLVR performance gain on 24 external benchmarks stems from genuine reasoning transfer or environment-induced distribution leakage.
筛选备注
No direct RLHF/preference optimization papers for Diffusion models were found in today’s set; selected top RLHF/RLVR algorithmic advances for LLMs and VLMs instead.