My Obsidian Blog

HF Daily Papers 2026-07-09

8分钟阅读

HF Daily Papers 2026-07-09

范围：优先 RLHF / preference optimization / alignment 在 Diffusion 或生成式 Diffusion 中的应用；其次为 RLHF 在 VLM / LLM 中的工作。排除 VLA、机器人、具身智能与 robot manipulation。

筛选说明

  • 本列表只基于 Hugging Face Daily Papers 可见元数据与摘要完成初筛和排序；不应将作者摘要中的主张视为已被独立验证的事实。
  • 中文 AI Summary 是摘要级判断，不等同于全文结论。
  • 相关论文数：3。
  • 全文精读候选数：3。

排序后的相关论文

1. OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generators

中文 AI Summary

摘要级判断：本文针对少步自回归视频扩散模型长视频生成中的误差累积和运动退化问题，提出基于真实视频上下文的在策略自蒸馏方法。作者声称该方法在多个模型上提升了视觉质量、运动动态和VBenchLong表现，并在用户研究中获得更高偏好率。

Abstract

We propose OPSD-V, an on-policy self-distillation paradigm for post-training few-step autoregressive (AR) video diffusion models. Existing few-step AR video generators can produce long videos with low latency, but still suffer from error accumulation and weakened motion dynamics during long autoregressive rollout. OPSD-V reduces long-horizon degradation while preserving the original few-step inference path. The key idea is to introduce real long-video data as temporal context during training and use it to provide dense trajectory-level supervision. Specifically, the student follows the exact inference-time rollout, generating each chunk conditioned on its own previously generated KV cache. In parallel, the teacher is evaluated at the same student-visited denoising states, but uses a cleaner AR-consistent temporal cache in which older history can be replaced by real-video context. This provides dense denoising-level corrective targets under on-policy AR cache dynamics, without changing the sampler, number of denoising steps, or inference-time cache mechanism. We apply OPSD-V to representative few-step AR video models, including Self-Forcing and LongLive. Experiments show consistent improvements in visual quality, motion dynamics, and VBenchLong scores. A user study with 10 participants comparing 20 video pairs shows that OPSD-V is preferred over the base models in 66.0% of overall-preference judgments (82.5% excluding ties).

排序依据与全文待核验点

  • 为什么相关： Explicitly studies post-training for few-step autoregressive video diffusion, using on-policy rollout states and dense denoising-level supervision to address long-horizon error accumulation while preserving inference behavior.
  • 全文待核验： Verify whether the reported gains come from the proposed on-policy supervision rather than differences in data, training compute, or teacher configuration.

2. Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 5/5
  • 潜在工程价值： 5/5
  • 作者： Zhenyu Hou, Yujiang Li, Jie Tang, Yuxiao Dong
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

摘要级判断：本文研究长时程智能体任务中的异步大语言模型强化学习，提出单次采样异步优化方法，并结合价值模型训练和严格的双侧 token 级裁剪来改善稳定性。作者声称该方法在SWE-Bench Verified、BeyondAIME和IMOAnswerBench上持续优于GRPO变体，并已用于训练GLM-5.2。

Abstract

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).

排序依据与全文待核验点

  • 为什么相关： Presents a concrete asynchronous RL post-training algorithm for LLMs, addressing single-rollout sampling, off-policy effects, value training, and token-level clipping with results on coding and reasoning benchmarks.
  • 全文待核验： Verify the exact baseline implementations, sampling budgets, and statistical significance of the claimed improvements over GRPO and its variants.

3. Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

中文 AI Summary

摘要级判断：本文针对多模态大语言模型学习触觉能力时容易损害原有视觉语言能力的问题，提出通过划分关键与休眠参数子空间来进行隔离式触觉对齐学习。作者声称该方法无需增加语言模型推理开销即可获得领先的视觉触觉推理性能，同时减少灾难性遗忘。

Abstract

Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of compact models forces a choice between acquiring the new sensory modality and preserving the established vision-language reasoning. We present Splash, a mask-isolated tactile alignment learning framework for MLLMs. Splash quantifies the significance of each pretrained parameter, and partitions the parameter space into a dormant and critical subspace. While the frozen critical subspace acts as a stable anchor to safeguard general visual knowledge, Splash updates the isolated dormant subspace to internalize tactile alignment towards LLMs. This selective, non-destructive expansion effectively prevents catastrophic forgetting and ensures non-destructive modality expansion. Extensive experiments show that Splash effectively achieves tactile reasoning without additional inference overhead in the LLM part, demonstrating state-of-the-art performance on visuo-tactile benchmarks, including SSVTP, TVL, and TacQuad, while preserving its original general-purpose capabilities.

排序依据与全文待核验点

  • 为什么相关： Alignment learning is the central contribution: it isolates dormant parameter subspaces for tactile adaptation in MLLMs while attempting to preserve existing vision-language capabilities.
  • 全文待核验： Verify how parameter significance and the dormant/critical subspaces are computed, and whether preservation is demonstrated on broad held-out vision-language tasks.

筛选备注

Other records were excluded because they were generic foundation-model work, lacked a substantive RLHF/preference/alignment/post-training contribution, or fell under the explicit robotics, embodied-agent, or VLA exclusions.

Edit

继续完善这篇笔记

编辑本文AI 协助

关系图谱