My Obsidian Blog

HF Daily Papers 2026-07-16

13分钟阅读

HF Daily Papers 2026-07-16

范围：优先 RLHF / preference optimization / alignment 在 Diffusion 或生成式 Diffusion 中的应用；其次为 RLHF 在 VLM / LLM 中的工作。排除 VLA、机器人、具身智能、robot manipulation、coding / software-engineering agents、long-horizon agents 与 agentic-RL。

筛选说明

  • 本列表只基于 Hugging Face Daily Papers 可见元数据与摘要完成初筛和排序；不应将作者摘要中的主张视为已被独立验证的事实。
  • 中文 AI Summary 是摘要级判断，不等同于全文结论。
  • 相关论文数：5。
  • 全文精读候选数：3。

排序后的相关论文

1. ⭐ Registers Matter for Pixel-Space Diffusion Transformers

中文 AI Summary

研究了寄存器 Token（register tokens）在像素级扩散 Transformer（DiT）中的作用。基于摘要，作者发现 DiT 虽然不表现出传统 ViT 的高范数离群 Token，但寄存器仍能显著提高其在高噪声水平下的特征图质量，据此提出的 Register Guidance 优化了视觉结构和连贯性；此判断仅基于摘要。

Abstract

Vision Transformers (ViTs) are known to exhibit high-norm patch-token outliers that degrade feature map quality, a problem effectively mitigated by register tokens. As diffusion models increasingly adopt transformer architectures and move toward pixel-space training, they become closer in form to ViTs, raising the question of whether register tokens are also useful for Diffusion Transformers (DiTs). In this work, we show that DiTs differ from ViTs in a key respect: they do not exhibit patch-token outliers but still benefit from registers. Interestingly, registers are more effective in pixel-space DiTs than in latent-space DiTs. By analyzing intermediate representations, we find that register tokens produce cleaner feature maps at high noise levels, which may contribute to their effectiveness in pixel-space generation. We further observe that recent pixel-space DiT architectures implicitly incorporate register-like mechanisms, which may partially account for their strong empirical performance. Motivated by these observations, we propose Register Guidance, a technique that amplifies the contribution of register tokens responsible for improving visual structure and coherence.

排序依据与全文待核验点

  • 为什么相关： The paper investigates the use of register tokens in pixel-space Diffusion Transformers (DiTs). It discovers that registers produce cleaner feature maps at high noise levels and proposes Register Guidance to amplify their contribution to visual structure and coherence. This is highly relevant to Diffusion architectural and post-training optimizations.
  • 全文待核验： Verify the exact formulation and computational overhead of Register Guidance during inference.

2. ⭐ Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 5/5
  • 潜在工程价值： 4/5
  • 作者： Xinyu Tang, Gangqiang Cao, Yurou Liu, Yuliang Zhan, Xiaochong Lan, Yifan Li, Yuchen Yan, Han Peng, Zican Dong, Zhenduo Zhang, Tianshu Wang, Xinyu Kong, Zujie Wen, Wayne Xin Zhao, Zhiqiang Zhang, Jun Zhou
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

研究了无人类标注反馈的零样本强化学习（zero RL）在万亿参数（1T）规模下的扩展动力学。基于摘要，作者通过剪切重要性采样和训练推理比例纠正等系统优化稳定了训练，并发现万亿级模型自发涌现出了自我验证和并行推理等高级认知行为；此判断仅基于摘要。

Abstract

Reinforcement learning with verifiable rewards without human-annotated data, often referred to as zero RL, has emerged as a powerful paradigm for eliciting chain-of-thought reasoning. However, due to computational constraints, existing studies are largely restricted to small models, leaving the training dynamics and emergent capabilities at a large scale unexplored. To meaningfully explore this frontier, we aim to elicit high-quality reasoning behaviors from the model. However, we find that naive scaling often suffers from poor readability, token redundancy, and a lack of adaptive reasoning depth. To address these challenges, we present a stable and efficient training pipeline, incorporating algorithmic and system optimizations such as clipped importance sampling, training-inference ratio correction, and mixed-precision control. Our experiments offer three key findings that validate the “bitter lesson” of scaling: (1) scaling to 1T parameters significantly enhances sample efficiency and performance ceilings; (2) the training process progresses sequentially through an initial discovery phase followed by a sharpening phase; and (3) the model spontaneously develops advanced cognitive behaviors, including anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety, rendering hand-crafted heuristics redundant. Evaluated on seven mathematical benchmarks, Ring-2.5-1T-Zero achieves competitive performance. Additionally, to assess CoT quality beyond final-answer correctness, we propose a structured evaluation framework across three dimensions: comprehensibility, reproducibility, and efficiency, where our model demonstrates clear advantages in producing structured and concise reasoning traces. By sharing our observed emergent phenomena, we hope to provide the community with deeper insights into scaling behaviors, particularly at the 1-trillion scale.

排序依据与全文待核验点

  • 为什么相关： An investigation of zero-data reinforcement learning (zero RL) scaled to a 1-trillion parameter model. It details algorithmic and system optimizations (clipped importance sampling, training-inference ratio correction, and mixed-precision control) to solve training instability, and documents emergent cognitive behaviors.
  • 全文待核验： Verify the mathematical formulation and system scaling limits of the ‘training-inference ratio correction’ and ‘clipped importance sampling’ in their 1T pipeline.

3. ⭐ ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 4/5
  • 潜在工程价值： 4/5
  • 作者： Qingyu Zhang, Qianhao Yuan, Hongyu Lin, Yaojie Lu, Xianpei Han, Le Sun, Xiang Li, Ming Xu, Jiarui Li, Xiuyin Zhao
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

提出了一种基于“由短到长”同策略蒸馏（ShortOPD）的方法，用以恢复结构化剪枝 LLM 的自由文本生成能力。基于摘要，该方法通过识别教师确认的重复后缀，动态分配生成步长，将恢复效率提升数倍并极大节省训练时间；此判断仅基于摘要。

Abstract

Structured pruning is a hardware-friendly way to compress LLMs, but it is mostly validated on multiple-choice recognition tasks, while the same compressed checkpoints can collapse on the free-form generation that deployment actually requires. Two observations trace this gap. First, greedy pass@1 nearly vanishes after compression, yet pass@k recovers substantially under repeated sampling: useful generations are demoted, not erased. Second, the recoverable regime fails mainly through suffix repetition. Recovery should therefore train on the compressed model’s own on-policy states with dense token-level supervision, which On-Policy Distillation (OPD) provides by reusing the pre-compression model as a frozen teacher. However, long on-policy rollouts spend early recovery budget on low-information repetitive suffixes, delaying loss descent. To mitigate this waste, we propose \shortopd, a short-to-long OPD schedule that detects teacher-confirmed repetitive suffixes, treats the surviving prefix as each rollout’s effective length, and allocates future rollout budgets to the effective lengths the policy can currently use. Across math, code, and open-ended generation, \shortopd\ raises the compressed model’s score to about 9times its unrecovered value and 1.6—4.4times standard recovery recipes (SFT w/o KD, KD, and SeqKD), and it matches a fixed 8192-token rollout horizon within two points using a quarter of the training time (8.5 vs.\ 35.9 hours) and 71% fewer rollout tokens. We hope this recipe helps move structured pruning beyond marginal gains on perplexity and multiple-choice benchmarks, a step closer to deployment-ready generation quality.

排序依据与全文待核验点

  • 为什么相关： Proposes Short-to-Long On-Policy Distillation (ShortOPD) to recover structured pruned LLMs. Reuses the pre-compression model as a frozen teacher to guide the student on its own on-policy states, optimizing token-level supervision efficiency.
  • 全文待核验： Verify the exact heuristic or threshold mechanism used to detect ‘teacher-confirmed repetitive suffixes’ during on-policy rollouts.

4. Length Penalties Make Chain-of-Thought Less Monitorable

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 4/5
  • 潜在工程价值： 3/5
  • 作者： Bryce Little
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

探讨了长度惩罚强化学习对思维链（CoT）推理可监控性的负面影响。基于摘要，实验表明长度惩罚虽缩短了推理并保持了多选准确率，但显著削弱了思维链对偏见提示（biasing hints）显式提及的真实度，降低了检测偏见影响的概率；此判断仅基于摘要。

Abstract

Length-penalized reinforcement learning can shorten chain-of-thought reasoning while hiding an influence that drives the model’s answer. In our experiments, training with length penalties does not stop misleading hints from steering models, even though the models’ chains of thought mention the hint much less often. A token-accuracy evaluation would count these runs as successful because they use fewer reasoning tokens with little accuracy loss; it would miss whether the remaining trace still shows what drove the answer. We train Qwen3-4B and Qwen3-14B variants with different target chain lengths, then evaluate them with biasing-hint interventions on held-out MMLU-Pro-R and four transfer benchmarks. Compression sharply cuts reasoning tokens, preserves most multiple-choice accuracy, and leaves hint influence near baseline. At the strongest target, lower-bound faithfulness falls to 63.1% of baseline for Qwen3-14B and 69.4% for Qwen3-4B; the raw rate at which a monitor catches hint use falls from 69% to 49% and from 60% to 48%. To separate length from content, we randomly delete sentences from uncompressed baseline chains until the remaining text matches the compressed length. Even after this length matching, compressed chains disclose the hint 7-35 percentage points less often than baseline chains that we shorten at random, for both Qwen3 sizes and all five evaluation distributions. Compression therefore does more than shorten reasoning, preferentially removing the cues a monitor needs to see what influenced the answer. Together, these results reveal a compression-monitorability frontier in which cheaper reasoning can preserve answers while making the influences behind them harder to detect.

排序依据与全文待核验点

  • 为什么相关： Analyzes the effect of length-penalized reinforcement learning on Chain-of-Thought reasoning. It demonstrates a compression-monitorability frontier where penalizing length preserves accuracy but heavily degrades the monitorability and faithfulness of reasoning paths.
  • 全文待核验： Verify how ‘lower-bound faithfulness’ and ‘monitor catching rate’ are mathematically formulated and measured under bias-hint interventions.

5. OvisOCR2 Technical Report

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 3/5
  • 潜在工程价值： 3/5
  • 作者： Shiyin Lu, Yinglun Li, Yu Xia, Yuhui Chen, An-Yang Ji, Jun-Peng Jiang, Qing-Guo Chen, Jianshan Zhao, En Lin, Haijun Li, Cheng Qin, Zhao Xu, Weihua Luo
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

介绍了 0.8B 级端到端文档解析模型 OvisOCR2。基于摘要，训练流程结合了有监督微调、在 4B 分支上进行多组件奖励设计的强化学习（RL），以及向 0.8B 模型的同策略蒸馏，实现了高精度的端到端 Markdown 解析；此判断仅基于摘要。

Abstract

We introduce OvisOCR2, a 0.8B document parsing model. OvisOCR2 is designed as an end-to-end parser: given a document page image, it generates a Markdown representation in natural reading order, covering text, formulas, tables, and visual regions. We build a data engine that combines filtered real-document annotations with synthetic pages whose rendered images and Markdown targets are derived from the same HTML source. The training recipe includes supervised fine-tuning, reinforcement learning on a 4B branch with a multi-component reward design, on-policy distillation into the 0.8B model, and model fusion. On OmniDocBench v1.6, OvisOCR2 achieves a state-of-the-art overall score of 96.58, placing an end-to-end model at the top of this leaderboard previously dominated by pipeline methods and highlighting the potential of end-to-end document parsing. On PureDocBench, OvisOCR2 also achieves the highest Avg3 score of 75.06. Beyond these two public benchmarks, we evaluate OvisOCR2 on an in-house benchmark designed to cover a broader set of long-tail and challenging scenarios. OvisOCR2 obtains the best overall performance among the compared methods, providing further evidence of its generalization and robustness. OvisOCR2 is available at https://huggingface.co/ATH-MaaS/OvisOCR2.

排序依据与全文待核验点

  • 为什么相关： OvisOCR2 uses reinforcement learning on a 4B parameter branch with a multi-component reward design, followed by on-policy distillation into a smaller 0.8B end-to-end document parsing model, representing a niche RL/post-training pipeline for VLMs.
  • 全文待核验： Verify the mathematical formulation of the ‘multi-component reward design’ and the model fusion step.

Edit

继续完善这篇笔记

编辑本文AI 协助

关系图谱