My Obsidian Blog

HF Daily Papers 2026-07-22

11分钟阅读

HF Daily Papers 2026-07-22

范围：优先 RLHF / preference optimization / alignment 在 Diffusion 或生成式 Diffusion 中的应用；其次为 RLHF 在 VLM / LLM 中的工作。排除 VLA、机器人、具身智能、robot manipulation、coding / software-engineering agents、long-horizon agents 与 agentic-RL。

筛选说明

  • 本列表只基于 Hugging Face Daily Papers 可见元数据与摘要完成初筛和排序；不应将作者摘要中的主张视为已被独立验证的事实。
  • 中文 AI Summary 是摘要级判断，不等同于全文结论。
  • 相关论文数：4。
  • 全文精读候选数：3。

排序后的相关论文

1. ⭐ Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing

  • 方向： RLHF-Diffusion
  • 研究阅读价值： 4/5
  • 潜在工程价值： 5/5
  • 作者： Xinjie Zhang, Peng Zhang, Shicheng Zheng, Jinghao Guo, Zhaoyang Jia, Yifei Shen, Xun Guo, Yuxuan Luo, Jiahao Li, Wenxuan Xie, Fanyi Pu, Xiaoyi Zhang, Kaichen Zhang, Zongyu Guo, Tianci Bi, Dongnan Gui, Zhening Liu, Zimo Wen, Zihan Zheng, Senqiao Yang, Xiao Li, Jinglu Wang, Bin Li, Yan Lu
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： https://github.com/microsoft/Mage
  • 项目页： https://microsoft.github.io/Mage/

中文 AI Summary

基于摘要判断，该论文推出了 Mage-Flow 4B 级原生分辨率 Flow Matching Diffusion Transformer 架构，包含 base、RL 对齐（Diffusion-NFT）与 4-step Turbo 等变体。作者声称其 RL 对齐提升了提示词跟随、文字渲染与审美质量，且 Turbo 模型在单卡 A100 上达到亚秒级/1 秒级生成与编辑。

Abstract

Large-scale visual generators are increasingly capable but costly to train, fine-tune, and deploy. We introduce Mage-Flow, a compact 4B-scale generative stack for efficient text-to-image generation and instruction-based image editing. The stack is built from two co-designed components: Mage-VAE, a lightweight high-fidelity latent tokenizer, and a Native-Resolution Multimodal Diffusion Transformer trained with rectified flow matching. Mage-VAE uses one-step diffusion-style encoding and decoding with anchor-latent regularization, preserving the reconstruction quality of strong public VAEs while reducing tokenization cost by more than an order of magnitude. Together with native-resolution packing and stack-level CUDA kernel fusion, the stack supports flexible-resolution training and improves end-to-end training throughput by about 2.5times. Built on this foundation, we develop a complete model family with Base, RL-aligned, and Turbo variants for both generation and editing. Diffusion-NFT improves prompt following, text rendering, aesthetic quality, and editing fidelity, while few-step distillation with adversarial perceptual guidance produces 4-step Turbo models for low-latency inference. Despite its compact scale, Mage-Flow and Mage-Flow-Edit achieves competitive performance across standard generation and editing benchmarks. More importantly, the Turbo variants make high-resolution generation and editing practical for interactive use: at 1024^2 resolution on a single NVIDIA A100 GPU, Mage-Flow-Turbo generates an image in 0.59s, and Mage-Flow-Edit-Turbo edits an image in 1.02s, while maintaining a small memory footprint. These results show that careful tokenizer—backbone—system co-design can deliver strong high-resolution generation and editing within an efficient 4B model family.

排序依据与全文待核验点

  • 为什么相关： Directly presents Diffusion-NFT (a post-training RL-alignment variant for rectified flow matching DiTs) alongside distilled turbo variants, delivering concrete engineering value and insights into RL alignment for diffusion image generation and editing.
  • 全文待核验： Verify the exact policy gradient / reward formulation used in Diffusion-NFT and its computational overhead during RL fine-tuning.

2. ⭐ SciForma: Structure-Faithful Generation of Scientific Diagrams

  • 方向： RLHF-Diffusion
  • 研究阅读价值： 4/5
  • 潜在工程价值： 4/5
  • 作者： Yuxuan Luo, Peng Zhang, Xinjie Zhang, Xun Guo, Zhouhui Lian, Yan Lu
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

基于摘要判断，针对单标量 reward 在多维度结构约束下难以有效对齐的问题，本文提出了 M-DPO（多维结合偏好优化），将图表生成分解为组件、箭头、文本三个结构轴并自适应引导梯度。作者声称该 Post-training 方法显著超越现有开源模型及 GPT-Image-1.5。

Abstract

Structural fidelity is essential to scientific methodology diagrams. To communicate research logic, these diagrams must faithfully render components, directional relations, and textual annotations. Since a single error, such as a reversed arrow or an unreadable equation, can invalidate the entire figure, structural fidelity is inherently conjunctive: correctness on one axis cannot compensate for failure on another. Current open-source models fail to satisfy this criterion. Supervised fine-tuning (SFT) learns plausible layouts but cannot reliably ensure structural correctness, while scalar reward-based post-training obscures which structural dimension has failed. To address this, we introduce SciForma, a framework for the structure faithful generation of scientific methodology diagrams. Specifically, SciForma decomposes diagram quality into three structural axes: Component, Arrow, and Text, guided by a structural inventory. Built on this foundation, we curate SciFormaData-700K for structured training and SciFormaBench-2K for logic-verified evaluation. To close the gap left by SFT, we develop Multi-Dimensional Conjunctive Preference Optimization (M-DPO), which enforces simultaneous correctness across all axes and adaptively routes gradients to the most deficient dimension in post-training. The same structural inventory also enables iterative editing at inference time to correct residual errors. This combination allows SciForma-9B to exceed all open-source baselines and GPT-Image-1.5 on both SciFormaBench-2K and AIBench, bringing open scientific diagram generation close to proprietary-level structural fidelity. Our code and data will be available at: https://github.com/microsoft/SciForma.

排序依据与全文待核验点

  • 为什么相关： Introduces Multi-Dimensional Conjunctive Preference Optimization (M-DPO) for generative diagram models, addressing the failure modes of standard scalar reward RLHF in multi-axis preference alignment.
  • 全文待核验： Verify how M-DPO formulates and routes multi-dimensional DPO losses across different structural reward signals.

3. ⭐ Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning

中文 AI Summary

基于摘要判断，本文分析了异步 RL 中策略滞后导致的策略漂移与训练不稳定问题，提出了陈旧度自适应信任域（SAT）方法，通过基于陈旧度的核缩放动态收缩 PPO 剪切区间。作者声称在 Qwen3-30B 上结合 SGLang 与 Megatron 验证了其能有效稳定高 lag 下的训练。

Abstract

Asynchronous reinforcement learning improves throughput by decoupling rollout generation from optimization, but staleness is an inevitable byproduct compounded by policy lag, engine delays, and mixture-of-experts routing. From a trust-region perspective, this mismatch is critical: training-inference divergence governs approximation error in finite-horizon bounds, whereas PPO clipping only gates sampled outward updates, acting as a sampled surrogate rather than a full-policy constraint. As a result, high-staleness updates remain weakly controlled in the asynchronous regime where stale rollouts matter most. We introduce the Staleness-Adaptive Trust Region (SAT), which uses the detached sampled log-ratio as a practical staleness proxy, identifies high-mismatch tails within each batch via staleness-based kernel scaling, and contracts only the sign-selected endpoint of the nominal PPO interval. This preserves baseline behavior on ordinary tokens while enforcing more conservative updates on newly intercepted outward bands. We prove local interval containment and pointwise pessimism relative to PPO, showing how the adaptive rule reshapes update geometry under heterogeneous staleness. We evaluate SAT in a decoupled asynchronous RL setup built on Qwen3-30B-A3B-Base, using SGLang as the inference engine and Megatron for training. In this setting, SAT-GSPO w/ R3 achieves the best observed AIME24 avg@8, reaching 35.83 at lag 1 and 34.79 at lag 8, while SAT-GSPO reaches 34.17 at lag 1. Adaptive clipping and routing replay act as complementary stabilizers targeting mismatch tails and routing inconsistency, respectively. Overall, aligning clip intervals with staleness heterogeneity effectively stabilizes asynchronous RL.

排序依据与全文待核验点

  • 为什么相关： Provides a novel trust-region technique (SAT) specifically designed for asynchronous RL (PPO/GSPO) under policy lag/staleness, demonstrated on Qwen3-30B-A3B-Base with Megatron + SGLang.
  • 全文待核验： Verify the mathematical derivation of SAT’s pessimistic update interval and its empirical stability across varying lags.

4. H^2SD: Hybrid Hindsight Self-Distillation

  • 方向： RLHF-VLM/LLM
  • 研究阅读价值： 4/5
  • 潜在工程价值： 3/5
  • 作者： Qiye Cai, Yichuan Ma, Linyang Li, Peiji Li, Yongkang Chen, Qipeng Guo, Yicheng Zou, Tao Gui, Xiaocheng Feng, Bing Qin
  • Hugging Face： 论文页
  • arXiv： 摘要页 · PDF
  • 代码： 未提供
  • 项目页： 未提供

中文 AI Summary

基于摘要判断，本文针对 RLVR 中稀疏标量奖励缺乏 Token 级信用分配的问题，提出了混合事后自我蒸馏（H^2SD）框架，对成功轨迹用 Teacher 概率调制更新幅度，对失败轨迹引入 Hint 引导反向 KL 蒸馏。作者声称在推理 Benchmark 上优于标准 RLVR 和 OPSD 方法。

Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation. However, most RLVR methods assign a scalar outcome reward to an entire trajectory, resulting in sparse supervision and limited token-level credit assignment. On-policy distillation (OPD) provides denser supervision by distilling token-level distributions from a stronger teacher model, but requires an additional teacher and typically assumes a shared vocabulary. On-policy self-distillation (OPSD) removes this dependency by conditioning the same model on privileged information to construct a teacher policy. However, directly matching the teacher distribution may cause information leakage and unstable optimization. RLSD avoids direct matching by using the teacher signal only to modulate update magnitudes, but it cannot provide an explicit correction direction when the sampled reasoning fails. To address this tradeoff, we introduce H^{2}SD, a hybrid hindsight self distillation framework that uses the teacher differently according to trajectory correctness. For successful trajectories, the teacher receives the student response confirmed as correct together with a rephrasing instruction, and its probabilities on the original response tokens are used to modulate update magnitudes without changing the direction determined by the reward. For failed trajectories, we condition the teacher on a reference hint containing key reasoning steps and a verified answer, and minimize the reverse KL divergence from the student to the teacher. Experiments on multiple challenging reasoning benchmarks show that H^2SD consistently outperforms representative RLVR, OPSD, and RLSD baselines while maintaining stable optimization and favorable generation efficiency.

排序依据与全文待核验点

  • 为什么相关： Proposes H^2SD, combining outcome rewards (RLVR) with hindsight self-distillation (OPSD/RLSD) on success and failure trajectories for reasoning LLMs.
  • 全文待核验： Verify how the hindsight hint conditioning is constructed without introducing label leak during optimization.

筛选备注

4 scoped papers retained (2 RLHF-Diffusion, 2 RLHF-LLM). Non-scoped agentic, embodied/robotics, document-parsing, and non-RL papers were excluded per instructions.

Edit

继续完善这篇笔记

编辑本文AI 协助

关系图谱