HF Daily Papers 2026-07-15
范围：优先 RLHF / preference optimization / alignment 在 Diffusion 或生成式 Diffusion 中的应用；其次为 RLHF 在 VLM / LLM 中的工作。排除 VLA、机器人、具身智能、robot manipulation、coding / software-engineering agents、long-horizon agents 与 agentic-RL。
筛选说明
- 本列表只基于 Hugging Face Daily Papers 可见元数据与摘要完成初筛和排序；不应将作者摘要中的主张视为已被独立验证的事实。
- 中文 AI Summary 是摘要级判断，不等同于全文结论。
- 相关论文数：2。
- 全文精读候选数：2。
排序后的相关论文
1. ⭐ Read It Back: Pretrained MLLMs Are Zero-Shot Reward Models for Text-to-Image Generation
- 方向： RLHF-Diffusion
- 研究阅读价值： 5/5
- 潜在工程价值： 5/5
- 作者： Runhui Huang, Qihui Zhang, Zhe Liu, Yu Gao, Jie Wu, Hengshuang Zhao
- Hugging Face： 论文页
- arXiv： 摘要页 · PDF
- 代码： 未提供
- 项目页： https://huangrh99.github.io/SpectraReward/
中文 AI Summary
摘要提出 SpectraReward 框架，将预训练多模态大模型转化为免训练的图像生成强化学习奖励模型，通过计算图像反推 Prompt 的平均对数似然评估对齐度。此外，Self-SpectraReward 实现模型自生成与自评估的闭环改进，在多个扩散模型和 RL 算法上取得了显著表现。此判断仅基于摘要。
Abstract
In this paper, we propose SpectraReward, a training-free reward function that turns pretrained MLLMs into off-the-shelf reward models for image-generation reinforcement learning. Instead of asking the MLLM to judge a generated image or answer decomposed verification questions, SpectraReward measures how well the original prompt can be recovered from the generated image through a single image-conditioned, teacher-forced forward pass. We use the average image-conditioned prompt log-likelihood as the reward, directly reusing the MLLM’s pretrained image-text alignment ability without preference labels, reward-model fine-tuning. We further introduce Self-SpectraReward, a special case for unified multimodal models where the policy’s own understanding branch serves as the reward model for its generation branch, forming a closed-loop self-improving framework without external reward models or external knowledge. Extensive experiments validate SpectraReward through a broad image-generation RL study covering two diffusion models, three RL algorithms, nine reward MLLM backbones from four MLLM families spanning 4B to 235B parameters, and five out-of-distribution text-to-image benchmarks. Results show that both SpectraReward and Self-SpectraReward significantly and consistently improve generation performance and outperform prior MLLM-derived reward training methods. Further analysis reveals that larger reward MLLMs are not always better, while Self-SpectraReward can match or surpass much larger external reward models, suggesting that reward-policy alignment is a key factor for effective image-generation RL. Project Page: https://huangrh99.github.io/SpectraReward/
排序依据与全文待核验点
- 为什么相关： Directly proposes a training-free reward function (SpectraReward and Self-SpectraReward) using pretrained MLLMs as off-the-shelf reward models for reinforcement learning in text-to-image diffusion. This directly targets preference optimization and alignment in generative diffusion systems.
- 全文待核验： Verify the mathematical formulation of the average image-conditioned prompt log-likelihood and how backpropagation or policy gradient is calculated relative to the policy network under both two-stage and self-improving settings.
2. ⭐ Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering
- 方向： RLHF-VLM/LLM
- 研究阅读价值： 4/5
- 潜在工程价值： 4/5
- 作者： Yanjie Zhang, Yafei Li, Rui Sheng, Zixin Chen, Yanna Lin, Huamin Qu, Lei Chen, Yushi Sun
- Hugging Face： 论文页
- arXiv： 摘要页 · PDF
- 代码： 未提供
- 项目页： 未提供
中文 AI Summary
摘要介绍一种名为 ChartCynics 的双路径代理框架，用于解决 VLM 面对欺骗性图表时的虚假视觉表征问题。该框架解耦感知与验证，并引入通过 Oracle-Informed SFT 和 Deception-Aware GRPO 优化对齐的代理总结器，在图表问答任务中表现优异。此判断仅基于摘要。
Abstract
Despite the success of Vision-Language Models (VLMs), misleading charts remain a significant challenge due to their deceptive visual structures and distorted data representations. We present ChartCynics, an agentic dual-path framework designed to unmask visual deception via a “skeptical” reasoning paradigm. Unlike holistic models, ChartCynics decouples perception from verification: a Diagnostic Vision Path captures structural anomalies (e.g., inverted axes) through strategic ROI cropping, while an OCR-Driven Data Path ensures numerical grounding. To resolve cross-modal conflicts, we introduce an Agentic Summarizer optimized via a two-stage protocol: Oracle-Informed SFT for reasoning distillation and Deception-Aware GRPO for adversarial alignment. This pipeline effectively penalizes visual traps and enforces logical consistency. Evaluations on two benchmarks show that ChartCynics achieves 74.43% and 64.55% accuracy, providing an absolute performance boost of ~29% over the Qwen3-VL-8B backbone, outperforming state-of-the-art proprietary models. Our results demonstrate that specialized agentic workflows can grant smaller open-source models superior robustness, establishing a new foundation for trustworthy chart interpretation.
排序依据与全文待核验点
- 为什么相关： Uses Deception-Aware GRPO (Group Relative Policy Optimization) for adversarial alignment of an Agentic Summarizer in a VLM-based chart question-answering framework.
- 全文待核验： Verify the exact reward formulation and group comparison design used in Deception-Aware GRPO, and how adversarial data is sampled during training.
筛选备注
Focused strictly on RLHF/preference alignment for generative diffusion and VLMs. Excluded coding, robotics, general RL paradigms, and search-guided generation works that lack explicit reinforcement learning / preference alignment contributions.