Optimizing AI Model Selection for Real-World Performance

Businesses must ensure their application-driving AI models perform effectively in real-world scenarios. Predicting these scenarios can be challenging, complicating evaluations. The updated RewardBench 2 benchmark offers organizations clearer insights into a model’s practical performance.

The Allen Institute for AI (Ai2) introduced RewardBench 2, an enhanced version of its RewardBench benchmark, designed to provide a comprehensive assessment of model performance and alignment with enterprise objectives.

Ai2 developed RewardBench with classification tasks that evaluate correlations via inference-time compute and downstream training. RewardBench focuses on reward models (RMs), which judge large language model outputs by assigning scores or “rewards” to guide reinforcement learning with human feedback (RHLF).

RewardBench 2 is here! We took a long time to learn from our first reward model evaluation tool to make one that is substantially harder and more correlated with both downstream RLHF and inference-time scaling. pic.twitter.com/NGetvNrOQV

— Ai2 (@allen_ai) June 2, 2025

Nathan Lambert, a senior research scientist at Ai2, told VentureBeat that the original RewardBench functioned well initially, but evolving model environments demanded updated benchmarks.

“As reward models grew more sophisticated and use cases more complex, we saw, alongside the community, that the first version didn’t fully address real-world human preference complexities,” he explained.

Lambert noted that RewardBench 2 improves evaluation scope and depth, incorporating diverse, challenging prompts and refined methods to better reflect human judgment of AI outputs. It features new human prompts, a tougher scoring system, and additional domains.

Leveraging Evaluations for Model Assessment

Reward models evaluate model performance, but alignment with company values is critical. Misaligned RMs can amplify issues like hallucinations, reduce generalization, or overly favor harmful responses during fine-tuning and reinforcement learning.

RewardBench 2 spans six domains: factuality, precise instruction adherence, math, safety, focus, and ties.

“Enterprises can use RewardBench 2 in two ways based on their needs. For RLHF, they should integrate best practices and datasets from top models into their pipelines, as reward models require on-policy training. For inference-time scaling or data filtering, RewardBench 2 helps select the best model for their domain with correlated performance,” Lambert said.

Lambert emphasized that benchmarks like RewardBench allow users to assess models based on priorities most relevant to them, rather than a generic score. He noted that performance is subjective, heavily tied to user context and goals, with human preferences often highly nuanced.

Ai2 launched the original RewardBench in March 2024, calling it the first reward model benchmark and leaderboard. Since then, new methods like Meta’s FAIR reWordBench and DeepSeek’s Self-Principled Critique Tuning have emerged for smarter, scalable RMs.

Super excited that our second reward model evaluation is out. It's substantially harder, much cleaner, and well correlated with downstream PPO/BoN sampling.

Happy hillclimbing!

Huge congrats to @saumyamalik44 who lead the project with a total commitment to excellence. https://t.co/c0b6rHTXY5

— Nathan Lambert (@natolambert) June 2, 2025

Model Performance Insights

With RewardBench 2, Ai2 tested both existing and newly trained models, including variants of Gemini, Claude, GPT-4.1, and Llama-3.1, alongside datasets and models like Qwen, Skywork, and Tulu.

Findings showed larger reward models excel due to stronger base models. Llama-3.1 Instruct variants topped the benchmark, with Skywork data aiding focus and safety, and Tulu performing well in factuality.

Ai2 noted that while RewardBench 2 advances multi-domain, accuracy-focused evaluation for reward models, it should primarily guide enterprises in selecting models best suited to their specific needs.