Home
OpenAI Slams AI Benchmark: Nearly a Third of Questions Flawed, Pass Rate Soared to 80% in 8 Months
OpenAI has publicly challenged the industry's leading benchmark, SWE-Bench Pro, in a blog post, claiming that roughly 30% of its 731 public test tasks contain evaluation flaws. Developed by Scale AI, SWE-Bench Pro is designed to assess the coding abilities of large language models and AI agents. Because it closely mirrors real-world enterprise development and enforces strict anti-cheating measures, it has become a widely trusted benchmark in AI software engineering.
OpenAI highlights a key signal in the blog post: the pass rate for top-tier models on this benchmark jumped from 23.3% to 80.3% in just eight months. Such rapid progress seems suspicious, and OpenAI argues that the benchmark can no longer accurately measure a model's real-world software development skills. The issue likely stems from flaws in the evaluation itself, not a genuine leap in model capability.
Two review paths cross-validated reveal that nearly 30% of tasks are deemed "unqualified."
To verify this, OpenAI launched two parallel review processes. The data-point analysis uncovered 200 failing tasks, or 27.4% of the 731 public tasks. Meanwhile, manual annotation flagged 249 failing tasks, or 34.1%. Cross-referencing both methods, OpenAI estimates that roughly 30% of SWE-Bench Pro tasks contain defects, falling into four categories: overly strict tests, inadequate prompts, narrow test coverage, and misleading prompts.
OpenAI also shared a typical example: one task asked for a single space at the start of a line when converting content to Markdown, but the hidden test expected two spaces. As a result, even if the model followed the stated requirement, it would be marked incorrect. This mismatch between explicit instructions and hidden requirements directly skews the evaluation of a model's true ability, and explains the unreasonable surge in pass rates.
Withdrawal of adoption recommendation and call for rebuilding the AI evaluation system
Based on this analysis, OpenAI has officially retracted its earlier recommendation to use SWE-Bench Pro. The company believes that future benchmarks should be created by experienced software developers specifically for AI evaluation, rather than repurposing test logic designed for human developers. If an industry benchmark has nearly 30% of its tasks flawed, the entire AI evaluation system's credibility comes into question. Shifting the focus from score-chasing competitions to genuine engineering capability assessments may be the next critical step in AI software engineering evaluation.
Related article
Baidu Q2 Revenue Hits 31.3 Billion Yuan as AI Business Surpasses Half for Second Straight Quarter
Baidu’s second-quarter 2026 financial results show total revenue of 31.3 billion yuan, with core business revenue at 25.2 billion yuan. AI-driven revenue now represents 50% of total business income, surpassing the halfway mark for two consecutive qua
OpenAI overtakes Anthropic in enterprise market share as AI spending grows
Recent figures from Ramp, a corporate credit card and expense management platform, reveal that OpenAI has reclaimed over 40% of the U.S. enterprise AI market, overtaking Anthropic. In May, Anthropic held a 41% share against OpenAI’s 39%; by July, Ant
Epson Unveils AX6 Cobot Featuring Compact Design and No-Code Programming
The AX6 force- and power-limited robot arm is designed to be lightweight and easy to use. Source: EpsonEpson Robots this week expanded its six-axis robot line with the AX6 collaborative robot. It said the new cobot is is designed with power- and forc
Related Special Topic Recommendations
Prompt
AI Prompt Testing Tools for Better Output Quality
2026 Latest Best Top-rated AI Prompt Testing Tools for Better Output Quality! XIX.AI has curated a powerful, game-changing collection of highly efficient tools that undergo rigorous real-world tests to ensure consistent top-tier results. You’ll find detailed free vs paid comparison insights, comprehensive rankings, and expert guidance to help you identify the must-try solutions that boost your workflow significantly. Explore now to unlock your AI edge!
11 tools
xix.ai
Business
AI Customer Insight Tools for Product Positioning
2026 Latest Best Top-rated AI Customer Insight Tools for Product Positioning! XIX.AI has curated a powerful, game-changing collection that goes through real-world tests and weekly updated rankings. It helps you uncover customer needs quickly to boost writing efficiency, create optimal SEO content, and craft perfect social media messages without wasting time. Explore now to Discover your perfect tool and Unlock your AI edge.
10 tools
xix.ai
Meeting Assistant
Zoom AI Note Takers for Client Calls, Internal Syncs, and Decision Tracking
2026 Latest Best Zoom AI note takers ranked by performance and usability. XIX.AI has curated a top-rated list of powerful tools that help you take accurate notes during client calls, streamline internal team syncs, and keep track of key decisions effortlessly. Get a free vs paid comparison along with real-world tests to find the must-try solution that boosts your productivity significantly. Explore now!
12 tools
xix.ai
Data Analysis
AI Spreadsheet Analysis Tools: Ask Questions and Get Charts Instantly
2026 Latest Best Top-rated AI Spreadsheet Analysis Tools are here on XIX.AI! This curated list features powerful, game-changing solutions that allow you to ask questions directly in spreadsheets and get accurate charts instantly, boosting your productivity significantly. We conduct real-world tests and provide a free vs paid comparison along with weekly updated rankings to help you find the perfect tool for streamlining data tasks. Explore now to Unlock your AI edge!
19 tools
xix.ai
design
AI Wireframe Tools for Product Designers
2026 Latest Best Top-Rated AI Wireframe Tools for Product Designers! XIX.AI has curated a powerful, game-changing collection of must-try tools that go through rigorous real-world tests. You’ll find detailed weekly updated rankings, free vs paid comparison insights, and all the key info you need to choose the perfect tool to boost your design efficiency and unlock your AI edge. Explore now!
8 tools
xix.ai
SEO
Best AI SEO Tools for Keyword Research
2026 Latest Best Top-Rated AI SEO Tools for Keyword Research are here on XIX.AI! This curated list features powerful, game-changing tools that go through rigorous real-world tests to deliver accurate data. Get a free vs paid comparison, see detailed rankings, and discover must-try options to boost your content strategy and improve search visibility. Explore now to find your perfect tool!
8 tools
xix.ai
Comments (0)
0/500
OpenAI has publicly challenged the industry's leading benchmark, SWE-Bench Pro, in a blog post, claiming that roughly 30% of its 731 public test tasks contain evaluation flaws. Developed by Scale AI, SWE-Bench Pro is designed to assess the coding abilities of large language models and AI agents. Because it closely mirrors real-world enterprise development and enforces strict anti-cheating measures, it has become a widely trusted benchmark in AI software engineering.
OpenAI highlights a key signal in the blog post: the pass rate for top-tier models on this benchmark jumped from 23.3% to 80.3% in just eight months. Such rapid progress seems suspicious, and OpenAI argues that the benchmark can no longer accurately measure a model's real-world software development skills. The issue likely stems from flaws in the evaluation itself, not a genuine leap in model capability.
Two review paths cross-validated reveal that nearly 30% of tasks are deemed "unqualified."
To verify this, OpenAI launched two parallel review processes. The data-point analysis uncovered 200 failing tasks, or 27.4% of the 731 public tasks. Meanwhile, manual annotation flagged 249 failing tasks, or 34.1%. Cross-referencing both methods, OpenAI estimates that roughly 30% of SWE-Bench Pro tasks contain defects, falling into four categories: overly strict tests, inadequate prompts, narrow test coverage, and misleading prompts.
OpenAI also shared a typical example: one task asked for a single space at the start of a line when converting content to Markdown, but the hidden test expected two spaces. As a result, even if the model followed the stated requirement, it would be marked incorrect. This mismatch between explicit instructions and hidden requirements directly skews the evaluation of a model's true ability, and explains the unreasonable surge in pass rates.
Withdrawal of adoption recommendation and call for rebuilding the AI evaluation system
Based on this analysis, OpenAI has officially retracted its earlier recommendation to use SWE-Bench Pro. The company believes that future benchmarks should be created by experienced software developers specifically for AI evaluation, rather than repurposing test logic designed for human developers. If an industry benchmark has nearly 30% of its tasks flawed, the entire AI evaluation system's credibility comes into question. Shifting the focus from score-chasing competitions to genuine engineering capability assessments may be the next critical step in AI software engineering evaluation.
Baidu Q2 Revenue Hits 31.3 Billion Yuan as AI Business Surpasses Half for Second Straight Quarter
Baidu’s second-quarter 2026 financial results show total revenue of 31.3 billion yuan, with core business revenue at 25.2 billion yuan. AI-driven revenue now represents 50% of total business income, surpassing the halfway mark for two consecutive qua
OpenAI overtakes Anthropic in enterprise market share as AI spending grows
Recent figures from Ramp, a corporate credit card and expense management platform, reveal that OpenAI has reclaimed over 40% of the U.S. enterprise AI market, overtaking Anthropic. In May, Anthropic held a 41% share against OpenAI’s 39%; by July, Ant
Epson Unveils AX6 Cobot Featuring Compact Design and No-Code Programming
The AX6 force- and power-limited robot arm is designed to be lightweight and easy to use. Source: EpsonEpson Robots this week expanded its six-axis robot line with the AX6 collaborative robot. It said the new cobot is is designed with power- and forc
2026 Latest Best Top-rated AI Prompt Testing Tools for Better Output Quality! XIX.AI has curated a powerful, game-changing collection of highly efficient tools that undergo rigorous real-world tests to ensure consistent top-tier results. You’ll find detailed free vs paid comparison insights, comprehensive rankings, and expert guidance to help you identify the must-try solutions that boost your workflow significantly. Explore now to unlock your AI edge!
11 tools
xix.ai
2026 Latest Best Top-rated AI Customer Insight Tools for Product Positioning! XIX.AI has curated a powerful, game-changing collection that goes through real-world tests and weekly updated rankings. It helps you uncover customer needs quickly to boost writing efficiency, create optimal SEO content, and craft perfect social media messages without wasting time. Explore now to Discover your perfect tool and Unlock your AI edge.
10 tools
xix.ai
2026 Latest Best Zoom AI note takers ranked by performance and usability. XIX.AI has curated a top-rated list of powerful tools that help you take accurate notes during client calls, streamline internal team syncs, and keep track of key decisions effortlessly. Get a free vs paid comparison along with real-world tests to find the must-try solution that boosts your productivity significantly. Explore now!
12 tools
xix.ai
2026 Latest Best Top-rated AI Spreadsheet Analysis Tools are here on XIX.AI! This curated list features powerful, game-changing solutions that allow you to ask questions directly in spreadsheets and get accurate charts instantly, boosting your productivity significantly. We conduct real-world tests and provide a free vs paid comparison along with weekly updated rankings to help you find the perfect tool for streamlining data tasks. Explore now to Unlock your AI edge!
19 tools
xix.ai
2026 Latest Best Top-Rated AI Wireframe Tools for Product Designers! XIX.AI has curated a powerful, game-changing collection of must-try tools that go through rigorous real-world tests. You’ll find detailed weekly updated rankings, free vs paid comparison insights, and all the key info you need to choose the perfect tool to boost your design efficiency and unlock your AI edge. Explore now!
8 tools
xix.ai
2026 Latest Best Top-Rated AI SEO Tools for Keyword Research are here on XIX.AI! This curated list features powerful, game-changing tools that go through rigorous real-world tests to deliver accurate data. Get a free vs paid comparison, see detailed rankings, and discover must-try options to boost your content strategy and improve search visibility. Explore now to find your perfect tool!
8 tools
xix.ai
