Andreessen-backed Vals Targets AI Benchmarking Leadership

Benchmarking has become the industry standard for how AI companies validate their models’ capabilities, leverage favorable metrics to differentiate from competitors, and market their superiority. In short, strong benchmark results almost always translate into positive public relations.

Unfortunately, companies have also discovered ways to game legacy benchmarking systems — many of which are outdated and not designed to assess the capabilities of modern AI models.

Vals, a startup founded in 2024, aims to rectify this flawed system. In less than two years, the company has established itself as a significant player in the tech sector, securing a seed round led by 8VC and Bloomberg Beta last year. More recently, following a period of rapid expansion, it raised $40 million in a Series A round led by Andreessen Horowitz.

Rayan Krishnan, the company’s 25-year-old co-founder, previously interned at Palantir and, as an undergraduate at Stanford, worked for Microsoft and the university’s renowned artificial intelligence lab. Krishnan explains that Vals was born from his observations that benchmarking was failing to keep pace with the rapid advancements in the industry it was meant to measure.

“We were seeing a bunch of new, very capable models come to market quickly, and the academic benchmarks [were] not keeping up with that frontier advance,” Krishnan notes. With AI being integrated into every aspect of society, benchmarks should serve to verify that models can actually perform what companies claim they can do, Krishnan adds.

Last week, the young founder gave me a tour of his company’s two-floor office on San Francisco’s Folsom Street — an old brick building that, a century ago, housed a large brewery. Instead of producing beer, this historical structure now hosts various startups aiming to shape the future of the tech industry.

“Historically, I think evaluation has been done to evaluate intelligence in a very abstract way,” Krishnan explains. “Like, do models know enough information to be able to take a bar exam type test?”

Here, Vals seeks to differentiate itself. While many benchmarking systems offer publicly available tests (which can allow companies to train their models against those tests, effectively cheating on their exams), Vals does not publicly disclose its specific test materials. Instead of measuring an AI model’s general knowledge, Vals also evaluates models on their ability to complete complex tasks associated with specific industries such as law, finance, and coding.

“What we’re doing is actually looking at what are the real impacts of the models,” Krishnan says. “Can they do work that produces a product of the same quality as a human within every domain?”

The goal is to check not just for positive outcomes but also for negative ones, he explains. The hope is to analyze how, “if these models ran wild in the world, what the negative implications would be.”

The capabilities Vals measures are expanding. In addition to more traditional industries, the startup continues to explore unique domains. “We have a benchmark on recursive self improvement. We’re doing some work in mental health, cybersecurity, biosecurity, and even law of armed conflict to models to understand how to apply the Geneva Convention,” Krishnan shares.

Companies pay Vals to test their models, a concept that may seem counterintuitive. Why would a company pay to discover its model isn’t performing well? However, having an effective measurement helps companies troubleshoot and improve over time. Krishnan compares their revenue model to how a student might pay the College Board to take the SAT.

In turn, these evaluations are becoming key decision-making factors for companies looking to acquire new AI models.

The startup recently revealed that its revenue is currently eight times what it was last year. Its staff is also expanding. Vals, which started the year with only eight people, has already tripled to a team of 25. Krishnan stated that as the startup grows, the plan is to relocate to a significantly larger office and bring on an additional 10 to 15 people. The company also recently launched a program focused on providing model evaluations to federal agencies.

Krishnan views his company’s benchmarking system as the future of how AI companies think about growing their businesses and establishing public trust.

“AI companies are starting to go public. SpaceX went public. Anthropic is slated for later this year. I suspect OpenAI will be public soon. I think as AI models become a core part of the economy and are diffused more broadly, the types of benchmarks and evaluations that we do are going to drive their usage and be a central part of how these companies submit public filings or talk about the prospective investments they’re going to make in AI,” he said.