Nvidia Dominates Gen AI Benchmarks, Outperforming Two Rival AI Chips

Nvidia's general-purpose GPU chips have once again dominated one of the most widely recognized benchmarks for assessing chip performance in artificial intelligence, this time focusing on generative AI applications such as large language models (LLMs). The competition was relatively one-sided.

Systems from SuperMicro, Hewlett Packard Enterprise, Lenovo, and other companies, each equipped with up to eight Nvidia chips, secured the majority of the top spots in the MLPerf benchmark test organized by the MLCommons, an industry consortium. This test, which measures the speed at which machines can produce tokens, process queries, or output data samples—known as AI inference—was the fifth in a series of prediction-making benchmarks that have been conducted over the years.

This latest iteration of the MLPerf benchmark included new tests tailored to common generative AI tasks. One test evaluated the performance of chips on Meta's open-source LLM, Llama 3.1 405b, a substantial model widely used in the field. Another test introduced an interactive version of Meta's smaller Llama 2 70b, designed to simulate chatbot interactions where response time is crucial. This test specifically measures how quickly the system can generate the first token of output, reflecting the need for rapid responses to user prompts.

A third new test assessed the speed of processing graph neural networks, which handle complex relationships among entities, like those in a social network. These networks have become increasingly vital in generative AI, exemplified by Google's DeepMind unit's use of graph nets in its AlphaFold 2 model, which made significant strides in protein-folding predictions in 2021. Additionally, a fourth test gauged the speed at which LiDAR sensing data can be compiled into an automobile's road map, using a custom neural net developed by MLCommons from existing open-source technologies.

The MLPerf competition involves computers built by Lenovo, HPE, and others, adhering to stringent requirements for the accuracy of neural net outputs. Each system reports its top speed in producing output per second, with some benchmarks measuring average latency, or the time taken for a response to come back from the server.

Nvidia's GPUs excelled in nearly all tests within the closed division, where the software setup rules are the strictest.

However, AMD, with its MI300X GPU, claimed the top score in two Llama 2 70b tests, achieving 103,182 tokens per second, which was significantly better than Nvidia's newer Blackwell GPU. This winning AMD system was assembled by MangoBoost, a startup specializing in plug-in cards that enhance data transfer between GPU racks, and LLMboost, their software designed to improve generative AI performance.

Nvidia contested the comparison of AMD's results to their Blackwell scores, pointing out the need to adjust for the number of chips and computer "nodes" used in each system. Dave Salvator, Nvidia's director of accelerated computing products, emphasized in an email to ZDNET:

"MangoBoost's results do not reflect an accurate performance comparison against NVIDIA's results. AMD's testing applied 4X the number of GPUs – 32 MI300X GPUs – against 8 NVIDIA B200s, yet still only achieved a 3.83% higher result than the NVIDIA submission. NVIDIA's 8x B200 submission actually outperformed MangoBoost's x32 AMD MI300X GPUs in the Llama 2 70B server submission."

Google also entered the competition, showcasing its Trillium chip, the sixth iteration of its in-house Tensor Processing Unit (TPU). However, it significantly lagged behind Nvidia's Blackwell in a test measuring query response speed for the Stable Diffusion image-generation test.

The latest MLPerf benchmarks saw fewer competitors challenging Nvidia compared to previous rounds. Notably absent were submissions from Intel's Habana unit and Qualcomm, both of which had participated in past years.

Despite this, Intel had reason to celebrate. In the datacenter closed division, Intel's Xeon microprocessor powered seven of the top 11 systems, outperforming AMD's EPYC server microprocessor, which secured only three victories. This marks an improvement for Intel compared to previous years.

The 11th top-performing system, tasked with processing Meta's massive Llama 3.1 405b, was built by Nvidia without using an Intel or AMD microprocessor. Instead, it utilized the integrated Grace-Blackwell 200 chip, combining Nvidia's Blackwell GPU with its own Grace microprocessor in a single package.