Gimlet Labs’ Elegant Approach to Solving AI Inference Bottlenecks

Stanford adjunct professor and successful startup founder Zain Asgar has raised an $80 million Series A for a company that tackles the AI inference bottleneck in a notably smart way. Menlo Ventures led the round.

The startup, Gimlet Labs, says it has built the first and only “multi-silicon inference cloud”—software that lets an AI workload run simultaneously across different types of hardware. It can split tasks among traditional CPUs, AI-tuned GPUs, and high-memory systems alike.

“We basically run across whatever different hardware is available,” Asgar told TechCrunch.

“A single agent may chain together multiple steps, and each requires different hardware: Inference is compute-bound; decode is memory-bound; and tool calls are network-bound,” writes lead investor Menlo’s Tim Tully in a blog post about the funding.

No chip yet does it all, but as new hardware rolls out and aging GPUs get redeployed, “the multi-silicon fleet is ready—it’s just missing the software layer to make it work.” That’s what Tully believes Gimlet Labs offers.

If the current trend of deploying more compute continues, McKinsey estimates data center spending could reach nearly $7 trillion by 2030. Asgar says apps today use existing hardware only “somewhere between 15 to 30 percent” of the time.

“Another way to think about this: you’re wasting hundreds of billions of dollars because you’re just leaving idle resources,” he said. “Our goal was basically to try to figure out how you can get AI workloads to be 10x more efficient than ever, today.”

So he and his co-founders—Michelle Nguyen, Omid Azizi, and Natalie Serrino—set out to build orchestration software that slices up agentic workloads so they can run simultaneously across all kinds of hardware.

Gimlet Labs says it reliably speeds up AI inference by 3x to 10x for the same cost and power. Gimlet claims it can even split the underlying model so it runs across different architectures, using the best chip for each portion.

The company has already partnered with chip makers NVIDIA, AMD, Intel, ARM, Cerebras, and d-Matrix.

Gimlet’s product—delivered as software or through an API to its own Gimlet Cloud—isn’t for everyday AI app developers. It’s aimed at the largest AI model labs and data centers.

The company publicly launched in October with what it said was eight-figure revenue out of the gate (at least $10 million). Asgar said his customer base has more than doubled in the last four months and now includes a major model maker and an extremely large cloud computing company, though he declined to name them.

The co-founders previously worked together at Pixie, a startup that built an open-source observability tool for Kubernetes. Pixie was acquired by New Relic in 2020, just two months after launching with a $9 million Series A led by Benchmark. (Pixie’s technology is now part of the open-source organization that oversees Kubernetes.)

After Asgar randomly ran into Tully about a year ago and also received angel investments from Stanford professors, VCs started calling. After launch, a term sheet landed on Asgar’s desk. When VCs heard Asgar was looking at offers, “we got a pretty big swarm of funding,” and the round was quickly oversubscribed, he said.

With the previous seed, the startup has now raised a total of $92 million, including from a slew of angels like Sequoia’s Bill Coughran, Stanford Professor Nick McKeown, former CEO of VMware Raghu Raghuram, and Intel CEO Lip-Bu Tan. The company currently employs 30 people.

Other investors include Factory (which led the seed), Eclipse Ventures, Prosperity7, and Intel Capital.