Pruna AI Unveils Open-Source AI Model Optimization Framework

Pruna AI, a European startup focused on developing compression algorithms for AI models, is set to release its optimization framework as open source this Thursday. The company has been working on a framework that incorporates various efficiency techniques such as caching, pruning, quantization, and distillation to enhance AI model performance.

John Rachwan, co-founder and CTO of Pruna AI, explained to TechCrunch that their framework not only applies these methods but also standardizes the process of saving, loading, and evaluating compressed models. This allows users to assess any potential quality loss and the performance improvements achieved through compression.

Rachwan likened Pruna AI's role to that of Hugging Face, which standardized the use of transformers and diffusers. "We are doing the same, but for efficiency methods," he stated, emphasizing the standardization of how these methods are applied and managed.

Major AI labs have already adopted similar compression techniques. For example, OpenAI has used distillation to develop faster versions of its models, such as GPT-4 Turbo. Similarly, Black Forest Labs created Flux.1-schnell, a distilled version of their Flux.1 model. Distillation involves a "teacher-student" approach where a larger model's outputs are used to train a smaller, more efficient model.

Rachwan pointed out that while large companies often develop these tools internally, the open-source community typically focuses on single methods. "But you cannot find a tool that aggregates all of them, makes them all easy to use and combine together," he said, highlighting Pruna AI's unique value proposition.

In addition to the open-source version, Pruna AI offers an enterprise edition with advanced optimization features, including an upcoming compression agent. Rachwan described this agent as a tool that automatically finds the best compression combination for a model based on user-specified performance and accuracy requirements.

Pruna AI's pro version is billed by the hour, similar to renting a GPU on cloud services like AWS. By optimizing models, users can significantly reduce inference costs. For instance, Pruna AI managed to compress a Llama model to one-eighth its original size with minimal quality loss, demonstrating the potential cost savings.

The company recently secured a $6.5 million seed funding round from investors including EQT Ventures, Daphni, Motier Ventures, and Kima Ventures. Pruna AI views its compression framework as a strategic investment that can pay for itself through reduced operational costs.