HomeTsinghua and Tencent Hunyuan Win MLSys2026 MoE Inference Challenge with 4.1x NPU Speedup
Tsinghua University's Storage Lab and Tencent's MegEngine AI Infra team recently claimed the global championship in the MoE Model Inference Optimization Challenge at MLSys2026, a leading machine learning systems conference.
To tackle inference performance bottlenecks in the mixture-of-experts (MoE) architecture with trillion-scale parameters running on heterogeneous NPU chips, the joint team designed a full-chain optimization solution for the official model and NPU hardware. By employing the E-Shard strategy for expert-level task partitioning, PSUM 3D tensor batch readout, the GEMV path distributing outputs across multiple banks for concurrency, and using scalar engines to reduce initial data transfer latency, they successfully resolved low data transfer efficiency and repeated activation transfers at the operator level.
In parallel, the team reorganized the on-chip data layout for the attention module and integrated core Transformer operators, achieving bit-level high-precision alignment.
Figure 3: Diagram of the MoE optimization architecture, featuring E-Shard expert partitioning, continuous DMA, PSUM/GEMV concurrency, cold start pipeline, and prefetch control.
Additionally, the team developed an agent-based inference operator optimizer named "Knight". Through an automated closed-loop process of proposal, code implementation, and iteration, it dramatically broadened the optimization search space. As a result, the end-to-end inference time dropped from 14.91 seconds to 3.56 seconds, a 4.1x speedup; single-step decoding latency fell from 12.63ms to 5.45ms, and DMA engine utilization during weight loading climbed to around 80%.
Outperforming teams from leading global universities like Stanford and MIT, this achievement highlights the deep expertise of Chinese teams in adapting large models to underlying systems and optimizing operators. It also provides a valuable engineering blueprint for deploying trillion-parameter MoE models on future super-node computing platforms.
Related article
Agility Robotics Selects Nyobolt for Rapid Charging of Digit 5 Humanoid
Nyobolt supplies fast-charging, long-lasting battery technology for Digit 5. Source: AgilityPower management remains a critical hurdle for humanoid robot developers. Agility Robotics Inc. announced a collaboration with Nyobolt to develop a rapid-rech
Apple iOS 27 Update: Siri Enters Dual-Mode Era with ChatGPT Free Switch
iOS 27 introduces a fundamental overhaul of Siri’s architecture, centered on a new standalone application that moves beyond traditional voice-only interactions.Designed with a chatbot interface, the app allows users to type prompts for seamless AI co
Soul Founder Unveils Open Core AI Tech as 4,700 Students Redefine Gen Z Socializing at Meta Creation Camp
As artificial intelligence reshapes social media, Gen Z’s approach to meaningful connection is evolving. On February 27, 2026, data from Qichacha revealed that the founder of Soul is actively driving the "2025 Soul App Metaverse Creation Camp College
Related Special Topic Recommendations
Design & Art
Top AI Illustration Tools: Create Editorial Scenes and Cover Art Faster
2026 Latest Top-rated AI Illustration Tools: Our curated list features the best powerfully designed tools for creating stunning editorial scenes and eye-catching cover art quickly. Each option has gone through rigorous real-world tests, offering a free vs paid comparison to help you find the perfect match for your workflow. Try these must-try solutions today to unlock your AI edge and boost your creative productivity. Explore now at XIX.AI.
10 tools
xix.ai
Education and Learning
Top AI Tutors for Exam Prep: Practice Math, Science, and English
2026 Latest Top-Rated Best AI Tutors for Exam Prep are here on XIX.AI! Our curated list features powerful game-changing tools that offer free vs paid comparison, real-world tests, and weekly updated rankings. They help boost your writing efficiency, simplify complex topic learning, and unlock your academic potential. Explore now to Discover your perfect exam prep tool and Start creating today to Unlock your AI edge!
10 tools
xix.ai
Finance
AI Budget Planning Tools for Personal Money Management
2026 Latest Best Top-Rated AI Budget Planning Tools for Personal Money Management are here on XIX.AI! This curated collection features powerful, game-changing solutions that help you conduct real-world tests, understand spending habits, and create personalized plans to boost productivity. Enjoy a free vs paid comparison along with detailed rankings. Must-try tools designed to simplify money management and unlock your AI edge. Explore now!
11 tools
xix.ai
Health & Wellness
AI Sleep Tracking Assistants: Improve Recovery with Better Bedtime Habits
2026 Latest Best Top-Rated AI Sleep Tracking Assistants curated for ultimate recovery. This fully updated weekly guide features powerful tools that deliver real-world tests, free vs paid comparison, and detailed rankings to help you find the must-try solution that boosts sleep quality and supports better bedtime habits. Explore now at XIX.AI to Unlock your AI edge.
10 tools
xix.ai
Data Analysis
AI SQL Copilots for Product and Revenue Analytics
2026 Latest Best Top-Rated AI SQL Copilots for Product and Revenue Analytics! XIX.AI has curated a powerful, game-changing collection of must-try tools that undergo real-world tests through weekly updated rankings. These tools help users boost productivity significantly by accelerating data analysis, streamlining complex queries, and generating accurate insights to drive smarter business decisions. Explore now to Discover your perfect tool for unlocking your AI edge!
14 tools
xix.ai
Education and Learning
Best AI Homework Helpers That Teach: Learn Concepts, Not Just Answers
2026 Latest Best Top-Rated AI Homework Helpers That Teach Core Concepts Not Just Answers Powerful Tools for Students Must-Try Curated List Weekly Updated Free vs Paid Comparison Real-World Tests Accurate Rankings to Help You Unlock Your Full Learning Potential Explore now to Discover your perfect tool for smarter studying.
10 tools
xix.ai
Comments (1)
0/500
![BrianMartinez]()
That's a massive leap! 4.1x NPU speedup for MoE inference? I'm curious about the techniques they used. Is it better quantization or sparse attention? Either way, congrats to the team! 🎉
Tsinghua University's Storage Lab and Tencent's MegEngine AI Infra team recently claimed the global championship in the MoE Model Inference Optimization Challenge at MLSys2026, a leading machine learning systems conference.
To tackle inference performance bottlenecks in the mixture-of-experts (MoE) architecture with trillion-scale parameters running on heterogeneous NPU chips, the joint team designed a full-chain optimization solution for the official model and NPU hardware. By employing the E-Shard strategy for expert-level task partitioning, PSUM 3D tensor batch readout, the GEMV path distributing outputs across multiple banks for concurrency, and using scalar engines to reduce initial data transfer latency, they successfully resolved low data transfer efficiency and repeated activation transfers at the operator level.
In parallel, the team reorganized the on-chip data layout for the attention module and integrated core Transformer operators, achieving bit-level high-precision alignment.
Figure 3: Diagram of the MoE optimization architecture, featuring E-Shard expert partitioning, continuous DMA, PSUM/GEMV concurrency, cold start pipeline, and prefetch control.
Additionally, the team developed an agent-based inference operator optimizer named "Knight". Through an automated closed-loop process of proposal, code implementation, and iteration, it dramatically broadened the optimization search space. As a result, the end-to-end inference time dropped from 14.91 seconds to 3.56 seconds, a 4.1x speedup; single-step decoding latency fell from 12.63ms to 5.45ms, and DMA engine utilization during weight loading climbed to around 80%.
Outperforming teams from leading global universities like Stanford and MIT, this achievement highlights the deep expertise of Chinese teams in adapting large models to underlying systems and optimizing operators. It also provides a valuable engineering blueprint for deploying trillion-parameter MoE models on future super-node computing platforms.
Agility Robotics Selects Nyobolt for Rapid Charging of Digit 5 Humanoid
Nyobolt supplies fast-charging, long-lasting battery technology for Digit 5. Source: AgilityPower management remains a critical hurdle for humanoid robot developers. Agility Robotics Inc. announced a collaboration with Nyobolt to develop a rapid-rech
Apple iOS 27 Update: Siri Enters Dual-Mode Era with ChatGPT Free Switch
iOS 27 introduces a fundamental overhaul of Siri’s architecture, centered on a new standalone application that moves beyond traditional voice-only interactions.Designed with a chatbot interface, the app allows users to type prompts for seamless AI co
Soul Founder Unveils Open Core AI Tech as 4,700 Students Redefine Gen Z Socializing at Meta Creation Camp
As artificial intelligence reshapes social media, Gen Z’s approach to meaningful connection is evolving. On February 27, 2026, data from Qichacha revealed that the founder of Soul is actively driving the "2025 Soul App Metaverse Creation Camp College
2026 Latest Top-rated AI Illustration Tools: Our curated list features the best powerfully designed tools for creating stunning editorial scenes and eye-catching cover art quickly. Each option has gone through rigorous real-world tests, offering a free vs paid comparison to help you find the perfect match for your workflow. Try these must-try solutions today to unlock your AI edge and boost your creative productivity. Explore now at XIX.AI.
10 tools
xix.ai
2026 Latest Top-Rated Best AI Tutors for Exam Prep are here on XIX.AI! Our curated list features powerful game-changing tools that offer free vs paid comparison, real-world tests, and weekly updated rankings. They help boost your writing efficiency, simplify complex topic learning, and unlock your academic potential. Explore now to Discover your perfect exam prep tool and Start creating today to Unlock your AI edge!
10 tools
xix.ai
2026 Latest Best Top-Rated AI Budget Planning Tools for Personal Money Management are here on XIX.AI! This curated collection features powerful, game-changing solutions that help you conduct real-world tests, understand spending habits, and create personalized plans to boost productivity. Enjoy a free vs paid comparison along with detailed rankings. Must-try tools designed to simplify money management and unlock your AI edge. Explore now!
11 tools
xix.ai
2026 Latest Best Top-Rated AI Sleep Tracking Assistants curated for ultimate recovery. This fully updated weekly guide features powerful tools that deliver real-world tests, free vs paid comparison, and detailed rankings to help you find the must-try solution that boosts sleep quality and supports better bedtime habits. Explore now at XIX.AI to Unlock your AI edge.
10 tools
xix.ai
2026 Latest Best Top-Rated AI SQL Copilots for Product and Revenue Analytics! XIX.AI has curated a powerful, game-changing collection of must-try tools that undergo real-world tests through weekly updated rankings. These tools help users boost productivity significantly by accelerating data analysis, streamlining complex queries, and generating accurate insights to drive smarter business decisions. Explore now to Discover your perfect tool for unlocking your AI edge!
14 tools
xix.ai
2026 Latest Best Top-Rated AI Homework Helpers That Teach Core Concepts Not Just Answers Powerful Tools for Students Must-Try Curated List Weekly Updated Free vs Paid Comparison Real-World Tests Accurate Rankings to Help You Unlock Your Full Learning Potential Explore now to Discover your perfect tool for smarter studying.
10 tools
xix.ai
That's a massive leap! 4.1x NPU speedup for MoE inference? I'm curious about the techniques they used. Is it better quantization or sparse attention? Either way, congrats to the team! 🎉
