Meituan Open-Source Audio Model Sets New Voice Cloning Benchmark

Audio generation is experiencing a fundamental shift from multi-stage cascade architectures to end-to-end models. To overcome the information loss and error accumulation inherent in the traditional "Mel spectrogram" intermediate representation used in TTS systems, the Meituan LongCat team has officially released and open-sourced LongCat-AudioDiT (available in 1B and 3.5B parameter versions). This model successfully surpasses previous performance limits in zero-shot voice cloning by performing direct waveform latent space modeling.

Core Architecture: Moving Beyond Mel Spectrograms

LongCat-AudioDiT discards the conventional multi-stage pipeline of "acoustic feature prediction + neural vocoder," establishing instead a streamlined minimal architecture built upon a Wav-VAE (Waveform Variational Autoencoder) and a DiT (Diffusion Transformer).

Efficient Wav-VAE: Utilizing a fully convolutional design, it compresses 24kHz waveforms by a factor of 2000x to an 11.7Hz frame rate. Through non-parametric shortcut branches and multi-objective adversarial training, it ensures the reconstructed waveform maintains precise time-frequency structure while delivering excellent natural listening quality.

Semantic-Enhanced DiT: The model innovatively fuses the original word embeddings from the UMT5 text encoder with its top-level hidden states. This compensates for phonetic details lost in high-level semantic representations, significantly boosting the intelligibility of the generated speech.

Inference Optimization: Precisely Correcting Voice Drift

To further enhance generation quality, the team implemented two critical technical refinements:

Dual Constraint Mechanism: This technique identifies and corrects the persistent "training-inference mismatch" problem in flow-matching TTS. By forcibly resetting the latent variables in the prompt area during inference, it completely resolves issues of speaker voice drift and instability.

Adaptive Projection Guidance (APG): APG replaces traditional classifier-free guidance (CFG). It can accurately filter beneficial components within the guidance signal while suppressing components that cause audio degradation, significantly improving speech naturalness without inducing spectral "over-saturation."

Performance: SOTA-Level Cloning Accuracy

In benchmark tests on the Seed dataset, LongCat-AudioDiT demonstrates dominant performance:

Similarity (SIM): The 3.5B model achieved a score of 0.818 on the Seed-ZH test set and 0.797 on the challenging Seed-Hard sentence set, outperforming notable models like Seed-TTS, CosyVoice3.5, and MiniMax-Speech.

Accuracy: It ranks among the industry's top performers across key metrics, including an English WER of 1.50% and a Chinese difficult sentence CER of 6.04%.

Remarkably, LongCat-AudioDiT achieves superior results compared to multi-stage trained models using only single-stage training on pre-processed ASR transcription data. The associated research paper, source code, and model weights are now fully open-sourced and available on GitHub and HuggingFace.

Project Links:

GitHub: https://github.com/meituan-longcat/LongCat-AudioDiT

HuggingFace: https://huggingface.co/meituan-longcat/LongCat-AudioDiT