HomeAlibaba's Tongyi Lab Open-Sources Fun-CineForge, Solves Multi-Speaker Dubbing Challenge
Traditional AI voice dubbing often falls short in high-stakes productions like films and animation, where capturing nuanced emotional peaks and perfectly synchronized lip movements is paramount. To tackle this core industry challenge, Tongyi Lab has officially launched and open-sourced the groundbreaking film-grade, multi-scenario multimodal dubbing model—Fun-CineForge .
Bridging the Audio-Visual Gap: A Four-Pillar Framework for Seamless Sync
Rather than relying on basic text-to-speech, Fun-CineForge is engineered to master four critical dimensions of professional dubbing:
Lip Sync: Ensures synthesized speech aligns with on-screen character mouth movements with exceptional precision.
Emotional Expression: Infuses the voice with authentic human-like emotion by analyzing facial cues and contextual instructions.
Voice Consistency: Maintains a stable, recognizable vocal identity for specific characters across complex multi-speaker dialogue scenes.
Time Alignment: Enables millisecond-accurate insertion of dialogue, even when the speaker is off-screen or partially obscured.
Core Innovation: Pioneering "Time Modality" and a High-Fidelity Dataset
The technical leap of Fun-CineForge stems from its unique "data + model" co-design philosophy:
The CineDub High-Quality Dataset: Tongyi Lab has also open-sourced the automated CineDub dataset construction pipeline. Utilizing a chain-of-thought error correction mechanism, it reduces transcription error rates for Chinese and English text to approximately 1% - 2% and slashes speaker diarization errors to as low as 1.2%.
Four-Modality Fusion Architecture: The model pioneers the integration of a "time modality", jointly modeling visual inputs (lip shape and expression), text (dialogue and emotional context), and audio (voice reference). This fusion allows for exact synchronization in challenging scenes, including those without visible faces.
Demonstrated Excellence: Pioneering Authentic Multi-Character Dialogue Dubbing
Benchmark results demonstrate that Fun-CineForge substantially outperforms baseline models like DeepDubber-V1 across key metrics: word error rate (WER/CER), lip synchronization (LSE-C/D), and voice similarity. A landmark achievement is its first-of-its-kind capability to handle duet and multi-person dialogues with precision, showing remarkable robustness in video clips up to 30 seconds.
GitHub: https://github.com/FunAudioLLM/FunCineForge
HuggingFace: https://huggingface.co/FunAudioLLM/Fun-CineForge
ModelScope: https://www.modelscope.cn/models/FunAudioLLM/Fun-CineForge/
Related article
China Telecom Invests in Mianbi Intelligence, Raises Capital to 713,000 Yuan for LLM & Data Infra
The "national team" and the leading figure from Tsinghua University in the large model space are deepening their strategic alignment. On March 1, 2026, according to the latest business registration data from Qichacha, Beijing Mianbi Intelligent Techn
Taotian Group Accelerates AI-Native Restructuring, Grants Interns Free Token Quotas
TaoTian Group recently introduced the "AI Productivity Plan," designed to accelerate the integration of AI technology into e-commerce operations and R&D workflows through resource allocation and tool subsidies. The program is now available to all int
Glean targets enterprise AI infrastructure in land grab
The race to dominate enterprise AI is accelerating. Microsoft is embedding Copilot into Office, Google is integrating Gemini into Workspace, and both OpenAI and Anthropic are selling directly to corporations. Meanwhile, nearly every SaaS vendor now i
Related Special Topic Recommendations
writing
Best AI Xianxia & Wuxia Assistants: Write Epic Cultivation Progression & Martial Arts Choreography
Discover the 2026 best AI assistants for crafting epic xianxia & wuxia tales. XIX.AI's curated list features top-rated, game-changing tools to master cultivation progression and martial arts choreography. Compare free vs paid options with real-world tests. Unlock your creative potential and start writing today!
10 tools
xix.ai
code
AI Mobile App Coding Tools: Generate Cross-Platform Flutter & React Native Code from Prompts
Discover the 2026 best AI mobile app coding tools for Flutter & React Native. Our curated, top-rated list features powerful, game-changing solutions that generate cross-platform code from prompts. Compare free vs paid options with real-world tests. Unlock faster development and build better apps. Explore the rankings on XIX.AI now!
10 tools
xix.ai
code
Best AI Chrome Extension Generators: Create Custom Browser Add-ons with Zero Coding Experience
Discover the 2026 best AI Chrome extension generators on XIX.AI. Our curated list features top-rated, must-try tools that let you create custom browser add-ons with zero coding. Compare free vs paid options, see real-world tests, and unlock your productivity. Explore the latest rankings and find your perfect tool today!
10 tools
xix.ai
Text-to-speech
Best AI Multilingual TTS: Generate Authentic Native-Accent Speech in 50+ Languages
Discover the 2026 best AI multilingual TTS tools for authentic native-accent speech in 50+ languages. Explore our top-rated, curated rankings with free vs paid comparisons and real-world tests. Find your perfect voice tool on XIX.AI and unlock global communication today.
10 tools
xix.ai
Meeting Assistant
Best AI Meeting Automation Tools for Smarter and Faster Collaboration
Discover the 2026 latest top-rated AI meeting automation tools for smarter, faster collaboration. Our curated list features powerful, game-changing solutions to automate notes, summaries, and action items. Compare free vs paid options with real-world tests and weekly updated rankings. Unlock peak team productivity. Explore the best picks now at XIX.AI.
10 tools
xix.ai
Prompt
AI Prompts for Infrastructure-as-Code: Deploy Terraform & Docker Configurations Safely
Discover the 2026 latest top-rated AI prompts for Infrastructure-as-Code. XIX.AI's curated selection helps you safely deploy Terraform & Docker configurations, automate cloud setups, and boost DevOps productivity. Compare free vs paid options with real-world tests. Explore now and unlock your AI edge.
10 tools
xix.ai
Comments (0)
0/500
Traditional AI voice dubbing often falls short in high-stakes productions like films and animation, where capturing nuanced emotional peaks and perfectly synchronized lip movements is paramount. To tackle this core industry challenge, Tongyi Lab has officially launched and open-sourced the groundbreaking film-grade, multi-scenario multimodal dubbing model—
Bridging the Audio-Visual Gap: A Four-Pillar Framework for Seamless Sync
Rather than relying on basic text-to-speech, Fun-CineForge is engineered to master four critical dimensions of professional dubbing:
Lip Sync: Ensures synthesized speech aligns with on-screen character mouth movements with exceptional precision.
Emotional Expression: Infuses the voice with authentic human-like emotion by analyzing facial cues and contextual instructions.
Voice Consistency: Maintains a stable, recognizable vocal identity for specific characters across complex multi-speaker dialogue scenes.
Time Alignment: Enables millisecond-accurate insertion of dialogue, even when the speaker is off-screen or partially obscured.
Core Innovation: Pioneering "Time Modality" and a High-Fidelity Dataset
The technical leap of Fun-CineForge stems from its unique "data + model" co-design philosophy:
The CineDub High-Quality Dataset: Tongyi Lab has also open-sourced the automated CineDub dataset construction pipeline. Utilizing a chain-of-thought error correction mechanism, it reduces transcription error rates for Chinese and English text to approximately 1% - 2% and slashes speaker diarization errors to as low as 1.2%.
Four-Modality Fusion Architecture: The model pioneers the integration of a "time modality", jointly modeling visual inputs (lip shape and expression), text (dialogue and emotional context), and audio (voice reference). This fusion allows for exact synchronization in challenging scenes, including those without visible faces.
Demonstrated Excellence: Pioneering Authentic Multi-Character Dialogue Dubbing
Benchmark results demonstrate that Fun-CineForge substantially outperforms baseline models like DeepDubber-V1 across key metrics: word error rate (WER/CER), lip synchronization (LSE-C/D), and voice similarity. A landmark achievement is its first-of-its-kind capability to handle duet and multi-person dialogues with precision, showing remarkable robustness in video clips up to 30 seconds.
GitHub: https://github.com/FunAudioLLM/FunCineForge
HuggingFace: https://huggingface.co/FunAudioLLM/Fun-CineForge
ModelScope: https://www.modelscope.cn/models/FunAudioLLM/Fun-CineForge/
China Telecom Invests in Mianbi Intelligence, Raises Capital to 713,000 Yuan for LLM & Data Infra
The "national team" and the leading figure from Tsinghua University in the large model space are deepening their strategic alignment. On March 1, 2026, according to the latest business registration data from Qichacha, Beijing Mianbi Intelligent Techn
Taotian Group Accelerates AI-Native Restructuring, Grants Interns Free Token Quotas
TaoTian Group recently introduced the "AI Productivity Plan," designed to accelerate the integration of AI technology into e-commerce operations and R&D workflows through resource allocation and tool subsidies. The program is now available to all int
Glean targets enterprise AI infrastructure in land grab
The race to dominate enterprise AI is accelerating. Microsoft is embedding Copilot into Office, Google is integrating Gemini into Workspace, and both OpenAI and Anthropic are selling directly to corporations. Meanwhile, nearly every SaaS vendor now i
Discover the 2026 best AI assistants for crafting epic xianxia & wuxia tales. XIX.AI's curated list features top-rated, game-changing tools to master cultivation progression and martial arts choreography. Compare free vs paid options with real-world tests. Unlock your creative potential and start writing today!
10 tools
xix.ai
Discover the 2026 best AI mobile app coding tools for Flutter & React Native. Our curated, top-rated list features powerful, game-changing solutions that generate cross-platform code from prompts. Compare free vs paid options with real-world tests. Unlock faster development and build better apps. Explore the rankings on XIX.AI now!
10 tools
xix.ai
Discover the 2026 best AI Chrome extension generators on XIX.AI. Our curated list features top-rated, must-try tools that let you create custom browser add-ons with zero coding. Compare free vs paid options, see real-world tests, and unlock your productivity. Explore the latest rankings and find your perfect tool today!
10 tools
xix.ai
Discover the 2026 best AI multilingual TTS tools for authentic native-accent speech in 50+ languages. Explore our top-rated, curated rankings with free vs paid comparisons and real-world tests. Find your perfect voice tool on XIX.AI and unlock global communication today.
10 tools
xix.ai
Discover the 2026 latest top-rated AI meeting automation tools for smarter, faster collaboration. Our curated list features powerful, game-changing solutions to automate notes, summaries, and action items. Compare free vs paid options with real-world tests and weekly updated rankings. Unlock peak team productivity. Explore the best picks now at XIX.AI.
10 tools
xix.ai
Discover the 2026 latest top-rated AI prompts for Infrastructure-as-Code. XIX.AI's curated selection helps you safely deploy Terraform & Docker configurations, automate cloud setups, and boost DevOps productivity. Compare free vs paid options with real-world tests. Explore now and unlock your AI edge.
10 tools
xix.ai
