Sesame Unveils Base AI Model Behind Viral Virtual Assistant Maya

Sesame, the innovative AI company behind the strikingly lifelike voice assistant Maya, has just made waves by releasing the base model that drives her capabilities. Dubbed CSM-1B, this model boasts a size of 1 billion parameters, a term that refers to the individual components making up the model. Released under an Apache 2.0 license, it's open for commercial use with minimal restrictions, as announced on the AI development platform Hugging Face.

CSM-1B functions by converting text and audio inputs into "RVQ audio codes." RVQ stands for "residual vector quantization," a method that transforms audio into discrete tokens, or codes. This technique is also utilized in other cutting-edge AI audio technologies, such as Google's SoundStream and Meta's Encodec. At its core, CSM-1B leverages a model from Meta's Llama family, combined with an audio "decoder" component. A specialized version of CSM-1B, after fine-tuning, powers the voice of Maya, according to Sesame.

Describing the model as a "base generation model" on its Hugging Face and GitHub repositories, Sesame notes that it's designed to produce a variety of voices but hasn't been refined for any specific voice. While it has some ability to handle non-English languages thanks to "data contamination" in its training set, its performance in this area is likely subpar. Interestingly, Sesame has kept the details of the training data under wraps, leaving us curious about what went into building this model.

One aspect that raises eyebrows is the lack of robust safeguards. Sesame operates on an honor system, simply encouraging users and developers to avoid using the model to replicate someone's voice without permission, produce misleading content like fake news, or partake in any "harmful" or "malicious" activities. I personally tested the demo on Hugging Face, and within a minute, I had cloned my voice. It was a breeze to generate speech on any topic, even sensitive ones like the election and Russian propaganda.

Consumer Reports recently highlighted the concerning lack of "meaningful" safeguards in many AI-powered voice cloning tools, which could lead to potential fraud or abuse. Sesame, co-founded by Oculus co-creator Brendan Iribe, caught the public's eye in late February with its assistant tech that nearly escapes the uncanny valley. Both Maya and Sesame's other assistant, Miles, exhibit realistic human-like traits such as taking breaths, speaking with disfluencies, and being interruptible mid-speech, similar to OpenAI's Voice Mode.

Financially, Sesame has secured undisclosed funding from heavyweights like Andreessen Horowitz, Spark Capital, and Matrix Partners. Beyond voice assistants, the company is also venturing into prototyping AI glasses intended for all-day wear, equipped with their custom models. This move shows Sesame's ambition to push the boundaries of AI technology further into our daily lives.