Superior Alternative to OpenAI's AI Transcription Tool: Less Hallucination, More Accuracy

OpenAI's Whisper, an AI-powered speech recognition and transcription tool released in 2022, has been found to frequently generate hallucinations or fabricate information, raising concerns about its potential to cause harm in critical settings. A researcher at the University of Michigan discovered hallucinations in 80% of the audio transcriptions from public meetings that he reviewed. Similarly, an engineer who analyzed 100 hours of Whisper transcriptions reported hallucinations in about half of them, and another developer found them in nearly every transcript he created, totaling 26,000 instances. While minor errors in transcription are common, experts noted that Whisper's level of hallucination is unprecedented among AI transcription tools. OpenAI claims that Whisper, an open-source neural network, achieves near-human accuracy in English speech recognition. It's widely used across industries for tasks like transcribing interviews, translating content, and generating video subtitles. The widespread use of Whisper could lead to the spread of fabricated text, misattributed quotes, and other misinformation across various platforms, with the impact varying based on the original content's nature. Whisper is integrated into some versions of ChatGPT, used in call centers, voice assistants, and cloud platforms from Oracle and Microsoft, and was downloaded over 4.2 million times last month from HuggingFace. Of particular concern is the increasing use of Whisper-based tools by medical professionals to transcribe patient consultations. The AP spoke with over a dozen engineers, researchers, and developers who confirmed that Whisper often fabricates phrases and entire sentences, sometimes including racial commentary, violent rhetoric, and even imagined medical treatments. "Nobody wants a misdiagnosis," emphasized Alondra Nelson, a professor at the Institute for Advanced Study. Although OpenAI advises against using Whisper in high-risk domains like decision-making contexts where accuracy flaws can lead to significant issues, the tool's marketed accuracy makes it attractive to various industries looking to streamline operations, often overlooking potential risks. The issue isn't limited to long or poorly recorded audio; researchers found hallucinations even in short, clear audio samples. They estimate that this could result in tens of thousands of incorrect transcriptions across millions of recordings. Christian Vogler, director of Gallaudet University's Technology Access Program and deaf himself, pointed out that individuals who are deaf or hard of hearing may miss these hallucinations "hidden amongst all this other text." These findings highlight a broader issue in the AI industry: the rush to market products for profit, especially in the absence of comprehensive AI regulations in the US. This is particularly relevant amid debates over OpenAI's for-profit vs. non-profit status and recent leadership statements that downplay AI risks. An OpenAI spokesperson told the AP that the company is continuously working to reduce hallucinations and values the researchers' findings, incorporating feedback into model updates. In the meantime, we suggest considering Otter.ai, a reliable AI transcription tool trusted by journalists, which recently added support for six new languages. While a longtime user reported a hallucinated statistic in a new AI summary feature, the error did not appear in the transcription itself. Otter.ai advises users to review and edit transcriptions for accuracy, especially for critical tasks, noting that accuracy can vary due to factors like background noise, accents, and conversation complexity. For iPhone users, iOS 18.1 with Apple Intelligence now offers AI-powered call recording and transcription, though ZDNET's editor-in-chief Jason Hiner describes it as "still a work in progress." OpenAI recently announced plans to provide more tools to its 250 million ChatGPT Plus users.

Related article

Comments (18)