Could AI be a suitable tool in the dubbing process of movies?

Last Updated on May 24, 2024

The last couple of years have really ramped up in AI capability and debate. It even became one of the biggest issues that the artists fought for when the writers and actors held union strikes when Hollywood studios started to implement more of the technology to cut certain corners. People all over social media would blow up at Disney for attempting to slip by some rather synthetic-looking extras for a basketball scene in a Disney+ movie titled Prom Pact. Even films like Civil War and Late Night with the Devil, which have been shown admiration and support from audiences for the filmmakers’ ingenuity with their creative visions, came under fire for using AI technology for producing images. A new article from The Hollywood Reporter has admitted that “everyone in Hollywood uses AI, but they are scared to admit it.” Naturally, no matter how big or how small the usage is, it’s a hot-button issue when people’s jobs are taken away and creativity is stifled with a bunch of ones and zeroes composing from an imitation of style. According to THR, David Kavanagh, who is an executive officer of the Federation of Screenwriters in Europe (FSE), has said that some sectors of the industry are in danger of being dismantled altogether. Kavanagh stated, “Dubbing and subtitling employment in Europe is finished. It’s hard to see how they will survive this.” He would cite a technology in Europe being used to produce lip-synced dubs in multiple languages, even using versions of the original actor’s performance. The technology can alter the actor’s mouth into matching the newly dubbed dialogue in addition to using an AI voice pulled from the actors’ own.

It was revealed that this Saturday in Cannes, indie distributor XYZ Films, who released popular foreign movies like The Raid films, will present a sizzle reel of AI-translated trailers for some international films. These films include a Nordic sci-fi movie, UFO Sweden, a French comedy thriller, Vincent Must Die, and the Korean action hit Smugglers. These trailers will be featuring the TrueSync dubbing technology from L.A.-based company Flawless. Additionally, Flawless and XYZ will be pitching the tech for a way to cheaply produce high-quality English-language dubs that will make them more accessible for viewers who do not wish to contend with subtitles. The companies — Flawless, XYZ Films and Tea Shop Productions — are planning to release UFO Sweden worldwide for what is being touted as the “first large-scale theatrical release of a fully translated film using AI.”

There have already been some AI “reshoots” used for the movie Fall, where Flawless used their TrueSync technology to remove F-bombs from the movie. It was explained, “The Flawless team in post-production changed more than 30 F-bombs throughout the movie into PG-13-acceptable epithets like ‘freaking’ along with a few other lines of dialogue. […] Employing the same principles used to create ‘deepfakes,’ TrueSync alters the mouth movements of the actors to match the alternate dialogue being spoken (a process the startup calls ‘vubbing’). Mann realized the Flawless engine could also be used to clean up the F-words in his movie.”