Amazon Prime Video’s latest trick is natural lip-sync for dubbed audio

I have watched plenty of stuff on streaming platforms where the translated audio doesn’t feel natural on the faces of the actors appearing on the screen. It just looks odd and takes away the fun from watching dubbed films and TV shows. I would prefer to watch content with the original audio and see the translated captions, rather than deal with the asynchrony between the audio and facial expressions. Amazon finally has a solution.

The company is launching a new lip-sync technology on Prime Video that will use AI and VFX to sync the lip movement of actors so that they appear to naturally speak the dialog, even if it has been translated and dubbed from another language. The technology is making its debut with the first and second seasons of Maxton Hall, and will also be used when the third season premieres on the platform.

“The technology synchronizes actors’ lip movements with human-dubbed audio, creating a seamless viewing experience that allows audiences to remain fully immersed in the story. AI and VFX technologies power the process behind the scenes, responsibly applied to deliver a better customer experience,” says Amazon. For now, support is limited to English only.

Thankfully, Amazon assures that it is going to add the AI-powered lip sync technology to more films and TV shows in the near future. Amazon won’t be the only platform trying to make human lips sync with dubbed audio. In an exclusive interaction with Digital Trends last year, YouTube gave us a quick demo of how it is using AI-powered lip-syncing technology so that the translated audio appears naturally on the human face.

Back then, YouTube Product Lead, Autodubbing, Buddhika Kottahachchi, told me that the lip-sync system “modifies the pixels on the screen to match the translated speech.” It seems Amazon is bringing something similar to Prime Video. Let’s just hope the AI stack doesn’t end up ballooning the cost of streaming subscriptions, once again.