Google launches AI video mouth-to-mouth technology: changes the way people speak according to the translation language, but it is not open to use

CTOnews.com, May 11, at the 2023 I / O developers conference, Google announced that it is developing an experimental technology called Universal Translator.

As its name implies, the tool aims to translate videos from one language to another while preserving the overall tone and atmosphere. This means that the technology can not only translate audio from one language to another, but also imitate the speaker's voice, tone and facial expressions, and the character's speech video will change the mouth shape synchronously according to the pronunciation of the target language.

As shown in the figure above, Universal Translator first detects words and translates them. It then checks the tone of the speaker and what they emphasize. After combining these two aspects, it will generate the voice of the target language. Finally, it synchronizes the mouth shape of the speaker in the video according to the pronunciation of the voice generated by AI.

Google said it would restrict access to Universal Translator, considering that the tool could be abused to create fake videos. As a result, only Google authorized partners can use it for constructive projects, and ordinary users cannot use it.