Microsoft presents the Rumi project to improve AI understanding through multi-mode.
CTOnews.com, August 5, Microsoft Research recently unveiled a Project Rumi project that combines text, audio and video data to improve the understanding of artificial intelligence systems through a multi-mode approach to better understand human intentions.
Artificial intelligence systems have made great progress in recent years, especially in the field of natural language processing (NLP). However, the existing NLP AI mainly depends on text input and output, ignoring human intonation, facial expressions, gestures and body language and other clues in natural communication, so there will be deviations in understanding.
In AI technical terms, these cues are collectively referred to as paralanguage (paralinguistics/ paralanguage).
In order to solve this problem, Microsoft Research has developed Project Rumi, a novel framework that aims to enhance the understanding of artificial intelligence through multimodal paralanguage cues. The project consists of two main parts: multimode paralanguage encoder and multimode paralanguage decoder. CTOnews.com attached the video as follows:
