The gold standard for an AI conversation system with sentiment detection is “multimodal sentiment analysis,” which is a collection of algorithms. These methods are critical for human-centered AI systems because they can automatically analyse a person’s psychological condition based on their speech, voice colour, facial expression, and posture. The method could lead to the creation of an emotionally intelligent AI with human-like characteristics that recognises the user’s feelings and responds appropriately.
Current emotion estimation methods, on the other hand, only consider observable data and ignore information contained in non-observable signals such as physiological signals. Such signals are a potential gold mine of emotions, with the potential to greatly increase sentiment estimation ability.
In a new study published in the journal IEEE Transactions on Affective Computing, physiological signals were added to multimodal sentiment analysis for the first time by researchers from Japan, a collaborative team comprising Associate Professor Shogo Okada from Japan Advanced Institute of Science and Technology (JAIST) and Prof. Kazunori Komatani from the Institute of Scientific and Industrial Research at Osaka University. “Humans are very good at concealing their feelings. The internal emotional state of a user is not always accurately reflected by the content of the dialog, but since it is difficult for a person to consciously control their biological signals, such as heart rate, it may be useful to use these for estimating their emotional state. This could make for an AI with sentiment estimation capabilities that are beyond human,” explains Dr. Okada.
To evaluate the level of enjoyment experienced by the user during the chat, the researchers studied 2468 encounters with a dialogue AI acquired from 26 participants. After then, the user was asked to rate how fun or dull the talk was. The researchers employed the “Hazumi1911” multimodal dialogue data set, which incorporated speech recognition, voice colour sensors, facial expression and posture detection, and skin potential, a type of physiological reaction sensing, for the first time.
“On comparing all the separate sources of information, the biological signal information proved to be more effective than voice and facial expression. When we combined the language information with biological signal information to estimate the self-assessed internal state while talking with the system, the AI’s performance became comparable to that of a human,” comments an excited Dr. Okada.
These findings suggest that detecting physiological signals in humans, which are usually hidden from view, could pave the way for highly emotional intelligence AI-based dialogue systems, resulting in more natural and pleasant human-machine interactions. Furthermore, by recognising changes in daily emotional states, emotionally intelligent AI systems could aid in the detection and monitoring of mental disease. They could also be useful in education, where AI could determine whether a student is enthusiastic and excited about a topic of discussion or bored, resulting in changes in teaching method and more efficient educational services.
