International workshop on audio-visual affective prosody in social interaction & second language learning

(AVAP 2015)

Bordeaux, 5 and 6 March 2015

Yoshimasa Ohmoto

How the affective internal states is expressed and interpreted in communication: two feasibility studies towards affective agents

Graduate School of Informatics, Kyoto University, Sakyo-ku, Kyoto, Japan
co-authors: Takashi Ookaki, Masakazu Abe, Masahiro Yoshino, Jun Furutani, and Toyoaki Nishida

We want to develop an embodied conversational agent that is regarded as a social partner, not just multimodal interface. However, the mental stance of people when they interact with agents is usually different from when they interact with humans. Therefore, in some cases, it is difficult for people to speculate on an agent's emotions and for an agent to persuade people. This is a problem to solve for realizing affective communication with agents.
We take two approaches toward the problem: first, we obtain and analyze human behavior in discussion to find factors which influence humans' affective experiences using a method named Synthetic evidential study (SES for short), and second, we try to induce a positive mental stance of humans by agents' nonverbal behavior in a game situation. In this communication, we introduce two feasibility studies of them.

SES is a novel technology-enhanced methodology for combining theatrical role play and group discussion to help people learn by spinning stories by bringing together partial thoughts and evidences. We are developing SES Support System which combines a game engine and an augmented conversation technology. In this communication, we introduce SES Support System and findings from a simplified SES workshop as feasibility study. As a result of this study, we suggest that subjective perspective influences interpretation of the internal states of other persons.
In another study, we hypothesized that agents could induce the intentional stance by making a person simulate the agent's subjective perspective related to the interaction. We made two agents: a "trial-and-error agent" that performed goal-oriented actions using multimodal behavior and a "text display agent" that displayed its behavioral intention via text. In this communication, we introduce the experiment in which we confirmed the differences of participants' reactions depending on the agents' behavior. The results showed that participants continuously tried to communicate with the trial-and-error agent, which did not respond to the participant's actions except when necessary for performing the task.

[1] Nishida, T. et al. 2014a. Synthetic Evidential Study as Augmented Collective Thought Process -- Preliminary Report, ACIIDS 2015, accepted.
[2] Nishida, T., Nakazawa, A., Ohmoto, Y., Mohammad, Y.: Conversational Informatics.A Data-Intensive Approach with Emphasis on Nonverbal Communication, Springer (2014)
[3] M. Roubroeks, J. Ham, and C. Midden, “When artificial social agents try to persuade people: The role of social agency on the occurrence of psychological reactance,” International Journal of Social Robotics, vol. 3, no. 2, 2011, pp. 155.165.
[4] D. C. Dennett, The intentional stance. MIT press, 1989.