National Science Foundation, Monitoring emotions while students learn with AutoTutor, 2003-2008, $1,250,000. Art Graesser is PI.

The intellectual merit of the proposed research is to build and test learning environments that coordinate complex learning and learner emotions. The project augments an existing intelligent tutoring system (AutoTutor) that helps learners construct explanations by interacting with them in natural language and helping them use simulation environments. AutoTutor has an animated conversation agent and a dialog management facility that attempts to comprehend the learner’s contributions and to responding with appropriate dialog moves (such as short feedback, pumps, hints, prompts for information, assertions, answers to student questions, suggestions for actions, summaries). AutoTutor has been developed for computer literacy and qualitative physics; it is available on the web and on desktop applications. The tutorial dialogue of AutoTutor will be enhanced in the proposed research by incorporating signal processing algorithms and sensing devices that classify various facial patterns and affective states of learners. The proposed research has three specific objectives: (1) To assess patterns of facial activity that arise while interacting with AutoTutor and to identify pedagogically significant subsets of these patterns, (2) to investigate whether learning gains and learner’s impressions of AutoTutor are influenced by dialog moves of AutoTutor that are sensitive to the learner’s affective states, and (3) to test and augment theories that systematically integrate learning and emotion into educational technologies.

This proposed research investigates strategies, processes, practices, and environments that are likely to assist the learner in the active construction of knowledge, particularly at deeper levels of comprehension and problem solving. It is already well documented that knowledge construction is enhanced by one-on-one tutoring activities that encourage the students to articulate explanations of complex systems, as well as manipulating such systems in simulation environments. Simply put, students learn by telling and doing. These constructivist principles have already been implemented in AutoTutor and have been shown to produce robust learning gains. However, constructivism is not entirely limited to cognition, discourse, action, and the environment because emotions are inextricably bound to the learning process. An agile learning environment that is sensitive to a learner’s affective state presumably enriches learning, particular when deep learning is often accompanied by confusion, frustration, boredom, interest, excitement, and insight. Therefore, attempts will be made to integrate state-of-the-art affect-sensing technology with AutoTutor. Emotions could potentially be classified on the basis of facial expressions, gross body language, spoken voice stress analysis, haptic sensing, galvanic skin response, and heart rate, but the primary focus in this research will be on facial expressions (together with knowledge states and discourse patterns). In addition to advancing research on complex learning, emotion, and sensing technologies, the proposed research will advance theories and models in the fields of cognitive science, discourse processing, computational linguistics, artificial intelligence, data mining, distance learning, and information technologies.

The broader impacts of the proposed research activities are to advance education, intelligent learning environments, and human-computer interfaces. It is widely acknowledged in the field of education that students rarely acquire a deep understanding of the material they are supposed to learn in their courses. Students normally settle for shallow knowledge, such as lists of concepts, a handful of facts about each concept, and disconnected definitions of key terms. Students lack the deep coherent explanations that organize the shallow knowledge and that fortify the learner for generating inferences, solving problems, and applying their knowledge to practical situations. Yet model-based reasoning is essential for learners of science, technology, engineering, mathematics, and other disciplines that are in high demand in today’s workforce. The proposed research will attempt to fortify future learners with enhanced dynamic reasoning, automated cognitive assessment, and intelligent handling of emotions. A learning environment that monitors learner emotions is also likely to be more motivating and personally relevant to the learner.