«Far East Journal of Experimental and Theoretical Artificial Intelligence Volume 1, Issue 2, 2008, Pages 87-125 Published online: August 12, 2008 This ...»
Far East Journal of Experimental and Theoretical Artificial Intelligence
Volume 1, Issue 2, 2008, Pages 87-125
Published online: August 12, 2008
This paper is available online at http://www.pphmj.com
2008 Pushpa Publishing House
EXTRACTING INFORMATION FROM
NATURAL LANGUAGE INPUT TO AN
INTELLIGENT TUTORING SYSTEM
MICHAEL S. GLASS and MARTHA W. EVENS Department of Mathematics and Computer Science Valparaiso University Valparaiso, IN 46383, U. S. A.
e-mail: email@example.com Computer Science Department Illinois Institute of Technology Chicago, IL 60616, U. S. A.
e-mail: firstname.lastname@example.org Abstract We have constructed a new module to process student natural language input to CIRCSIM-Tutor, an intelligent tutoring system designed to help medical students learn to solve problems involving the negative feedback process that regulates blood pressure in the human body.
CIRCSIM-Tutor spends most of its time engaging the student in a natural language-based dialogue. The new input understander uses an information extraction approach that is robust enough to handle freeform student input. We describe an evaluation of CIRCSIM-Tutor by forty-two students at Rush Medical College, with particular emphasis on the performance of the input understander.
Keywords and phrases : natural language understanding, intelligent tutoring system, information extraction, finite state parsing.
This research was partially supported by the Cognitive Science Program of the United States Office of Naval Research under Grant No. N00014-94-1-0338 to Illinois Institute of Technology and by the National Science Foundation under award No. 0634049 to Valparaiso University. The content does not reflect the position or policy of the government and no official endorsement should be inferred.
Communicated by Shun-Feng Su Received January 25, 2008; Revised May 2, 2008 MICHAEL S. GLASS and MARTHA W. EVENS Introduction We have constructed a new module designed to understand natural language student input to CIRCSIM-Tutor, an intelligent tutoring system that helps medical students learn to solve problems involving the negative feedback loop that regulates blood pressure in the human body.
It has been tested on forty-two students in a standard physiology laboratory at Rush Medical College.
CIRCSIM-Tutor is a language-based tutor. In fact, the designers of this system were motivated by the belief that the use of language enhances learning, that learning new scientific concepts is inextricably enmeshed with learning the language of that science, and that trying to explain new phenomena in words is the best way to understand them.
Thus the system primarily engages the student in dialogue. It begins by describing a disturbance to the cardiovascular system, such as a hemorrhage or a pacemaker malfunction; it asks the student to enter qualitative predictions about the responses of seven physiological variables to this perturbation; and then it launches a tutorial dialogue.
The new input understander was designed to be robust enough to handle free-form student input and fast enough that the system can respond to the student in under two seconds. Thus we chose an information extraction approach using a cascade of finite state machines.
The new module was one of several improvements so that this version of CIRCSIM-Tutor became the first to be used with large classes of students. Later in this paper we describe the evaluation of our system in a regularly scheduled class laboratory exercise in November, 1999.
Interactive dialogue-based tutoring systems are less common and harder to build than other kinds of computer-aided instruction. But there is scientific evidence for endeavoring to create a dialogue-based system.
There is evidence in the education literature that human tutoring is highly effective compared to other forms of instruction, and there are studies that show that putting theories into words and giving explanations aids learning.
Studies demonstrating the value of tutoring
one-on-one tutoring and make it clear that it is an effective kind of teaching. Cohen et al.  conducted a metastudy of 65 controlled evaluations of school tutoring programs, concluding that tutored students outperform control subjects to a high degree of statistical significance.
Bloom  reported good results comparing classes of elementary school students taught in a normal classroom using mastery learning methods against students tutored individually or in groups of two or three together. Bloom found that the average tutored student performed two standard deviations above the average normal classroom student.
Michael and Rovick  demonstrated that one-on-one tutoring is effective even for bright and motivated adults like those at Rush Medical College. Students tutored for one hour showed significantly more improvement between pre-test and post-test than a control group that spent the same time on task reading carefully chosen relevant text. A recent paper by VanLehn et al.  shows that tutoring produces greater improvement than reading under a variety of circumstances.
Studies demonstrating the importance of language in learning
Several recent studies have demonstrated the importance of natural language in learning. There is reason to believe that merely making students talk or write has value. That is, independent of whether the system understands what they say, making students articulate ideas improves retention and understanding. Chi et al.  demonstrated experimentally what they call the “self-explanation effect”, where learners are prompted to explain back what they just learned to a neutral listener. Based on this notion, Aleven and Koedinger modified a geometry tutor to prompt for student explanations of their proofs [1, 2]. Their experiments showed that encouraging self-explanation by students improved learning outcomes. However without two-party dialogue--students did not receive feedback on their explanations from the geometry tutor---students left the majority of the explanation boxes empty. Aleven et al. later produced a tutor  that attempts to understand student input and to provide feedback to improve the quality of that input. In the realm of tutors that talk but do not listen, Di Eugenio et al. [18, 19] have demonstrated that more sophisticated natural language tutorial utterances produce better outcomes than the same knowledge expressed in template-constructed unsophisticated language.
MICHAEL S. GLASS and MARTHA W. EVENS Graesser [34, 36] dissected 66 one-hour tutoring sessions with untrained human tutors. Analysis of pre-tests and post-tests showed that tutoring was effective despite the fact that these tutors rarely used sophisticated tutoring strategies. The tutors asked many deep questions;
the students also asked deep questions at the rate of eight per hour, which is much more often than in ordinary classroom teaching. Graesser et al.  concluded that it is these deep questions and the attempts to state them and answer them that account for much of the learning.
There is also evidence that dialogue itself is an important factor in learning. In an ingenious series of experiments, Fox Tree  has shown that people learn more from overhearing a dialogue than from overhearing a monologue with the same content.
Other experiments in adding natural language interaction to tutoring systems From the beginning of research on intelligent tutoring systems, many researchers believed that natural language was essential to tutoring.
Carbonell’s  SCHOLAR geography tutoring system asked questions and produced the first mixed-initiative dialogue with a computer. Collins and his collaborators [15, 16] continued to stress natural language interaction as this project grew. Brown and Burton’s SOPHIE system [7, 8], which tutored students in basic concepts of electricity, was also based on natural language interaction. Wilensky’s Unix Consultant , which is more of a coach than a tutor, also used natural language interaction extensively, based on a semantic grammar for Unix concepts.
The most pronounced difficulty with SCHOLAR and other early dialoguebased tutoring systems was a lack of multi-turn coherence to the dialogue. Typically a question would not be related to the previous or following question.
Cawsey  studied tutoring in the domain of electrical circuits and focused on the interactive nature of tutor explanations. In building a system to emulate this kind of tutoring she created a number of dialogue plans for tutoring. Cawsey’s perception of the interactive nature of explanations in tutoring helped us to recognize the same kind of behavior in transcripts of human tutors, prompting the development of multiturn planning in CIRCSIM-Tutor .
EXTRACTING INFORMATION FOR AN ITS 91Interest in natural language based tutoring systems has increased in the last decade. While ITS’96 had no papers on tutoring systems based on natural language dialogue except Freedman and Evens  from the CIRCSIM-Tutor project, the situation has since changed. Both the Conference on Intelligent Tutoring Systems and the Conference on Artificial Intelligence in Education regularly feature multiple sessions and ancillary workshops on natural language processing in tutoring systems. The center of much of this new activity was the NSF sponsored Circle Project, joint between the University of Pittsburgh and Carnegie Mellon University. VanLehn directed development of the Atlas Project [27, 59, 60, 68]. Atlas-Andes  is a physics tutor that combines a model tracing approach with natural language dialogue with the goal of improving the student’s ability to solve problems in qualitative reasoning.
Atlas uses Freedman’s Atlas Planning Environment (APE) [23, 24, 25], a hierarchical, opportunistic, reactive planner, to plan natural language dialogue. Rosé’s CARMEL parser [59, 60], which serves as the input understanding component of Atlas-Andes and other Circle projects, is intended to handle a wide range of student input, especially studentgenerated explanations. Lane and VanLehn  have built a dialoguebased tutor for program design. Moore’s emphasis on the role of planning in generation, especially in the generation of dialogue, has been of great importance in text generation in general , with significant contributions to tutorial dialogue generation especially. Her Beetle system  illustrates what good planning can contribute to naturalsounding dialogue. Heffernan and others  have done significant work building a framework for more easily creating dialogue-based tutoring.
Graesser and others have produced another set of dialogue based tutoring systems . The input understanding in his AutoTutor uses latent semantic analysis  for robust processing of complex student utterances. Student answers that are not easily recognized by more conventional means (usually the longer ones) are evaluated by their similarity to a collection of archetypal answers to the question. Some of the archetypes contain known misconceptions or partially correct answers. AutoTutor was originally designed as a computer literacy tutor, but it has now been ported to other domains such as physics .
MICHAEL S. GLASS and MARTHA W. EVENS There are now dialogue-based computer tutors that use spoken input and output, for example SCoT at Stanford  and ITSPOKE at Pittsburgh . Litman’s system, ITSPOKE, which is essentially AtlasAndes with spoken input and output, has been shown to produce better and faster learning outcomes than Atlas-Andes. These systems still need the kind of language understanding and generation mechanisms developed by CIRCSIM-Tutor and other typed-language systems, as well as the kinds of dialogue strategies used in human tutoring.
How the CIRCSIM-Tutor project got started
Our goal from the beginning was to emulate a subset of expert human tutoring. Our primary source of data on both tutor and student behavior is a set of human-tutored keyboard-to-keyboard tutoring transcripts that the CIRCSIM-Tutor project has accumulated. These tutoring sessions were held with the student and the tutor in different rooms communicating only through the computer keyboard and screen in order to match the conditions under which the students would interact with the computer tutor; this process also simplified the collection of the session transcripts. We have 75 transcripts from expert tutor sessions, mostly one hour in length [33, 42, 43, 44]. In this paper we use raw transcript extracts, so that the reader can understand the kind of input that the system encounters. We have sometimes cleaned up the portion typed by the human tutor to make it easier to read. References to transcript abstracts are of the form “K51-tu-17-3” meaning transcript 51, turn 17, sentence 3. It is the tutor’s turn. In all 75 transcripts the tutors are Joel Michael and Allen Rovick, professors of physiology at Rush Medical College; the students are first-year medical students. In addition to the data from human tutoring we also use examples from logs of medical students using various revisions of CIRCSIM-Tutor Version 2.
Before this project started, in addition to tutoring students face-toface themselves, Michael and Rovick had already built several CAI systems for students of physiology: CIRCSIM, ABASE, and GASP [50, 61, 62, 63]. These systems have been very successful and are still being widely used, but Michael and Rovick decided that natural language input and output are essential to making better systems.
This paper outlines the problems we set out to solve, explains our solution, and describes an evaluation with a class of medical students. In
EXTRACTING INFORMATION FOR AN ITS 93the next section we discuss the operation of CIRCSIM-Tutor. Then we describe the operation of the new input understanding component, showing the types of phenomena it is designed to address. We describe the results of evaluating CIRCSIM-Tutor with regard to learning gains and with regard to the performance of the input understander. Finally we outline further input understanding phenomena we might address in the future.
The Operation of CIRCSIM-Tutor