Vu Le, Gheorghe Tecuci, and Mihai Boicu

1
Overview

Research Problem

Abstraction of Reasoning

Learning and Tutoring Agent Shell

Lesson Design and Generation

Test Learning and Generation

2
Research Problem

• Problem: building tutoring system for expert

problem solving knowledge is hard:
– It requires a team of subject matter experts, knowledge
engineers, instructional designers and software
developers.
– It takes more than 100 hours to produce an hour of
instructional material.
– Most of the existing tutoring systems are for “well-
defined” domains.

• Solution: Agent shell that can

– Quickly learn directly from the expert even in “ill-
defined” domains.
– Use the learned knowledge to help solve problems
– Rapidly construct an ITS to tutor the problem solving
expertise the way it was taught.
3
Overview

Research Problem

Abstraction of Reasoning

Learning and Tutoring Agent Shell

Lesson Design and Generation

Test Learning and Generation

4
Problem-Reduction/Solution Synthesis Paradigm

The reduction representation of a class of

problems is a quadruple (P, S, RO, OS) P1 S1
where: RO
i
P - class of problems;
S - solutions of problems;
RO - reduction operators, each reducing P1 S1 … P1 S1
1 1 n n
a problem to
sub-problems and/or solutions,
SO - synthesis operators, each
A problem P1 is solved by:
synthesizing the solution P2 S2 … P2 S2
1 1 m m
 successively reducing
of a problem fromitthe
to simpler
solutions of
itsproblems through the application of the
sub-problems. SOj
reduction operators;
 finding the solutions of the simplest P3 S3 … Pp3 Sp3
problems; 1 1

 successively combining these solutions

through the application of synthesis 5
Application Domain: Intelligence Analysis

6
Complexity of the Reasoning Trees
Assess whether Al Qaeda has nuclear weapon
Total Nodes =
1758

7
Abstraction of Reasoning Tree
Concrete Abstract
Level Level
P1 Abstract
AP1
Strategy 1
Problem Solving Rd1
ARd1
Strategy 1
P2 P3 AP2 AP3

Rd2 ARd2

P4 P5 P6 AS1

Rd3 Rd4 Rd5

Abstract
Problem Solving Strategy 2
Strategies 2, 3, 4 S1 S2 S3

 The reasoning tree is partitioned and abstracted based on

the used problem solving strategies.
 The abstract tree consists of a hierarchical set of abstract
strategies learned from concrete reasoning trees.
 The abstract strategies are tutored to the students. 8
 Reasoning Tree and Its Abstraction for Tutoring
Assess to what extent the
Concrete piece of evidence favors Abstr
the hypothesis.
Tree act
(1758 Consider the relevance and the
believability of the piece of
Tree
nodes) evidence (217
Assess to what extent the Assess the nodes
piece of evidence favors believability of the )
the hypothesis, assuming piece of evidence
that the piece of evidence
is believable

Assess to what extent the piece of Assess to what extent the piece
evidence EVD-Dawn-Mir01-02c of evidence EVD-Reuters-01-
favors the hypothesis that Al Qaeda 01c favors the hypothesis that
considers deterrence as a reason Al Qaeda desires to obtain
to obtain nuclear weapons. nuclear weapons.

Q: What factors determine how a piece Q: What factors determine how a piece
of evidence favors a hypothesis? of evidence favors a hypothesis?
A: Its relevance and believability. A: Its relevance and believability.
22
Assess to what extent the piece of evidence Assess the Assess to what extent the piece of Assess the
believability Abstra
EVD-Dawn-Mir01-02c favors the hypothesis evidence EVD-Reuters-01-01c favors believability
that Al Qaeda considers deterrence as a of EVD- of EVD- ct
the hypothesis that Al Qaeda desires to
reason to obtain nuclear weapons, assuming Dawn- obtain nuclear weapons, assuming that Reuters-01- Reduct
Mir01-02c ion
that EVD-Dawn-Mir01-02c is believable. EVD-Reuters-01-01c is believable. 01c 9
Overview

Research Problem

Abstraction of Reasoning

Learning and Tutoring Agent Shell

Lesson Design and Generation

Test Learning and Generation

10
Architecture of Learning and Tutoring Agent Shell

Subject matter expert Instructional

Student
Knowledgeengineer designer

OntologyImport,
Elicitation, Learning
andRefinement Student
Disciple Tutoring
Mixed-initiative LearningandTutoringAgentShell
andAutonomous
ProblemSolver Lesson Designand
Mixed-initiative,
Test Learning
Multi-agentFramework
RuleLearning
andRefinement
Learningof Abstract
ReasoningStrategies
Knowledge
Management
Domain System Pedagogical
KnowledgeBase KnowledgeBase KnowledgeBase

11
Overview

Research Problem

Abstraction of Reasoning

Learning and Tutoring Agent Shell

Lesson Design and Generation

Test Learning and Generation

12
 Tutoring: Lesson Design and Generation

Design
Operato Designe
rs d
Lesson

Abstra
ct tree

 Abstraction-based lesson design through Drag-n-Drop operations

 The instructor controls how the lesson is to be taught, adds explanations, definitions
 Automatic generation of lesson’s script
 Script-based lesson generation adapted to the knowledge base 13
Lesson Fragment
Hypothesis support from a piece
of evidence Abstract
reduction
strategy

Lesson Automatically
section on generated
Evidence illustration of
the abstract
strategy

14
Lesson Fragment
Hypothesis support from a piece
of evidence Abstract
synthesis
strategy

Automatically
generated
illustration of
the abstract
strategy

15
Evaluation of Generated Lessons
Expert Novice
students analysts students analysts
12 12

10 The examples facilitate MAAI-06 10 The examples facilitate CS681-06

the understanding of the the understanding of the
8 presented topic 8 presented topic
6 6

4 4

2 2

0 0
strongly disagree disagree neutral agree strongly agree strongly disagree disagree neutral agree strongly agree

The tutoring system helps The tutoring system

me to learn the addressed students helps me to learn the
topic 10
addressed topic
students
8 8

6 6

4 4

2 2

0 0
strongly disagree disagree neutral agree strongly agree strongly disagree neutral agree 16
strongly agree
 Evaluation of Tutoring
Subjective Prior Knowledge
students students Subjective Post Knowledge
CS681-06 10
10
students Objective Test-Based Evaluation
9
9 Hypothesis 9
Information content Credibility of
8 assessment 8 and credibility
8
reported evidence
7 through evidence 7 7

6 analysis 6 6

5 5 5

4 4 4

3 3 3
2 2 2
1 1 1
0 0 0
None Very low Low Medium High Very high None Very low Low Medium High Very high None Very low Low Medium High Very high

Credibility of the Credibility of

reporter of a piece tangible evidence Grades of Generated Tests
students of evidence students
9 9

8 8 students
7 7 7

6 6 6

5 5 5

4 4 4

3 3 3
2 2 2
1 1 1
0 0 0
None Very low Low Medium High Very high None Very low Low Medium High Very high 50- 60-69 70-79 80-89 17
90-100
Overview

Research Problem

Abstraction of Reasoning

Learning and Tutoring Agent Shell

Lesson Design and Generation

Test Learning and Generation

18
Test Generation

 Automatic generation of tests based on the content of the knowledge

base and the learned lessons.
 There are 3 different types of test: omission, modification and
construction tests. 19
 Test Learning

 The agent learns a test rule by generalizing the example (including the
explanations and the hint) based on the corresponding reduction rule
from the domain knowledge base.
 Different knowledge bases result in different tests without instructor’s
involvement. 20
Discussion

21
Backup slides

22
Reasoning Tree Abstraction

abstracted to
Reasoning Tree Abstract Reasoning Tree

instantiation-of generates

Problem Classes Abstract Problem Classes

Reduction Rules abstracted to Abstract Reduction Classes
(based on abstraction
Solution Classes rules)
Abstract Solution Classes
Synthesis Rules Abstract Synthesis Classes

The abstraction rules are automatically learned when the

subject matter expert abstracts a concrete reasoning tree.

23
Abstraction Rule Learning
Assess to what extent the piece of
evidence EVD-Dawn-Mir01-02c Assess to what extent the
favors the hypothesis that Al Qaeda piece of evidence favors
considers deterrence as a reason to the hypothesis.
obtain nuclear weapons.
Consider the relevance and
Q: What factors determine how a piece abstracted to the believability of the piece of
of evidence favors a hypothesis? evidence
A: Its relevance and believability.
Assess to what extent the Assess the
Assess to what extent the piece of evidence Assess the piece of evidence favors believability of the
EVD-Dawn-Mir01-02c favors the hypothesis believability of the hypothesis, assuming piece of evidence
that Al Qaeda considers deterrence as a EVD-Dawn- that the piece of evidence
reason to obtain nuclear weapons, Mir01-02c is believable
assuming that EVD-Dawn-Mir01-02c is
believable.

instantiation-of generated-from
• Assess to what extent the piece of evidence ?O1 favors the
hypothesis that ?O2 considers deterrence as a reason to
obtain nuclear weapons.

• Assess to what extent the piece of

associated with evidence favors the hypothesis.
(construct • Consider the relevance and the
believability of the piece of evidence
abstraction rules) • …

24
Reasoning Tree Abstraction Generation
Assess to what extent the piece of
evidence EVD-Reuters-01-01c
Assess to what extent the
favors the hypothesis that Al Qaeda
piece of evidence favors
considers self-defense as a reason
the hypothesis.
to obtain nuclear weapons.
Consider the relevance and
Q: What factors determine how a piece
of evidence favors a hypothesis? abstracted to the believability of the piece of
evidence
A: Its relevance and believability.
Assess to what extent the piece of evidence Assess to what extent the Assess the
Assess the piece of evidence favors believability of the
EVD-Reuters-01-01c favors the hypothesis believability of
that Al Qaeda considers self defense as a the hypothesis, assuming piece of evidence
EVD-Reuters- that the piece of evidence
reason to obtain nuclear weapons, 01-01c
assuming that EVD-Reuters-01-01c is is believable
believable.

instantiation-of generates

• Assess to what extent the piece of evidence ?O1

favors the hypothesis that ?O2 considers
deterrence as a reason to obtain nuclear weapons. • Assess to what extent the piece of
evidence favors the hypothesis.
• Assess to what extent the piece of
evidence favors the hypothesis,
abstraction rules assuming that the piece of evidence is
believable
• Assess the believability of the piece of
evidence

• Consider the relevance and the

believability of the piece of evidence

25