'\.,,-.

v;,::

- ~';';,' '-,'" ._,

J. -.. '

DOCUMENT RESUME
- ~'. . .,
'

ED 315 423 AUTHOR TITLE

TM 014 423 Facione, ~eter A. Critical Thinking: A Statement of Expert Consensus for Purposes of Educational Assessment and Instru~tion. Research Findings and Recommendations. bmtlr1can Philosophical Association, Newark, Del. l12p., Some pages have broken type. Reports - Research/Technical (143) MF01/PC05 Plus Postage. Cognitive Ability, *Critical Thinking, ~Curriculum Development; Cu. r rcurun Evall1ation~ ~Delphi Technique; ~Educational Assessment; Qualitative Research ~Experts; *Panel Consensus Technique
90

SPONS AGENCY PUB DATE NOTE PUB TYPE EDRS PRICE DESCRIP'l'ORS

IDENTIFIERS ABSTRACT

Using a qualitative research methodology, kn~wn as the Delphi MeUlod, an interactj,ve panel of experts was convened to work toward a consensus on the role of critical thinking (CT) in educational asse~sment and instrUction. In Delphi research, experts participate in several rounds of ~uestions that require thoughtful and detailed responses. Panelists work toward consensus by sharing reasoned opinions and reconsidering the opinion~ with regard to comments, objections, and arguments offered by other experts. A total of 46 scholare, educators, and leading figures in CT theory and CT assessment res~arch were gathered for the panel meetings. About half of the ranelists were primarily affiliated with philosophy departments: the others were affiliated with education, social sciences, or physical sciences. Recommendations resulting from the discussion rounds address the cognitivp skill dimension of CT, the diSpositional dimension of CT, and specific recommendations on CT instruction and assessment, including development of a CT curriculum. A discussion of commercially available CT assessment tools, a bibliography With an emphasis on assessment, and a set of letters which chronicle the progcess of the Delphi research group are appended. (TJH)

*~~~~~~*~~*~~~~~~~~~~*~~~~~~~~~*~***~*~*~**~~~*~~~*~~~~**~~*~*~~~~~**** Reproductions supplied by EDRS are the best that can be made *

from thp. originaJ document. *~~~~~~~~~~~~~~~~*~~*~~~~**~~*~*~~~~~*~~*~~~**~~~*~~~~***~*~**~~~~***~*

.... ,:,. ~.,~...:.:: ·

:):;;i:~;;~·.,.-,
-

·,,'

•• ,,,-,

•••

'"'1

\.

:f'~'i~
.. ,:
Office of Educa',onal Aaseareh and Improvemen, eDUCATIONAL RESOURCES INFORMATION CENTER (ERIC)

-;.•'o "

._!

u.s.

DEPARTrtIIHT

OF EDUCATION

"PERMISSION TO REPRODUCE THIS MATERIAL HAS BEEN GRANTED BY

r Tills

r Monor
•

0"11,"8,,"g "

document l1li8 bean raptoouced as recolved from 'lie person or orllan'za"ol'

~1(:.~

8 . ;'/±CIOtv[

changes neve been made '0 Improvo reproductIon Qualo)y

FOlnts of vIew or oponlonsstated In tlll$ cocument 110 nO' nflceSSIl"ly represent ofloclal OERI pOsItion or poliCy

TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC),"

CRITICAL TIIINKING:

A STATEMENT

OF EXPERT CONSENSUS

FOR PURPOSES OF EDUCATIONAL ASSESSMENT AND INSTRUCfION

Research Findings and Recommendations Prepared for the Committee on Pre-College Philosophy
of the

American Philosophical Association

Peter A. Facione, California State University, Fullerton

by

(c)

1990

P.

A.

Pac-ione

~,

:.~~ .
,

BEST COpy AVAiLAbLt.

'_,1

;"~.t:

t-

' .••

y1> .•..

....

',

.::~~~h~:= .~.,
,

-'.:.:./'~

.Q_ONTENTS

I II III IV

The Critical Thinkillg Movement and CT Assessmer",t Research Methodology and Purpose

1 4
8

The Cognitive Skill Dimension of Critical Thinking

The Dispositional Dimension of Critical Thinking 20 Procedural, Laudatory and Normative Uses the Term "CT" ..21 Dispositions of the Good Critical Thinker 27 Further Recommendations The CT Curriculum. The CT Assessment Tool
e

The CT Goal

on CT Instruction and Assessment ..27

, ,.
I ••••• , , ••••• 0

•••••••••••••

e.

••••••••••
0 0 •••••••••••••••

00

••

eo

••••

'fhe CT Instructor

30 30
32

28

II

•••••

VI

The Delphi Research Panel ..

•••••••••••••••

••••••••••••••

33

TABLES
TABLE 1 TABLE 2 TABLE 3 1ABLE 4 TABLE 5 TABLE 6 TABLE 7 Consensus Statement Regarding Critical Thinking and the Ideal Critical Thinker Project History Consensus List of CT Cognitive Skills and Sub-Skills Consensus Descriptions Affective Dispositions
0 ••••• 0

•••

3 7

12

of Core CT Skills and S~b-Skills.13 of Critical Thinking 26

Consensus Statement on Teaching and Assessing CT Skills.29 Participating Critical Thinking Experts 35

APPENDICES Appendix A: Commercial CT Assessment Tools Appendix B: CT Bibliography with Emphasis on Assessment Appendix C: The Delphi Research Letters
00 ••

36 40 51

),.',', ~.~...;.

~
:

..: ~~._~~_:~:t:/~·?!·~·f'(:·r' ),~~." .

V·t·_--'··

"

_': .•

A STATEMENT OF EX!"FRT CONSENSUS FOR PURPOSES OF EDUCATIONAL ASSESSMENT .IU~D INSTRUCTION .

CRITICAL

THINKING:

Critical 7hinking Hgyement and ~

to Cali fernia,

As~essment

From New Jersey leaders in the critical reform. requires

and from Newfoundland to Florida, major

thin~dng movement have advocated that effective

educational education at

They have argued that curricular,

and meaningful strategies

pedagogical
50

and asses$ment

all levels

of edl.tcation be coordinated skills and habits

as to foster associated

in students

those

cognitive

of inquiry that

with critical to be critical in genera~

thinking. thinkers

They have made the is vital for

case

educatinQ students

the student~

themselves

and for society

(Ennis, 1962,

1981, 1980; Pa.ssmore, 1907; Schievella, 1980, 1988; Gardner,

19~B; Shef'fler,

1973; Lipman, 1977; Siegel, Beyer, 1985; Costa,

1993; Arons, 1983; 1985;

1985; Quellmalz, 1983, 1995; Scriven,

StE:!rllberg, 1985; Ruggiero, 19a5; Paul, 1988 (a) and (b); etc.). The arguments After accor-d that advocates of inquiry, disjointed decades for critical thinking have been successful. the eighties exactly witnessed a growing

of rela ti ve neglect, of education lies

the heart

where traditional was -in the processes of

"

of a liber&l education learning skills

always said it

and thinking rather

than in the accumulation The c:ritical thinking

t ,

and senescent

information.

~ r

movement gained momentum throughout papers courses. lesson led to the development In elementary

the decade. level

Conferences

and position

of college

critical

thinking (eT) revised

and secondary

schools

(K-12) teachers

plans to incorporate

CT objectives. CT staff

In the span of a few years development programs major success when became

publishing CT textbooks growth industries.

and offering

The CT movement enjoyed

4
"':..

'_I'_~;:-~~::JZ~~:::';:-'I':"._.,;-~,,::::._r. __ -:~;

~.::~.~-'~' _'_~'v.,.:",,_ ..• ~.~ .:

~f;;:···
,
~, '

."_. .

universities

introduced CT requirements into their

general education

programs and state curricular

departments of educcationtargeted CT in their standardized testing programs. By the

frameworks and their

decade's end CT could no longer be cha£loacterized as a cottage industry. With success come questions: Not: new ones necessarily, but, because

of the e.:pectations which have been ri!lised and the investments being proposed, vel(ing ones. IntUitively, CT instruction should focus on how CT pedagogyshould dispositions

students approach a. qLlestion and reason about it. develop in sbJdents those cognitive skills which characterize the good critical

and affective

thinker.

Rather than or in addition CT assessment should

to targeting whether a given answer is correct, target arriving the qLlali t y of the critical at that answer. Thuf:;,fer

thinking the students put into all OTtheir succeosses, T experts C

find they mL(stccmtinue to address some fundamental academ:lc concen,s. Whnt exactly are those skills What are someeffective
if and

dispositions whic:hcharacterize

CT'?

ways to teach CT? And how can CT, particularly or statewide requirement, Ue

it becomesa campus-wide,district-wide

assessed?
When these academicquestions are asked by the individual professor ar teacher ~eeking to introduce CT into her own classroom, they are difficult political district ':-f"lough.But t:he questions take on social, fiscal, and

dimensions when asked by cempuacurriculum committees, sc:hool offices, boards of education, and the educational testing This is not to say that and

publishiwJ

indLl~tr'es. i

the experts find these

ql..lestiunsinslll'""lnoLlntc\ble. n the contrary, O

CT experts have worked with

their" colleagues in the education communityon some remarkable projects.

For- e;.:ample, alifornia C

a.ndNewJersey have established .... ays of curr-icular frameworks and statewide testing

intradLU:ingCT into their

" 'v "_"

"

- '->.

.!~:.: ~~;~(~.:.~: ... •..

. :;'::;:.\

.....~..

r: ~':";-"\~;-: ......-:- ..~,.

-::--::'< - ,

'"'\'>:'~
:·t

programs. enrolls

The twenty-clampus California

State

University

system,

which fer

hundreds of thousands

of students, general

has established education

a process

the approval

ef CT COLtrSesfor its role

requirement. the of CT, with

Givan the central

played by philosophers

in art.ic:ulating

value, both indi vidual clnd social, in designing college efforts that to introduce

of CT, in analyzing the concept

lev6l1 academic programs in CT, and in assisting CT into the K-12 curriculum, Association, it is little its

wonder
01'

the American Philosophical

through

Committee

Pre-College

Philosophy, has ta'~en an interest

in the CT movement and its committee asked this state of CT

impact en the profession. investigator

In December of 1987 that

to make a systematic

inquiry into the current

and CT assessment.

TABLEt
CONSBNSUS STATBMBNT RBGARDIN"G CRITICAL THINKING· AND THB IDEAL CRITICAL THINKER We undentand critic:al thinking to be purposeful, self-regulatory judgment which results in interpretation, analysil, evaluation, aad inference, as weU as explanation of the evidential, conceptual, methociologic:ai, criteriological, or conteztuai considerations UPOD which that judgmeDt is based. CT is essential as a tool of inquiry. As such, CT is a liberating force in education and a powerful resource hi ODe'. personal and civic: life. While not synonymous witb good thinking, CT is a pervasive and self-rectifying human phenomenon. The ideal critical thinker is habituaDy inquisitive, weU-informed,lrusUul of reason, open-minded, flexible, fair-minded in evaluatioD, honest in facing persoD8l biases, prudent i.a making judgments, wiI1iDgto reconsider, cleu about issues, orderly in complex malten, diligent ill seeking relevant informatioa, reasonable in the selection of criteria, focused in inquiry, and penistent in seeking resu'ta which are 81 precise u the subject and the circumstances of inquiry permit. Thus, educating good critical thinken means working toward this ideal. It combines developing CT skills with nurturing those dispositions which consistently yield useful irudghts and which are the basis of a rational and democratic society.
As Table 1 st.lggests, a panel of CT experts dimensions: cognitive a key result of inquiry is the artiCUlation terms of two Section II of by

of a conceptualization skills and affective

of CT it

dispositions.

. :.. - ..
';'-,"

this

reper t describes the

the

Delphi research

methodology_

Section dispositional the are

III

address dimension both

skill

dimension Research

of CT, and Sect.ion findings are

IV the

of CT. text

presented

throughout

report,

in the

and in tab'.llar

form.

Six recommendations most sensibly

presented

in Sections
rationale.

III and IV so they

Nine c.\ddltionc\l recommendations and assessment care presented

-. ..

can be reloted

with their to

which pertain in Section

specifically

CT instruction

V.

II --

Researr:JJ_ Methodology

and PUr:I3ose

The Committee

on Pre-College

Philosophy CT whom this controversial

suggested investigator issues

several

persons

with
as

special plJrt

e~~pertis"e in CT and inquir'y into the

might contact

of the

known to lie decided the to

at the
employ

heart the

of the

professl.on's

concer·n. research the

This investigater methodology

power"ful q~lalitative

known as

Delphi Method. panel ef and work fir~ ~

The Delphi experts. toward group

Q

Methed reql.lil'·ss Thes:;e persons

consensus

formaticn
be willing

of an interactive to share their

must

expertise Using the soon teck

r~solution

of ma.tters of opinion. othurs, the Delphi panel by their

of

~~perts

to nominate
persons, special

shapp.

In all forty-six
coUeagLles ,l\ssessment project.

widely recognized

professional in CT ~nstruction, in this Delphi

to have or

e)~perienc:e and expertise commitment to cOI,scientir:~s

theory, were

made tt.e
not

partj~lpate effort,

If it
is

f(.)r their

(for' which this

il1ve'5.tiga.tor
" repor"t

e:.:tremeiy

dPprec:ia.tivp~, the

consensus

exprp.ssed

in thi.s

cou ld :lot

h~VE~been

re;>~ched. participate in several rounds of Achieving a of

III Delphi research q\.testions ,:on'sensus which call of expert

eHperts

for~ thoughtful opinion

using

and detailed

responses.

the

Delphi Method is

not a matter

"...

voting cr tabulating toward consensus reconsider offered

quantitative their

data.

Rather the expert

panelists

work

by sharing

reasoned

opinions and being willing to and argument~ ~xpresses

them in the light of the comments, objections experts. In Delphi research,

by other

once an expert

an opinion, even a dissenting

....

one, it becomes a factor

!.~I the mix and

flow of all sL,bsequent argLlment and thought. influence

TC't ircumvent undue c status, each round are to the

carising from any given expert·s p.-cfessional is initiated through that by the p.r:.;ject director
...~.

Q·f questions

and all responses circulates

coordinated entire

p~~'$on. The project and synthesized

director

panel directq.:.;.,6tations aut.hpys removed. panelists of their

responses,

with the name!:;

of their

Th~ expert p,t:rsuasiveness The project direction fruitful assists sources

t.hemselves, through the thoughtfulness written responses,

and

shape the line of inquiry. which respond· to the toward director useful

director

endeavors

to frame questions

panel debate resolution. the panelists of relevant

is taking and lead the conversation the project

As the inqLliry proceeds, with bibliographies informaticn.

and alerts

them to other

As areas

cf accord or disagreement of

.:.nerge these preliminary

are presented

to the panel in the form of drafts follow-!..,pQuestions. c..~tGrcninesthat

oJ

findings or crucial

The process sufficient accord

tenninatr:?s when the project has been reached

director

for ar'eas of consens~

to be made public. disagreement

Delphi

findings als~ include descriptions statemp.nts A clear of minority opinion. and ciccurate

o.f residual

and

conceptualization

of CT is absolutely tools and effective that and

essential CT

for the development of valid CT assessment ill~trllctional div~rgent programs.

'Ht.h this in mind, and· recognizing

conctOlptualizations o·f CT have hindered curricular

, -" '.' =~ .'

~,

! ; ~.

..

:':'.\~'

.......

--

..

,' ."

...........

assessmf~nt efforts,

early

in the Delphi process

the panel decided its of a clear and devoted their

by coming to

most worthwhile ~Qntribution could be the articulation correct conceptualization toward that of CT. end. The expert panelists

major effort ,:onsensus instruction richness

The eHperts hope~ that interested

they could offer

....

educators

in CT assessment clarity,

or

a conceptualization to warrant their

of Cl' of sufficient attention.

accuracy and

serious

To balance the theoretical themselves cri tical describe what a generally

with the practical,

the experts

asked

educatad

college lower division

level to

thinker

should be able to do.

However, they did not attempt thinker. It soon became an ideal.

the typical

college level critic:al were actually

f~"ident that that

the experts

articulating

It may be the

no person is -{:'-tlly adept at found to be central all the affective

all the skills

and sub-skills

experts

to CT. It may be that dispositions

no person has fully a good in ways that This gives

cultivated critical

which characterize their lives

thinker.

Also humans compartmentalize and evident in some areas

CT is more active no more reason system

than in others.

to abandon the effort knowing no friendship

1'Me experts'

to infuse is perfect purpose

CT into the educational gives one reason in putting the ideal to

than that

de'Spair of having fr~iends. before the educatiol'

community is that

it should serve

as a rich and

worthy goal guiding CT assessment educational levels.

and curriculum development at all

l

_",~::~\~>:·:~·.t . '..
.
.,';., .

.i .••••

~/:.-::-;r.~~ :".'..~-"''''.:>~:..~~~~-~,,/t.;:-,.; ". .."~... .... ·

"

~:~.~

_,_,

_.

__

..;.....o..___'__""""""".

"T""~~

""___

_.~

••

TABLB2 PROJECT HISTORY Round 1 (P"b. 11, 1988) and Round 2 (Mar. 14, 1988) initiated the Delpbi process. In both roundB panelists were invited to nomb18t~ other experts to join in this research project. The experts reached consensus on tbe working assumption that "the CODcept f cr could be made operational to the e~eDt that important parts of CT o could be auessed ¥&lidly and reliably." The experts agreed to be!iD their analysis 01 CT by "identifying the core elements of cr which might reuonabl, be expected

cr

at tbe freshman and sophomore general education college level." The rationale for this decision was tbet the college level theoretical c:orustmctof CT could reasonably be used to guide wbat might be said about cr at the K-IZ level. Also the panelists Doted that most of the participatibl ezperts had greater ezpedence at the college level than in Ie12 education. Round 3 (MI.'Y 1988) wu an opea·-ended invitation for experts to write their own 4, list of the operations which they conceived ~f,Jf central to CT. The first syntbesis of this as input was presented for expert review in Round 4 (Sept. 23, 1988). This synthesis focused on the skill dimension of CT. Round 4 invited responses regarding each skUI and sub-skill identified, a proposed [and ultimately rejected] input/output model of CT operations, a list of closely related cognitive operations which might or might not be

distinguished from cr, a general statement regarding what a skiU is and how one is taught, and ill list of caveats and cautions regarding CT instruction aud assessment.

Rountl SA (Feb. 28, 1989) reviewed the definitions and classification of CT cognitive skiDsin the light of expert responses to Rouad4. RDuud S8 (also Peb.28, 1989) proposed sblltements regarding the dispositional dimension of CT and about its possible normative Cf)nnotations. RoundSC (Mar. 10, 1989) aslced forspeclfic recommendations regarding (:f instruction ana assessment, and offered a revision of the general statement (tn teaching and assessing 8 cogYlitive skill. RdUDd S included several quotatioDScuiled from the panelists' earlier responses and invited comments and reactions. The f~Xperts' comments regarding the "arious quotations included in each round added greatly to the project directors understanding of the experts' overaU views. From these and the responses to specific Round SA, SB and SC questions, the project director assembled u draft report of all Delphi findinl~, including recommendations. Round 6, (Sept. 25, 1!'89) circulated that draft and gave the CT experts the opportunity to express their views or make comments for inclusion in the final report, which went through its last revisions in Nov. 1989.

,
. ··f

III

--

The Cogp.ltlve Skill

Dimension of Cnticsl

ThiokiDR

FINDING: As indicated in Table 1, the expert~ find good critical thinking to include both a skill dimension and a dispositional dimensinn. The experts find CT to include cognitive skills in (1) interpretation, (2) analysis, (3) evaluation, (4) inference, (5) explanation and (6) self-regulation. Each of theso six is at the core of CT. Associated Nith each are criteria by which its execution can be meaningfully evaluated. However, no attempt is made here to specify those criterii since ample criteriological discussions exist in the literature.
Concernednot to generate misunderstandings, the experts offer cautions about the analysis of CT in terms of skills experts warn that and sub-skills. many The

good CT is not rote, mechanical,unreflec:tive, They caution not many

disconnected e){scution of sundry cognitive processes.

to lose sight of the whole while attempting to attend well to its parts.

RECOMMENDATION 1: All CT instruction should aim at developing good critical thinkers -- persons who can integrate suc~essful execution of various skills in the CT enhanced classroom Nith the confidence, inclination ~nd good judgment to use these pawerful tools in their other studies and In their ev@ryday lives. Persons who have proficiency in CT skills but fail to use them appropriately are most unlikely to be regarded as good critical thinkers. RECOMMENDATION 2: Those Nno seek to infuse CT into the educational system to be guided by a holistic conceptualization of what it means to be a 900d critical thinker. That some aspects of CT, particularly features Hithin its skill dimension, are more readily targp.ted by existing educational assessment strategies should not distort the conceptualization of CT nor truncate full-blown CT instruction.
The exper·t.scharacterize CT s~dlls.

certain

cognitive

skills

as

central

or core

The more one achieV'es proficiency in these skills,

the more are not,

to

worthy one is of being regarded as

adept at CT.

The experts

however, scayirltJ trat

perceived
c..~S

a person must be proficient

at every skill

be

.,

h,,~viflg CT ability. trying

Considering the panel's pl~"poses and of necessary a.ndsuf ficient Thus, in view of the

IIlC'thorJology,

to an~lyze CT in terms

conditions would have hau strcJng negativf~ utility.

11

~~:!~
·-t .

..

pl"ec.:isifm ~~hic:ht.he questicJn permits,

the

panel,

ear-ly in the Delphi The panel

not a

p,."oc.:ess,dec:ided to strive

for a c:onsensus on the core skills.

without whic:h a person is surely

was not asked to name skills

cr"itical thinker.

Responses unerumcus cepJ ral. to

to Rounds

4 and 5A reveal

the experts

to be virtually
as

(N)95Y.) on ir":luding analysis, CT. BI_\t in response th:= rnctu-srcn

evaluatic:m,

and inference

to Round 6 one assessment

of

expert
it

str·or.I:Jly
was

dissented
prop~rly

regarding a pdrt of

interpretation,

arguing that ncted

c:ommunication,

~ot CT.

The same expert

that

anal ysis, a'6 defined in this These

pointE'

repor-t, overlaps
difficulties for

with reading CT assessment,

between

and listening. particularly

r'C:lise bvious o

as on~ alt~mpts

to make finer

differenliations

CT and

cUlnmllnice\tion or- between n:~adin'J-·'5ens~. Regat"cl.ing

an.alysis-in-the-CT-sense self-r"egLll~tion with the

and analysis-in-the"I thin~;: this

to to Round 6

e~<pert said,

hi where te-·:>ting must mertJe

another
6.SSeSsmE'nt

teaching.1I

In response
as compar"ed

e:~pr::'r-·t pc:dnted out

that,
of

the

others,
In

self'-r-etJulc,tion self-r-eguldtion

dppear-s to be a skill
one c'pplies
or:~'s

a. differ"ant

kind or- level.

the

other

CT skills This

to

one's own CT, by, for

CT an interestingly the meta-cognitive Llsing strong

'"i'xc:.'\mple, t?vci\lL\c:sUfll;J

o\~n inferences.

gives

r-,c;.!Ct.'t·',;i,,~Lhd.r"~I.:b:w.

However, as this iflr~kes it

e>:pert noted,

~Sf'l'?ct of self--rsfJ,-\la.tion

e:<tr'emely difficult irlstr-uments.

t.o assess Nonetheless,

the

~t-.and<::\rd kinds

of

paper- a.nd pencil

cOlls~nsu<;:)

(N87~·~)t-·Y.ist:s tl1at interpretation,

an:;, ,:2fltl'"dl

e:::plc:\flation and self::ice the

retJ'-'le.~tion
t~bles

l o CT.

[For-

deta.iled results
for

response

un p":\ge 10 ~)f the Delphi letter

Round 5A in Appendix C.l

9
: I

-.~ I, ~

.t,

FINDING:

:.

~.'
_.

... -,

There is consensus that one might improve ane's awn CT in several Nays. The experts agree that one could critically tUamine and evaluate one's awn reasoning processes. One cauld learn ho~ to think mare objectively and logically. One could expand one's repertoire of those mare specialized procedures and criteria used in different areas of human thought and inquiry. One could increase one's base of information and life experience.
It was readily apparent that the experts do not regard CT as a body along

of l::nowledgeto be delivered with others. applications The panel sees in all areas

to students

as one more school subject

CT, like reading and writing, as having and learning. And, as several pointed

aT life

out, CT instruction,

like reading and writing, can occur in programs rich content or in programs ~Jhichrely on the events for developing one's CT.

with discipline-specific:

in everyday life as the basis

One implication the experts draw from their analysis of CT skills is this: "while CT skills themselves transcend specific subjects ar disciplines, exercising them successfully in cert~in contexts demands domain-specific knowledge, same of which may concern specific methods and techniques used to make re~sonable judgments in those specific contexts.
II

FINDING

Although the identification significant these skills ways, specific

and analYSis of CT skills or disciplines,

transcend,

in

subjects

learning and applying ~~nowledge. This

in many contexts

requires

domain-specific

demain-specific principles

knowledge includes understanding

methodological practices that The are

and competence to engage in norm-regulated judgments in those conceptual,

at the core oT reasondble explicit

specific-contexts.

mention of "evidential,

methodological, in connection with if CT is

I.:r'j, t.er"iologiLcd,

or c:onte)ttuC\l"considerations

expla.natiun rE~infon:es this point. r,:onceived of simply a.s a list

...:.. :".

Too much of value is lost

of logical operations

and domain-specific: of information • application can

knowledge is clJf"lceivedof simply as an aggregation Inquir'y into the nexus of reasonable

judgment and actual

10
...... ...',

13

prcdl.\c:e new appreciations

of the nec:essity

of robust

c:onc:epts of both CT

and domain-specific knowledge in educ:ation.

RECOMMENDATION 31 Since becoming adept at CT involves le~rning to use CT skills effectively in many different contexts, the experts insist that "one cannat overemphasize the value of a solid liberal education to supplement the honing of on~'s CT skills and the cultivating of one's CT dispositions."
The experts c:lassified c:aution that CT skills c:an usefully be grouped and sub-

in a number of legitimate from this

ways.

Hence, the sub-c:lassification as to

whic:hresulted

Delphi research

should net be interpreted

necessaril y e>:cluding all others. be in agreement with this experts areas

Indeed, while dec:laring themselves various participating

sub-c:lassification,

have also published their of overlap

own sub-classific:ations.

There are

in the c:lassifit:ation

system which emerged from the eac:h skill and sub-skUl simply to forc:e eac:h

Delphi r-esearch.

However, while charac:terizing differentiations

is important, c~reating ar-bitrary and every sub-skill is rJeither l1eceStsary

s.oloe

to bec:ome c:onceptually disc:retlrt from all the others nor useful. In prac:tic:al c:ontexts the exec:ution of others. Thus, order of the

skill~ or sub-skills

may presuppose

Delphi Hsting is nClt intended to imply the endorsement of any psychologic:al, logical or epistemological prescribing any educational the skills or-der or skill-sequenc:e, nor as

taxonomy or skill-hierarc:hy. and sub-skills whic:hthe experts identify

Table 3 lists as being at

the core of CT. No claim is being made that breadth or detail.

the list Beyond their are

e~;hal.lsts the concept of CT in either inclu!iion in CT, ma.nyof the skills valuClble, if not vital, for other

a.nd sub-skills

identified

important

ac:tivities,

suc:h as

c:ommunicatingeffectively_ other technical

Also CT skills

can be applied in c:oncert with

or if Iter-personal skills

to any number of spec:ific:

11

14

,r:O ..

>

••

. ~.

·:11

concerns

such as programming comput.ers, defending clients, strategy. managing an office,

developing a figure

winninc;J sales

or helping a friend this

aut what might be wrong with his car. mean by characterizing is also fair particular success these CT skills

In part

is what the experts and purposet-ul. or a for The It.

as pervasive

to say that

a particular

skill, such as evaluation, is essential

sub-skill,

such as developing reasons, such as properly various skills

in a given endeavor,

diagnosing illness. and sut'''·skills

expert.s are not concerned that used. other It is not a problem that endeavors.

are widely elements in if

the skills

might be essential

On the contrary,

it would be extrl:!mely disconcerting the educational and learning. useful cognitive

they were not, since t.he case depends un eT's utility The experts across

for infusing CT into almQst all areas

system

of life

are clear

on the point that

not every

TABLB3
CONSBNSUS LIST or CRITICAL THINKING COGNlTIVB SKILLS AND SUB-SKILLS

t. Interpretation

• Categorization • Dec:od.iD& Significance • CarifyiD& Meaning • Identifyir.g Arguments • ADalyzir.& Arguments

2. Analysis

• Bxamiaingldeu

3. Bvaluation 4. Inference

• Assessing Caims • AssessiDg Arguments • Queryinl Bvidence • Conjecturing Alternatives • Drawina Conclusions

S.Expianation

• Justifying Procedures
• Presentia& Arguments 6. Self- RegulatiOJl. • Self-examination

• Statial Results

• Seif-eorrec:tioD
-/)"':.-,'.0; .

.t5

process CT skill. order'

should

be thought

of as

CT.

Not every

valuable

Lhinking skill of higher-

is

CT is one among a faPail y of clasel y rela ted along with, for e>:ample, problem~sol"ing, Unfortunately the the

forms

thinking,

decision overlaps and

making, cmd creative complex relatior'ls=hips thinking ha.ve yet to

thinking. ,ollsnongll a

conceptual

vari.ol.~s forms

of higher-order However
n ,

be examined

satisfactorily.

that

does

not

imply that
the

one cannot CT --

develop

a careful

and accurate fully adequat~

conceptualization

of

target,

a conceptualization

to '~trpCse,

which is to guide In addiUofl find

remar-kable

CT assessment on the
on

and instruct.ion. listings in Table 3, the Delphi e>:perts skills and

to acccrd consensus

the

descriptions

of ea.ch of the in Table as 4.

sub-·:;kills.
as':>c.Jciatt;ld

These

descriptions

a.re presented
i:\re

The e:·,a.mples
Some

4~ith ea.ch sub-skill

intended

clarifications. instructional tools to

reader~ ~ight see

assessment staff

in the~ suggestions Others

of possible

or initiate

strategies.

might see i~bout the has

in them the curri<...ular

development

conversations

implications.
and sl.lb-skill

H'.lwever',

t.he ;:t.:\nel's c:or,sensus

Lo do with the

skill the

descrip t ior~s, ctnd does

not

necessarily eHtend to

examples.

------------------TABLE 4
CONSENSUS DESCRIPTIONS 1. INTERPRETATION: OF CORE CT SKILLS AND SUB-SKILLS
To comprehend and express the meaning or significance of a wide variety of experiences, situations, data, events, judgments, conventions, beliefs, rules, proced~res or criteria. to apprehend or appropriately formulate categories, distinctions, or frameworks for ~nderstanding, describing or characterizing information. * to describe experiences, situations, beliefs, events, etc. ~o that they take on comprehensible meanings in terms of appropriate categorizations, distinctions, or frameworks.
For ~~impl~: to retognize
d

1.1 CATEGORIZATION;

~roblem

and

defin€

its

character

13
." -: '\_

...

16

. .'~:~'~;{_i~.
: ,~.
(
~"

,;. ~. . ;_:

without prejudice to inquiry; to detfrmine a useful way of sorting and subMclassifying information; to make an understa,'dable report of what one experienced in a given situation; to cla~sify data, findings or opinions using a given tlassification schema.

to deteot, attend to, and describe the informational content, affective purport, directive functions, intentions, motives, pur~oses, social significance, values, views, rules, procedures. cr.iteria, or inferential relationships expressed in convention-based oommunication systems. suoh as in language, social behaviors, drawings, numbers, graphs, tables, charts, signs and symbols.
For example: to detect ~nd describe a person's purposes in asking a given question; to appreciate the significance of a particular facial expression or gesture used in a given social situation; to discern the use of iro~y or rhetorical questions in debate; to' interpret the data displayed or presented using a ~articular form of instrumentatlon.

1.2 DECODING SIGNIFICANCE:

to paraphrase or make explicit, through stipulation, description. analogy or figurative expression, the oontextual. conventional or intended meanings of words, ideas, concepts, statements, behaviors, drawings, numbers, signs, charts, graphs, symbols, rules, events or ceremonies. * to use stipulation, description, analogy or figurative expression to ~emove confusing, unintended vagueness or ambiguity, or to design a reasonable procedure for so doing.
~or example: to restate what a person said using different words or expressions while preserving that person's intended meanings; to find an example which helps explain something to someone; to develop a distinction which makes clear a conceptual difference or removes a troublesome ambiguity.

1.3

CLARIFYING HEANIN'G:

2. ANALYSIS: To identify the intended and actual inferential relationships among statements, questions, concepts, descriptions or other forms of representation intended to express beliefs, judgments, experiences, reasons, information, or opinions. to determine the role various expressions play or are intended to play in the context of argument, reasoning or persuasion. to define terms. to compare or oontrast ideas, concepts, or statements. to identify issues or problems and determine their component parts, and also to identify the conceptual relationships of those parts to each other and to the whole.

2.1 EXAMINING IDEAS:

* * *

For example: to identify a phrase intended to trigger a sympathetic emotional response which might i~duce an audience agree with an opinion; to examine closely related proposals

to

14

17

I
,'_: .'

regarding a given problem an~ to determine their rOints of similarity and divergence; given a complicated assignment, to determine hOH it might be broken up into smaller, more manageable tasks; to define an austract concept.

* given a set of statements, desoriptions, questions or graphic representations, to determine whether or not the set eKpresses, or is intended to express, a reason or reasons J.n support of or contesting some olaim, opinion or point of view.
For example, given a paragraph, determine whether a standard reading of that paragraph in the context of how and where it is published. would suggest that it presents a claim as well as a reason or reasons in support of that claim; given a passage from newspaper editorial, detprmine if the author of that passage intended it as an expression of reasons for or against a given claim or opinion; given a commercial announcement, identify ~ny claims being advanced. along with the reasons presented in their ~upport. 2.3 ANALYZING ARGUMENTS:

2.2

DETECTING ARGUMENTS:

given the expression of a reason or reasons intended to support or oontest some claim, opinion or point of view, to identify and differentiate: (a) the intended main conolusion, (b) the premises and reasons advanced in support of the main conclusion, (c) further premises and reasons advanced as backup or support for those premises and reasons intended as supporting the main oonolusion, (d) additional unexpressed elements of that reasoning, such as intermediary conclusions, unstated assumptions or presuppositions, (e) the overall structure of the argument or intended chain of reasoning, and (f) any items contained in the body of expressions being examined which are not intended to be taken as part of the reasoning being expressed or its intended
background. For example: given a brief argument, paragraph-sized argument, or a position paper on a controversial social issue, to identify the author's chief claim, the reasons and premises the author advances on behalf of that claim, the background information used to ~upport those reasons or premises, and crucial assumptions implicit in the author's reasoning; given several reasons or chains of reasons in support of a particular claim, to develop a graphic representation which u s e+u l l y ch er e c t er i z e s the inferential flow of that reasoning.

representations which are acoounts or descriptions of a person's perception. experience, situation, judgment, belief, or opinion; and to assess the logical strength of the actual or intend inferential relationships among statements, descriptions, questions or other forms of representation.

3. EVALUATION: To assess the credibility of statements or other

3.1 ASSESSING CLAIMS:

to recognize the factors relevant to assessing the

1·5

",
•f''':,.

.
_. I':~:~\,: •• •

18
.:,01.'

';~/~.'~~ .

·.,

.'

degree of credibility to ascribe tQ d source of information or opinion. to assees the contextual relevance of questions, information, princi.ples, rules or procedural directions. to asseES the acceptability, the level o~ confidence to place in the probability or truth of any given representation of an experience, situation, judgment, belief or opinion.

* *

For example: to recognize the factors which make a person a credible witness regarding ~ given event or credible authority on a given topic; to determine if a given principle of conduct is applicable to decidin~ what to da in a given situation; to determine if a given claim is likely to be true or false based en what one knows or can reasonably find out.

to judge whether the assumed acceptability of the premises of a given argument justify one's acoepting as true (deductiy~ly certain), or very probably true (inductively justified), the expressed conclusion of that argument. to antioipate or to raise questions or objections, and to assess whether these point to significant weakness in the argument being evaluated. to determine whether an argument relies on false or doubtful assumptions or presuppositions and then to determine how crucially these affect its strength. to judge between reasonable and fallacious inferences; to judge the probative strength of an argument's premises and assumptions with a view toward determining the acceptability of the argument. to det·ermine and judge the probative strength of an argument's intended or unintended consequences with a view toward judging the acceptability of the argument; to determine the extent to which possible additional information might strengthen or weaken an argument.

3.2 ASSESSING ARGUMENTS:

* *

g i Ii en:· p 1 rl 1 2 ~I,

For example: given an argument to judge if its conclusion follows eith~r with certainty or with a high level of confidence from its premises; to ch~ck for identifiable formal and informal f~llacle~l given an objection to an argument to evaluate the l~ ~;:~1 ferce of that objection; tc evaluate tn~ qua!l~~ ~nj applicability of analogical argumerl!.sl 'r: ~'.ldCJe the log.i;:?1 strength of uguments based C,' r~·· ...0 Q t ~~t 1 C c11 's 1 t '.1 e1 t ion s C' r causal rea -:;c n i n 9 ; l 0 j LI d 9 e if a given ~rgument !~ relevant or dpplicable or has implirations for the s i t ue t i on clt t:and; to determine h o a possible new data mlyh~ lead logic~lly to the further confirmation or distonfirmation of a

reasonable conclusions; to form conjectures and hYrotheses; to cOIlsider relevant information and to educe the consequences flowing from data, statements, principles, evidence, judgments, beliefs, opinions, concepts, descriptions, questions, or other forms of r~presentation,

4: INFERENCE: To identify and secure elements needed to draw

16

19
...
'

in particular, to recognize premises which require support and to formulate a strategy for seeking and gathering information which might supply that supPort. in general, to judge that information relevant to deciding the aooeptability, plausibility or relative merits of a given alternative, question, issue, theory, hypothesis, or statement is required, and to determine plausible investigatory strategies for aoquiring that information.

4.1 QUERYING EVIDENCE:

For example; when attempting to develop a persuasive argument in support of one's opinion, to judge what background inform~tion it would be useful to have and to develop a plan which will yield a clear answer as to whether or not such information is available; after j~dging that certair. missing information would be germane in determining if a given opinion is more or less reasonable than a com~eting opinion, to plan a search which will reveal if that informatiun is available.

* to formulate multiple alternatives for resolving a problem, to postulate a series of suppositions regarding a question, to project alternative hypotheses regarding an event, to develop a variety of different plans to aohieve some goal. to draw out presuppositions and projeot the range of pO~5ible consequences of decisions, positions, policies. theories, or beliefs.

4.2 CONJECTURING ALTERNATIVES;

For e~ample: given a problem with technical, ethical or budgetary ramifications, to develop a set of options for addreSSing and resolving that problem, given a set of priorities with which one mayor may not ~gree, to project the difficulties and the benefits which are likely to result if those priorities are adopted in decision making.

to apply appropriate modes of inference in determining what position. opinion or point of view one should take on a given matter or issue. given a set of statements, descriptions, questions or other forms of representation, to educe, with the proper level of logical strength, the~r inferential relationships and the consequences or the presuppositions which they support, warrant, imply or entail. to employ successfully various sub-species of reasoning, as for example to reason analogically. arithmetically, dialectically, scientifically, etc. to determine which of several possible conclusions is most strongly warranted or supported by the evidence at hand, or which should be rejected or regarded as less plausible by the information given.

4.3 DRAWING CONCLUSIONS:

*
*

For example: ~t~tistical an empirical

to carry out experiments and to apply appropriate inference techniques in order to confirm or disconfirm hypothesis; given a controversial issue to examine

17

informed opinions, consider various opposing views and the reusons advanced for them, gather relevant information, and formulate one's own considered opinion regarding that issue; to deduce a theorem from axioms using prescribed rules of inference.

5: EXPLANATION: To state the resul ts of one 's re.asoning; to justify that reasoning in terms of the evidential, conceptual, methodological, criteriological and contextual considerations upon which one's results were based; and to present one's reasoning in the form of oogent arguments. 5.1 STATING RESOLTS: to produce accurate statements, descriptions or representations of the results of one's reasoning activities to analyze, evaluate, infer from, or monitor those results.

so as

For example: to state one's reasons for holding a given view; to write down for one's own future use one's current thinking about an important or complex matter; to state one's research findings; to convey one's analysis and judgment regarding a work of art; to state one's considered opinion on a matter of practical urgency.

to present the evidential, conceptual, methodological, criteriological and oontextual considerat~ons which one used in forming one's interpretations, analyses, evaluation or inferences, so that one might accuratelY record, evaluate, descrihe or justify those processes to one's self or to others, or so as to remedy perceived deficiencies in the general way one executes those prooesses.
For example: to keep a log of the steps followed in working through a long or difficult problem or scientific procedure; to explain one's choice of a particular statistical test for purposes of data analysis; to state the standards one used in evaluating a piece of literature; to explain how one understands a key concept when conceptual clarity is crucial for further progress on a given problem; to show that the prerequisites for the use of a given technical methodology have been satisfiEd; to report the strategy used in attempting to make a decision in a reasonable way; to design a graphic display which represents the quantitative or spatial information used as evidence.

5.2 JUSTIFYING PROCEDURES:

to give reasons for accepting some claim. to meet objections to the method, conceptualizations, evidence, criteria or contextual appropriateness of inferential, analytical or evaluative judgments.
For ex~mple: to write a paper in which one argues for a given ~osition or policy; to anticipate and to respond to reasonable criticisms one might expect to be raised against one's political viewsl to identify and express evidence and counter-evidence intended as a dialectical contribution to one's own or another person's thinking on a matter of deep personal concern.

5.3 PRESENTING ARGUMENTS:

* *

18

21

,. ..: .'

8: SELF-REGULATION: Self-consciously to monitor one's cognitive activities, the elements used in those activities, and the results educed, particularly by applying skills in analysis and evaluation to one's own inferential Judgments with a view toward questioning, confirming, validating, or corr~cting either one's reasoning or one's results. to reflect on on~'s own reasoning and verify both the results produced and the corr~ct application and execution of. the cognitive skills involved. * to make an objective and thoughtful meta-cognitive self-assessment of one's opinions and reasons for holding them. to judge the extent to which one'e thinking is influenced by deficiencies in one's knowledge, or by stereotypes, prejudices, emotions or any other factors which constrain one's objectivity or rationality. to reflect on one's motivations, values, at~itudes and interests with a view toward determining that one has endeavored to be unbiased, fair-minded, thorough, objective, respeotful of the truth, reasonable, and rational in coming to one's analyses, interpretations, evaluations, inferences, or expressions.

8.1 SELF-EXAMINATION:

For example: to examine one's views on a controversial issue with sensitivity to the possible influences of one's personal bias or self-interest; to review one's methodology or calculations with a view to detecting mistaken applications or inadvertent errors; to reread sources to assure that one has not overlooked important information; to identify and review the acceptability of the facts, opinions or assumptions one reliej on in coming to a given point of view; to identify and review one's reasons and reasoning processes in coming to a given conclusion.

where self-examination reveals errors or deficiencies, to design reasonable procedures to remedy or correct, if possible, those mistakes and their causes.
For example: given a methodological mistake or factual deficiency in one's work, to revise that work so as to correct the problem and then to determine if the revisions warrant changes in any position, findings, or opinions based thereon.

6.Z SELF-CORRECTION:

19

22

IV -- The Dispositional Dimension of Critical Tbinking

As is evident, and self-correction, thinking. Indeed each particularly there are in the descriptions of self-examination to critical

di5pOs~.tional components skill, if it is

cognitive

'to be exercised disposition to do so. to

appropriately, In each have the case

can be correlated a person who is

with the proficient that skill,

cognitive

in a given skill even if at

can be said

aptitude using

to execute the skill.

a given moment the deal habits more many nf mind, good

per'son exper-ts

is not

But there to the

was a great traits,

wished s ... in regard ~y

personal

<i\tt:it:udes or affective cri tical thinkers.

dispositions

which seem to

characterize

FINDING: Although the language here is metaphorical, one would find the panelists to be in general accord with the view that th~re is a critical spirit, a probing inquisitivenesn, a keenness of mind, a zealous dedication to reason, and a hunger or eagerness for reliable information which good critical thinkers possess but weak critical thinkers do not seem to have. As water strengthens a thirsty plant, the affective dispositions are necessary for the CT s~,ills identified to take root and to flourish in students. RECOMMENDATION 4: Modeling that critical spirit, awakening and nurturing those attitudes in students, exciting those inclinations and attempting to determine objectively if they have become genuinely integrated with the high quality execution of CT skills are, for the majority of panelists, important instructional goals and legitimate t sr get s for e d u cat ion a las s e ssm en t • H 0 1'4 eve r, t ~Ie ex per t s tl arb 0 r n 0 illusions about the ease of designing appr~priite instructional programs or assessment tools. Procedural, Laudatory
The ey.perts dispositions expressed a.re part have
d

and Normative Uses vf the Term YeT"

regarding the list of affective This consensus affective cognitive is

consensus

which characterize in Ta.ble 5.

good critical

thinkers. these the first.

However, whether

or not way that from the

dispositions skills are,

of the

meaning of "C'F" in the the exper·ts

was an issue

which divided

It became evident

20
'::.: ."
.

23

.... !-.•

>",;,;.

that

various

experts

mean different

things when they used the term "eT" in components. two-thirds majority who
to

r·eference to its The ~eepest

hold

possible

dispositional

division is between the nearly

that

the term neT" includes in its dispositions

meaning a reference

certain

affective

dnd the roughly one-third skills

minority who hold that but not to affective

neTIl t-efet-s only to cognitive

and di~positions, put this issue

dispositions. differ-snt Responses,

The project

director

to the panel in several

ways, sometimes directly

comment.s C\nd arguments

and at other wer-e shared,

times more obliquely. as were the objections and

counter-arguments

which they engendered.

In the end the panel remained with opPOSing positions continued.

divided both numerically and in depth of feeling, b"='!coming more strident

In Round
and entr'enc:hed as

the debate

513, of those

expr-essing an opinion, the majority (611..) dispositions dispositions

the

Inaintairl t.hat

the affective

constitute

part

of the meaning of

"CT.II They drgue tha.t these

the ver"y concept th~t of

flew from, and are implied by, dispositions habitually are.

These

CT, much as being

cognitive
but

e~<pE"rts r.trgue

adept at

CT skills

not using them thinker at all.

c.\pprcpr··iatelydisqualifies
Thus,

one from being called a critical

p.... ocedural

in addition
LtSe "C'T"

to

!.Asing "CT" in its

sense. but

sense,

these

panelists

als

l)

in i ts Ieuaecorv

They find it sensible ather

to say, "This

perSc.111 is a cTitical

thinker,

this

person is so mentally lazy,

(.lcJse-mirlded, unwilling to check the facts dt-guments that

we silTlply use

and unmoved by reasonable thinker.'1 of hew well a person the parser. because the

canuo t call

him a critical
approval

The laudatory
dpplie'E>

CJf "C'I'" can suggest

h~r

CT skills

or it can convey praise dispositions.

for

pe.... stu·. has the p.... oper ~ffective m.=-.j~Jrity w-as eloqLtsnt regarding .:Jffective dispositions

While the two-thirds

the importance of finding ways to instill in the final analysis they were una.ble

in students,

21
.

:-~':

.. .
'

..';~ ,,:¥,"

_, ~l'· .'.....

~.:.

~ ~'. ,"~

:;::,",

.,
to persuade the
0

ther

third of their

expert colleagues

to view these
was, however, the

di:.positio.1s as essential persuasive affective Table 1.)

In Round :S8
C\

to the concept I.Jf CT.

The majority l.\5ing

in bringing about virtual dispositions to describe

unanimity regarding the paradigm critical

thinker. (See

minority (30~) insist

on using

"eT" in a strict
judgmental of critical

procedural ~ense, that is as referring prcc~ss.

thir)/dng

only to a certain true

They distinguish from what is true

sharply between what: is of good criticdl

thinkers.

Their primary thinkers are

c.oncern is with the CT skills. people who have those skills

They argue that

and certain

good ct-itical

valuable habits

as well.

If they

ar·e good cl'-itical thinkers,

then they llse their' CT skills

appropriately

because good cr-itical thinkers t1ispositions listed

in Table 5.

also have some or all of the affective But those dispositions are not what is is not

mednt by 'leT.'1
a critical

They argue that

one would not want to say a sophist

uses CT skills

I:hinker simply b~caLlse the sophist

ends.

fer deceptive

or sel.'f-iJlter'~sled

The sophist,

they would lTI~intain, is a c:ritical

senss). The strict

thinl-:er --

but not an good one <in a.n ethical

do not find it
lJecdUSI:l

pr·o,:eduralist,s

sensible

deny that

a person

is a critical

thinker simply

the

person,
LIp

while skilled in CT, fails

to check the

crtJdibll it 'I uf <':)c.Il.ln:es,

'Jives

too soon when asked to wor'k a challenging problems, or because of

(.Jrr...:;t.lem, lacks confidence in using ,...~asan to approach everyday i'jnores painfu.l facts. These experts hold that such a person,

hi.~ CT ..akills, shol_lldbe called a critical

(ill

thinker -skills).

but not a good one,

t~rms of his effl?ctive

Llse

of those

As <;.uggest,~dabove, lhere

are two senses the phrase

of the term "good" which thinker.1I One

might be op~".ating when one uses sense

applif!:'s

"good critical

to the thinker's

effectiveness

and r·esponds to the question,

22 "

25

"How well is thinker's

this

person

using

CT?n to

The secorld the question,

sense

applies

to

the use' of wished CT might

morality

and responds to clarify

the

"Is this

person's experts that

CT ethical?" to convey,

In order

whic.:hsense to

of

ngood" the

Round 4 asked

panel

respond to
to

a proposal

have

a normative

dimension

in addition

a skill

dimension and a affec:t.ive

disposi tional

dimension.

FINDING: The mistaken notion that CT has a normative component is rejected by the expert panelists. It is an inappropriate use of the term to deny that someone is engaged in CT on the grounds that one di sapproves ethi call y of "Ihat the person is doi ng. What "CTn means, why it is of value, and the ethics of its use are best regarded as three distinct concerns.
The majority of experts
nCT.II <527.) forc:efully

reject

the to

proposed say
What

normative use of
someth~ng is, person's
su"ffers skillS

They hold that thing are to say

it

is one thing

and another

how it ought

t,e be used.
the

and attitudes ethic:al

what the)'

c:\re, even if

person

from certain

inadequac:ies.

Onl)' a small group

r.1~rma ve sense. ti

(17%), argue

in favor

of using all
of

"C'F" in a use
nCT" in

This minority laudatory

of experts, sense,

whom also

its

commonly

understood

hold that
nor-oms and a

the

true

meaning of
For

"CT" e~~tends to example, skills to they cause

a c:er-tain set

of ethical say that

social

values.

WI.Juld

be willing to or
of

defense

attorney client

who uses ought not the

CT

a mistrial title

win ac:quittal critic:al to

for

a guilty

be dignified

with the

thinker.

By the same token, to mislead is since in the not

pro'!>er;utor who use~

jll(

CT skills

contrive

a way

a gullible a critical
appear

Y into

corivrc t.inq and punishing

Sirrce

an innocent

psrson
cg.nd

thiph:~r.
lad:: the

nt:?itl-~er SLtfficiently fiber to

valLte truth

both

to

,nor'al

e!-~chewdelibE:?rate deception neither attorney

practice

of

their

socially

Import.an t; professions,

should

be accorded

26

.,""" ;""

. e

· :":

'"~"'~I .~., "-"."'

"

"

~~:

the moral approval which calling them c:ritical

:

thinkers in another

would imp y. 1
way

-.

The debate consensus s)(press personal

turned out to be instructive

as well. should

The

(74h in support

and 4h opposed) was that support and appreciation

this

report

the experts' and social

fullest

of the immense

importance of CT. a solid consensus force about the importance of CT as a as a powerful

The panel shares

tool of inquir'y, as a liberating

reSOLlrce in one's

in education,

personal

life,

and as a vital

component in a rational any panelist purposes. would However the use

democ("atic society.

It is 9>:tremely unlikely that or unjust

condone using CT for immoral, deceptive, personal

dre not

and civic value of CT and sensitivity acceptable grounds far

to the morality of its

building a normative dimension into the as misguided and

meaning of the term nCT". potentially destr-uctive

Some aven saw such an effort

of the CT movement. Giving "eT" a normative limitations

011

twist could, the'l" argue, lead to unwarranted and b:l unjustifie.\ble ideological being a "thinking" person. restrictions

on open inquiry

the very conc:ept of this conjures earlier. up is

The totalitarian critical

specter

t.he antithe'Eli.s of the liberating

spirit described

Dispo~"iti()"s

of' the Good Critical

Thinker

FINDING: To the experts, a good critical thinker, the paradigm case, is habitually disposed to engage in, and to encourage others to engage in, critical judgment. She is able to make such judgments in a wide range of contexts and for a wide variety of purposes. Although perhaps not al~ays uppermost in mind, the rational justification for cultivating those affective dispositions which characterize the paradigm critical thinker are soundly grounded in eT's personal and civic value. CT is known to contribute to the fair-minded analysis and resolution of questions. CT is a powerful tool in the search for kn~"ledge. CT can help people overcome the blind, sophistic, or irrational defense of intellectually defective or biased opinions. CT promotes rational autonomy, intellectual freedom and the objective,

24

.:~\~;:>'".' ..

~"\1'''

••

'--'

••

.<.~(:.; ',.,

: .... :'

<'..,~'~;;
..

.\.

......

reasoned personal

and evidence-based investigation and sQe~al issues and tonesrns.

to possessing CT skill~ affective

of a very

Nide

range

of

Thus, in addition

the goed critical or habits

thinker of mind.

c:an be charac:ter"izsd by certain

dispositions

These dispositions, listed in Table 5 belew, flew from two sources:

characteristics
ttllnkers, CT in its

which the experts judge to hold true of good critical

dispositions the expert£:i judge of the experts to be part (61X) regard of can be of

and t.he affective

fullest. realization.
listed

The majority

the dispositions
CT.

in Table 5 as part
(8::3X favor) in that

of the

conceptualization
thinkers

A consensus eHists

good critical

characterized

as exhibiting thesE: dispositions.

TABLBS AFFBCTIVB DISPOSITIONS OF CRITICAL THINKING

APPROACHBS TO LIPS AND LIVING IN GBNBRAL: • iDquisitiveness with regard to a wide range of issues, • concem to become 8JId remain generally weU-informed,

• alertness to opportunities to use CT, • trust in the proc:eues of reasoned inquiry, • self-confidence in ODe'. OWD abUity to reason, • opea-mindedneu reprdin& divergent world views, • ftezibiUty ill considering altemaUves and opinions, • understaadiD& of the opinions of other people, • fair-miDdedaess ill appraiaillg reuonin& • honesty in faciag one'. own biases, prejudices, stereotypes, eloc:entrie or sociocentriL tendencies, • prudence in suspendiag, making or altering judgments, • wiUiapess to reconsider and revise views where honest reOection sugests that change is warranted.
• cladty in statiol the question or concem, • orderliness in working with comple:Kity,

APPROACHBS TO SPBClFIC ISSUBS, QUBSTIONS OR PROBLBMS:

• diligence in seeking relevant iDformation,

• reuouableaess in selectinl and applying criteria, • care in focusiDS attention on tbe concem at hand,

• persisteDc:e thou ... difficulties are encouatered, • prectlion to the depee permitted by subject and circumstances • 2S
• 0

,...

~ ::-

~~;. \_:~.. .... .,...

28

...... ~
.....
' '

:'~

The experts epistemological, different Beliefs

are not saying that political, cultural

a person whose metaphysical, view of the world is thinker. to being

or religious

from ane's own is, ipso 'facto, net a good critical

are not atoms each of which is at any moment subject independently. Beliefs form intricately attention

reconsidered systems

interconnected

of thought. those

To focus critical mor'e central

on any of them,

particular'ly

or fundamental to one's own view of the throughout cne's entire belief system.

world, ca.r. cause

reverberations

Thus in ad\iocating CT the panelists confor·inity. Indeed, just

the values

are not urging ideological argued that an over-emphaSis on

as many experts

of CT could lead to trQuble, others

warn that

an over-emphasis

on the skills dispositions

dimension of CT to the exclusion of the affective might have the unfortunate inflexible result of making some students

closti-minded, intellectually

and dogmatic.

RECOMMENDATION 5: Just as with the cognitive dimension of CT, when conceiving of the education or assessment of critical thinkers, it is important to consider ways of developing materials, pedagogies, and assessment tools that are effective and equitable in their focus on these affective dispositions. The cultivation of thesp. dispositions is particularly important to insure the use of CT skills outside the narrow instructional setting. Persons who have developed these affective dispositions are much more likely to apply their CT skills appropriately in both their personal life and their civic life than are those who have mastered the skills but are not disposed to use them. As with the intend that experts
are

listing

of cognitive

skills

ea.rlier,

the

panel does

not

each disposition
characterizing c:rP:icdl

be considered

.a necessary forth

condi tion. The

the concept cf

the ideal.
they

In setting

the paradigm
all per

thinker,

intend

to e>:press

a gOnl taw~l~d which

might strive.
SOl

These '.ir·tLteS require

not commcmlv found

a meaSLlr-e of matLlrity
in college

io=t.fld
twelfth

,al development

saphomor-es or

gradt:?(·s.

Yet to u~l<;'IY

embad:ing

en the practices

and disciplines

which

2f

_""

".:';',';

:~.

"

will

lead to

these vir-tL,es

WOL\ld be an even more profound mistah:e.

RECOMMENDATION 6: From early childhood people should be t~ught, for example, to reason, to seek relevant facts, to consider uptions, and to understand the views of others. It is neither impractical nor unreasonable to demand that the educational system teach young people the habits of mind which characterize the good critical thinker, reinforce thos! practiceG, and move students well down tre path toward their attainment.

Y -- Further

Recommendations

on CT Instruction and Assessment

Severa.l pedagogic:aland assessment implications follow from the dispositional if dimensionof CT, implications which might not be apparent dimensionof CT. The education of students to execute a set of

educ:at.orsfocused only on the skill

good critical

thinkers is more than training

c:ognitive skills. evaluate critically uncritically

For example,in terms of pedagogy,modelinghow to that information which students would normally accept

and encouraging them to do the same can do wonders for CT ability. With this confidence themselves. Just as

developing their' confidence in their students are muchmore likely

to try thinking for

instn.ll:tion should not focus on shills only, assessment which focus on

s~dlls onl)r' may give a misleading or

inc:ompletepictu.re of someone's

,=,trengt.hs as a critkal

thinker.

The CT Goal

RECOMMENDATION 7: educational, personal aca~emic goal of CT should be furthering skills and affective

Brcause CT helps students wlth a wide range of and CIVIC concerns in a rational way, the instruction, regardless of the educational level, students in the development of their CT cognitive dispositions.

27

TABLB6

CONSENSUS STATBMBNT ON TBACHING AND ASSBSSING CT SKILLS

A CT skill, like any skill, is the ability to engage in aD activity, process or procedure. In general, baviDl a ,kill includes being able to do the right thing at the right time. So, being skilled at CT involves knowin&. perhaps impUcidy or witbout tbe ability to articulate this knowlecl&e, both a set of procedures and wbeu to apply those procedures. BeiDg skilled also involves having some degree of proficiency in eJ:ecuting those proc:edures and beiDg wiUiag to do so wben appropriate. ReSecting on and improviDl one's CT skills involves judging.ben one is or is not performing weD, or 81 weU u possible, and consideda& way. of improving one's perfOrDIaDce. LeamiDg CT involves acquiring the ability to make such self-reDective judgments. skiDs, can be taugbt in' a variety of ways, sucb aa by making the proc:edures expUc:it, describiag how they are to be applied and executed, explaininland modeMI their correct use, and justifying their app6c:atioD. Teaehiaa c:opitive skills also involves exposing leamen to situatioDi wbere there are good reuons to exerc:ise the desired, proc:edures, jud&ing their performanc:e, and proYidia& the leamen with constructive feedback t'eprdia& both their proficiency and ways to improve it. Instruction might start with situatic.ns that ale aditidaUy simple, but should culminate in situations that are realisticaUy complex. Padicularly ill the case of CT, the leamen must contribute a solid measure of personal effort, attention, practice, desire, and, u they learn how, self-monitoring. Teaching skiDs involves motivating learnen to achieve higher le~els of proficiency and, particularly in the case of CT, independence. It also involves coaching learners on how tbey can achieve those goals. Skills, particularly CT captive In tbeory there are several ways persons can be judged to be more or leu proficient in a given L"'T skill or at the integrated use of related CT sIriUs. One way is to observe a person over time performing those activities, processes or Pnk.'"'edures generally regarded as presupposing that skiD (or proper execution. ODe then mDlo... ( judgment regarding the degree to wbieb tbe person possesses the general skill in

question. A second way is to compare the outcomes (if any) that result from elecuting a given skiD apillit some set of criteria. A third way is to query persons and receive their descriptions of the proc:edures aDd judgments they are using as they exercise that skiU, would use if tbey were to perform that skiD, or did use when tbey performed that skill. A fourth w.y is to campa'ie tbe outcomes (if any) tbat result from performing another tuk against some set of criteria, where the performance of that otber task
has been sbown to correlate strongly with exercising the skill of interest. However, that such correlations exist between any otber task and CT, or any of its sub-skiUs, bas yet to be established in the research literature.

Bach of the four ways of CT assessment has limitations as well 85 strengths. No matter which ways are used, it is important to ensure that the assessment conditions foster an attitude in which the subjects are disposed to use their skills as well as they ean, and are not constrained or inhibited from doing so. In our view it is highly advantageous to gatber evidence regarding CT performance in many situations, using several assessment methods, so as to compile a composite picture of the subject and to cross chtN:k the results of an one wa of assessment.
'"

. ":.y ....
,

;).~ ..

"";.'

.~ .. o_

·_' :', r

~: "

.'~.:.'

Either to transform

CT into one subject strictly

field among others, to domain-specific

or to subject: and

narrow the range of CT applications content, would be to truncate its

utility,

misapprehend its

nature

diminish its
CT can be

valu~~. Within the overall clearly distinguished

curriculum the goal of learning

from the goal of learning domain-specific two goals can b£' distinguished, ways to learn the

content. experts subject

And yet,: while these

do not wish to deny one of the best c:ontext.

CT is within a

RECOMMENDATION 8: Direct instruction in CT and assessment of CT should be an eKplicit parts of any course granted approval for purposes of satisfying CT requirements, whether that course is a CT course per se Dr a course in a given subject field. The primary academic criterion in the evaluation of a proposed instructional program for purposes of achieving the CT goal should be whether the program will further the development of students' CT skills and dispositions.
The CT Curriculu.

Given that

CT has, in many cases, schools

became a college genera.l education

reqLdrement, secondary college preparatory beyond its

can be expected to begin to develop However, the value of CT extends inquiry tool. well

CT programs.

importance as a university-level personal

CT is vitally A

impor"tant in the

and civic life of all citizenry

members of society.

significant: pet-centage of the school, or ii they graduate, education.

will not gr-aduate from high of post-secondary

will not have the benefit

RECOMMENDATION 9: Thus, CT instruction should not be reserved only for those who plan to attend college. hor should it be deferred until college, since it is not likely to be effective if it were. RECOMMENDATION 10: Explicit attention to the fostering of CT skills and dispositions should be made an instructional goal at all levels of the K-12 curriculum. The cultivation of CT dispositions and an insistence on giving and evaluating reasons, should be an integral part of elementary school education. In middle schools and high schools,

, '.~

-,

':,..

..

-'

~.-.

_;.:."

..

instruction on various aspects and applications 04 CT should be integrated into all subject area instruction. Specific courses in CT and an advanced placement examination program in CT for college bound students should be developed. Although for gDod reasons at the postsecondary level CT programs are generally associated with departments of philosophy, no academic unit should be restricted in principle from participating in an institution's CT program, provided that the overall institutional program in CT equips stlldents to apply CT to a broad range of educational, personal and civic subjects, issues and problems.
There is growing evidence of economic, of those industrialized the successes, both scientific and

democracies which emphasize demanding educational standards is for career and a key

academic assessment and ~et firm prof,~ssional fac:blr advancement.

Assessment that

counts

unquestionably

in promoting academic achievement.

RECOMMENDATION 11: Thus, mlnlmum CT proficiency eKpectations should be set for each educational level, including promotion in grads, high s:hool graduation, college entrance, and graduate school admission.

The development of which teachers contrast to

valid and reliable

assessmeot

strategies students'

from

can dr.:lw reasonable domain-specific writing>, is

inferences

about;

CT, in

their

knowledge or other

~cademic: abilities

(such as reading or whether for

essential. classroom

CT assessment strategies,
or"

Llse in the

individual at

for

broader

pL~rposes, must

!lot simply rewar'd arriving recognize achieving correct to It

correct

answers.

They must. however good CT. The challenge our of

answers by way of let what is easily

CT assessment is not the .fullness of CT.

meaSLlred restrict those

sense of

would be shameful if

a.ssessment instruments design and caused

",."hieh fl-lCUS CJnl

on CT skills

dr-ov e OLlr CT cur r icular .... goad CT to

the

d1sposition.:'\1 co-npcmerrt e of

be neglected.

RECOMMENDATION 12: In evaluating the acceptability strat~qy or in~trument one should consider content validity, reliability, and fairness.
30
,.

of a CT validity,

assessment constrLlct

.: ;......

~::~~

< .....

r'.r

f~.;~;

"

./~
i~·

(I) 011

Content Validity:

The strategy

or instrument

should be based understanding of should

an appropriate

conceptualization

of CT and a clear targets.

which aspects be evaluated matter

of CT the assessment to insure that correctly

Each task or question

responding to that

item is not a

of rote

learning or information recall. educational targeting purposes,

Whether for' the classroom should include

or for broader strategies fer

CT assessment

CT's dispositional

dimension as well as its

ccgni ti ve skills

dimension. Validity: In acceptable CT assessment that each task

(2) Construct

or

qLtestian should have been evaluated correctly responses do so on the basis euoe the result

to insure

students inadequate

who answer or wrong strategies or

of geod CT and that

of weak or inadequate

CT. Entire

speci fie: item~"i n which good CT leads o right answers, should not be ,-,sed. (3) Reliability: In acceptable

to wrong answers, or poor CT to

CT assessment that

each task

or question

should l1ave been evaluated generally different do better persons on that

to insure

good critical

thinkers If

item than weak critical

thinkers.

are involved in evaluating

the results,

for example of the different

gr-r:tding essC\ys or judging presentations, judges should be cross-checked r-sliable, that is, generally to assure

the evaluations that their

findings are However, it is with the

consistent

with one another.

an open question

whether the levels and affective

of achievement associated dispositions are positively correlate

dHfer-ent CT sub-skills con·-elated.

oUler

Empirical r-esearch on how the sub-skills

with each Thus, at

and with vari.aus dispo'Sitions has yet to be undertaken. lime, due cQLltion should be exercised reliability 3l regarding

this

how to interpret

t..~.:hnic:alflleCl'·.iLtrl~S t:est-form uf

in the case of paper and

"

.,.....~.

::.~/ ....
'.

~ ··t,;,

"'-.'
••• F" •

"~~."

pencil CT 4=1SSeSsment instruments.

(4)

Fairness:

CT ~ssessment

should not unfairly

disadvantage

or domain-

advantage specific

groups of students

on the grounds of reading ability, as including the evidential, contextual

knowledge [broadly understood

conceptual,

methodological, cri tericlogical, with technical vocabularyl,

considerations, life

or familiarity

gender or age related differences

e)~periellce, ethnicity norms, or differences tasks and questions

or socio-economic status, in cut tLlral assumptions. in some assumed context, Thus, guaranteeing

in social loca tes CT

CT assessment either that

subject-specific, all students,

ever-yday life, regardless assessment

or fictional.

of their

individual backgrounds, will ceme to the CT equal basis in terms background knowledge, is impossible. However, these

on a perfectly

reading ability,

lii=e e)~perienc:est etc. strategy

e}~<3,"inil,g assessm~nt the factors

or· instrument

to be sure that

do not unf~irly influence the results

is prudent and reasonable. variables, one may

Althc,ugtl one cannot eliminate the influence of these bet ,alble to neutr-ali.::e or control The fainleo:;s crit.erion discipline-specific for their affects.

applies

both to discipline-neutral Within curricular is encouraged, regarding programs

a.nd

CT assessment..

discipline-c::.pec:ific:CT assessment
one ttl be fair concept.s,

since it is possible subject-specific

for

in one's presumptions

criteria, The content in

methodologies, evidence, information and terminology. is to fac:tor out the discipline

challenge of such assessment ordert.o c"\cc.:ess

the strength

or weakness of the CT. It is worti., noting also makes similar assumptions topic content.

I:hdt disc:ipline-neutnll

CT assessment contexts

regarding

the everyday

which form its

RECOMMENDATION 13: CT assessment should occur frequently, and it should be used diagnostically as well as summatively. Different kinds of

a5

-:..t.,

. '':~

..

instruments should be employed, depending en which aspect of CT is being tdrgeted and where students are in their learning the introductory stage, the practice stage, the integration stage or the generalized transfer stage. Although the veteran CT instructor is abl~ to assess studJnts continuously, CT assessment should be made explicit to reinforce its worth in the eyes of the students, their families, and the public. It should be made explicit to support the goals of educators seeking to improve the curriculum. And it should be made explicit to properly inform educational policy formation.

The CT Instructor
RECOMMENDATION 14: Teaching CT is most effective if the instructor models CT dispositions and the proper use of CT skills in the very process of instruct~on. Regardless of the subject area, students should be encouraged to be curious, to raise objections, ask questions, point out difficulties in the instructor's position. These objections and questions should be clarified, interpreted, and examined objectively. Students should be given reasons for doing things a certain Nay, rather than being dogmatically told how to do them. Instruction should bridge the gap between the subject and the student's own experience. In the case of CT instruction, the topics of discussion should not be restricted to factual matters Dr academic subjects, but sh~uld include issues which have normative, moral, ethical or public policy dimensions.
The ideal CT instruc:tor variety of subject subjects areas. as will integrate instruc:tion in CT in a directly skills. to She
tEl

She will teac:h specific content for the application

CT skills of those these

using these

will help students var"iety of contexts.

elaborate,

transfer

and generalize
tEl

skills

She will create of CT.

classroom

and school teaching

environment and her with undertake.

which is supportive interactions

She will model CT in her She will provide to learn about, activities her

with colleagues. sL(bjects student's

shdF'e

stUdents to

thought-provoking She will engage on, artic:ulate, contrasts studeltt'!:i

,

and projec:ts requiring

in social

them to reflect and each

and di<:;cuss justifications,

e>;planations

in how they e>:ecuted var-ious CT tasks. o . PI'"gress, ~chievement or proficiency

She will evaluate

in CT continuously.

RECOMMENDATION 15: For CT to infuse the K-12 teacher "training" should give way to teacher

and college "education."

curriculum, If teachers

33
...
,

~ .;:......

'

~re to model CT, so must those who have an instructional role in teacher preparation or staff development. In all instructien, and particularly in CT instruction, both faculty and leaders of faculty development should model CT. They should foster the students' confidence in their own powers of reason, rather than dependency on rete learning. They should nurture in students open-mindedness, attention to alternatives, and as much precision of thought as the subject and circumstances permit.

VI -- The Delphi Research panel

The Delphi research participation theory of forty-six findings reported scholars, research. here result from the in CT

educators

and leading figures

and Cl' assessmant

Roughly half the panelists

are are

primarily affilia.ted affiliated

academically
(22'Y.),

with Philosophy (S2X), the others the Social Scienc,es (207.),or the

with Education (6/.).

Physical Sciences
It

would be a mi.stake to construe

as

participation

in this

research
Thus,

project

implying that

a person

agrees

with all the findings. of panelists

where r..:onsensus is r sported

a minority

hold divergent

views.

Where near unanimity is reported acc.ord wiLh hew t,he specifics reviewing the f(Jrty-si~ draft

a some panelists

may may net be in full

are expressed.

In the end, however, after in Round 6, only one of the

Delphi findings presented

optecl to be listed

explicitly

as a pa,,..tic:ipant only, but not as

wuppor t; ing the

document. fi>: an important of forty-six in matters moment in time. possessing

These Delphi findings moment when the

It is a
speCial

efferts

experts
relating

experience

and knowledge

to CT converged with a viaw

be

tOWai""'lj discovering

i.f SCllne lne~asure

of general accord could into the decade

project

found.

As

we mQ"e from t! If:? S'-tcc(~sses

of the eighties

of the
the

ni:l'~ties;, the person';; wl'\o par'ticipdted

in this

hope that

finding'-:,> e;,~pe:-'tconsenSLtS reporled of

thinkiny and h~lp
sh~pe the futUre

her'ein will advance critical

and CT asseSSlilent.

of CT instruction

..

.;.~.~.

.... ~., -

...J ,

'_) ~'1

TA8LB1
J'

PARTICIPATING

CRITICAL THINKING EXPBRTS Ball State University University of Washington Howard Community College, MD

Jonathan Adler
David Annis AmoldAroDl James Bell Barry K. Beyer Charles Blatz Rob Brady Neil Browne

Philosophy

Psychology Bducatioa PhUosophy Economics

Philosophy Pbysics

Brooklyn CoUege

Philosophy

John Hoaglund Kenneth Howe Ralpb H. Johnson Stuart Keeley

Bez CemmellSOD Arthur L Costa Stan Dundon Robert H. Ennis James B. Freeman Jack Furlong Eugene Garver H. Scott Hestevold David Hitchcock

Bduc:ation Philosophy Bducation Philosophy Freshman Studies

Critical Thinking

CT Assessment

American College Testing (ACT) Sacramento State University Cal. Polytechnic Uaiversity, SLO University of Dlinois

George Mason University University of Toledo Stetson Univenity BowHng Green State University

Anthony Lawson Matthew Lipman

Philosophy Philosophy Philosophy_, Education Philosophy Psychology Zoology Philosophy

Education Pbilosopby Philosopby

Stuart Miller Brooke Noel Moore Wayne Neukberger Richard Parker Richard D. Parry Richard Paul Philip Pecorino William Rapaport Pasqual Sc:hievella Zack Seec:b
Anita Silvers Stephen Norris

David S. Martin John Martin Gary Matthews

Psychology

Assmt. and Bval.

Education Philosopby

Philosophy

Philosophy Philosophy

GaDaudet University University of C"mcinnati U. Massachusetts, Amherst Towsen State University CSU L'1tico Oregon Department of Education Memorial University of Newfoundland CSU .. Otico

Hunter CoUele, CUNY Transylvania University Saint John'. UDivenity University of Alabama McMaster University Christopher Newport College University of Colorado University of Wmdsor Bowling Green State Univenity Arizona State University Montclair State College

Ricbard Stiggins Robert J. Swartz

Steven Tigner

Carol Tucker Perry Weddle Mark Weinstein Peter WmoQard

~',~

Robert Wengert

Social Sciences Computer Science Council of Critical Analysis, Port Jefferson, NY Behavioral Science Palomar College Philosophy San Pranclsc:o State University Northwest Regional Educational Laboratory, Portland Philosophy U. Massachusetts, Boston Philosopby University of Toledo CT Assessment Bducational Testing Service (BTS) Philosophy Sacramento State University Philosophy University of Illinois Institute for CT Montclair State College Education' University of KentuckY
t

Agnes Scott CoUege Sonoma State University Oueensborough C. College SUNY Buffalo

35

.....

~~... .

:,.~

38

_:.: • ~~.: _., :1 .~.

••

••

_:f~-·_·:,

APPENDIX A
Commercially Ayailable OT Assessment Tools

Prepared for the APA Committee

on Pre-College

Philosophy

Delphi Researoh Project on CT Assessment

The Education Testing the academic abilities data, all within the of

Service

(ETS) Academic Profile wrtting,

Test

measures

CT, reading, of three

and using mathematical -a multiple scored). general It

context

major academic areas sciences. This is

humanities,

social

sciences with

and natural

choice instrument assumes that education --

an optional

critical

essay (locally all of their

studel1ts that is,

have completed most or completed the

sophomore college

year.

J
ETS also thinking has sections of its targeting logical reasoning and analytical (a) Law

on several

widely used instruments

such as the

School Admissions Test,

(LSAT) ilLogical

Reasoning" section, the Analytical

(b) Graduate sec:tion, (c) (d) National

Recol'"dE>,amination, (GRE)-AdvanceeJ Pl~~c:ement Test, Assessment of Educational questions,

General Test,

subject-matter Progress,
(e)

based CT questions, order thinking

higher

and porti.on, 1972the

lr.:lboratory-based

Forei.gn Service

Test,

in-basket

~\nd (f) Graduate Record E~<Clmination --

Advanced Test

in Philosophy to

1982.
8r
~c1I_lc\

ETS is

adding a section

of

Critical

Reasoning Questions

te f'1anagementAdmissions Test of information-seeking

(GMAT).

ETS has constructed

bro:.~nchil1g tests and-pencil test

and deciSion-making, some paperthe clinical practice

and same on computers. for the National

An example is

prepared

B~ard of

Respiratory

Care, Shawnee

36

... }

· ?"

1'1i5sion, I<ansas.

Stephen R~sear'ch developed

Norris

and R. King, through

the

Institute of

for

Educational has

and Development
the

at Memorial University
Observations
subjects made by

Newfoundland, Using to the

Test on Appraisi'lg fictional

of the situation claims

1983. are

backdrop

of a common but

asked

judge

the

relative

credibility

various

(:har·ac::ters.

The America.n College tlCollegid te CAAP is Assessment to of

Testing Academic

Pr'ogram

(ACT) in 1988 produced (CAAP). skills

the

Proficiency"

ACT describes

designed

measure

selec1:ed academic

including

reading,

writing,

oktthelllatics, ability The test

CT, and sciel1tific:

to is

reasoning.
evaluate,

1ne CT Test
an~ extend encountered questions. in a

mea=Ltr-es the
anJLhlI13I1tS."

"clarify,

analyze,

composed of passages

commonly ct,cic:e test

posts~?condary c:urricLtlLu-o

fallowed by multiple

P.~wl M. Ramin~;::: discLlsses

(VIr.TA) in The 1~t?adi1'lg Teacher.

the

"Valet:t

Inventory

of

CT Abilities" 348.

vol.

41, Dec. 1987, page

THE NINTH MENTAL MEASUREMENTSYEARBOO~~, N~1MY),l.ists (

commercially Many of

available
these U5t~ful

tests

in print described

c.,long with

reviews

and I'"'esearch

data.

c\t-e also Evaluating

a d r-eviewed

by NOt"'ris and Ennis Pdcific Gt-ove,

in thE:irCA, 1989.

CT, Midwest Public.ations,

#769

"Cor'rte l l Cri tiedl

ThlnkirllJ

Tes t, Levels

X and

Z." ~Ennis, !'Ullman,

T(::n~:o), 1961"-1983, Midwest Revil:!wed Vol. 43., in

Publica.tions, Psychological

Pacific:

Grdve,

CA.
1983

Educat ional

and by

i'teasureme'nts

pp. 1187-1197,

Modjeski

and Michael.

:.:_,,: ~ , ."

:'.\

.'

40

.~~.

,:,.1.
"

..

;:::~.~':' .
. -.

""

#390

"Ennis-Weir

Argumentation

Test,

Level

X: An Essay

Test

of Rational

..
I,

Thinking Ability,"
,.

<Robert

Ennis

and Eric

Weir) 1982, Illinois Urbana,

Thinking Project, Reviewed

University

of Illinois,

IL.

by Herbert Critical

Rudman, Michigan Essay Ennis Test:

State,

in NMMY. for

..
~,

*391 "Ennis-Weir

Thinking <Robert

An Instrument

Testing/Teaching.1I Publications, #1347 "Watson-Glaser and reviewed other #751 "New Jersey Insti tute tt1258 "Test research Test
for'

and Eric

Weir) 1983, Midwest

Pacific Critical

GrcvL!, CA. Thir.king Appr-aisal" in the 1942-80. Described of

by two persons roegarding of Reasoning the this

NMMY, many c:itations

instrument. 1983, Virginia Philosophy Council this test for for Shipman, Children .. Educational to

S~dlls," of

Advancement

of Inquiry For

Skills"
junior of

1979, Australian
high grades,

Research.

purports

evaluate skills

a t-ange

research,

study

and critical

thinking

in the
of

sciences. Cognitive
Processes"

#1061 "Ross Test

Higher

(John

Ross

and Catherine

Ross) 1976-79,
this

Academic Therapy sub-scores prehlises,

Publications.

For grades deductive

dnd

4-6,

test

inclLtdes

on analogie~, questioning

r-;~';:ioning, missing relevance #1~48 "Test

of

strategies,

informa tion. Skills" 1981, McGr"--,w Hill. For grade levl?ls 2-

of Cognitive

1", this ..:.,

(!"Ienary,

test

includes sub-scores
reasoning.

on sequencing,

analogies,

and verbal

#122 "BC\sic Skills

t"

Ass~ssment .. 1977-81, p,.:\ckdye is

Q

McGraw Hill.

Included

in the

sading

sub-score

on infer'sllce

and evaluation. evaluation. ages 6-12,

In the #1269 "Te-st of

w""iting package Problem

Solving"

is

a sub-score

on logical Inc. For

1984, LinguiSystem 3

·~~i·q .. .::~ .
.. '. -"'f.,

41

'.

this tests to events

a child's thinking and reasoning abilities of everyday life.

critical

It inclLdes sub-score!.:; on .• negative

why

explaining inferences, ques ticns, etc.

determining causes,

#272 "Corrective Reading Mastery Test" 1980, Sc:ienclaResearch

•

Associates, corrective deductions, inference,

Inc.

Designed to measure the effectiveness

of on

reading programs, this test classifications, analogies,

includes sub-scores inductions, statement

hypothesis/evidence.
1972-73,

#1302 "Deductive Reasoning Test .. (J. M. Verster) Institute syllogistic

National Foc:uses on Tor

for Personnel Research, South Africa.

problems and designed for for candidates and higher professions.

graduCl scientists te

#1010 "PSI Basic Skills Test for Business and Industry" Psychological Services Inc. Includes sub-scores

1981-1982, on problem

solving, decision making, reasoning and classifying.

t*106

"Ball Aptitude Battery" for occupational scores

the Ball Foundation.

Used to tests

persons

placements, this instrument

inclL-tdessubidea

on il1ductive reasoning, analytical

reasoning,

flLtsncy, and shape assembly.

3'

42

.. . :.1:,::".
A,

~ .
~"

. APPENDIX B Critical Thinking Bibliography with Emphasis on Assessmeu:t

Prepared for the APA Committee on Pre-College Philosophy Delphi Researoh on Critical Thinking Assessment
Adler, H., "Why 'CT' PrograMii:aWon't Work," Education Heek, Sept. 1986.

Annis, David B. and AnniS, Linda, "An Empirical Study of the Impact ~f Philosophy on Students' CT Ability," Teaching Philosophy, v3, pp. 145152, 1980. Arons, Arnold B. ·CT and the Baccalaureate v71, n2, Summer 1985. Curriculum," Liberal Education,

_______ , "Achieving Wider Scientific Literacy," Daedalus, Journal of the A.erican ACide.y of Arts and SCiences, v112, n2, p91-122, Spr. 1983. Azima, Kiavach, and Henry, Rebecca, "Teaching Students to Reason: An Application of Piagetian Psychology to College Teaching No. 76," Learning and Evaluation Service, Michigan State University, East Lansing, MI, 40p, 1980. Sociology and nssessing CT," Teuching Thinking Sociology, Skills: Theory v8, and in

Baker, P. J., "Learning p325-363, 1981.

Barun, Joan B., and Sternberg, Robert J., Teaching Practice, W. H. Freeman Publishing, 1987.

Bangert-Drowns, Rober, L., et, al., "Individualized Systems of Instruction Secondary Schoels," RevieN of Educational Research, v53, n2, p143-58, SUlllmer 1983. Beck, ~onald A., A Guide to Criterion-RefereDced Hopkins University Press, 1984. Beyer, Barry K., ftImproving Thinking Delta Kappan, v65, n7, p4R6-90, _______ , MImproving Thinking Skills Kappar" v65, n8, Apr. 1984. Test Construction, the Problem," Phi Delta Johns Phi

Skills -- Defining March 1984. -- Practical

Approache5,~

_______ , "Practical Strategies for the Direct Teaching of Thinking,Y in Veveloping Hinds: A Resource Book 10r Teachi~g Thinking, Arthur L, Costa, ed., Association for Supervision and Curriculum Development, Alexandria, VA, 1985. _______ , "A Suggested Format for Testing Record, v24, n i , p3-5, Spr. 1987. _______ , Practical Bacon, 1987. Strategies Thinking Skills," Social Boston, Science Allyn and

for the Teaching

of Thinking,

4C

43

~:~~
;J-'

Blatz, Charles V., "Contextuali~m and CT: Programmatic Educational Theory, v39, n2, 1989.

Investigations,"

Block, R. A., and TaylClr, S. V., "Cognitive Skills: Enhancement and Assessment Issues, Presented to the American Psychological Association, Toronto, Canada, 1984. Blumberg, Fran, et. aI, A Pilot Study of Higher-Order Assess.ent Techhiques in Science and Hathelatics Tested Tasks -- Part II, Final Report, National Progress, Princeton, NJ, Nov. 1986. Thinking Skills -- Part I and PilotAssessment of Educational Evaluations and v44, p67-70, 1986.

Bransford, J. D., et aI, "Teaching Thinl:ing: Evaluating Broadening the Data Base," Educational Leadership,

Branson, Stimmann Margaret, "CT Skills -- A Continuum for Grades 3-12 in History/Social Science," Social Studies Revie~, v25, n2, p24-32, Winter 1986. Brandt, Ron, "On Philosophy in the Curriculum: A Conversation Lipman," v46, Educational Leadership, p 38, Sept. 1988. with Matthew

Braungart-Bloom, Diane S., "Assessing Higher Thinking Skills through Writing," lOp. Presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA, April 1986. BrONn, J. lo, "On Teaching Thinking Skills in the Elementary Schools," Phi Delta Kappa», v64, p709-714, 1983. and Middle of

Bryden, David P., "What Do Law Students Learn? A Pilot Study," Journal Legal Education, v34, n3, p479-506, Sept. 1984.

Carlson, E. R., "Implications of Cognitive Theory and Research for Teaching CT, Presented to the Annual Meeting of the American Psychological Association, Toronto, Canada, 1984. Chaffee, John, "Viewing Reading and Writing as Thinking Processes," 9p, Presented to the Annual Meeting of the American Educational Reoearch Association, Chicago, IL, Mar-Apr. 1986. Chance, Paul, Thinking in the Classrool: College Press, 1986.
A Survey

of Programs,

NY, Teachers

Cierzniak, Susann~ Lipetska, The Question of CT: An An~otated Bibliography, 64p. Exit Project, University of Indiana at South Bend, ED 260 069, April 1985. The specific focus of this work is CT in tha 5econdary schools. Cornbleth, Catherine, "Assessing Skills and Thinking in Social Studies," po~ition paper prepared for Study Group on the national Assessment of Student Achi evement, and ci ted in Appendi x B of thei r report, "The Nation's Report Card" eTM 870 049), Journal announcement: RIEJUL87 CClsta, Arthur L., ed. Developing Hinds: A Resource Book for Teaching Thinking" Association for Supervision and Curriculum Development, Alexandria, VA, 1985.

. .\ ,.' .
.~,:~-.~

41

,-

- , .-.:'

,~,.' ....

_______ , "Thinking: How Do We K~cw Students are Getting Better at It?" ms., Dept. 04 Education, Sacramento State University, CA., 1989. Cost.a, Arthur lo, and Marzano, Robert J., "Teaching £ducati(Jnal Leadership, v45, p29, Oct. 1987. D'Angelo, Edward, The Teachirlf1 11f CT, Amsterdam, the Language
of Th i nk i nq ;"

B. R. Gruner

N. V.,

1971.

Deshmukh, M. N., "Teaching the Unteachable: Some Pedagogical of Creativity," PsychQ-Lingua, v15, n1, p33-40, 1985. Dewey, John, HON He Think, Drake, James A., Teaching Publishers, 1976. D. C. Heath and Co., Baston, CT, Danville, 1933. Printers

Considerations

Il, Interstate

and

Duck, Lloyd E., "Seven Cardinal Principles for Teaching Higher-Order Thin~:ing,1I Social Science Recorl'J, v24, n i , p3-5, Spr. 1987. Elman, Sandra E., and Lynton, Ernest A., IIAssessment in Professional Educition," 24p. Presented to the National Conference on Assessment Higher Education, AAHE, Columbia, SC, Oct. 1985. Ennis, Robert H., 'A Concept
1962.

in

of CT," Harvard A.erican

Educational Educational

RevieN, Research Educational

v32, n1, Journal, ResAarch v1,

-------, "Operational Definitions," n3, p183-201, Hay 1964.

_______ , IITesting for CT: state of the Art," American Assoc., San Francisco, CA, 1968.

_______ , "Rational Thinking and Educational Practice," in Philosophy and Education, Jonas Soltis (Ed.), Eighteenth Yearbook of the National SOCiety far the Study of Education, Chicago, NSEE, p143-183, 1981. _______ , "Goals for a CT/Reasoning Curriculum, Illinoi~, Champaign, IL, 1984. _______ , "Probl~ms in Testing Informal Logic v6, nr , p3-9. 1984. _____ -_, IIA Logical Basis for Measuring v43, p44-4B, 1965. Illinois CT Project, Ability," U. of Infor.al

Logic CT Reasoning CT Skills,"

Educational

Leadership,

_______ , "A Logical Approach to Measuring CT Skills in the Fourth Grade,lI Illinois CT Project, Champaign, IL, draft, April 1985. _______ , "A Taxono~v Of CT Dispositions and Abilities," in Teachi~9 for Thinking, Joan Baron and Robert Sternberg (Eds.), Freeman, Ne~ York, NY, 1987. _______ , "A Conception of CT with SORle Curriculum Suggestions," He"Jsletter on Tea,hing Philosophy, American Philosophical Assoc., p1-5, Summer 1987. 42
!~~ "-),[:7' :
. '.','"
," -".

,;.-~.::.,"

,'

J .••.;~

.'

~'.'

~-:.; .

<:

___ ..

, "A Bibliography of Testing CT,'! CT He~s, Center for the Reasoning Arts, CSU Sacramenta, v'o, nl, Sept.-Oct. 1987. Class Reasoning Test,

Ennis, Robert H, Gardiner, William L, et aI, Cornell University of Illinois, 1964, .......... Cornell Conditional __ , IL, (date ';') Reasoning

Test Illinois CT Project, Champaign, CT Test Level X, Midwest

Ennis, Robert H., and Millman, Jason, Cornell Publications, Pacific Grove, CAl 1985. _____ .. Cornell _, CT Test Level

Z, Midwest Pub., Pacific Grove, CA, 1985.

CT essay Test, Midwest

Ennis, Robert H., and Weir, Eric, ennis-Heir Publications, Pacific Grove, CA, 1985.

Ennis, Robert H., and Norris, Stephen P., "CT Testing and Other CT Evaluation: status, Issues, and Needs," in Issues in £valuation, Algina, James, (Ed.), Ablex Press, New York, NY, 1988(7). Facione, Peter A., "Toward a Theory of CT," Liberal Fall 1984. _______ , "Testing College Level CT," Liberal 1980. ___ .. , "Teaching abuut Fallacies," Summer 1987. Teaching £ducation, v70, p253-261, Fall

£ducation, Philosophy,

v72, p221-232,

vl0, pp. 211-217, CT Ne~s,

on Tests,

_______ , "Some Current Concerns p3 8< 9, 198B.

Regarding

HOTs Assessment,"

v7, n2/3,

_______ , "Assessing Inference Skills," £RIC Clear;nghouse Heasurelents and £valuation, TM 012917, Mar. 1989.

------_. ------_.
-------,

"Reasoni and Premises,"

CT He~s v7, n4, pl, 7f, 1989. v7, n4, p12, 1989.

"A Quick List of CT Tests," CT He~$, "Some Definitions

of CT," CT He~s, v7, n4, plO, 19~q.

alld

_______ , "Strategies for Multiple Choice CT Assessment," in CT at Colleges Universities, David Hitchcock, (Ed.), Vale Press, Newport News, VA., forthcoming. Follman, John, "Contemporary CT Bibliography," CT HeNS, Center for the Reasoning Arts, CSU Sacramento, vo, n2, Nov.-Dec. 19~7. Fraser, Berry, Test of Inquiry Skills, The Australian Measurement, Hawthorn, Victoria, 1979.

Council for Educational

Frederiksen, Norman and Ward, William B., "Measures for the Study of Creativity in Scientific Problem Solving," Applied Psychological
43
'."f,... ;

.~
-

.. ,':,,'.....

46

.. ' ....... ~:~ ...

~: ..
\

. '

~', . ..

".

:- ' ..

Heasurelerlt,

v2, pl-24,

19?B.
II

Fr itz, Pau 1 A., and We aver , ·Ri ch a r d L. II, Tea chi n 9 C T S ki 1 lsi nth e P II b 1 ic Speaking Course: A Liberal Art5 Perspective," 36p, Presented at the Annual Meeting of the Spee~h Communication Assoc., Chicago, IL, Nov. 1984. Gardner, H., Frales of Hind: Socks, NeN York, 1983. The Theory of Multiple Intelligences, of CT, Teachers Alerican Ithaca, NY, Basic College,

Glaser, Edward M., An Experilent Columbia University, 1941.

in the Develop.ent

Glaser, R, IIEducation and Thinking~ The Role of Knowledge," Psychologist, v39, p93.-104, 1984. 8rinols, Ann Bradstreet, (ed.) CT: Reading Cornell University Press, 1984. Across

the Curriculu.

Haars, Venant J., and Mason, Emanuel J., "Children's Understanding of Class Inclusion and Their Ability to Reason with Implication," International Journal of Behavioral Develop.ent, v9, n1, p45-63, March 1986. Haney, Walt, "Testing Reasoning and Reasoning Educational Research, v54, n4, p597-654, Harnadek, Anita, CT Troy Mich., MidNest about Testing," Winter 1984. 1976. of Speech HA, Nov. Revie~ of

Publications,

Hey, Ellen, liThe CT Movement and Its Implications Communication,1I paper to Speech Communication 1987. Hermstein, R. J., et aI, IITeaching Thinking v41, p1279-12B9, 1986 Hitchcock, David, (Ed.), CT at Colleges Press, Newport News, Virginia.

for the Teaching Assoc., Bosten, Alerican

Skills,"

Psychologist, Vale

and Universities,

forthcoming,

Howard, George S., and Englehardt, Jean L., "Teaching Rival Hypotheses in Experimental Psychology,1I Teaching Psychology, vll, n1, p44-5, Feb. 1984. Hunter, Jacqueline; Jones, Lester; Pre-College Program," Journal
e t , al., IITeaching Cognitive Skills of Learning Skills, vl, n2, p24-26,

in a 1982.

Kean, Michael H., "Testing Future Challenges, Future Respol"ses,1I Presented at the Annual Heeting of the National Ccuncil of Measurement in Education, San Francisco, CA, April 1986. K~arney, C. Philip, et al, Assessing Higher Order Thinking Skills,lI ERIC Clearinghouse on Tests, Measurement, and Evaluation, Princeton NJ, pB6 Apri 1 1986
II

Kneedler, Peter E., "Testinq CT at Grade Eight,1I Social n2, p7B-89, Winter 1986.

Studies

Rev i e«, v25,

'.'

_______ , IIHistory-Social Stience Assessment Record" v24 n 1 p8-9, Spr.' 1987.

in California,1I

Social

Scier/ce

Lawson, Anton E., liThe Effects of Causality, Response Alternatives, and Context Continuity on Hypothesis Tssting Reasoning,1I Journal of Research in Science Teaching v20, n4, p297-310, April 1983. Lee, Braham, and Oakhill, Jane, liThe Effects of Externalization on Syllugistic Reasoning,1I Quarterly Journal of Experi.ental Psychology HUlan £xperilf'ntal Psychology, v36A, n3, pSI9-30, Aug. 1984. Lipman, Matthew; Sharp, Anne Margaret; and Oscavan, the Classrool, Universal Diversified Services,
- .... 1

Frederick, Philosophy Inc., 1977. Press, 1978.

in

(ed.> G,o~ing

Up ~ith

Philosophy,

Temple University Leadership,

-------,

lIeT: What Can It Be?1I Educational

v46, p38, Sept. 1988.

Lohman, David F., "Predicting Mathemathanic Effects in the Teaching of HigherOrder Thinking Skills," Educational Psychologist, v21, n3, pI91-208, 1986. Mandic, Peter, and Stojakovic, Peter, IIContemporary Yugoslavia: A Review of Research", Conte.porary v9, n3, p207-13, July 1984. Educational Educational Psychology in Psychology,

Martin, David S., "Restructuring Teacher Education Programs for Higher-Order Thinking Skills," Jour~al of Teacher Education, v40, p2, May-June 1989. Marzano, Robert J, and Jesse, D., IIA Study of General Cag,itive Operation on Two Achievelllent Batteries and their Relationship to Item Difficulty,1I Unpublished paper, Mid-Continent Regional Educational Laboratory, Aurora, Colorado, 1987. Marzano, Robert J, et. aI, Dimensions of Thinking: A FrameMork for Curriculum and Instruc~ion, Assoc. for Supervision and Curriculum Development, Alexandria, VA., 1988 Marzano, Robert J, and Costa, Arthur L., "Question: Do Standardized Test5 Measure G~neral Cognitive Skills? Ans~er: No," Educational Leadership, v46, pp 66-71, May 1988. McKee, Saundra J., IIImpediments to Implementing p443, Oct. 1988. McPeek, John, Critical NY, 1981. Thinking and Education, CT," Social St. Martin's Education, v52t

Press, New York, in Literature

Mebane, John S., IITeaching Interpretive Skills Through Testing Cla5s," ExerCise Exchange v32, nl, p7-10, Fall 1986.

Modjeski, Richard B., and Michael, William B., "An Evaluation by a F'anel of Psychologists 0,' the Reliability and Validity of Two Tests of eTn Educational and Psychological Heasurelent, v43, n4, p1187-97, Winter 1983. (The test5 reviewed were the "Watson-Glaser CT Appraisal" and the

48

._
~};~.~/.
c .«,
•• '0:". '..t.,'

;~;.:..

~',

-..

"Cornell CT Test."] Morante E. A., and Ulesky, ~. "Assesument Leadership, v42, p71-74, 1984. of Reasoning Abilities,H Educational

Moss, Pamela A., and Petrosky, Anthony R., "A Proposal for Heas~ring Critical Thinking," A revision of paper p,'esented at the Annual Meeting of the American Educational Research Astoc., Montreal, Canada, April 1983. Pittsburgh board of Public Education, PA, Report No: T~a3-2. 23p, Sept. 1983. Moore, Brooke, and Parker, Richard, "T~e Written Debate -- A Technique Honing CT Skills," HeNsletter ~n Teaching Philosophy, American Philosophical Association, p8, Sumt!r 1987. for

Morgenstern, C. F., and Renner, J. W., "Measuring Thinking with Standardized Science Tests," Journal of Research in Sci~nce Teaching, v21, pt39-648, 1984. Meyer, Chet, Teaching Bass, 19B6. Students to Think Critically, San Francisco, JosseyEducational

Nickerson, R. S., -Kinds of Thinking Taught in Current Programs," Leadership, v42, p26-36, 1984. Nickerson, R. S.; Pprkins, D. N., and S.ith E. E, The Teaching Lawrence Erlbaum Associates, Hillsdale, NJ, 1985. Norris, Stephen P. Test on Appraising Observations, Research and Development, Memorial University

of Thinking,

Institute for Educational of Newfoundland, 1963.

_______ , "The Choice of Standard Conditions in Defining CT COlpetence," Educational Th~ory, v3S, p97-107, 1985. _______ , "Evaluating CT Ability," p13S-146. Spr. 1986. History and Social Science Teacher, v21, n3,

_______ , "Controlling for Background Beliefs When Developing Multiple-Choice CT Tests," Educational Heasure.ent , v7, n3, p5-11, Fall 1988. _______ , "Research Needed on CT," Canadian 37, 1988. ~ournal of Education" v13, p12S-

_______ , "Verbal Reports of Thinking and Multiple-Choice CT Test Design," Technical Report No. 4.17, Champaign, IL: Center for the Study of Reading, University of Illinois. (ERIC Doc. No: ED302826.) _______ , "Effect of Eliciting Verbal Reports of Thinking on CT Test Perforlance," Journal of Educational Heasure.ent, v27, nl, 1990. _______ , "Informal Reasoning Assessment: Using Verbal Reports of Thinking to Improve Multiple-Choice Test Validity," in Infor.al Reasoning and Education, D. N. Perkins, J. Segal, and J. F. Voss (Eds.), Lawrence Erlbaum Publishing, Hillsdale, NJ, in press.

46
".
i·'

):·\:1'·,:
,"'~'. < •

.. . ~.t,";.
<

'.!~ •.

------_.

"Can We Test Validity

for CT?" Educational

Researcher, CT, Midwest

in press. Publications, and

Norris, Stephen P., and Enni~, Robert H. Evaluating Pacific Grave, CA, 1989.

Norris, Stephen P., and King, Ruth, "ObservAtion Ability: Determining Extending its Presence," Inforlal Logic, v6, n3, p3-9, 1984.

Norris, Stephen P., Phillipe, L. M., "Explanation of Reading-Comprehension: Schema Theory and CT Theory, Teaching College Reading, v89, n2, p281-306, 1987. Norris, Stephen P., and Ryan, James, "Designing a Test of Inductive Reasoning," in Proceedings of the International Conference on Argulentation 1986, Van Eemeren, Frans H., and Grootendorst, Rob (Eds.), Foris Publications, Dordrecht, Holland, 1988 (7). Passmore, J., "On Teaching to be Critical," The Concept of Education, Routledge and Kegan Paul, London, p192-~11, 1967. Patrick, John J., "Core Content, CT, and Civil Values: Issues on Education in the Social Studies," in Trends and Issues in Education, 1986. Flaxman, Erwin, (Ed.), Council of ERIC Directors, ERIC Clearinghouse on Urban Education, New York, Jan. 1987. Paul, Richard W., "Teaching CT in the Strong Sense: A Focus on SelfDeception, World Views, and a Dialectical Mode of Analysis," Infor.al Logic, May, 1982. _______ , "The CT Movement: 1985. _______ , "CT Research: v42, p46, 1985.

A Historical

Perspective,"

National Educational

Farul,

Winter

A Response

to Stephen Norris,"

Leadership, Center for CT

_______ , CT Handbook Rohnert Park, CA,Sonoma and Moral Critique, 19B6.

State UniverSity

_______ , "DialogiLal Thinking: Critical Thou~ht Essential to the Acquisition of Rational Knowledge and Passions," Teaching Thinking Skills, Perkens, et. aL (eds.), Lawrence ErlbaulII Associates Inc., 1987. _______ , "CT in North A~erica: A New Theory of Knowledge, Learning, and Literacy," 115. Center for CT and Horal Critique, Sonoma State U, Rohnert Park, CA., 1989. _______ , "35 Strategies for Infusing CT Principles and Applications," ms. Center for CT and Moral Critique, Sonuma State U., Rohnert Park, CA, Jan. 1989. _______ , "CT: What, Why, and How," ms. Center for CT and Moral Critique, Sonoma State U, Rohnert Park, CA, April 1989. (Draft for CT: Educational l.perative HeN Directions for Co •• unity Colleges, Jossey-Bass.) Pecorino, Philip,

"cr

Bibliography,"

HeNsletter 47

on Teaching

Philosophy"

50

American

Philosophical

Association,

plS, Summer 1987.

Per kin s, D. N., II Po 5 t - Prim ary'Ed ucat ion Has Li ttl elm p ae ton Infor m aI Reasoning," Journal of Educational Psychology, v77, p562-571, 1985. _______ , "Reasoning As It Is and As It Could Be: An Empirical Pe~spective," Presented to the American Educational Research Assoc., San Francisco, CA, 1986. _______ , KnoNledge as Design: Critical and Creative Thinking Learners, Hillsdale, NJ, Erlbaum Assoc., 1986. for T!achers
and

Quellmalz, Edys S., "Designing an Integrated Instructional and Assessment System for CT SKills," Presented to the American Educational Research Assoc., Montreal, Canada, 1983. _______ , "Needed: Better Hethods for Testing Higher-Order £ducatioDJl Leadership, v43, p29-35, 19R~. Ramirez, Paul H., "Valett Inventory v41, p34B, Dec. 1987. of CT Abilities," Thinking Skills," Teacher,

The Reading

Raths, Louis Edward, et. al., Teachi~g for Thinking Theory and Application, Columbus, Ohio, C. E. Herrill Books, 1967, (2nd ed.) 1986, Resnick, L. W, Education ind Learni~g to Think, Issues National Academy Press, 1987. Lawrance Erlbaum Uorld,

_______ , (ed.> Cognition and Instruction~ AssDciates Inc., (in press).

and Agendas,

Ross, Elinor P., "Teaching--Not Just Testing--The v24, n1, p84-BB, Oct. 1984.

Main Idea," Reading

Ross, John A., and Haynes, Florence J., Teaching Problel-Solving The Ontario Institute far Studies in Education, Ontario, Canada, 20Bp, 1982. _______ , "Development of a Test of Experimental Problem-Solving Skills," Journal of Research in Science Teaching v20, p63-7S, 1983. Rud , A. G• Jr., Educational
u

CT an d p~. -ColI e9e Ph ilos 0 Ph Y r " Pre sen ted tot he Am eric an e Research Assoc., San Francisco, CA, 1986. Thinking Across the CijrriculuI, Harper and RON, New National

RugQiero, V. R., Teaching York, 1988.

Sabini, John, and Silver, Maury, "CT and Obedience Forul, p13-17, Winter 1985,

to Authority,"

Scherer, Donald, and Facione, Peter A., "A Pre/Post Test for Introductory Logic Courses," Hetaphilosophy, vB, n4, p342-347, Oct. 1977. Schievella, P. S., Critical Analysis: Press, NeN York, NY, 1968. Language and Its Functions, Humanities Station NY.

_______ , Hey! Is That You, God? Sebastian

Pub. Co, Port Jefferson

'.'

5 ~I

4t

=,«:: ~~ ..
'

Scriven, Michael, .

~Critical

for Survival,P

National

Forul, p9-12, Winter 1983 • Nedlands,

, ~Multiple-Rating Items," draft ms., U of Western Australia, Western Australia, Oct. 1988. Sheffler, lsrael, Reason and Teaching, Bobbs-Merrill, 1973.

Shemesh, Michal, and Lazarowitz, Reuven, "The Interactional Effects of Students' Cognitiv~ Levels and Test Characteristics on the Performance of Formal Reasolling Tasks," Research in Science and Technological Education, v6, n1, p79-89, 1988. Shipman, Virginia, Ne~ Jersey Test of Reasoning Skills, Institute for the Advancement of Philosophy for Children, Montclair state College, NJ, 1983. Siegel, Harvey, "CT Nov. 1980. _______ , Educating
as

an Educational Rationality,

Ideal,"

The Educatio~al

Forul,

p7-23, 1988.

Reason:

CT, and Education,

Routledge,

Smith, E. L., "Effects of Training and Use 'f Specially Designed Curriculum Materials on Conceptual Change Teaching and Learning," paper to National Assoc. for Research in Science Teaching, Washington DC, 1987. Smith, Charles W., "Verbal Behavior and Cla$sroom Practice," 36p, Presented the International Conference on Thinking, Cambridge, MA, Aug. 1984. Sternberg, Robert J., Press, 1982. Handbook of Hu~an Intelligence, Cambridge University at

____ - .., "Teaching CT Part I: Are We Making Critical _ Kappan, v67, p194-198, 1985. _______ , "Teaching CT Part II: Possible p277-280, 1985. _______ , Beyond UniverSity Solutions,"

Mistakes?1I Phi Delta Phi Delta Kappan, Cambridge 19, act. v67,

IQ: A Triarchic Theory of HUlan Press, New York, 1985. of CT Programs,1I Education

Intelligence,

_______ , IIIn Defense 1986.

Heel Commentary,

_______ , "Teaching CT: Eight Easy Ways to Fail Before You Begin," Phi Delta Kappan, v68, p456-459, 1986. ______ , (Ed.) Teaching Thinking Skillsl Theory and Practice 1987. Institute of

_______ , "CT: Its Nature, Measurement, and Improvement," Education, Wc?,shington, DC, 37p, 1986.

National

Sternberg, Robert. J., and Baron, Joan B., "A Statewide Approach to Measuring CT Skills," Educational Leadership, v43, n2, p40-43. act. 1985.

4£
,'.I.'

..

..., :~'
.

""".

.. 'i.:

Sternberg, Robert J., and Bhana., K., "Synthesis of Research on the Effectiveness of Intellectual Skills Programs: Snake-Oil nemedies Miracle Cures," Educational Leadership, v44, p60-67, 1986. Sternberg, Robert J., and Wagner, Richard K., Understanding Intelligence: Mhat's in It for £ducators, National Commission on Excellerice in Education, Washington, DC, 82p, July 1982.

or

Ste~art, B. L., "Testing for CT: A Review of the Resources," Rational Thinking Reports HUlber 2, UniverSity of Illinois, Champaign-Urbana, IL, 1987. Stovel, John E., "Document-Based Questions Record, v24, n1, p11-12, Spr. 1987. and Testing for CT," Social Science

Swartz, Robert, "What is CT and Hc~ Can We Teach It?, ms. U. Hass. Boston, June, 1989. Talbot, Jan, "The Assessment of CT in History/Social Science Through Writing," Social Studies Revie~, v24, n2, p33-41, Winter 1986. Thompson, B., and Melancon, Skills," Psychological J. G., "Validity of a Validity Measure of CT Reports, v60, p1223-1230. 1987.

Tucker, Carol B., "Examples of Question Types Used to Test Reasoning," Educational Testing SerVice, Princeton NJ, Apr. 1986. _______ , "Some Multiple-Choice Formats Used to Test Reasoning," Testing Service, Princeton NJ, Apr. 1986. _______ , "Working Paper on Defining CT for Purposes Testing Service, Princeton, NJ, Summer 1988. of Testing," Educational Educational

Van Der Bogert, H. Rebecca, "The Evolution of the Concept of CT: A Literature Review," Ms. Harvard University, 1988. Watson, Good~in, and 8laser, Edward M. Hatson-Glaser CT Appraisal, Psychological Corporation, San Antonio, TX, 1980. The

Weddle, Perry, "CT in California: The Department of Education Testing Program in Social Studies," History and Social Science Teacher, v21, nJ, p14714Q, 151-154, 156-157, Spr. 1986. Whimbey A., and Lochhead, J., "You Don't Need a Special 'Reasoning Test' to Implement and Evaluate Reasoning Training," Educational Leadership, v43, p37-39, 1985.

50

The Delphi Research Lett~

Over course of this research sent to the experts participating College Philosophy Delphi Research

sixteen "Delphi Letters" were in the APA Committee on Pre-

Project on CT assessment. interactive Delphi

Eight letters constituted research rounds. clarifying

the specific

Tlle eight others were for purposes of planning, providing information, and sustaining

procedures,

involvement.

Feb. Mar. Mar. Apr. May May

11, 1988 1, 1988 14, 1988 14, 1988 4 , 1988 18, 1988
28,

ROUND 1 Information ROUND 2 Plans and Procedures ROUND 3 Procedu.re e and Involvement Information Procedures ROUND 4 Plans and Involvement Information ROUND 5A ROUND 5B ROUND 5C Information ROUND 6 and Involvement and Involvement and Involvement and Information and Plans

June
Sept.

1988

1, 1988

Sept. 23, 1988 Nov. Feb. Feb.

Mar.

22, 1988 7 , 1989 28, 1989

6,

1989 1989 1989 1989

Mar. May

10,
9,

Sept. 25,

51
~ :-'~ .. ::-

- . ,,~~.r '
--:Lrj,

California State University, Fullerton

Fullerton. California 92634 Oepartment of Philosophy
(714) 773-3611

Feb. 11, 1988 ~D"~D

.1

Dear Colleague, Gary ~!atthews, Chair of the APA Committee on Pre-College Philosophy, asked me to head up an ad ~oc comrnitt.eeon testing critic~l thinking. To get the project rolling. he .suggested severa names of people interested in the question of ~~w to validly and r~liably test critical thinking skills. I adQe~. a few more. Here are some of the particulars. 1) By using a modified Delphi approach, I think the necessity for actual meetings can be largely, if not entirely eliminated. Co~mittee members' contributions will involve sending their reasoned and timely responses to question~, given their particular background and expertise. 2) Thare are, no 'doubt',a great number of other people who are interested in the question of testing critical thinking and who have valuable expertise which would help us in in dealing with this question. They should also be invited to participate. 3) Since our charge is rather vague, I propose that among the fi~st things we should do is agree on priorities. To do this, using the Delphi process, let me lead off with some assumptions anct questions: First assu~ption: Most of the members of our group will come at the issue of testinq critical thinking with the orientation of philosophers or logicians who teach at the post:-secondary level. rather than as K-12 educators, psychologists, or personnel directors (all of whom also have legitimate theoretical and practical interests in assessing critical thinking). Given the interests of the Amer~can Philosophical Association, this ~ acceptable. Second assumption: Critical thinking can be defined operationally to the extent that it can beco~e a dependent variable in a valid and reliable assess~ent tool.
QUESTIONS:

1. Do you agree with the two assumptions? am~r.d/clarify the~? Why?

How would you

2. To which educational level (from Kind~rgarten through postBaccalaureate> should the committee give priority? Why so? 3. After looking the attached preli~inary w~uld yo~ recom=end be added to our co~mittee?
rhe Ca"'orn.a Slale l.Jnl~erS'I)'

list, whom else

Appendix
I •

C: Delphi

Researoh Let5t~!'s, PAGB52

"~

....~ .
I •. _-,·C

.;;:~:~. . ,) ..

,~

For the Delphi process, which is very interactive, to function optimally, reasonably quick turn around is needed. Let's target a Feb. 29 post~ark. Is th~t possible for you? Please send responses, suggestions, comments, etc. to: Testing

Ad Ho~ APA committee on Critical Thinking c/o Dr. Peter A. Facione Professor of Philosophy and Education Dept. of Philosophy . California State University, Fullerton Fullerton, CA 92634

Like math and composition, at ~any colle~es throughnut the country CT is being built into the curriculum. For example, the California State University system, which enrolls hundreds of thousands of undergraduates, has implemented a system-wide critical thihking requirement as part of its general education package. !f those of us who teach critical thinking were able to agree on a way or ways it could te tested, what a positive contribution that could be to the quality of that curriculum. I sincerely hope you will agree to ~acome an active participant in·what ~rornises to be a most in·t~resting and important effort. I've Ln c Lude d some reference m(lteria~.from Tha r~inth Mental Heasure~an~ Yearbook regarding published instruments which purport ~easure critical thinking and/or related cognitive skills. to

Pete Facione ce. G3ry Matthews, University of Massachusetts, Steve Tigner, University of Toledo, Ohio ATTCHMENTS: Amherst

Preli~inary List of Committee Members Quick Survey of Published Instruments Bibliography on Testing Critical Thinking

TO BE DEVELOFED:

Append1x
'
""

c:

Delph! ReBea~ch Lette~a,

PAGE53

-.

... . ,.
\',

California State University, Fullerton

Fullerton, California 92634

Dopa~ment of
(714) 773-3611

Philosophy March 1, 1988

Dear Colleague, tarl¥ :esponses to the first Delphi round are very encouraging! Some of you know what I'm talking about, others, new to the effort, need to be brought up to speed. Let me back track a bit and explain. I am writing to you because you were nominated by one or more of your colleagues as a person intereste~ in the question of testing critical thinking. You are being invited to participate in the work of an ad hoc American Philosophical Association SUbcommittee concerned with the problem of testing critical thinking. In its boldest form, our aim is to find ways to validly and reliably test critical thinking, or find solid reasons why such a goal is not achievable. Using the Delphi process, I will serve as coordinator of the effort. At the moroent we are very near the beginning of Phase 1. Please review the preliminary plan outline~ below and decide to beco~e actively involved in what promises to be an intriguing effort to shed light on an important pedagogical and profession concern. ad hoc APA Sub-Committee on Testing Critical Thinking Plan

Draft Preli~inary

Phase 1: Start the Inquiry The five objectives of this phase are: (a) Initiating the Delphi process. This is a method of achieving reasoned consensus among a group of experts with regards to a given problem or issue. The core of the strategy is to maka inquiries, g~ther each expert's responses and their reasons, then summarize and share those with the group. After "hearing" what other experts think, people ha1e the opportunity to refine their responses or defend those responses. The interactive rounds continue until reisoned consensus is achieved (or communications· break down). (b) Developing the "List of Experts" who will take part in this inquiry. Many of you were nominated by those who replied to my first letter, (2-11-08). That letter was sent to an original group of ~bout twenty-fiv~ experts and interested persons suggested by the APA comnittee that conceived of this project and asked me to coordinate it. A~ lny time if someonE is no long~r int~rested in continuing, just drop ~e a note. I plan to send out ~pdated rosters of participants periodically. Ve are now up to fifty.

Appendix C: Delphi Research Letters, PAGE 54

,~...... .,--~.~,
T~ '.,1"'.
t"''''''IIit,·,;J,

5 \. 4
1-'1

rt ....

It.

"I

',lOt •.

· ,.;~~.' .,
'.1

.
(c) Developing a bibliography on testing critical thinking. Work on this is progressing well. Many of you have sent me items to include, and I appreciate that. I will send you a copy of the bibliography later this semester. (d) Planning the subse~~ent phases in our process of responding to the general question of whether or not critical thinking (whatever CT is) can be validly and reliably tested at some educational level or levels. If the consensus is "Yes, at level X," then we will focus on the question: How? If the consensus is "No, at It:.::;t not at level X," then we will focus on, "Why not?" (e) Agreeing on basic assumptions •. In the 2-11-88 letter two assumptions were put to the group: Assumption 11: "Most of t~~ members of our group [of experts participating in the Delph1~process] will come at the issue of testing critical thinking with the orientation of philosophers or logicians who teach at the post-secondary level, rather than as K-12 educators, psychologists, or personnel directors (all of whom also have legitimate theoretical and practical interests in assessing critical thinking). Given the interests of the American Philosophical Association, this [is an] acceptable [orientation]." Assumption #2: "Critical thinking can be defined operationally to the extent that it can become a dependent variable in a valid and reliable assessment tool." People were asked (1) if they agreed with the two assumptions as stated, or whether they would reject them or rephrase them somehow. Naturally, people were invited to explain why. They were also asked (2) to identify the educational level (K-post baccalaureate) to Which our committee should give priority, and why they would recommend that level. Both questions have gener~.ted controversy, as you will see in my next letter. If you haven't had the opportunity to respond to these questions, you will be invited to respond when the first round of the Delphi is raported back to you. Delphi is not about vote-counting, it ai~s at reaching agreement on the basis of reasons and common assumptions. In the Delphi method people are supposed to share their premises, not just their conclusions. As conceived at the moment, our work can be divided into 4 phases. These are not in stone! I welcome your suggestions, a~endments, alternatives, atc. WE WILL USE THE DELPHI PROCESS TO AGREE ON OUR PLAN OF InQUIRY. Because the plan should be. a~ended as a result of your input, only goals, not detailed objectives, have b~en developed so far for the next three proposed phases.

......

58
Appendix C: Delphi Research Letters,
,..,~..-:: '::"~f~,~,: ..

PAGE65

,.·r: ..A; ...

,f,;'r';'-':

...
.

~ ..

" ' ':. :~',I'

,-"-,'

".

Phase 2: Define "Critical Thinking" The goal of this phase is to define "critical thinking" with sufficient clarity and precision to ask and answer the question of whether or not CT can be tested. Is CT fundamentally a set of skills, concepts, procedures, attributes, behaviors, outcomes, dispositions, aptitudes, or what? Even if we canno~ reduce CT to an equivalent operational definition, how might we express what CT is with sufficient operational precisions to permit us to justifiably infer things about the relative CT abilities of students? Phase 3: Recommendations The goal of this phase is to communicate our findings about what CT is and whether there is an adequate way of characterizing CT operationally so as to permit its being tested as some educational level. Depending on our results in Phase 2, we will recommend either that programs aimad at testing CT be abandoned, or:that they be focused in certain ways. If this is the dire~1ion Phase 1 takes, then we will also try to come to consensus on recommendations regarding the relative importance of different kinds of CT sub-skills and possible strategies for accessing and measuring these subskills. Phase 4: Design and Validation of Model Testing strategies Contingent on the results of earlier phases, the goal, if it were considered achievable in principle, would be to construct and evaluate different approaches to testing CT at some appropriate educational level or levels. Ve might find ourselves breaking into sub-committees to achieve this goal, although all work will have to be guided by the agreements reached in earlier phases and as well as by the special e~pertise of those who understand the intricacies of designing, piloting, norming and validating educational tests at specific educational levels. As I mentioned, you have been nominated as person who might be interested in this project and could make a strong contribution to the work of this ad hoc sub-committee. I hope you will agree to participate actively, because, 'as you must know, ti,e quality and utility of our effort is directly related to the invol~e~ent of concerned persons like yourself. '

Sincerely,
~,......:;...---

Pete Facione

5a
Appendix
'-.:.\ . ,<~:~;.~!.:,~ ,

C:

Delphi

Reeearch

Letters,

PAGE 56

'.' ,

.~ .
, ".'. ',' ',"

...•.

.'

'.

"

'

'" ,:4, .' ~ .'

"'

..

~'~'\: ,
. >

.~"

Ii

IIfIIE:Ie

pHS·Wt

3A

::::t=

98'=

, Fullerton, California Department of Philosophy

(714) 773-3611

California

Sta te University,
92634

Fullerton

March 14, 1988

ROVIJ'J) ~

Dear Testing

CT Colleagues,
t;'"....a:;!' __

Let's give Phase I, Round 2 Z5

shotl

Thanks for your responses to the first round of questions. Nineteen of the twenty-six or so who received the original 2-11-88 lette: for Round 1 were able to respond. Round 2 invites everyone (which now includes just over 50 people) to review the results of Round 1 and con~ent on the agreements and controversies that are emerging. . ,
, M

Round 1 focused on three issues: (1) The composition of our ad hoc committee in view of the interests of the A3erican Phllosophical Association, (2) the assumption that critical thinking can be operationally defined, and (3) the educational level to which we should give priority. In regard to the composition of our committee, we are in decent shape, particularly since our group has been greatly expanded as per your recom~endations. In regard to an operational definition of CT, we generally agree on the possibility but many would add various caveats. In regard to the educational le7el to which we should give priority, we have disagreements. The following pages cover each of the three questions in turn. You'll find restatements of the original questions and several representative quotes and su~maries of your comments. AfTER EACH SUMMARY, A SECOND ROUND QUESTION WILL BE PUT TO YOU. The new questions take the for~ of stating a position and as~ing your opinion, now that you have had an opportunity to consider what our colleagues have to say. In all, there are three ne~ questions. IF YOU COULD GET YOUR RES PONS ES TO HE \11THHI TEN DAYS OF RECEIVIUG THI S LETTEIh. THAT lJOUL,D ~1 GREAT! (I wish we all had electric Qail, or unli~~ted phone budgets, but •.. ) Several people noted that our task was huqe, yet were willing to give it a try. In contrast, one pernon wrote a major critique of the entire enterprise. This person argued that trying to test CT was a serious mistake. s~ that his opinion is ~ot lost in the shuffle, at the end of this package I have provided extensive quotations from his letter. If you fin~ yourself in agree~ent with his v i evs , t he n let me know and we will take up any "prior questions" we must • .If you don't agree, then we will press on along the path we are charting for our s e 1'I es .
Th 1n kyo
IJ 1n

ad

'J -l n c e

for you r pat't ~

~ pat ion •

,,60,

"\"1 I "" ........~

t' ..,1.

l ·'I~·I

...

'J. ,. ..

1''/

Appendix C: Delphi Research Letters,

PAGE57
,,, .. ,'tJ.

',r, r_ , ~'I,t:. •
,

".

~.(~

, ,:: ..

',.1"

..

PHASE I, ROUND 2, FIRST ISSUE CURRENT STATUS: CLOSE TO CONSENSUS CONPOSITION OF THE AD HOC COMliITTEE: lie vere asked if we a!/reed v i th the assumption that, al though most of the membe:s of our ad hoc commi ttee would come at the issue of testing with the orientation of philosophers or logicians who te~ch at the college level, this orientation still would be acceptable in terms of the interests of the American Philosophical Association. Almost everyone agreed, however some qualified their responses in terms of our collective professional interests and abilities, or in terms of educational level to which we should give priority. Here are representative responses: "I agree." "I have no problems with this assumption."

"I don't see why this is a problem. First we're concerned about students acquiring the thinking skills required for college work, ••• Second, we're concerned that they learn the standards of good reasoning: I do not believe we need to know a lot about psychology to;achieve this purpose." "I agree, with reservations. We need to avoid tunnel vision. It is acceptable that most members be philosophers, but there should be a generous sprinkling of 'outsiders' for the insights they will bring and to give our findings greater credibility outside the APA." "We ar e what we are! This is an appropr.iate place to begin. lie are starting from what we know best and with what we can deal with most easily. This is not to suggest that we shall forever ignore other orientations, or that we really know that we can define all aspects of CT operationally." "I agree, this is acceptable; but it is unnecessarily many of the tests are created by cognitive and educational think some of them should be included •••" narrow. Since so psychologists, I

"It should not be too quickly assumed that those who t.each at the postsecondary level are therefore knowledgeable and competent with regard to testing at the elementary school level." "I agree, but we should make a r-erious effort to inform ourselves of approaches to CT in pre-colle~e and non-academic settings ••. Assuming our primary focus is everyday reasoning skills, we should not allow college CT instruction to be fundamentally different from pre-college CT instruction nor to become idiosyncratically colored by our own traditions. One person disagreed but did not give a reason. proof-reader!" And one urged "Pete, get a

..

ROUND ~L QUE?TIOtl OrIE: In view of the above co~ments, and in view of the additional names added as the result of your recommendations, can we ~gree that th~ ad hoc comaittae, as listed on the att3ch=ent, is sufficiently w~llconstituted for us to move on with our main task? As you can see, it still has its original orientation toward philosophers teaching at the college level, but it also includas se~aral people from other relevant disciplines backgrounds, including psychology and education.

APpendix C: Delphi Researoh Letters,

PAGE58

'~~~,

,- ... _"- ~'A "_~ :~

.,:.

'.

PHASE I, ROUND 2, ISSUE 2 CURRENT STATUS: AGREEMENT SEEMS ACHIEVABLE OPERATIONALIZING CRITICAL THINKING: We were asked if we agreed witb the assumption that CT can be defined operationally to the extent that it can become a dependent va.riabl in a v'llid and reli.able essessaen t tool. e Here, too, most people agreed and were ready to get on with the work. some crucial ambiguities, concerns and caveats emerged. Here are some responses. Yet,

"I expect we will argue about the details of any definition proposed, but I do not object to the assumption that we shall need some such instrument if we are to get any comparisons of interest." "This is a tautology because of the to the extent phrase. how we should leave it •••" Perhaps this is

"Sounds ok. •••I'm not a st a t i st i c i an. so I'm not quite sure what dependent vari~ble means -- but if you'are asking whether CT can be tested, then, yes, I agree!" "I don't understand what this assumption is supposed to meanl"

"I would agree only if we amended it to say at least some components at CT can be defined operationally to the extent that they can become dependent variables in a valid and reliable assessment tool •.• I do not accept as analytic the proposition that CT can be defined operationally •.• I think some important aspects of CT, such as making judgment calls and weighting nuances may resist operational definition." "As the term oper'ltional definition is generally used by philosophers and edu~ation researchers, I do not think CT can be operationally defined •.. but I do think that part of the operational spirit can be employed in formulating reduction santences (that do not reduce!)." "I agree, but there will probably have to be a variety of sub-definitions because CT is not one thing, but many. It somewhat resembles IQ in that.
II .:',',

"Thera are several definitions of CT themselves more to operational definition anywhere, we will have to become clear in distinguished from other kinds of thinklng "Is this a normative, definitional,

floating around ..• Some lend than others •.•. If we are to get our own minds as to how CT is to be •••" or planning assumption?"

conceptual,

ROUliD ~ QUESTION T'.IO: W'ithout hanging ourselves up on the word "operations," can ·,.,e agree tha t: (1) Even if CT cannot be reduced en t i rely to an equ i valent set of opers t ions (or performances, behaviors, processes, outcomes, or skills,) (2) it is possible to conceptually anJlyze CT so as to describe a set of relevant and important CT operations, such that (J) ~sing these descriptions, (competent) investigators could, on J consistent basis, gather sufficient evidence to draw conclusions, with high degrees of confidence, regarding the relative CT abilities of a group of people, [everything else being equal, of course) •

,~ u
')
"wi

Al)pendix C: Delph! Research Letters,

~~ ...
"":OJ. , ~,,""'.

PAGE59

........

~.

. ,v~~.

.
PHASE I, ROUND 2, QUESTION 3 STATUS: CONTROVERSY

"

';

PRIORITIES: We were asked which educational level (kindergarten through post-baccalaureate) should be the priority for our committee. And why so? Responses were split. Here are some representative examples:

A person with considerable experience in the area of C'l testing wrote: "I think we should concentrate on high school at first. Since t~is is a sUbcommittee of the pre-college committee, all [levels] above that [are] ruled out. Furthermore, the younger the population, the more difficult the problems. Let's start with the easiest ones first -- and they are very difficult." By contrast, it was argued, "Priority should be given to the post-secondary level. One should examine end-products first, and then work backwards if needed. Find out fi,rst if the car doesn't run before attempting to determine whp.re the problem is. If a good test of CT revealed no CT deficiencies on the part of graduating seniors -- (no doubt a counter-factual assumption) -- then I would think the APA might not wish to pursue the issue down in K-12." Noting that we are a sub-committee ot the APA pre-college committee one person argued: "\Ie must give priority to K-12; that is our mission." However, the person who will assume the chair of the APA pre-college committee for the next three years wrote: "It makes sense to start by playing from the APA membership's greatest experience and strength, which is surely college freshman level logic." Taking note of the interests of the AFA, one person argued, "Since our ad hoc committee is convened within the structure of the APA, our focus should perhaps be primarily on the improvement of post-secondary education .••" However, this person also suggested, " ••• that our assessment tool should be usable in secondaz:y schools as well as at the post-secondary level. ••
II

Some people did not offer an opinion, but did note important distinctions. For example: "There are really two areas. One is the whole K-12 integration of thinking skills into the curriculum. The other is the single CT course, typically the approach followed in post-secondary education. The single college CT course offers exceptional opportunities for measuring gains in thinking skills, while the effort to incorporate thinking skills into the [K-12] curriculum may offer much greater potential for actually improving student skills." Another person, experienced in the pre-college arena, wrote: "It may be necessary to think of four tests, one for grades K-3 (one should not have high expectations for reliability at this level); one for grades 4-8 <the level at which testing might have the maximum impact, even though the maximum impact for the teachi.'lg CT might be at K-3); one for grades 9-12, and one for 13-16." of Some were tan t at Ive : "Perhaps we should gi';e priority to CT at the college level, at least to start with, since the large ~ajority of APA members telch at the college level. Later we might wish t o broaden our focus."

Appendix C: Delphi Research Letters, PAGE 60

:~' .. .,
..
.:.'1/ ....

t)J

"t ~::. ,
00

'.' ,~'.~:-

.'

.;.~.

..

Some were direct, "College and university level." others were focused, bUL concerned not to overlook anything important, "College level -- but someone ought to look at the high school level." One person declared for the college freshman level and ~rgued against going any higher saying, '''rhereare few if any thinking skills pos ses+ed by people beyond this level not also possessed by well-prepared college freshman. Graduate school and professional life chiefly consists in the ability to persistently apply these skills in more and more recondite subject matters. Some narrowed the range, but still left us with a choice: grades 9-12 and freshman/sophomore level in college." "I would say

Another argued we should give priority to the introductory baccalaureate level saying, "First, it is the area where most philosophy departments have numerous classes actually being taught. Second, it is taught at a level which will have the nest connections in other areas and at other levels." To get started let's give priority to the college Do you agree? If not, is your disagreement based on pedagogical and theoretical concerns or on concerns relating to our charge as a sub-committee of the APA committee on pre-college philosophy?
freshman/sophomore level.

ROUND !t. QUESTION THREE:

Please ill to ill your responses ts ~ ~i thin ten days.

1'hanks.

6~i
Appendix C: Delphi Research Letters, PAGE 61
. r\

.:".~-"' : -,

_-

."

".'.'

....

,~

·,UI· ._ , .. ~ ..

."';'.1".

"

PHASE I, A CHALLENGE RAISED AGAINST THE ENTERPRISE One person wrote: "I would like to comment that teaching CT skills ••• is a far mote significant matter than testing for them. Since CT is not an inherited trait, as is intelligence, the quality and extent of the CT is dependent on and proportional to the degree to which chi'dren have learned to, or have been taught to, think critically about their experiences and knowledge. "Our educational system ••• is an abysmal failure. Most students ar e unable to recognize assumptions, not alone question or examine them .•.• Yith rare exceptions, they show an almost total absence of recognition of even the simplest of 10gical/mathematical/linquisti~/philosophical/scientific facts and concepts needed to be able to think critically. They have been so nurtured in a world of superficial "education" ••• that to think of testing them on the basis of that "education" is an exercise in futility or at best an attempt to determine how inadequately they think critically as opposed to how much -- which if taken literally amounts to the same thing. "I am willing to contrirute my expertise to the teaching of CT. I have been doing so for over thirt' years. But until I see considerably more evidence of students being able to think critically without such teaching, I s~p. little point in testing them for the insignificant amount of critical acuity they may have acquired haphazardly. " ••• Teaching CT must precede and supersede testing for CT. Testing for CT cannot be cons~dered to be an enterprise separate from teaching it. Testing for such skills and concepts presumes prior teaching of them. Psychologists and K12 teachers as well as other educators show interest in CT. The problem remains, however, that most of them have only superficial, naive, and conflicting concepts, of what critical thinking entails. Even we philosophers can't agree on what it is ••• "From my understanding of the term, testing for CT means testing based on what I teach CT is •••• Any tests that I would, and have designed, are predicated upon the version of CT I have taught. "My co~ments will undoubtedly reflect those which you will receive from my colleagues equally concerned with the problems facing our educational system, particularly as they relate to the teaching of CT skills and concepts." REMIUDER: If you believe there are issues (suggested by the above or otherwise) which our ad hoc sub-committee must address before we can m01e ahead, please let me know. On the other hand, the above challenge may represent a view which is not widely shared, or may raise questions which, in your view, do not fall within the scope of our work or do not warrant our attention at this time. If that is the case, then, for the present, no response on this item is necessary. Thanks again for 10ur participation .

,. ,-t..(e
-

/ »-"

. .1. '

»: -...

.....

Appendix C: Delphi Research Lettera,

'.J

PAGE62

65
•, _,.'-'. ~ 11·'-'

,.,:,,~.: ...-,.'

!'In .

...,

California State University, Fullerton

Fullerton. California 92634 Department of Philosophy
(714) 773-3611

Apr i I 14,

1988

Dear Colleagues, Thanks for your help with the addresses. I also appreciate the notes, suggestions and other helpful comments many of you are sending. So far 17 responses to RJund Tvo have been rec eived . It would help if we had more, particularly since it there may still he a split over Question Humber 3, the one:about which le~el to pursue first. Drop me a note, with your reasons and opinions. Thanks. One of the responses to Round Two proposed an alternative to actually trying to come up with our own CT assessment instrument -- a goal some of us hope to achieve, but others of us are extremely skeptical about. This person suggastad that we articulate the best list of CT skills we can, then let people go their own way with regard to building testing instruments. I'll expand on that idea when I sum~arize Round Two responses. I Mention it here because perhaps we all should be thinking ahead and trying to chart the most reasonable path for ourselves. Some of you ha~e electronic mail. I don't, at least not yet. So, Illl be using snail mail and talephona to try to reach you for particular questions or clarifications. If you want to phone in your views on the questions in Round T~o you can reach rne at 714-773-3742 (office) 08:30-10:00 HIJ or 09: 30-11: '0 TTh (PDT). If those ti~es are incon~enient, call the depart~ent secretary at 7l~-773-3611 and lea7e your phone number. 1111 get back to you. I ~ill be attending the First National Conference on Assessing Thinking in Baltimore on May 6 and 7. This confarcnce is sponsored by tha Maryland State Department of Education and the Association for Super~ision and Curriculu~ Da~elop~~nt. Although plans are to discuss all educational levels. the ~articipating organi:ations, (over 35 professional associations. centers and qo~ernment agencies) ~re concerned pri~arily with K-12 education in so~e way or anothe~. I'll report on what pro~i5~s to be a ~ost interesting gathering. ~lthin th~ ~aek you should racei~d two 1te~s intsr~3t. One is a plrtial bibliography CT with emphasis on t~sting CT. the other 15 an upd a t ed listing of sone of the ex i s t i ru t-:sts vt.xch purport to =easur~ CT or closaly r~llt~d r~asoning s~llls.
s i nc e r ' ".

Appendix C: VelPn1 Researoh Letters. PAGS 83

The Caitfornia Siale
UOIY(USlly

2Z~~

",
i.,J

-".'

•..

~.'

....,.
E __

p:

#4

#4#*'96'

_ide±;

:d5J'f::£!IJUIiI:IJ:aAMNf>$!UE

'u,

'''I'

California
Fullerton,

CalifornIa

State University, Fullerton

92634

Department of Philosophy (714) 773-3611

RO(/NI>

May 4, 1988 Dear Colleagues, I hope this letter finds you happy and well. It's time for Round 3 of our Delphi -- the round where we finally get to the heart of the matter: What is CT? Also, ;,or those who are new to our effort, this letter includes a brie~ overview of who we are and what we are about. The last three pages summarize our Round 2 results. Please send Round J respon~es by June 25. As a result of nominations in Round 1 and Round 2, sixty persons, inclu1ing some of the most eminent names in the field, are now invite~ to participate in this effort. By way of bQckground, in January the American Philosophical Association Committee on Pre-College Philosophy asked me to chair an ad hoc sub-committee on testing critical thinking. Beginning with an initial group of APA nominees and asking them for additional recommendations, the "sub-committee" has grown to include people from a variety of academic disciplines and professional affiliations. Our unifying concern is in testing CT. However, we do not necessarily share the same conceptualization of what CT is nor do we necessarily agree on how it might best be tested. It is to resolve precisely these two things that we have undertaken the Delphi process. In Round 1 (Feb. 11, 1988) and Round 2 (Mar. 14, 1988) the focus was on establishing group membership and agreeing on preliminary working assumptions -such as the assumption that CT could be operationalized to the extent that valid and reliable assessments of importa~~t and relevant CT skills could be made. Starting with Round 3 we will focus on what those CT skills are and eventuall~ we will decide on recommendations regarding testing, based on any Delphi consensus we achieve. To assist with the conceptual work that must be done, I developed and circulated two items, a list of existing CT tests and a CT-Testing Bibliography (Apr. 19, 1988). If you want copies of any of materials mentioned or if you wish to have a copy of the mailing/membership list, just drop me a line. Last week I presented a workshop on testing CT at Sacramento State. At that time Perry Weddle agreeci to publish the CTtesting bibliography and the list of CT tests in a fall issue of CT News. So, please get any corrections, additions, or deletions to me as soon as you can . AUD llOW OU TO ROUliD 'l'HREE! Appendix C: Delphi Research Letters, PAGE 64

",

****
QUESTION:

ROUND THREE

****

WHAT OPERATIONS ARE CENTRAL TO CT?

Response requested ~

Jun£ 25.

The sole task for Round 3 is for each of us to come up with that list of operations [or p~rformances, behaviors, processes, outcomes or skills] which we understand to be at the core of the concept of Critical Thinking. Although many of us have published on this question, since there are potentially sixty in our group and since I have no assistants, it will be most helpful to me if you would take the time to distill your views and send a list of what you interpret the central CT operations to be, fell free to indicate which are the more gelleral and which are the sub-operations. Naturally you are welcome to include justifications for the items on your list. In thinking about this, please keep in mind that in the two preliminary Delphi rounds w~ hav~ narrowed our focus for now to CT '~opera tions unders tood as performances behaviors processes outcomes ~ sk~lls which could be tested validly and reliably at the college freshman/sophomore level. But keep in mind that we very likely will extend the question downward to K-12 later.
II
I I

In Round 4, which r will initiate in sept., you will be given combined lists and invited ~o comment on the wisdom of excluding, retaining, or amending the descriptions of specific items. If the results of Round 3 are clean enough, Round 4 will also invite you to begin rating items in terms of how ~ore or less important, crucial, central, integral etc. they are to the concept of CT. If you do not intend to respond to Round 3, for whatever reasons, please drop me a lin@. so that I can keep track of participation levels. Matt Lipman suggested that we might have -an excellent chance of working with the APA and the Assn. fer Infor~al Logic and CT tc secure the use of the Wingspread Conference Center. Please let me know if you think it would be productive to get together in that setting. How might a conference be organized to most effectively use our valuable time? What kinds of issues, problems, tasks might we address? What kinds of solutions or desirable results might we achieve by meeting which couldn't be achieved (at all or as well) using the Delphi? since Round 3 asks the "big" question, to respond. please take the time

Sincerely,

-;:;~~

Appendix C: Delphi Research Letters,

--J

PAGE65

68

SUMMARY OF RESPONSES TO ROUND TWO Overall response rate: 23 of a possible 51. Question! 9i Ro~nd ~ asked if cur sub-committee was sufficiently well constituted to move on with our task. As you can infer, the consensus answer was "Yes." A small number of additional names from psychology and education were recommended and strong cases for adding ther were made. That is how we moved up to sixty members -- presuming the new nominees agree to join in. I'll send you an updated list soon. Question ~ of Round l asked if we agreed with this claim: Even if CT cannot be reduced entirely to an equi valent set of operations [or performances, behaviors, processes, outcom~s, or skills,] (2) it is possible to conceptually analyze CT so as to describe a set of relevant and important CT operations, such that (3) using these descriptions, [competent] investigators could, on a consis tent basis, gather suff icient evidence to draw ecncl usions , with high degrees of confidence, regarding the relative CT abilities of a group of people, leverything else being equal, of course].
II

:.:~

(1)

II

There was sufficient consensus on this to move ahe~d. The m3jority of responses (18 were strongly to moderately positive, 2 were negative and 3 did not respond to this question.) However, to ?ovoid mis unders tanding, let us keep i.. mind tha t the above statement should not be interpreted to imply that constr~ct validity can be determined strictly in an a priori manner (by simply coming up with our list). Nor should the words "relevant and important" be interpreted to imply that we can come up with an exhaustive list. Nor should the purpose in (3) be interpreted to mean that we have set our sights on actually writing a CT test, or, for that matter come to any agreement about what recommendations regarding testing we are likely t~ ~ake. Most of the positive responses (13 of 18) were Vf;.ry short, "Yes," or "Agree," or "Yes, this is an acceptable working hvpothesis." Here are two of the longer positive responses I found interesting: "I c1gree with the statemcant offered. I am compelled to note ttl'l' the tasks described in the statement are going to be quite difficult to c~mplete. There will nnt be universal acceptance. There will be criticisms (legitimate and illeqitimate) cf the results tor a long time to come and many will .un a course similar to criticisms of attempts to defirle and measure intelligence." 'I(J) V~S. pvpn thnuMh ~.~ -- ~~~~:h ! ~~li:~~ ~: :~= ~:~i:7~ ~ ~roadly sa tisfying reduction... (2) agreed here, (J) agreed here. Theref ore, yes. And rather than get hung up on this questivn, I'd prefer to jump right in and see whether we can do it. If we can, terrific! If we can't, well, then the doubting T's will have a field day; but I'm prepared to take that risk ••." One of the n"g..ltiveresponses was expressed this way: "I'm. sorry, but I can't help but qet hung up on 'operations' -- the term so psychologizes ard jargonizes ••• the question. Why not go for Iprinciples'? ••" The strongest negai;ive w~.;. registered by a Derson who said, "I fear this
Appendix
:'':''
',.:
< ;~~ "

c:

Delphi Re8~arch Letters,

.J

69

PAGE66
<,»,

.. ...
•

",:',\.

. ., ~; ', <:-." .....

"
.' .

,:'1>
,~

..

,,(,>

. ~! II

proposal to define CT operationally may have the effect of ruling out, by definition, one major position on CT, a position with which many in the CT field mali' in sympa thy, at leas t to some degree." This' per con was concerned be that defining CT as a list of skills would focused on weak-sense-CT and missed stronq-sense-CT which relates to a person's character -- "being self-critical, sPJkinq to overcome blind spots, biases, prejudices .•• , [being] critics of une's society, ... [seeking] what is of value in auother's position •.•" The person asked, "Can these character traits be defined operationally?" In con tras t, note this rnspcnse t "Yes, (1 agree wi th the statemen t] particularly if we confine ourselves to abilities and ignore dispositions ..•" Even though agreeing that "A subset of the processes that constitute CT can be assessed using the multiple-choice format that I assume is being sought," one person expressed serious concern saying, "I fear the creation of an instrument promising more than it can deliver -- an instrument touted n~t for what it is, an assessment device measuring certain important, but rudimentary, CT activities, but rather as a valid and reliable assessment tool for CT." He maintained that "As is so commonly true when discussing assessment, the instrument and its characteristics would then dominate the social construction of what is being assessed, in ~his case CT. The initiation of CT activiti~s, generation of appropriate CT strategies, and defense of a tentative reasoned judgment are.not susceptible to the type of assessment legislators are willing to finance or faculty are willing to undertake. The extensive writing or oral argument required to demonstr~te CT, as I understand it, are not practical inclusions in an assessment instrument." Four who agreed with the approach mentioned important factors which relate to co~struct validity. One mentioned the role of background knowledge in CT, another the role of divergent assumptions, another the relationship of CT skills to reading skills, and a fourth spoke to the need to validate any list of CT operations we might agree. There is much in the research literature about these problems. Steve Norris, in particular, has been working on strategies to respond to precisely these kinds of problems. Although I mentioned all the negatives, the positive responses were far more numerous than the negatives. A consensus to move ahead exists. But we must not forget the warnings and concerns of our colleagues. A great deal depends on what we ~ome ~ with when we actually sit dowll to an§wer the question for ~ound ~ since both the positives and the negatives were based on our ideas about what CT i~. Rou . ~ question I asked if people would be willing to agree my proposal that to get started by giving priority to the college freshman/sophomore level. The responses ran: 14-yes, 3-no, 2-both, 3-abstain, and one that I could not figure out. Since the question was about priority and was not intended to exclude working at the K-12 level, which is, after all, what the APA Precollege Committee is charged with doing, I believe we have sufficient consensus to focus initially on the lower division post-secondary level. Here are some "Yes, give priority to the frosh/soph level" comments: "Most if not all of the CT we teach is directed at this level" "I doubt philosophers should take the lead .••when it comes to K-12." "I'm still unpersuaded to reverse my form.erlv expressed views -Appendix C: Delphi Rese&rch Letters, PAGE 67
"0 ., ,

'">:

.. \:

'~'

\,j
c

,10

',,-,:.:._~_

-.._.'S

":.'-

~E--//:

- -." I,~_:

--

~~.~,

, ,~

intro. undergr.aduate level first, until we get square there, where we live." "I feel rather strongly that we should begin on tho post-secondary level. I agree with those who say that that is where the strength of our membership lies and that that is where the vast majority of our members teach." "Yes. That's where most of the pedagogical action is; that's where the students we're interested in testing and have relatively easy access to are to be found. Later, if we succeed at all here, we can extrapolate to other levels." I agree in the Iigh t of wha t was said in Round 1." "Why not? It's what we know best, and we can always move on to other levels later."
II

'On the negative side: "[Your] recommendation seems arbitrary and not consistent with the fact that this is a pre-college committee." "No, based on the name of the committee." "I would prefer to begin at the K-J level. My posi tion and opposition is based upon pedagogical and theoretical concerns which I assign a higher priority to than to political concerns related to the officers and membership of the APA •••• I am willing to accept that the sub-committee begin with the college level but the project will have to be extended downward then •••" Two people expressed the concern that this was a difficult question to answer until one knew the purposes for testing. [A point well taken.] Another suggested that our goal should be to make "contrihutions to the criteria for a college test ••• [but that] individuals should be encouraged to make up their own [assessment inst~uments] and try them out, obtain correlations with other tests and with outcomes, and then subsequently co~pare notes with one another as to what worked and what didn't."

Thanks to all who responded to Round 2. '{our letters were most interesting and thoughtful. Don't forget -- Round 3 by June 25!

·~tK~ ..
.
;-(.-~-

Appendix C: Delphi Research Letters, PAGE 68 --, -

71

~.\

to

:f~.~~·;. :.\
;"

.~

California State University, Fullerton

Fullot1on, California 92634 Department of Philosophy (714) 773·3611

May 18, 1988 Dear Colleagues, I hope this letter finds you well, happy and looking forward to a restful mentioned Conference as well a~ productive su~~er. In my last letter I

that I would be attending on Assessing Thinking"

the "First National and that I would and

in Baltimore

be sending along a. report. the promised

The conference

was very rewarding

report is attached. the current list of persons invited to We will have to freeze

I've also attached respond

to Round 3 in our Delphi process. According

the list at this point.

to what I've read about the

Delphi method, once the central disruptive

debate is joined, it can be

to try to add people who have not the benefit of With the circulation that point . of tha

earlier rounds in the dialogue. Rc~nd

J

question we have reached

. Many thanks to those who have already sent there responses to Round 3. Don't panic if you haven't yet, though. The target

date is June 25.

Have a good summer. I look forward

to hearing from you.

Sincerely,

<> p... <_.~

~

Pete Facione Appendix C: Delphi Research Letters, PAGE 69

The California Slate Unt~or5ily
.,.(

.....

.. ,

'7'~ i:.
• •.• • • _ •. ,. -_ •. f'

; -it -.' ..' -."

~ "... .1

. . .... -'

.
California State UniveraJty. Fullerton
Fullerton. california 92~

_ ,-'-;r .
: v-

-,.;--

~~~;'::~".

.:

Department of PhilosophV

(714) 773-3611

,June 28. 1988

Dear Colleagues, Thanks for the large number of interesting responses to the Round 3 of our critical thinking Delphi process. Judging from

the length and sophistication of what many of you sent. it should take me a fair amount of time to organize and synthesize the material and t.hen to frame fruitful questions for our next round.

r expect to
3.

be working on this for several weeks. so if for

some reason you haven't had,the opportunity to respond to Round please know that your ideas are more than welcome. Have a good summer and thanks again for so much high quality participation -- it's very encouraging.

Sincerely,

Pete Facione

, .:..c-: ..~

a-

Appendix C: Delphi Research Letters, PAGE 70

=;» ... ~-

The CJhfornia Stale Unl'tor~iry

-';'.'

· .-_J.

California Stale University, Fullerton

Fullerton, Califor

nia 92634

Onpartmeut of Philosophy
(714) 773-36'11

1.-

s e p t , 1,

1988

Dear

Oelphi I'm

Colleagues, through skills. Right play letter the The now them 25 responses quality analyzin&, for two your weeks. is to drop whatsoever Dolphi she rounds. not persons after ~ff the Delphi to Round and care 3, which evident a~kcd fer your responses the

working ~ore CT

list

of

scope, I'm ~ack

in your and

is most ideas Expect so

impressive. that I con

4

synthesi~in9, reaction and

organi4ing

rectification.

the

Round

in about rule made I've no least

A Delphi mailing invitations person's lIst

operationaL if they to respond have

adopted contact three he or

receiving It's fair to

~ay D

to at

sustained here

silence

signals Phil Te~ts,

wishes shared.

to be It (c) lists

included. (a) Critical (d)

Included Reasoning Newsletters Also, you and and might

is somethin~ Logic

Pecorino Cb) Related and

Informal Journals also

Texts,

Logic and

Texts,

reLnted to write and ask It

to CT,

(~) Bell, of

CT Centers Howar~ his of it 132

OrganiZations. College,

want
21044

to James for a copy

Community page Guide on

Columbia,
Maryland

Maryland,
~~

!~~ igr
how of CT Is

Scientists.

Includes backgrounds

a wealth and

material lists

seen and

from

different

disciplinary

a number

ideas

resources. Thanks ogain for so much be high hearing quality from partiCipation me soon. in Round 3 -it's

very

encouraging.

You'll

Appendix
The California
"-

c:

Pete

Facione

Delphi Research Letters,

PAGE71

Slale Uniyersily

",::~,~-

74

California State University, Fullerton

Fulierton. California 92634

L")epartmentof Philosophy

ROUND ~
. .--,

Sept. 23, 1988

l ,

Dear Delphi Colleagues, Round 4 seeks verification of a list of CT skills and sub-skills. Please accept, reject, amend, and comment on the group's responses to round 3. Remember, the goal in this phase of our project is to arrive a't; n accord regarding the skills we understand to be central to CT. a Your responses to round 3 yielded 200+ ~ages. Some sent previous publications, some sent lists and commentaries, some wrote new pieces of clearly publish~ble quality. ,You tended to approa9h the question of identifying core CT skills and sub-skills four ways: (a) by appeal to your own experience and understanding, (b) by citation and comment on what other CT authorities (including others in our Delphi list) have written or said, (c) by describing the key characteristics of persons who have internalize CT, and Cd) by consideration of what should reasonably be taught or included in a CT curriculum. In addition to differences of opinion, there were' variations in disciplinary orientation, vocabulary, and emphasis. There were also differences in the specificity, depth, and scope of responses. Some were extremely general, others very specific. Distilling your 'opinions, positions, views, ide~s, lists, descriptions, explanations, examples, counter-examples, caveats, . credos, and course outlines was one of the most intellectually interesting and stimulating experiences I've ever had the pleasure of attempting. Although I've been '~lching and researching CT for two decades, I noticed that my ~wn vi~ws on the range and character of CT expanded greatly as a result what you contributed in round 3. In naming and describing CT skills for Round 4, I intend to rely on standard English usage and to avoid technical or disciplinespecific vocabulary. Your responses emphasized generic skills. Distilling your responses, I name and describe six generic CT skills and give two or three sub-skills under each. Clearly additions or deletions might be needed. To avoid prejudicing your responses at this crucial time, I do not indicate the numbers of persons who may have agreed on any given point. Areas of agreement and controversy will come out in round 4. In this round you are invited to make a number of kinds of responses to a variety of questions. However, because organizing the material and framing the' issues was such a delicate and complex task, I urge you to read the whole package and get an overview of the terrain prior to starting to reply. Thanks again for the high quality responses to Round 3. To insure we are talking· about the same things when we refer to CT skills, we'll need maximal participation in round 4. If you could consider this material and reply in 15-20 working days that would be wonderful. If you need more time, or want to discuss any aspect of this project, call me at CSUF (714) 773-3742, [office] or 372-3611 [dept.], or 993-1356 [home]. SincE:rely,,........-} [J '1 Appendix C: Delphi Research Letters. PAGE 72
_"',"'~-"

>~;)';.
"_
-_

::'

..~.~.

".

,"

.. ./ ....../.' ..........>. , .

. ..

.... ~"_

.....

--._.

,...... , ""'"'-'

: .'. .:~. ;.'.'

(

.....

'

APA Delphi Project

There

RESEARCH

IN PROGRESS

-- ~qr ~

PUBLICATION

PAGE 2

are five parts to Round 4: (1) The listing of CT sldlls and sub+e k i l l s • (2) A model diagramming the flow of CT skill input (3) A list of things some of you said CT is not. (4) A description of what is meant by "s k i l l ". (5) A list of caveats and comments you shared. Round

and output.

it. E!!:.!. h IT. ski

11 s

Com men t : I" e ' v e all see n e ): amp 1 e s, 1 i' e the d u c I: - r a b bit " 0 f how com pre 11ens ion I~ combines imposing an order on reality as well as discovering an order inherent in reality. Applying that lesson to the problem at hand, there are many ways CT skills and sub-skills could be organized. Even among those of you who essentially agree, the variations in your responses to round 3 illustrate this. After considering a large number of ways of organizing things, I've decided on the configuration you will find beloH. True, the skills and subskills could have been named using other labels or grouped in other ways. To this add that so me sub - 5 Id lIs ma~' c·om e i n top Ia yin m 0 ret han 0 n e .9 en era 1 area, some can operate on the products of others, some presuppose others, some are almost always employed while others may not come into play except under special circumstances, and you have an even more conceptually complex situatio~. Now add that some of us might· exclude one of the more generic groupings, to say not~ing of how we might want to add, su~tract, amend or rearrange the sub-skills, and the complexity of our task takes on greater magnitude. There were other ways to "see" the Round 3 data. So, beside asking yourself if each skill and sub-skill belongs in the list, and if anything central is missing, another question to ask is, like the duck-rabbit, can you see CT this Hay? Ins t r u c t ion 5 : Be l a N y ~IU irJ 1lIT j n d a· 1 l $ t n a 0 I n 9 and d e So c r l b j r, 9 U..!. '0' e C T skill~ and sub-skjlls. Read and cons11er the e~tjre Ijst, Then, ( 1) Co f1 sid e r i.!Lct ~ c h ~l: 1 1 uu: $ u b - s k 1 1..L,.. tM.r...L. tl.~.l!.::t_e..§: ti. ~9.J:L ylliJ.1.. . j Lt te. Ql. 2. ~U U 5 I: ill 2.!.. sub - s I: i.ll. and ~ Lf.. ~ I'f 0 LII dar 9 LI e ~ e l< r. 1LI d e lL... §.!i\te ~oLlr r..g_~ for ~luding 9111.. marl:ed :liB..::... ill "I a 1!1_ Qll2£.B.. 5 LI b s tan ti "e CI men d m e n t s tQ. the des c r J p tin. n s Qi.!!!.}:_ sub!k i..i.LL 2!1_Q.. e:~ 1..ll!l. tt!L'i. the .ch CI n q e i..§.. nee d e d..!.. dd !!l:t.. m iss i n q LU b -! i. J II! p A n am e_ Ws. ~ 5 c: r i Ite.. lli 1 0 cat e_ U. "I i t h in gru.. Qi the s i.lLU 5 I: ills. ~.!l.S ide r in g tb.. ~ ty tt - s k.111 $ \II i ~.bilL e CI. c h.. Q..i. t h e ?jlL U a I: ills! i n die ate H h i c: h sub - s I: i 11 5 LL g.!l.Y...1. a!lP.lli...~.~g_ ~.2.:!e d ~B.. aP...tt 0 tlL~!.5 I: i l.L_ ( 3) AIn e ns. the Li~i. Qi. silL Q. ! k i 11 ! • !.i YJUL del eli. ! s I: ill! i n die B t e ~c:,1. to. d q_ w iJl!.. its sub - s U IJ..L.. !i. ~9..!:!.. add s.. m iss i n g rr s I:i II! ~. and ~!?ic_ri beLl\.. QMlt c3 n d. des c: rib e Ui? u b - 5 k ill S, e :( 1 a i n h 0 1'1 2 1 see n t r a 1 tQ.. C T.. h 0 ~I il. d iff e r 5 fro m 2!!L Qi. the ail:.. 5 I: ill s Q.!l the cur r en t 1 i s t! and ~ \.. L.t m e r its b e iM 1 i s ted.. tl. the illtl. Pi. 2. ? I- ill rat her t han ~ 5 U b - s I:: ill. ill ~n ~j de r th ~. ~I g_ reg a r din 9 the r ole and £..Q..!n.P0 5 i ti 0 n gJ_ ~ hum an· s 0 aT I: n ~I PI 1 ssss: QiJ.1...L.. v e r i f L.. a men d \.. com men t QlL 9..!li:. asp e t: t Q.i t his. 1.§l_ l'Iak_g_.n:L needed edi torial chan~ e 1.6 ) C q m men t, u.~. LI 1'1 i.~_Ql!. the en t ire !J...a1. Q.i ski lIs, its 0 r g ani ;: t ion 0 a
I

u..

\:LliJ

it

'I

llil.U.

b.ll§.. ~

1 urge ~'ou to read the entire list of skills and sub-skills before beginning to respond to any specific item. Thanks!

Appendix C: Delphi ~esearch Letters, PAGE 73

'18
<>:

...

..'~'-,.

'

RESEARCH

IN PROGRESS
Core

-- ill

f].R.

PUBl.ICATION
Skills

PAGE :3

PROPOSED:

Critical

Thinking

TAXONOMY

~:;.E:,:prr-=,ssi ng 6. l'lord"t or i ng

r I1terprr:~t i nq I n f I:r·r i n q :3;. Ptnc:d yz i ng 4. 1:::>... c;\ 1 1.1 f.\ tin q

:l •
.~
"j

R~gul~ting,

Conj e c t LI r i r. g, L> r i;WJ i n q Cor If': 1 LI S ion !-,,s • Loc a tin 9 (~'''IJL\fIlen t 5, F','oU' sin 9 (:",. urnen t~:;. g VElrifying Clc:\ims, Assessing Logic"l Strength. St ~ tin 9 n e S u 1 t s , Df.~ S c rib i n q F'r"0 C E.Hj I.W ~ S •
(.!LH~r

Observing,
yin g

Dscoding,

Clarifyinq.

Reviewing,

Correcting.

DESCRIPTIONS
YES/NO

OF SKILLS AND SUB-SKILLS

1, IN T E R PRE.TIN G: T" C (I m pre ben d the s i 9 n i i i c a fl ceo l' a .. d e v a r i e t y 0 f Ii experiences, situations, judgments, beliefs, rules, procedures and criteria. 1.1 OBSERVING: To detect, attend and correctly perceive experiential input with particular focus on input that ~onveys or is intended to convey data, information, or inferential relationships. 1.2 DECODING: To detect, attend to and torrectly percelve the informational content, rules, procedures, criteria, and inferential relationships expressed in various c~nvention-based communication systems, such as language, social behaviors, dra~ing5, numbers, signs and symbols • • 1.3 CLARIFYING: Ta make explicit, through stipulation or description, the contextual, conventional andlor intended meanings of ijords, ideas, concepts, statements, behaviors, draWIngs, numbers, si~ns or symbols; to remove confusing vagueness and ambiguity; to facilitate communication.

2: INFERRING:
deteraine the

T~ secure infere~tial

ele~~nts needed relatioDshipr

to ~ake inferences bet~ee~ or floNing

aDd tQ
irol

statements,

descriptions or representations. 2.1 QUERYING: At any,point in the CT process, to r e c o qn i z e the need for evidence or information of some kind, and to formulate and execute a strategy for see~ing and gatherIng that eVIdence or' information. 2.2 CONJECTURING: To formulate alternatIves, to develop hypotheses, to postulate suppositi~ns, 2.3 DRAWING CONCLUSIONS: Given a set of statements, descriptions or representations, to educe their inferential relationships and to educe the consequences Hhich they support, warrant, imply or entail.

Appendix C: Delphi Research Letters, PAGE 74

'.;::.i. •. " ,

7~i

• ,:...;f1'; -. t~'
'.:'".

APA Delphi

Project

RESEARCH

IN PROGRESS
It

-- ~

trui PUBLICATION

PAGE 4

,.'

NOT ~: A 5 man y 0 f y 0 L' a r 9 LI ed, all C T s I: ill s, but par tic u 1 a r 1 y sub s ki 1 1 5 1ike 2. 1, 2. ~ and 2. 3 pre s un e u ~ n 0 I~1 e d q_e- bas e • A hum an' s knowledge-base i~ c~mposed of at least these things: (a) a ~I0 rId If i fHI \,1 h inc Iud e son e 's u nd e rs tan din gsa f hie ~Ihat i s rea 1 (a met a p h Y 5 i C 5), • .

*'

* how knowledge is gained and refined (an epistemology), • what is important or valuable (a value theory); (b).a lli..s.. b a s e , including one's cp i n i cn s , beliefs, experiences, ~tc. as filtered through the world view; (c) an inferlt!J..U.engine which includes * general rules for drawing logical inferences (a logic), * sets of procedures and criteria appropriate for making reasonable judgments within specific areas of human thoLlght and inquiry (discipline-specific ~ule;.)

Executing sub-skills 2.1, 2.2, and 2.3 as well as 4.1 and 4.2, 'more effectively can be achiev~d by learning how to think logically, by expanding one's repertoire of sets of procedures and criteria used in different areas of human thought and inquiry, and increasing ene's base of relevant data. An implication of this analysis of CT skills and sub-s~ills is that they transcend specific disciplines, but executing them demands background knowledge, some of which is specific to how one goes about making reasonable judgments in different realms. Becomin~ adept at CT ,involves learning CT skills and learning to use those CT skills more effectively in different contexts -- hence the importance of a liberal education to go along with one's CT ability.

*
J. ANALYZING: Tu ide~tlfy

the inierentlaJ ,~JationshJPs betNeeh st,tements, descriptions or representations which express experi~nces, situations, judgments, beliefs, or opinions. 3.1 LOCATING ARGUMENTS: Given a set of statements, descriptions or representations, to determine whether it does e~press or was intended to express a reason or reasons in support of some claim, opinion or point of view. ~ 3,2 PARSING ARGUMENT5~ Given a the e~pression of a reason or reasons in support of some claim, opinion or point of view, to identify: (a) the intended conclusion, (bl the premlses and reasons advanced in support of that conclusion, (e) additional unexpressed elements of that reasonIng, such as intermediary conclusions, unstated assumptions, and ld) for ehclusion, any items contained in body of expressions being parsed which are not intend to be ta~en as crucial to the reasoning being expressed.

4. EV~LUATIHG: representations;

10 assess

the

,redJbJlity

of statements,

and

tu asse~s

the Jtrength

descrIptions or of the expressed Inierentlal

Appendix C: Delphi ReseArch Letters, PAGE 75

.:l_(~~ :
·e,;';-:"-~';"'f. :
,

.'

....

'18

..,

_,

, • _oo. ~

. ·;~7(:
~ - '. 1

APA Delphi Project

reIativnshipJ

RESEARCH

IN PROGRESS -- ~

~BLICATION

PAGE 5

:'

.'

such statements, descriptions or representations. 4.1 VERIFYING CLAIMS: To assess the degree of confidence to place in a given statement, description or representation. 4.2 ASSESSING LOGICAL STRENGTH: To de t er ni ne the n a tur e and quality of e~pressed inferential relationships; to judge whether the assumed truth of the premises of a given argument justify one's accepting as true, or very probably true, the expressed conclusion of that argument.
betNeen

5: EXPRESSING:
the results

To state, describe of ODe's CT activJtJes

or represent to une'$ spIi aDd the ~ay ODe Nent about

or others prOdUCJDq

t ties«

r e s ul is ,

5.1 STATING RESULTS: To produce accurate statements, descriptions or representations of the results of one's CT activities so as to analyze, evaluate, infer from, monitor or remember those results, or so as to communicate them effectively to others. 5.2 DESCRIBING PROCEDURES, To produce accurate statements, descriptions or representations of hO~1 one applied and executed any CT skill or sub-skill so as to evaluate or monitor one's proficiency, or so as to co~municate to others about how one went perfcrming a given CT skill or sub-skill.

0: !'IONITORING: TL' regulate s l l e s s e c t s (If o u e ": ONrl CT a c t r v r t re s , the • eleaents used in those activJties, a~d the resul:s produced by those actJvities, particularly by applying the skills of analyzing" and

evaluating validating

to one's own inferring with a view toward confirming, and/or correcting the results. 6.1 REGULATING: To sequence one's execution of CT s~ills and sub-skills. 6 • 2 R E IJ IE 1'1IH G: Toe x ami n eon e "s 0 1'1 n C T act i 'I i t j e 5 and v e r 1 f Y both the result~ produced and the correct applIcatIon and e~ecutl0n of each CT Sl;i~l and s ub+s k i l l involved. 6.3 CORRECTING: Where errors are found in one's own CT e c t i v i t i e s , to c cr r e c t those errors and remedy their causes.

Appendix C: Delphi Research Letters,

PAGE76

•• :r .t.

*
ROLmd 4t

RESEARCH
2,

IN PROGRESS

-- NOT ~
Model

PUBLICATION
of

PAGE 6

.Part

An Input/Output

CT Skil,ls

Comment: "nlt?I"e'r:_':, no dmlyincJ thC\t ·the model 1 'va c omo LIp is hecwi ly I n f 1 Lle~11 c (~d, ·f or (j ood or i 11. by my r es~c:.r ch inc: (JcnPLI (?I- SC i ell c F.? cn("l d t a r i~., of i c i,:-d. i n t f? 1 1 i 9 f"J n C E • j The ~ r r Q ~'Jsin cJ i c ate t h P. U ire c t i o n 0 of t h r~ + 1 CH'J 0 f i 11P u t n d ClLIl:. P LIt as i t c i r' c: I_I i.l t e s b Q t vJ e e nan dis 0 per-ate d LIP0 n 1 by t; h t; s i:-: C T ~~ i 1 .1 s , her e 'c on c e i ve'd 0 f a s ~~ k LIfH: t ion s •

i.'

OLltPLIi.: in the form o+ in1~ornlc:\ti(::m'fr-onl the knr.::n·ller.fge basr~, the resLIlts of ol:hetCT fltn(::'l:icJI'H:', i.~ncl Lt s c s elf t·.he sb:;ps pc;r"fonnf::d dur-t n q other CT +unc t i cris , I':: .:p '" t:.:1!::~:;i n C.J l; "1f} 1'\ of 0 r' III ,;;,I: s t; h r.:1 Lou t; P LIt of 0 r- LIS e by ttl r.:' n e :.:t; C'I of U II C t i o I I , : +or' ~~ 1.:0'" ~1(]f? in l: h t;: kn ow l edC:lc-:--b se, a or' for tr an s;mi ss ion out; 0+ tt1E~ C, c:yc:l e. nlater-ial +r om e:·:prE?s~Jing ~.:md, Ins t.h~~ regLllc?tin~l and self-correc::ting +unc t i on , deterrnine~:s Idhere it shc.:tLIld C;}O nen:t •. It can r ou t e things i n t o c:,r ou t of iU1Y of the fiveC) 1:I'H:? I'" C T ski 1 1 s a r- 'l: tHo? !< n 0 vLl e d q e - b ~ sse • For e :.:a In P 1 e , rn 0 nit 0 r" i n q C c?11 loop materi al b a c k t hr ouqh ariv ski 11. ThLI5, it c ari send the resLIl t: t:>f r.Jnf.~'s own in"f_cgl.:cLr:l_9_ for revie\'1 by r ou t i nu them to ~Y.5"l.~1L~.~_to..g. e+or e b all o",d ng ~=11Jdr.'!?-Jf~~j_D.9. to trr.lnsmi t them to others. tJr. it c an r out e infcH'"ma'l:iol1 f.rom ths knol.-lled9fJ bas~~ to ~y_~.t~l.r!.t;.A.D.fl to help it ver"ify a c:1 i~ i III? 0 r t; 0 i.ni.. .[E..Og t 0 h f? 1 pit ~ t cj r a ~'J en c em c 1 LI i em LIsin q c r i t En- i 1:\ s p (: c i ·f i c ·c 0 ~ 9 i v end i sc i p 1 i n t~ •

In

t.his

model,

§:l~iE.r_g_??jn.g.receives

l'Jithin

thf.?

CT cvc l e , t!.!..Q.QJ.tgr-i.nr.trec:eives

CT skill~~ lI\c:\ny o·f t.1:i c:r.~E!nd so rmic h tllTle .. h e Lp i nn ~tLlLl£-.'fll~". b(:?C::(~lll1(? pl"o+ic:i£~rlt i~l., n~l1Iel'l ~_L'.~~::r':I~.t"_gtl..n_(:l.\. ~~n§ty'~-:j.l"..q,. t..~Y..c:3~1~~,:1.l:p~lll c.1!.1.t1. •. ~nJ".~~r. t"1 q r ec f.?i 'It? nti.\ l: t.?ri ~ r cu ted tot .. rj. 1 h em by th ~ (lion i t. or in q ;: LHICt i C.'r1 • They op(:'?r-c.~t(?011 'l:.l1i S rrtlatsri c.\l. And then th(~y s~nd tI1',: resLIl ts tr..) t11l:~ e:.: ,- U'.:~in g f LU'I t.. ion t o b E~ f 0'- mcd: ted f IJr del i 'II:,I rye 1Sf.H·j £:l r e • ·1I\f:? Sl~ p s c h skil.l!s In150 ou t pu c r.\ r·.~{:ord o+ t.lle steps thl;~y pf.:?rfor·lned in corni nc to those rt?~.;l.11tc:::. Th i u rE~cor-~ is:, cr uc i e l if the monitor-ir'tq +unc t i on is 1:0 ~'~r:Jr-k c(.J,"r-tJctly, s i ric a it mLl~t check not only ~'Jlh.~t ~'j~S C\chievr:d t.)I_ti.:

"'hc~ fC.Hlt"' ..

how it

w~s

achieved.

ThE? L!Jg.ttJ.!-"r_.ISJ~ R.~~~7U?. is a storehoLI.se +r cn. \'jhic:I-, interprf...?t:i.nq~ c:.mc.,.ly::.in<;.1 , ~'1r.l.lu,;.\tinq arid lnferrll1(;J t.lra~~ reSOLlr(:es. it st.cwes tl1f: output ct any CT function, wh&rr directed to do 50 by tIle monitorlng +unc c i on , The L.!J.f!.\.:!.~_rg.(~_q1l Q_~?..!-~: ~150 flJ.i.:t:?rs ra~'j ex t er n aI i rip u t; and. t h LIS. i n + 1 l.1E:::" '\ c.:i-J'= t h e ;'.0 t e 1 eo _i..!l9.. 0 f ~j h c:\ t ~'I e cu- e 0 b 5: eo v i rt 1':1, dec 0 d l. n 9 t ro r c J. t:\ r- .i. f Y i. n q •

r.!2L.

I nstrLlcti ens:

1~ IJ.92.:7. Lt. m!':~J::t-? ?'J.-~[I.?.tl ;; .s: 1~:? ~ t __ !~.C;_C;_l_.tr~.~\j:._~?

~;~.:_X.? 1.J;. ~J.s_~ I:A ~ ?_._ f
4.

!;,Q, ~C'.H:!?

§.i..!" S_g'_ ~lf\D.Y R~.qpJ_§'. t.i.Q.c.J. Ri_c;:.t.9_r_i_~l_ ~J_qd_E?L=. t1 f:?..l_p.!'l:-Il . in ~lns~!.~~.:s...'': .. ::1ndj..Dq ~J~.U)P'.,l_,~~).L C t:.~.l.f:~ t:.i..Qf].?hlll.s ..s, t:;§!.1 yg.l:!. s9~.9 '?.~1;. 1.:l~t~·1 t.t!.i. ~ C?!' ~~ WtC;Ltl:t- 9~:.t!l.lpr.9Y~:9. ~-:'L.'1. ~_f \':R.L.~ r:f-:..~:_O..!!!!!2.llrrQ. 9.t~.~~.~:.sJ_to_CJi_t...\. r;_C\_f'!. 'y~q~l. .r..DP.~~S.~ .. P mO_ fo'\l~.§.r 1~~:~i.Y..@.? 1

Appendix C: Delphi Research Letters, PAGE 77

"_-'-('.,

• ->',~

•
_-

80

..

~,'

AN

INPUT/OUTPUT CRITICAL

FLOW

MODEL SKILLS

OF

THINKING

....
•,---_
I

..,
I --~
I

..,
o
If)

"'0

w.

.-.
"

t1"

EXPRESSING
~

'g

K
N 0

5. ~.

W
L

o
II

E
0
G

'8.
a
~ ~

...
if

i' ....,

1 I \ .:
"

l>

ee m r.n m

.....--........ ..

c")
:r:

~;.,

INTERPRETING

E
B A S E
~'A

A II A L VZ 1 N G , )

INFERRING

'. ..-...:-_/
~

<+ (1)

~ r.+

,.

----.
MONITORiNG

I~'
16
l:tl
."

..

'1 CD

--------

c::
tt1

~ 0
tI:l
Q)

...

-.2

100'4

.....

n :D ~

.....
\.I Z

1. C

81

82

·~'.-~i:;"~~; ~,.:. .....

';

.. -{:(:·.APA phi Project ~ Del

';:~;;,'._'

:('.~~

RESEARCH
ROl.md

IN PROGRESS
~: F'ar-t

-- ~JLL EQa PUBLICATION

3, What t;I

PAGE S

:':~l;

i~

Not.

,'_ . j

Nany of Yf.1L.1 di sti nqu i shed CT ski 115 from oth~r c I o:;·f?ly 1"(~lci\ted t.h i nq s , 8elCl~1 i~) c:. list c+ l'Jhat various p er aon s sc:dci ~Ias n9~~.CT. ~.r1.l5t beci:\l.ISe s ome t h i nq is on this 1 isi: dl"JEH.' not ltlt::'iHl a P€uOc;HJi·, (H'HH1 n'c:d:. \.ISS cr in d o i n q the thin(~ nor· tI1c\;:' C\ P(=I""son rn i qb l; not do th~ thing befure, after or during CT. Instructions: ~;.g.ftJl£lJ]DJ;,.9.U

Comment:

t~.?. ~i;_i

eL~~g?~. £.QI.l§j_Qgu:. thi s Li...§.L\..

1.J.~.:._

~_~r:j.. x: i

i.~._,.~.fll~Llg, iL..

~.o.d

YES/NO

CRITICAL
I, Sen sin
q, ( 5e e i n q ,

THINKING

IS NOT
etc.)

TOL.lchi n q ,

He a t- i n g.

2. Reading.

Listening,

3. Speaking 4. Motivating,

or writing,
persuading, s~lling, petitioning, world around~
S
I:)

5. Interrog~ting, ~.
7.

cross-examining, inve$tiq~tinQ the

PhysicaLly

T t- 0 L.I 1 £:" - t:t 11CJ uti n g, b

pro b 1 F.? 111- 1:'11 vi n q , P L.I:::: 1e S s e 1. e c tin q , c h 0 cs i r, g.

(;I

1 v i nq ,
0

8 • 1) l~ c i -= ion .-In r.:d::i n g, 9 • P 1~\ 1n i n ~l, 1 10 . Fin din q

1 1.

~:: ere i 5 i n l)

n G~• s "d. 1 1 •

de fin i n g goa 1 5 and

0r
c"\ 5

b j P. C t i ve s ,
Ii\

c rib i ngam
0

e ~n i n 9 t IJ r bel i e f,

r t,

1 i 'ff?)
C i\

. 0 rev

ent s ,

1'"'

...

D C' -t end i n q

on n

pin

ion

a r- g u i n 9 a

= eo,
Ot··

t·lr~naql.nCJ,

admil1istratinq,

cr' gavurning

per£;':Jn~)

t h i ric s ,

13. Philosophizing,
14. Conducting

research

within

any
q,

particular
0

disciplin~,

1. 5 • E>: per i e~ c i n c , 11
16. Comrnun i c e t i n q

fer:] 1 i n q , u ar nq

ern 0 tin

r' emp .n 1:.111 ;: i n q ,

1 anyL.laqe.

'e~'

Appendix C: Delph! Rese8.Y'ch ~tters,

PAGE79
.,-

:f" .

~'~~~~:~:>~':~;?':':.\:..;. __., :,'/ ,.,t;" :.,",. -:

:;\r,,

83
'0\""""_ ',,',"_ ., ...... ' ........ r , "

' :'.~.

..

~:: ~

.·X~~{\APADel phi

Pr c Jec l

*
ROll

RESEARCH

IN PROGRESS

-- ~

trua PUBLICATION

PAGE 9

':~',<

n d 4 i, Par t is.. Ih e Con c ~i 91. !. 5 k i u_

Comment: It became clear from your responses that it would be useful to have an understanding about what a skill is. Although there may be very little disagreement about this, 'ome of you mentioned subtleties others mayor may not accept. Based on your contributions I 've written a little narrative. Ins t rue t ion s: B.! yo 5 e, e d iJ_ and com ole t e t h ~_ f 0 11 0 ''I in 9 n ar r a ti ve.J_ i A $kill Nhat i~ the ability

to do something

Nell.

Having That

a skill

includes skilled to apply procedures. li~ited involves at

knoNing

to d~, Nhen involves

to do it, and hON to do it.

knoNlng
a set of procedures,

is, beihg Nhen those but not teachJng

something procedures, Skills coaching,

judqih9

those

and being Jught

proficient in a variety and

or adept of Nays

at executing ihcludihg, Part of the they

car, I

to,
laking Skills drill,

demon~trating, explicit thr~ugh

h.

training. Nhen

the procedures
cah

and shoNJng
a cOlbinatioh

and hON

are employed. quided lore practice, or le$s

be learned

of obrervatioh, be judged as beJhg

JS

ahd self-,orr~ctJI in
a

Persons

can

proficient as it
JS

9iven

skil;.

The first

~ay of assessin~ Nay il to compare !o:e set

to observ~

the skill (if any) that

A third

bein~ from query they that

performed. executihq persons are

uSJng

A second
a given ski:l

the outcoaes of criteria.

result

against

~ay is to

and receive

their

descriptions that !kJll,

of the proced~res ~ould that sklils avaJlable use skill. if they Si~ce

and Judqments Nere to periorl

as they perform they

skill,

or did use ~hen

perfor~ed of aental Nays superlor are

~~ cannot

directly skills,

observe only

the perfor~ance
and

the Nay

Ne cah phYSlcal. Nho ~ould ••••••••••• assess ,

the second

thJrd

to those bec~use

CT.

The second

Nay is (lS ~ot)

to the thlrd

provided

that •••••••••••

84
~~.
'.":

Appp.ndix C: Delphi Research Letters, PAGE 80

. -/·....... ·-:\·.::;\_·x,,,...,_ '- .
..:

.~)

-;' ,.:..i_I!f : ':IV':.

»; , ,Y•.,;' .»~.

...

:~t'
,-.," ~ • -.e-;~"~:; .

,':

,T0

: .:f~··" .

_.0.-::,0'"

~ •••

A PAD e 1phi Pro j e c t

*
Round

RESEARCH
4: Part

IN PROGRESS
5,

-- [[L EWl PUBLICATION

and Cautio.fJ.?.

PAGE 10

Caveats

Comment A number of you sent comments. Some were intended as caveats or cautions, other as encouragement. I appreciate them all. I thought some might be good to sharei In many cases to save space I have paraphrased. might agree or disagree with what your coll~agues have ~aid. Or, reading 'list might prompt you to pass along a contrary view. Let's find out. Instructions agreement reflections.
QL

You this

Consider these ~i sagreement.

comments. Use these

~ ill.. remarks remarks, ~. sprinQ

l!LLJ. wistLt whether ill boards for 19J:!L 9J:!.!!..

AGREE/DISAGREE in mind that our goal in defining CT is to do some assessment. But you dor.'t just start testing people. As~essment needs focus and purpose •. 2 • I a g r e e I,d th 1'10r I: i n gat the col leg e I eve 1 0 n I y b e c a use I want to have an idea about what He ~hould be doing with kids in K-12. 1 think we can use what we say about CT for college frosh/soph to guide curriculum development in K-l. CT. Of course, we will havp to adjust reading levels, background knowledge expectations, and lots of other thing5 •. 3. Skills are not the same as operations. Resist behaviorism! 8ehaviors give evidence that a person has a skill or ability, but a skill is not a set of behaviors. 4. Don't trap yourself into using the jargon or vocabulary of anyone discipline (especially philosophy) when y'ou describe CT. S. AI tho ugh n 0 H 0 r d H ill esc ape e r i tic i Sin, don' t use d e du c t ion 0 r in d u c t ion. A v ai d s e ID ant i c spa ts • b. If CT is a set of attitudes as well as a set of skills that poses no problem for assessment because we can just develop ways to assess the CT attitudes, too. 7. C Tis th i n I: i n 9 5 k i I Is. Say in geT i s a 5 etc f attitudes may be a way of describing what people who are good dt CT ar~ like, but it is not a way of describing critical n' king itself. 8. Even if lie agree on ~Ih~t CT is, I'lestill have to face the problem that any student might get the right answer on a CT test but for the wrong reason, or might get a problem wrong but have done a good Job of CT, 9. When assessing CT He should not ~uplicate efforts with areas already well tested by existing instruments, such as covered by reading or intelligence tests. 1 (J. You don' t h a vet 0 t est ,? 'I er yin 9 1 e C T sub - s L ill t 0 decide that ~ person i~ qood at CT. 1 1. I I 0 0 I: e d a t that lis t 0 f e:~ per ;. ~1 and y 0 II h a v e all the big names I can think of, but you can't possibly exp~ct those people to agree. If they did agree, even on what CT is, that would really be something. Good luc~!
II II II II

1. Keep

.)

FEEL FREE TO DISCUSS PLEASE

':;'...,;"r..,. ~:~;,.
"

THIS MATERIAL

WITH COLLEAGUES,

IF YOU WISH. GROUP.

...

~.. .. ~.'"
. .,."\
\ '_"

DeM'T REPRESENT ANY OF THIS LETTER AS THE OPINION OF OUR DELPHI Appendix c: Delphi Rese~oh Letter8, PAGE 81
the
.
,;

.~

.',
i~

Ene:

tl

Thanl:s!

l~...

. ."

~').~'.; :. '.

~~:"",~.~~.""!.",,, ......;:)...~~.;.~.::: -;.._ ..~ . _"

...•

'

..

85

...

,,'....

.. ,i.

California Stat~ University, Fullerton

Department of Philosophy (714)773·3611

----------------------------------------------------------------------------Nov. 22, 1988

Fullerton. California

92634

Dear Delphi Colleagues,

During this Thanksgiving season and I want to express my gratitude to you for your generous participation in this research during 1988. The many responses to the very long and difiicult ROUND 4 have been most gratifying. There won't be any more "ugly-long" rounds like that, I promise. Let me pass along this quick review of what we accomplished in 1988. First, working under the auspices of the Amerl~an Philosophical Association Committee on Pre-College Instruction in Philosophy, we built the Delphi list of expertg. By your recommendations, during Rounds 1 and 2 (Feb. 11 & Mar. 14) we expanded the original APA list of about ten names to a working group which numbers around 45 active partici~ants and which, I am proud to say, includes many of the most important people in CT research today. During Round 1 and 2 we agreed that whatever CT is, we would be possible for us to make the concept operational to the extent that important parts of CT could be assessed validly and reliably. We also agreed to begin by identifying the core elements of CT expected at the Frosh/Soph. general education college level. Ve agreed to use this college level theoretical-construct of CT to guide what is said about CT the K-12 levels. Round 4 (Sept. 23) sought to verify the concept of CT which emerged from Round 3 (May 4). A quick look at the results of Round 4 is most encouraging! Along the way we shared journal articles, lists of existing CT tests, CT bibliographies and other items of mutual interest. 1he Delphi, however is not a SUbstitute for the fine work being done by journals, newsletters and the many centers for CT that have emerged in recently. While I work on analyzing the results of Round 4, I invite you to consider where do we go from here. Last spring I outlined a four phase Delphi project. When '~e achieve consensus on the core list of CT skills expected at the lower 'livision college level, we will have completed the two of the four phases. Originally phases 3 and 4 were described this way: Phase 3: Recommendations The goal of this phase is to communicate our findings about what CT is and whether there is an adequate way of characterizinq CT operationally so as to permit its being tested at some educational level. Dependinq on our results in Phase 2, we will recommend either that programs aimed at testing CT be abandoned, or that they be focused in certain ways. If this is the direction Phase 3 takes, then we will also try to come to consensus on recommendations regarding the relative importance of dif~erent kinds of CT sub-skills and possible ~trategies for ~ccessing and measuring those sub-skills. Phase 4: Design and Validation of Kodel Testing strategies Contingent on the results of earlier pbases, the goal, if it were considered achievable in principle, would be to cons truct and evaluat~ different approaches to testing CT at some appropriate educational level or levels.. \fe might find ourselves breaking in'~o'sub-commi ttees to achieve this g~al, although all ~r.k will
•.
'.

-a.' ..
,'

...•

.' " :':.

." " .• ~._ ..•• ,_"V.~;,: •.'" «,
"

Appendix C: Delphi Research Letters. PAGEB2

,.~._":":'.~~.'!..,, :'~
1 .• ."_'.
v : ~ •• :~,,~_

..1 .....

'

•••

•• _._

.. _··.

"~"'!.

~I).'"'_'

_.t· ~-~0::...... I .... '·

:~h(

••••

'

"

_.

_ •• :.

_.

, -. ':

'

have to be guided by the agreements reached in earlier phases and as well as by the special expertise of those who understand the intricacies. of desig:'ing, piloting, norming and validating educa tional tes ts at 'spe ci f ic educational leve ls. To date I've done little regarding preparing to communicate findings. Two CT newsletter~ inte~ested in publishing something our results have contacted me. Also, the Pre-College Committee session at the March 1989 APA meetings at UC Berkeley. At that I'll be outlining our Delphi process and what we have agreed on time. More suggestions are most welcome. our regarding scheduled a session by tnat

For many reasons I am extremely skeptica! about actually developing a good CT test using a Delphi process, Once we declare consensus regarding the theoretical construct of CT for the general education (lower division) college level our choices include at ieast these three, and maybe more. (1) We could move on to consider questions like these: Given what we understand CT to be at the college level, what does CT m~an a~ different grade levels in K-12? What is the relative importance of thE skills or sub-skills i~ our college-level CT construct in terms of testing, say jJnior high school students? How might one write a question which assesses a given sub-skill in, say 5th graders? (2) Having declared consensus on a conceptualization of CT for use at the college level, we might recommend that test makers at all educational levels be guided by our conceptualization. But we, ourselves, might decide to leave the matter of writing specific tests for specific age groups to others, better qualified than ourselves for developing and valida~inq such instruments. Those of us interested in specific grade levels could be put in contact with one another. (3) When we reach consensus on the CT concept as it applies at the ~ollege level, we might recommend examples of how questions framed to address these skills and sub-skills in college students or K-12 students. We could share these ~xa~~le questions and evaluate them. Those which we think a priori might be good ~o assess certain skills or subskills, could be included in recommeneat~ons we mak~ regarding CT assessment. These questions would not be a CT assessment tool, At best we might think of them as models of how to c~nceive of questions that might be included in a CT assessment. Nota: Even with questions which a priori se~m to address the proper concept of CT and avoid other difficulties (like relyifig on special background knowledge or esoteric vocabulary), there is still the problem of a posteriori verification. Steve Norris has done important work on how to overcome the "cons truc t -val ida tion problem, namely de: termi n ing, for any given test item, if students get right answers because of good CT s}~ills and wrong answers because of inferior CT skills.
II

Give the issues of what do to next some thought. I would welcome hearing from you on this. Have a joyous hvliday season and thanks again for contributing so generously during phases 1 and 2 of this project. Sincerely,
"
.' ..

,/ "'/~-.,
'

...

, ..-

....~---,

----

L.,..,_.

'-;':'.-~~,..

Appendix C: Delphi Research Letters,

PAGE. 3 8

.~':~{~;):;f\~)~:~~); ...;,:, .:~",,'::~'::,o.,=.:::,

.....

_'.'

..

'

87

lito .•.

. '"

•• ·-~.t -,.:

,\.

California
Fullorton,
Department of Philosophy (714) 773-36~1

..

California

-----------------State University, Fullerton

92634

..

Feb. 7, 1989
Dear Heroic Uelphi Colleagues,

34 people responded to Round 4. A great retur~, our biggest so far -both ill numbers of persons and in numbers of pages! 1 ..i i l l tally, analyze, distill and share "dlat people s s i d on each c hur ': of Round 4. lhal ..,ClY you will havQ the b~nefit of an overview of what others in the Delphi are thinking. But, intending lo spare all of us any more horrendously bur' dell S 0 mer 0 un d s, .., hen i teo mest 0 for mU 1 a tin CJ the n ext set 0 f que s t ion s , I ' 1 I h e e p t h (~RI a 5 S h 0 r t a 11d f CJ c: LIsed asp 0 s sib 1 e •
1 pclrticularly wanl to t h ank the I!'any people who sent me d e t a i l e d , thoughtful, (and even fu~tnoted) r~sponses. Several ran near ten page~ single spaced. ~ot inten~ing to diminish the value of the ~~ief "Yes/No" res p CIn s e s a s w e a p pro a c: h C tI n 5 ens "I SOli C rue j alp 0 i "t s , I mu s tat I: nOl'I 1 e c.J 9 e and co~vey my appreciation for the many extra hours of wor~ several of you i:i rep LIt t i (I gin. Al s c , l' vel e c1 r ned a 9 rea t d e a 1 fro m you r sen sit i v e and sen sib 1 e com IT,e Il t S • Yes, 1'It? are approaching consensus. Ny first two r e ad i nq s of the ir,put on koulld 4 is that we have a great deal Of accord on the list of I.:T 5~4ll~. a 1 tho LI g h l her e ..Ii lib esc 0 res 0 fad jus t f.\ e n l san dam'.? n d me 1I t S t 0 tJ emit de. Also, your comments on whdt a s~111 is and hON a cognitive s~ill can be assessed suggest l'Ie ar e c l o s e there too. But I'll be sumlllar.izing all that and more for you very soon. 1 n l II a "fI >: t f e~, ..I a e I: s I' lib e sen din g y c u a 4 e w qui c k s h 0 r t r 0 u 11d s • 1'1 y ~ 1 it n i 5 l 0 4 0 c use a c h tI r i e fIe t t e ron 0 ned i s ere tea s pee t 0 f 0 u r Ii0 r L \·1 (? '= h C,I U 1 d l r y t IJ com p l e t e s e 'I era 1 min i - r 0 u n d s t his s p r i n 9 • T II a l way l' 1 I b (? a b 1 I:! t o r ou 9 h 0 uta fir s t d r a flo f 0 u r rep 0 r t tot h e A P A f' r e - Colle 9 e Philosophy CUrTlmitlee during the SLimmer. lhinl:ing about recommendalions.

ve

I ' "t? l' n c: 1 "S l! u ~1 1 i s l CJ t t tr e p e 0 p 1 ~ who are par ti c i pal i n gin 0 urI) e 1p h 1 r f! S ~ C\ r c II ~I I" 0 j ~ l. Tile" R 1" " R2" .. R3 " and c kif" s YITI b 0 1 sin d 1 cat e the r 0 L\n d 5 t~ which the p~rson has ~~nlributed. The "l" symbol me~n5 the person c cnnun i c e t e d an interest to be involved, but has not r e sp on d e d yet to any rounds. Cur r e n t l v there are 56 !lames an the list uf people being invited t (1 r e 51p 0 n d • A f e ~I h a v e n eve r res p 0 n d e din d n y way. 5 0, '" hen 1 teo mest. 0 mJ I: i n y o u r fin i:i 1 rep 0 r t , 1 e x p [! C t the ret 0 b ear 0 II n d 5 0 0 f u S 1 fI the 9 r 0 up.
II

etc.

1 I b e i II t uu Ctl ..Ii t has v~ry SOOIl. lhanks again

1'

u mmc'\ r y for all

fRo u n d 4 and your worL Si nc;Ze~:.!..v

. ~I'''''''-\ \.

so me min i - r 0 u n d s 5,

(.,

f'eter

, - --=---A. Fa::ione ." .' .

.., ...~ .
,

: .'::l~::::
):

Appendix C: Delphi Research Letters, PAGE 84

Tho California Slato Universlly
" . _, .._, •. _.

.. ........ :.~~'.'_'.~~.::_. -~ - :::-.~.":."._'.'.:~-~.~';.:_~:.-i,.:...:..". -

,. ,,_ ~ .__.~ ,'. 1._..

'f

•. "

. ",

88

.,,'

':. ~:7 '.~~ •• ',.;; '.-.:.:~ :~.'

_",;2"·

U
: .' . '_ ;,:

California
Fullerton,

California

State University, Fullerton

92634

Department of Philosophy
(714) 7"73·3611

DELPHI ROUND 5-A

Feb. 28, 1989

De.ar Delphi ColleagueG, One of you wrote, "J'm beginning to think that to have done all that [we have doneJ and not have tried to define CT may turn out to be a mistake.
1I

Also in response to ROUND 4 three or four of YOlo' commented like this: "I have no major quarrel with any parts of your organization, [butJ your emphasis leaves out a major component of CT -- the dispositional component and the set of values i'1herent in being a critical thinker ••• I think it is a good working model of CT skills, but an incomplete picture of being a cri tical thinl~er.1I In view of the many positive responses to ROUND 4, yet sensitive to the concerns raised by comments like the abcva, ROUND begins by building on 5 our success in articulating a dec.ent first draft list of CT skills. In this letter we start right in on the question: For purposes oT ner ral education assess~ent at the college lo~er division level, ~hat do ~e experts reco ••end be included as a core critical thinking skill? In addition to asking YOLlrendorsement of a revised draft of the skills dimension of CT, this letter also shares some key ROUND results 4 and some of the many useful comments you sent. The next letter, ROUND 5-B, works on the two other aspects of CT you commented on in your ROUND responses 4 -- rHlmely eT's dispositional and the normative dimensions. Some of this also finds its way into the revised sJdlls statement -- see 6.1! RaG, J 5-C pic:ks up the remaining pieces of ROUND and asks your approval of an outline of our report to 4 the AF'ACommittee on Pre-College Philosophy. The table at the end of this letter shows that over 85'l. of us, (23 of 26), could lJe described as fundamentally in accor-d with our first listing of CT skills. The second draft you are now being asked to consider endorSing was prepared in view of the many helpful comments and suggestions you sent in. I am very optimistic about the revised statement of CT skills, first hecaLLse ROUND 4'~ dr'aft was approved by such a solid plurality, and second because your suggestions helped me substantially strength~n and enrich tha.t statement. Since we are very close on so many things, your approval or disapproval of the expressions of our views presented in ROUND should clarify 5 things enough for me to start putting together our report to the APA Committee on Pre-Collegp Philosophy. Where we have consensus our report will say so. ,Where we diverge, it will say that as well. I truly appreciate all that vou have already contributed, and I realize you you are all very busy folks. Yet I beg vcur' cont inued indulgenct'? Please respond to the three round 5 letters with all reasonable dispatch. All responses are welcome, no matter how brief or seler:ti vee Wih sincerest t
. 'J

;:?.i":~t-~'L'~~{ ',.,

;;}~~~t. \'~

ThO Califo:nll Stato Unlvorslty Appendix

.,j,~

C: Delphi
"

Research
.

r: ::~-.~ ~~~ ~-i\~!.:>~

~t".\~,: ,._:_;_~:.).

·-~L .••• ;:.\.:.

"

. .-:'

._

, t, ...

..

,.

:.....

.V_.....~., ~ . ....
._ .,._
"

Letters, PAGE,·85

gratitude,

,j

,..

." ,

,·.:--.h

_
.

89

..

....

...

".
"

",,,._,";'

......,',
*.''';'

. ..
'

~ .:

Half of this letter shares commenls regarding defining and testing CT made in response to ROUND 4. Before digging into our revised statement of CT SkIlls, you might jump to page 6 and look through the comments or examine the tabular results of our earlier Hork on page 10. I learned, for example, that not listed in our original statement was a skill the majority feels is part of CT -- arguing. And, given what this Delphi project is all about, how on earth could I have emitted the CT skill of analyzing an idea from the first draft? ROUND ~ PART

.L:.:.

rr SKILLS

INSTRUCTIONS: Consider the following a~ended description of CT skills along with the accompanying statements. Starting with the title and preamhle, make any needed changes, deletions, or addi~ions (editorial or substantive). After working through the descriptions and statements you will be asked specific questions regarding endorsement. Please respond to those question as well. Skills Dimensions ~ CriticJI Thinkin~

For purposes of general education assessment at the college, lower division level, we understand Ci to include the cognitive skills of interpreting, inferring, analyzing, evaluating, expressing and monitoring. Because of our collective conviction regarding their centrality to CT, He urge those persons interested. in assessIng the stills dimensions of CT focus on these six abilities. However, since CT can be subclassified in a number of legitimate ways, our subclassification should not be interpreted as an educational taxono~y nor as implying or presupposing any psychological, logical or episte~ological order or sequence of skills. While including those s~ills He take to be central to CT, we do not claim' that our list is exhaustive in either breath or detail. Critical thinking involves actively interpreting one's experiences and self-consciously ~aking and expressing one's analytical, evaluative ~nd inferential judg~ents regarding what to believe or do. As .such, critical thinking is a pervasive and multi-dimensiohal human phenomenon involving both dispOSitions and skills. Without di~inishing the vital i~portance of cultivating CT di~positions throughout the K-12 and post-secondary educational proce~s, He have here chosen to focus our attention on listing and describIng CT abilities. As a goal statement of .Hhat a generally educated college level critical thinker should be able to do~ He hope our cons~nSU5 description of CT Skills will assist in CT assessment and CT curriculum development both at the college and the K-12 levels. Among the ~any ways one ~ight improve one's CT arp. by reflecting on one's reasoning processes and learning hOH to think ~ore analytically, objectively and logically, by expanding one's repertOire of those· ~ore spcrialized pro c e d u res a d c r i t ~i· i a use din d iff ere n tar e a 5 0 f hum ant h 0 ugh tan din qui r y , and by increasing one's base of information and experience. An i~plication of our analysis of CT skills, however, is that CT skills per se transcend spp.cific disciplines, yet executing them successfully in certain contexts deman~s background knowledge, so~e of which ~ay be specific to hON one makes reasonable judg~ents in that context. Since beco~ing adept at CT involves learning to use CT skills effectively in many different contexts He cannot overemphasize the value of a solid liberal education to supplement the honing Df Dne's CT skills and the cultivating of on~'t CT dispositions.

I'

-::;~,.
::: .'.:\;\.J'.:; "." .... .., ..... <
__ •.. •

Appendix C: Delphi Researoh

. ',,\. .

Le"vt:l&.8,

PAGE86

90

.1

.
Hales of Core C1 Skills ahd Sub-~kills

(Oo.t.:-

1. Interpreting 2. Analyzing 3. Evaluating 4. Inferring 5. Expressing 6. Monitoring

Categorizing, Investigating, Decoding, Clarifying. Analyzing Ideas, Id~ntifying Arguments, Analyzing Arguments. Assessing Clai~e, Assessing Arguments. QUQr~ing, Conjecturing, Concluding, Developing Rea50ns. Stating Results~ Describing Procedures, Stating Arguments. SElf-examination, Self-correction.
of Core CT Skills

Descriptions

and Sul ·skills

1. INTERPRETING: To comprehend the meaning or explain the significance of a ' wide variety of experien:es, situations, data, events, judg~ents, conventions, beliefs, rules, procedures and criteria. 1.1 CATEGORIZING: To formulate categories, distinctions, frameworks or questions, and to describe experiences, situations, beliefs, events, etc., so that they take on comprehensible significance or meaning, as for exa~ple to recognize a problem and define its cnaracter without prejudice to inquiry. 1.2 INVESTISATIN6: To actively seek, attend to, discriminate and describe experiential input relevant to a given situation, problem or cencern; to gather input that conveys or is intended to convey data, information, or inferential relationships, as for example to gather evidence relevant to solving a proble~ in the light of how that problem is defined. 1.3 DECODINr,: To actively detect, attend to and correctly understand, the informational content, tffective purport, directive functions, intentions, purposes, symbolic significance, values, views, rules, procedures, criteria, or inferential relationships expressed by others in convention-based com m u n icat ion s y stells , s u e has i n I an g u age, soc:i alb eh a vi Drs, :'"' d n g s , u, number5, signs and symbols. 1.4 CLARIFYING: To explain, paraphrase or ~ake explicit, through stipulation, description, analogy or figurative expression, the contextual, convention~l or intended ~eanings of Hords, ideas, concepts, 5tate~ent~, behaviors, drawings, nu~bers, siqns, sYBbols, rules, events or cere~onies; to an extent proportionate with the purposes at hand, to use stipulation, description, analogy or figurative expression to remove confusing, unintended vagueness and a~b~guity, or to design a reasonable rro:edure for so doing. 2. ANALYZINS: To identify the intended inferential relationships a~ong statements, questions, concepts, descriptions or other for~s of representatlon intended to express beliefs, judgments, experi~nces, information, opinions. 2.1 ANALYZING IDEAS: to identify expressions used in co~munication and determine the role they are intended to play in arguing or persuasion, as for example to identify a phrase intended to trigger a sympathetic emotional response and induce an audience to agree ~ith an opinion; to identify related judgments, views, or concepts and to deter~ine the conceptual si~ilarities and differences between them; to identify issue$ or problems, determine their component parts, and identify the conceptual relationships of those parts to edch other and to the whole. 2.2 IDENTIFYING ARGUMENTS: Giv~n a set of state~ents, descriptions, qu~stions or representations, to deterline whether it does express or was intended to express a rea~on or reasons in support of or conte~ting so~e claim, opinion or point of vieM. 2.3 ANALYZING ARGUMENTS: Given the expression of a reason or reasons intended to support or contest some claiB, opinion or point of vieH, to Appendix C: Delphi Research Letters,
l.J

PA3E 87

..

_".

91

l~-

:.~.~t :.,.l. __

..

~ .. ~
'.

-;

... '

.....

identify: (al th~ intended main conclusion, (b) the premises nnd reasons advanced in support of the main conclusion, (c) further premises and reasons advanced uS backup or support for those pre~jses and reasons inlended as supporting the ~ain conclusion, (dl additional uneKpressed elements of that r e e s on i ne s u c h Jr. i n .. .... ......J c ..... ,t..w""... o .... un c t c t c d ...... ..t i cn s or "' "'dl' cnc l u i '·r. ~"'''''''~w presuppositions, (e) the overall structure of the argument or int~nded chain of reasoning, and (f) any items contained in body of expressions being examined which are not intend tc be taken as part of the reasoning being expressed or its intended bac~ground.
~,_...... .. ._.101.. 'WI

11_,

W•

.JW

.. '"

,
•

3. EVALUATING: To assess the credibility of statements, descriptions, questions or other representatio~s expressing experiences, situations, beliefs, judgments, or opinions; and to assess the strength of the expressed inferential relJtionships among su~h statements, descriptions, questions or other forms of representation. 3.1 ASSESS1N6 CLAIMS: To assess the degree of credibility to ascribe to a source of information or opinion; to a: .. s r the relevance " of questions, i n for mat ion, p r in ci p 1 e s , r u 1e s 0 r pro c e d v ,: e c t ion 5 t 0 a 9 iv e n iss u e D r cencern; to assess the truth or the level oJ" " idence to place in any given representation of an experience," situation, judgment, belief or opinion. 3.2 ASSESSLNG ARGUMENTS. To determine the nature and quality of expressed inferential relationships; to judge whether the assumed truth of the premises of a given argument justify one's accepting as true, or very probably true, the exprp,ssed conclusion of that argument; to anticipate and raise questions and objections, and then to assess whether these point to significant Heakness in the argument being evaluatedl to determine "hether an argument relies on false or doubtful assumptions or presuppositions to Judge how crucially these affect its strength; to Judge between reasonable and fallacious inferences; to judge the probative strength of an argu~ent's premises and assumptions with a view toward deter~ining the acceptability of the arguBent; to determine and Judge the probative strength an argument's intended and unintended implications with a vieH toward judging the acceptability of the argument; to judge the extent to Hhich additional information would strengthen or Heaken an argu~ent.
I :"

4: INFERRING: To identify and secure elements needed to make inferences and to deter~ine the inferential relationships between or flowing from statements, descriptions, questions, or other forns of representation on the basis of which inferences can be drawn. 4.1 QUERVING: to recognize the need for evidence or information of some kind, in particular to recogni:e Hhic~ statements, including those offered as premises, need justification, and to for~ulate and execute a reasonable strategy for seeking and gathering that evidence or infor~ation. 4.2 CONJECTURING: Glven a problem, question or point of view on an issue, to formulate multiple alternatives, develop hypotheses, or postulate suppositions, and to design reasonable st.rategies for deter~irting their plausibility, viability or rqlative merit; to objectively draw out the presuppositions and the consequences of decisions, positions, beliefs or vieHs Kith which one lIight agree or disagree. 4.3 CONCLUDING: GAven a set of statements, descriptions, questions or other forms of representation, to educe with the proper level of logical strength, their inferential relationships, both deductive and inductive, to educe the consequences or the pr~suppositions ~hich they support, warrant, imply or entail; to successfully emplo~ iar'ous sub-species of inductive or Appendix
• :":l.;~".'"'t:':.~-,;\,"-"-"_~"".,,,,.

: .::~~'f.-\~"
. ,.'

<~. ,l,'.·

C: Delphi

Research

Letters, PAGEB8

"

.'

92

. "(f,','

... G ...

:.. ~".~ J ..

','! '.~';l.' •

•. ~"

,~I

deductive rea~oning, as for example to reason analogically, arith~etically, dialectically. scientifically, etc. 4.4 DEVELOPING REASONS: Given a question to be answered or a position on an issue, use appropriate inductive or deductive modes Ot inference to articula~e reasons for answering the question one way as opposed to another, or for supporting or for opposing the position.
5: EXPRESSING: To state, describe or repre$ent to one'~ self cr to others the results of ene's reasoning and the way on~ went about producing those results. 5.1 STATING RESULTS: To produce accurate statements, descriptions or representations of the results of one's reasoning activities so as to analyze, evaluate, infer from, or monitor those results, or so as to accurately and effectively recall or represent those results to one's self or to others. 5.2 DESCRIBING PROCEDURES: To represent as cl~arly as possible how one cam e too ne •5 i n te r pre tat ion s , a n a 1y s e s , e val u a ti 0,"~~ r i n fer e n c e s , sot hat 0 ne 0 might accurately record, evaluate, describe or justify those processes to one's self or to others, or so as to re~edy perceived deficiencies in the general way one executes those processes. 5.3 STATING ARGUMENTS: To present argu~ents which communicate one's ground~ for accepting some claim, their logical force in supporting that claim, and, as nece5~ary, ~eeting objections to the pre~ises one relied on or the reasoning one employed.

To self-consciously regulate one's co~nitive activities, the elements used in those activities, and the results produced by those activities, particularly by applying analyzing and evaluating to one's(own inferring with a view toward confir~ing, validating, correcting or questioning either one's reasoning or ene's results. 6.1 SELF-EXAMINATION: To reflect carefully on one's own reasonIng and verify both the results produced and the correct application and execution of the cognitive skills involvedl to make a thoughtful meta-cognitive selfassess~ent; to reflect on the extent to wh:ch ~ne's thinking is in~luenced by deficiencies in one's knowledge, or by stereotypes, prejudlces, emotions or any other factors which constrain one's objectivity or rationality; to reflect on one's ~otiyations, values, attitudes and interests with a view toward deter~ining that one has endeavored to be unbiased, fair-~inded, thorough, uujective, respectful of the truth, reasonable, and ratienal in '-:oalingto one's interpretations, analyses, evaluations, inferences, or expressions. 6.2 SELF-CORRECTION: Where self-examination reveals errors or deficiencies, to design reasonable procedures to remedy or correct, if possible, those ~i5takes and their causes.
tttttttl

0; MONITORING:

BOONO 5-6, PART ~

ENOORSEMENT~

1) Do you endorse the above statement as useful for purposes of assess i ng tb~ skills dimension of CT at the lower division coLl ege level? 2) Would you be willing to l'ave yo~r name listed in association with the above description of CT skills as a contributing ~ember of the Delphi research project which generated it?

*****

Appendix C: Delphi Research

"

Lettere.

PAGBB9

f,·~,
.. I

''/'"i·:'

93

The responses to ROUND 4 included so~e telling ob~ervatiDns, thoughtful objections and tlell f o cu s e d c r i t i c i s ns , Sharing these WIth others In tlla lJQlphi qr o up is e s s en t i a l to t h e DL'lphi pr cc e s s , L'v e i nc l u de d l1r:, nJ an y 115 I S f Q iI sib I e and u ~) C u n bell n cJ P. r 5 too dOLI t sId o t h Po call t C! l: l af whatevr.r ROUND 4 item mt'lY hay£! prOr.Jpted t h e e , nather than use of your time reading positive comm~nts, of Hhich there were many, I've stuck chiefly to the critical one$. INSTRUCTIONS: Read and consider what our colleagues are t~lling us. In addition to the adjustments already incorporated into the above draft statements regarding CT and CT assesscent, Nhat other re5ponses and improvements should we make? In the jight of these comments and other concerns that come to ~ind, what specific rer.om~endations should we include in our report to the APA CO~Dittee on Pre-College Philosophy?
$ f L £ r T El'.. R £ S PO It ~~f S Q.!i.

£l T f S T S d1iQ. T F: S T 1 H GeT

• "A test cannot be considered in the abstract, without Horklng out its intended use and intended ~sers, the specific population to be tested, and the discrimlnations the test would be required to ~ake.,. The domain the test covers is governed by this conte~t, and concepts that constitute s~b~lassiflcations of the domain are arranged in different ways fro~ the Hays in ~hlCh they mIght be al ranged for a test WI th the s a e a name but a dl f ferent purpose... CT can be sub c l a s s i f I £?d in a nu~ber of leglti~~te Hays, WIth ~any of the same ele~cnts recurrIng in dlfferent plac~s jn dIfferent claSSIficatIon scheDcs ••• "
hardJy anything "There's tests. 1 've used ••• in pre-post best 1 knoH of • (Su t ••• J
II

He now testIng

need more for several

than i~proved years. It's

CT th e

• " ••• to ~ake sense of CT ~e nust thInker. The cognitive ~usl be dIscussed and both ~U5t be discussed in relatIon world. Curriculu~ and assessment r.ust be ••• There is no one right definItion of never confuse testing for ~lcro-skills Host CT tests are ~Icro-skill tests only. in a qualified Hay. The Delphi project confusing the part with the \1hole,u
I

lake sense of the critIcal in relation to the affective, to thelr roles in the real put into some broad context • CT, and in testing He should WIth testIng for CT itself. They are valuable, but only see~s Hell on its Hay tONard

"The [Round 4 list of skIlls] see~s fine to ~e. lhere are definite liAlts to ar~chair analYSiS, and untJl s~.e~ne actually ~tarts trying to .easure these thJngs, It i~ dIffIcult to kn~~ just ho~ t~ revise the list of CT skills and sub-s~ills." uPlease interpret ny responses r.autlously. ~uch time trying to definH CT. My business assess~ent once you 9uys arrive at clear and usei~l
I

i5

trying deflnltlons!"
0

1 have

spl.!nt not to ldo) .,' 9 1 ve n

II

We

tJ

us t f 0 c U son

the

pur p 0 ~ I? a 1\ d tar get

au d I ~ nee

fan

'; 1

Appendix C: Delphi Research Letters,

Pl~GE 90

94 \

.. .-'._:~:~I
. ~~.'";

CT test. Only should I say our

t

that way can we fine tune our definition "theoretical construct" Of CT. II

of

CT,

or

"Rather than limlt our conc:eption of CT to achieve a certain kinti of test, why not simply make more modest claims about the test? Why not say that you are tasting dSp'Cts of CT? This ~~~~~ ~ore justified. Otherwise the test becomes the tail that wags the dog ••• Be honest about what ~~ are a~d are not testing by multiple choice tests. L~t's not reduce rich and complex realities like CT to that which can be directly tested in th~ multiple-choice format."
f [Some of your descriptionsl are diificult to te~t without openended items ••• [For examplel it's difficult t.o t£st {or~ulation of strategies, He e~ams focus more on recognition of best ~tr~tegy,

liThe categorization syste~ ••• gives little guidance on what pr ec i s e l v to teach and test. For instance, under "evaluating" is lito assess the credibility ~f state~ents.N But what should be taught when doing this, and Hhat ~hould b~ tested when trying to find out what skills st~dert~ have? If teachers ~re supposed to act based on what He produce, I believe we need to incl~de criteria for assessing N credibility. Again I refer you tOt ••
t

conn~cted in those individuals Hho are successful critical thinkers to a pervasive self consciousness ~bout one's OHn thinking and reasoning processes. Such self consciousness should be deliberately cultivated in our students and should therefore be included somewhere in the overall description -- it is an intrinsic part of the (CTl process. It monit, s the selection, applitation and interlillkage of the va~jous relevant processes."
t

"eT is deeply

S;..:t .....T...::,tD __L..;:;"t __ C

f

lli!' a H S (S Q.1i £L.

A H ~ D f F 1 H 1 H 6'

rL

IIUntil we have a theory of reasoning (a cc~bined nor~ative theory of infor~al logic along Hith a descrlptive theory of cognitive processes) He will not be able to spell out CT skil!s non-arbitrary Hith ccnp Ie t ene s s and precision .... We ar e procfucjng ... a frallJeHork, [that isl a list of concepts used for underst~nding a do~ain. If that i5 so, it Hill play hell out of our attelDpts to assess CT skills. 1 doubt that we will be able to manage construct validity for any ccnventio"~l MC test with our [list of CT skills]. Vet, Hhat we are coming up Hith is p.~tre~ely valuabl~ if we focus on performence ass es s a\ e :1t • Our f r a II e Ii 0 r k g i v e sus a pas s a b 1 y g cod set 0 fer i t eria by which one would judge goud perfor~an,e on CT tasks. The criteria are devel~ped by experts -- us -- and we are currently judging whether He accept the~ or reject them, another step in the process of developing a goo~ perforlDance assess~ent. The next step would be to distribute typical essays, (good, bad and ugly ones) and aim at some consensu~ in telling the good ones from the bad ones."
I

"We

should

resist

the assertion

that

CT is dOl'llain dependent."

donel What

beginning to think that to have done all that [Me have and not have tried to define CT may turn out to be a mistake ••• lakes [the list ofl itlportant, indeed basic, intellectual skills

tlI'~

Appendix C: De Iphl Research Let tere , PAGE91

95

. ,~./;.;:

.) .• :..,<'~;..... _-: .... :

••

_ ....

_, __ '''._ ••

__ ,...• ' .

('.";?~,~ ;-;,~..

_._T

-_
[ in R 0 u n d 4 1 ali s t o)f cr i tit; a J t h ink i Il g 5 j.: i 11 s? I teo u 1 d ; ~ s t il S well be a list of rational thinking ski lIs, or logical thinking skills or h rg her 0 rd er cog r.it i ve ski 11 s • • •• I fa i 1 to see h 0 H th i 5 1 is t captures the force of the word "e r t t i e al ", [EtY'Dologicallyl the meaaninQ of "cr t t i c ar " is judging, evaluating, estimating the worth of something ••• A critical thlnker is so~eone ~hu renders an oplnion on an intellectunl product ••• by assessing the strengths and weaknes$es of that product, •• Doing s~ requires the capacity to elicit and apply standards, principles and crlteria, None of the [Round 4] list is really this s~ill. If we as~ed for a list of problem solving skills, Nould we get the same list [as in Round 411 If S~, then either there is no conceptual difference between the two, which 1 think is wro~g, or else the list fails to capture what is distinctive about C~ s~il1s. CT also connotes IIcrucial". tHere) the (Round 4J list fares better, because ••• these six ski 11 s are cruci al --i. e., essential for intellectual survival. The problem is that the list is ~o bread and wide-ranging that it is not clear ~hat intellectual skills have been excluded.1I If 0 n e ide n ti fie 5 C T as that w hie h m a I: sac e r·itJ s: alp er sen to be Hhat he is, t~en [yourJ narrow concept of CT is inadequate. CT is", an anSHer to the goneral proble~ of conformity, prejudice, narrow-mindedness, ~nd irrationality in the world. CT is what one does to achieve autonomy and independence of thought, to lesson one's prejudices, to broaden one's perspective, and tu become ~ore fitional,»
I
II

liThe ~ain overall Horry is that the categories are Iuch too When He get down to testing it will not be for something like "assessing loqical strength or IIclarifying" but very specific skills such as "rer:ognt~ing whether so~ething is a necessary or sufficient condition; recognizing the difference between if p, then q's and if q the p's etc. The lines of de~arcation are very unclear e.g. between psychological and logical (epistemological) criteria con~erning say observati on. II
t

broad.

ll

t III have no major quarrel with any parts of your organization, [butJ your emphasis leaves out a major compcnent of CT the dispositional component and the ~et of values inherent in being a critical thinker.,. I think it is a good working model of CT skills, but an incomplete picture of being a critical thinker.1I

"There is no attention ••• to the dispositions of CT. These... are as essential to CT as are information used in the ~~ocess~s.~
I t

••• the

characteristic knowledge and skills. I

suggest
I

liThe dispositions their addition.1I

are at least

as i e.p c r t e n t as the

IICT works by recognizing and criticizing sourcec; of inforllation, by drawing i~plications irc~ given ~aterials, identifying assumptions, noticing relationships of consistency, inconsistency, i~plications and contradiction, inferring interesting consequences, recognl~lng, analyzing and evaluating argu~ents and c~nstructing them as Mell. Of course, there's a lot ncr a to it.·

Appendix C: Delphi Research Lettera.

_'"
~:~~.~

PAGB 92

::<' -,\ ~.... " _!;!~•.

.,: ·.... 01

96

..~{:i'~r

.'

')

\',

"

-~~

'.-'.'~' ..
.... ,',c. J:

,·:V:

"I

."

f
.-:

"Arguing

is not li5te~

as a separate
h

skill!" another's purpose." this

f
. <

would

"No mention is made of understanding fall under interpreting, I suppcse.

* "I di~agree with the list of CT skills as described. Interpreting is obviously a cognitive operation. But it is not a CT operation. Expressing is essentially ION level communicatiun, not g~nerative in the sense of CT. Monitoring is 'meta-cognitive. The problem is that this description is so broad "critical" thinking gets lost in all the other kinds of thinking. This blurs the nature of CT beyond ~ecognition... CT is "judging the werth, accuracy or sigr.i·[jcance of something."
"There are several items I mess overall. They may be subsumed in some of the processes you have listed, but very few people will be conscious of them unless they are brought out explicitly. One is the capacity for arithmetical reasoning with ratio and di\ision it begins Hith word problems in 5th and 6th grade arithmetic and carries up to exactly similar reasoning with concepts such as density, composition, contraction, rates of change, in more sophisticated settings. It includes the ration reasoning that goes with scaling areas, volumes rates, etc. This capacity is profoundly important in any CT that involves nu~erical inforBation (whether it be scientific, economic, sociological, psychological ... I'ID talking about a."jthletic and not sathe.atics at the level of calculus or even algebra. CA second capacity to include is] "correlational reasoning." Finally I miss er.plicit inclusion of the process of translating symbols (e.g. graphs, numerical data, hjstogra~s) into words or Hords into corresponding sYlDbols. Such tr anslation is essential to lIuch CT."
f f

explicitly dra~n, irrelevant capacity relevant plausible imposition

"CThe concept of CT should also include] discriminating betHeen the fact~al or experiential input and the inferences identifying gaps in available infor~ation and identifying or superfluous infornation, and it should include the to consider so~e situation in the abstract and, by applying governing pr1~ciples or constraints, arrive at reasonable and conclusions about the outcomes that would result fro~ the of so~e change -- hypothetico-deductive reasoning,-

the "Include so~ething about ~pposlti~hal rpasonlng -- taking hypothetical part of one with whom one disagrees. Also include reas~ning -- reasoning fro~ suppositions and hypotheses,
t "Taxo~omy" is not a good word fer ~hat we've got; it's more like a list. H~vJny a taxonomy in biology and education is to have a hierarchical set of categories such that each subsequent step in the h1erarchy subsumes t~p. steps below it. We don't have that here.

U[ConjecturingJ

is creative seeking

thinklng

not CT. evidencel strong is a~biguous. ratl0nale for

* "[To have included We are not scientists.

t

and gathering 1 don't see any

NCRegarding

expressing,J

Appendix C: Delphi Research Letters,

PAGE93

';;h,. ~
,
"~.'

;'..;, ._.-. .

97

, ':.>:<:,i
.

,~?-':-:':.:I.:_~ \ .',;-:'~" ~~.

\1; .

,.- ....

.:,"

..

extending
II

the focus

to communicating

the results

of CT.

* Non e 0 f [y 0 u r s i :: 1 i s ted i t EJ m s are s ~ ill 5 - - 1 eta J Ion e subs~ills. They are general categories into Nhich many (at least do:ens of) distinguishable skills may be lumped. This ~s important.
TABLe

QL R£SPOHS£

1- INTERPRETING

QJi THE ROtlHO i rr S;~li.LS LIST Agree Disagree Unsure

2.

3. 4. 5. 6.

Observing Decoding Clarifying INFERRING Querying Conjecturing DraHing Conclusions Analyzing Locating Arguments Parsing Arguments Evaluating Claims Verifying Assessing Strength Logical Expressing Stating Results Procedures Describing Honitoring Reg u 1 " ti n g Reviewing Correcting
ReSPOHSES

.I.~

.,26 ., ..
.I.~

24 20

2 s 3

..

24 23 24 23 23 23 24
2S 26 21

., ..
1

"-

.,

3 1 1

3 2 3

., ..
,)

3
.)

2:S
23

21

21 i1

..

3 .. 3 .,

'" 3
3

24

'"

.,
of on

r.Jl..

'UHAT

To help delineate CT through co~parisons and contrasts, I offered a list a~tivitles that bore so~e family resemblances to CT. Each of the~ depends CT. Sut whether any of Has CT per se was ~he issue. Here's what you said: Isn't (Seeing, Touching, Hearing, etc.) 2. Reading, Listening, 3. Speaking or ~riting, 4. Motivating, persuading, selling, S. Interrogating, cross-examining, petitioning, 6. Fhysically investigating the world around, 7. Trouble-shooting, problem-solving, pu:zle solving, 8. Decision-making, selecting, choosing, exercisin~ will, 9. Planning, defining goals and objectives, 10. Finding or ascribing a ~eaning to art, life, or events, 11. Defending an opinion or belief, arguing a case, 12. Ma~aging, ad~inistrating, governing persons or things, 13. Philosophizing, 14. Conductlng research within any particular discipline, 15. Experiencing, feeling, emoting, or empathizing, 16. Co~cunicating u5i~g language.
1. Sensing,

£L ll. HOT"

U
16
13 14
1 .;

U.
4 3 4 4
4 3 7
5 7

ll.
2

Pu

rr 5
6 6 5
6

t!.1..

12 13
13

14

10 13 16

6 6 6
6 8

12 12
15

3 3 3
2

8 8 5
6

15

Appendix C: Delphi Research Letters,

PAGB94

9.8

~.~.'..

--~

" "

.....

Department of Philosophy (7"i4) 773-3611

"

California

Fullerton. California

State University,
92634

Fullerton

DELPHI ROUND 5-8

Mar. 6, 1989 Dear Delphi Colleagues,

This letter Tollows the outline in the Round 5-A letter -questions Tirst, background information second. The focus here is on fleshing out our. conceptualiza tion oT CT beyond the revised list of CT abilities presented in Round 5-A. Specifically this letter responds to your comments regarding those dispositional and normative dimensions many include when describing CT. The implica tions for K-12 and college level assessment, curriculum development and pedagogy oT including either of these dimensions are crucial for our efTort. You're asked for your ideas about this, too. To get things started, on the ne>:t ;lage you'll find a draft statement r'egarding the dispositional dimension oT CT. Following ".hat is u druft sta:ement regarding the normative dimension. These two draTt statements are based on your comments regarding ne~ding a fuller conc~ptualization of CT and an analysis OT commonly referenced concepts of CT which appear in the literature. Have at those draft statements. Amend, edit, comment, uccept, reject -- whatever- you think will help us be able to present an intellectually credible and educationally useful conceptualization of CT. Coming soen: Round 5-C focuses on our report to the APA Pre-College Philosophy Committee. It will include a proposed outline of that: report, showing how the various pieces of the assessment puzzle we've wor~:ed on for over a year now will be incorpora ted. It also picks up the two pieces from Round 4 which haven't been addressed yet, namely the Inpu t-output model (which we rather roundly rejectedJ, and the statement of what a cognitive sJ:ill is and how cognitive skills might be asses~ed. I'll be speaking on March 24 at 1:00 p.m. al the Pacific Division meetings of the APA in Oakland CA, sharing a little of what we've been doing and the direction things seem to be taking. If you happen to be in the neigtlborhood, please .stop by 50 we can visit. Knowing you are very busy, I beg mercy and ask you to reply as as is reasonably possible. A quidde note with general comrnerrt a is helpful, if you won't have time to go through things in careful detail. Thanks for your continuing support and involvement. sincerely, soon

Yours
~

Appendix C: Delphi Research Letters, PAGE 95

.\¥.. ~ '~'i~:::;'::'.
.'~. ~_• .n •

e_,.

Pete Facione

99

Part:, !l The Dispositional Dimension e£ CT INSTRUCTIONS Re-flect on the -following statement. Edit, amend, or revise as you see "'it. Some backgr·ound information is presented in the second half of t.his letter. Please respond to the following questions: 1) Should our- final report include a statement on the dispositional dimension of CT? Why, why not? 2) What implications or- recommendations for K-12 ,and college lower division level assessment, curr~culum dev~lopment and pedagogy follow from inc:.luding a dispositional element in our conceptualiz,3tion of CT?

CT -- The Dispositional
skills traits use. co~ceivin9 perso~al

Dimension

In addition c09~itive crucial ~bilities i~structional laterial~, dispositions. ilportant instructional and personal $kills tha~ discipline disposJtions, ~ seeki~g ~ curiosity ~ eagerness ~ openness • trust ~ honesty ~ willingness ~ prudence ~rle's position,
~ ~Jsdol

to its cog~itive and successful of CT, ~hen in CT, strategies the use Perso~s listed beloN ~ho knoN

dile~sio~f or intellectual Just

CT also

involves ~hich
an

certai~ are

di~po;itions, to its broad diDensio~ progral teaching in insuring sftti~g. traits are those

virtues
or

as ~ith to consider tools outside the likely

the cog~itive ~ays focus of developing

on

of CT assessaent ~hich

it is i3portant and assess~ent of CT cognitiye of C1 abilities Nho have are ~uch the skills
I~re

CT

The cultivation

dispositions

is particularly

the narrON ihtellectuaJ to apply

CT
virtues their

developed but lack

CT

the intellectual the C~9nJtiv~

to use them. thlnker

a clear

The crit~cal

is ~ne ~ho possesses

or

and cultivates virtues questions of

personal

traits Itatelent

Jntellectual of issues, and seeking

or problels,

in explorJng In seeki~g to seek and

issues and to consider

inforaatioh, vieNS or alternatives, ~ell-i~for.ed, and vieNS,

~r

in elploying

CT abilities, inquiry, i~ keeping ONn credible and one's reasoning sources, in taking cha~gin9 and to the

divergent

in the processes diligence

in

of reaioned appraising and lention or laking

~ perseverance,

and discipline use

and hu.ility to seek, in suspending and persIstence

judglents

in the use of appropriate, reasonably the subject and to relain per.its, issues

defensible relevant

relevant basic

criteria, issues • of cOlplex and processes. PAGE96 concern or problel

to

~ e, ort to address • precision, ~ orderliness

the extent

in the treatlent

Appendix C: Delphi Reaearch Letters,

100

I •.

:t=:·";~~X::'~'~;~,i:-:';C:~'~'~~"'''';''~;· -"-'-'';c, ."

". ~

t/;. :'"

- . --,. '.:~ .

~·-·r:

Part 2: The Normativ~ Dimension of CT INSTRUCTIONS Reflec:t on the following statement. Edit, amend, or revise as you see fit. Some background information is presented in the second half of this letter. Please respond to the follo.~ing questions: 1) Should our final report include a statement on the normative dimension of CT? Why, why not? 2) What implications or recommendc:\tions for ~~-12 and college lowt=~r division level assessment, curriculum development and pedagogy follol.~ from including a normative element in cur conceptualization of CT7

CT -- The Hor~ative In addition also involves Understanding descriptive lust be I~re that analysis, than values used, rather used, should used, used, to its cognitive certain norlative in laking ~e judge of CT. training Education ~hich insure freedol the blind, defective those sound rational
any

Dilension and dispositional ~hich 90vern this Ne are going and rational than dilensions, use. a purely cDlpanent education of a set of of those qenera~ions. a~( resolutio~ of vie~s in defense civic fairbeyond noraative society

skills features statelEnt In a free and lore lust that ~ilJ

~T

its proper

this

it vital

to include

in our c~nceptualization skills personal .i~dedness Properly of issue$, k~o~n Properly

a reflective

the inculcation the nurturin9 to futur~ of intellecfual analysis

dispositions. a~d political than

include be passed

and personal

the herita9~

CT co~tributes

to the fair-linded sopnistic, or biased. Nho ~ould reason even

or irrational together if their

to be intellectually

CT unItes disagree. CT prolDtes of

objectiY~ly final jud91ents

and intellectually

process,

and analyses Properly the objective Pr~perly equal abusive. Thus, assesslent developing the nor.ative respect

autonolY, or concern persons exclusively and qood society, in CT, and

intellectual Nhatsoever, ~ith faith, ~hen conceiving

freedol,

and

investiqation

issue

CT treats
or ~ithout open

all affected integrity progral strategies

sensitivity

and ~ith

and dignity in a free.

-- it is never and rational

self-interested. of CT ~ays of to ~hich insure

coercive.

or an instructional laterials, dilension teachinq

one ought assesslent

to consider tools

of CT ~ill

be developed

in students,

in addition

CT skIlls

and di$posit,ons.

Appendix

C: Delphi

Research

Letters, PAGE 97