Escolar Documentos
Profissional Documentos
Cultura Documentos
Educational
Evaluation
Studies in Educational Evaluation 32 (2006) 83-99
ELSEVIER www.elsevier.com/stueduc
*Faculty of Education Sciences and Psychology, University Rovira i Virgili, Catalonia, Spain
**Department of Electrical and Mechanical Engineering, University Rovira i Virgili, Catalonia, Spain
Abstract
The educational goals and objectives provide a reference for curriculum designers,
teachers, and educators, and they are required for evaluation or accreditation
processes. This article presents a systematic method to state the learning objectives
and analyze the consistency of a secondary school science program developed in
Spain. The method is developed through seven steps, starting with data collection
and organization, followed by induction and categorization of the objectives, and
finishing with an internal consistency analysis between stated objectives and the
program's learning activities and an external consistency analysis between these
objectives and program designers and users' perceptions.
Introduction
Over the last three decades, tendencies in science teaching and in development and
implementation of new educational programs have led to an increasing interest in
evaluation research and, more specifically, in educational evaluation. Also, the
responsibilities and resources allocated by society to educators have increased the need for
accountability.
APQUA is an educational project that develops materials for the community and
schools with the financial support of industry as well as private and governmental
foundations. It has been widely diffused in Spain, reaching more than 3000 teachers and
150,000 pupils in about 900 schools, and more than 7,500 adults (APQUA, 2004).
The main goal of APQUA is to develop greater awareness, knowledge, and
understanding about chemicals and their interaction with people's lives. The project intends
to provide people with the knowledge and tools that help them learn to obtain information
about chemicals, to make effective individual decisions, and to participate actively as
members of a democratic society. APQUA, thus, also aims to promote the use of scientific
principles, processes, and evidence in public decision-making (Medir & Abell6, 1996).
Figure 1 shows the organization of the APQUA project, which consists of the
APQUA Community Program for adults, the APQUA School Program 10-12 for primary
schools and the APQUA School Program 12-16 for secondary schools.
S. E1Boudamoussi et al./ Studies in Educational Evaluation 32 (2006) 83-99 85
ro ram&
APQUA PROJECT
./
PnblicPro
ram >
The APQUA School Program 12-16, which is the subject of this study, is made up
of modules. Each module contains several units, and every unit integrates several activities.
A list of modules is included in Table 1 (the ones selected for the present study are marked
with an asterisk). All modules deal with two main themes: risk and chemicals. They may
tackle different contents and different specific objectives, however, they all aim at
achieving the goals of the APQUA project (APQUA, 2003). In order to distinguish
between different levels of learning objectives, we will use in this study the words "goals"
for the project, "general objectives" for the programs of the project, "specific objectives"
for the modules of a program and "concrete objectives" for the units of a module.
Although APQUA is designed and developed from clearly defined goals, the general
objectives of the APQUA School Program 12-16, the specific objectives of its modules and
the concrete objectives of the units were either partially stated or not stated as shown in
Figure 2. This is so because for adapting the SEPUP materials to the context of Catalan
society and schools, the APQUA Project team field-tests various versions of the APQUA
modules before a final one is edited and distributed. Therefore, the learning activities are
modified many times and the specific and concrete objectives initially targeted may
become incomplete, non-structured or roughly stated.
~ h o o l Program 12-16~
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
r--- Modules
"........................... Units
iI Module I . objectives of
1 Module2 [] [] [] partially
il
1
I
I :
Module3
i [] [] [] stated
Research Purpose
This study develops a systematic method to evaluate the APQUA School Program
12-16 by stating the learning objectives (of the units, the modules and the program) and
analyzing the internal and external consistency of the program. The stages of this method
are built up, step by step, following a systematic and structured process, which can be
described as follows:
Program Evaluation
Methodology
The research methodology followed consists of four steps stating the learning
objectives for the APQUA School Program 12-16:
Sample definition
Information gathering
Objectives statement
Internal and external consistency analyses
Sample Definition
The particular structure of the APQUA School Program 12-16, organized into
modules and units (Figure 2) has shaped, in general, the method developed and, in
particular, the sample selection.
Four of the eight existing modules were selected: Solutions and Pollution,
Contamination of Groundwater in Vallfrondosa, Solid Waste Management, and Plastics in
our Lives (Table 1). The criteria followed included the topic's relevance, the reflection of
the program's educational approach, and the non-similarity between learning activities in
the selected modules.
The modules excluded were: What is a Chemical?, Risk Comparison, Hazardous
Waste Treatment, and Toxicology.
Data Gathering
Statement o f Objectives
The process of stating the general objectives was the inverted path of the program
design. It started from listing the learning activities presented in the units of the modules.
Then, it identified the educational contents (knowledge, abilities, values, and attitudes)
tackled in these units and induced the concrete objectives of the units and the specific
objectives of the modules. Finally, it ended up at inducing the general objectives of the
program (Figure 3).
General
objective
Learning
activities
Concrete Specific
ontents objectives objectives
Statement of Objectives
Consistency Analysis
Consistency Analysis
The internal and external analyses of consistency were carried out using
specification tables (Roegiers, 2000) and expert moderation (Thier & Daviss, 2001),
respectively.
~5. L I H o u a a m o u s s t e t a l . / ~51uates m L a u c a t t o n a l Lvaluatton 3 Z (ZUUO) ~3-!~!~ 39
A general table (see, for an example, Table 3), was built to show how the general
objectives of the program were generated from the specific objectives of the sample's
modules. This general table, together with Table 2, allowed checking the coherence
between every general objective and the learning activities.
The expert moderation involved 20 secondary school teachers and 6 project team
members in the process of stating the objectives. The secondary school teachers were
selected according to their experience in using the program, in general, and the modules of
the sample, in particular, and the project team members according to their involvement in
the program's development, in general, and the modules of the sample, in particular. The
experts' opinions and suggestions were obtained through questionnaires and interviews.
Five questionnaires were designed: four of them were related to the specific objectives of
the modules and one was related to the general objectives of the program. The experts who
answered at least one questionnaire were interviewed.
A group of three judges intervened throughout the whole research process in order
to evaluate and seek consensus for each of the results obtained.
The method developed (shown in Figure 4) consists of seven stages, each of which
has several steps and generates various versions of the program's objectives following a
spiral process. The seven stages of the method include data collection; data organization;
categorization; internal consistency analysis (using specification tables); application and
transfer; generalization, and external consistency analysis (using expert moderation).
S. E1 Boudamoussi et al./ Studies in Educational Evaluation 32 (2006) 83-99 91
Step 2.2. Induce and state concrete objectives for the units o f M 1 I : * I
Step 2.3. Identify concrete objectives common to all units o f M 1 ...........i'"'"'i'""~ C . . . . . teand . . . . . . I I
I I t 1 objectives of units I
Step 2.4. Induce and state specific objectives for M 1 ~__4____1_ ...... i I
4,
i
Stage 3. Categorization
Step 3.1. Define the main categories of s?ecific objectives stated
for M 1 and assemble them into these categories
i I ~- .................... i
. . . . . . . . . .
! i Version : i I
Stage 4. Internal Consistency (of modules) • "...L ................................................. ~ I
L 2 __ "~ - -
Step 4.1. Encode the learning activities, the contents, the concrete
and common objectives of the units and specific
objectives o f M 1
Version3
i
Step 4.2. Build a specification table forM 1
•I .ff Version 4
F i g u r e 4: Method of Stating Learning Objectives for the APQUA School Program 12-16
92 S. E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99
The first stage, Data Collection (Stage 1 in Figure 4), includes three steps which
consist of selecting a representative sample of modules (indicated on Tablel), reading and
analyzing documents of the program, and listing the learning activities for the first sample's
module (M1). An example of learning activities is shown in Table 4.
The second stage, Data Organization (Stage 2 in Figure 4), consisting of four steps,
leads to stating the first version of the specific objectives for module M1 (Table 8). This
stage starts by identifying the educational contents tackled in the learning activities (an
example is shown in Table 5) and induces from them the concrete objectives of the units
(an example is shown in Table 6). Then, it identifies the concrete objectives common to all
units (an example is shown in Table 7) and, finally, based on the concrete and common
objectives, Stage 2 generates the first version of the specific objectives for module M1 (an
example is shown in Table 8).
The concrete objectives common to all units are very important to decide which of
the concrete objectives of the units will contribute to the statement of the specific
objectives of the module. However, the statement of a specific objective rather uses the
information provided in the concrete objectives of the units. As shown in Table 7, only part
of this information is present in the common objectives.
The first version of the educational contents and concrete and common objectives of
module M1 is generated at this Stage 2 (Figure 4).
The third stage, Categorization (Stage 3 in Figure 4), clusters the specific objectives
of module M1 into main categories (Table 9). This categorization leads to the second
version of the specific objectives for this module.
I. Concepts
E.g.: To build operational definitions of basic concepts related to solutions and neutralization.
II. Scientific methodology, its transfer and application to real life and daily issues
E.g.: To observe and register, in an organized and systematic way, quantitative and qualitative data
concerning the color and concentration of prepared solutions, or solutions obtained from successive
dilutions and from acid-base neutralization.
The fourth stage, Internal Consistency (Stage 4 in Figure 4), analyzes the
consistency between the specific objectives stated for module M1 and its learning
activities. A specification table, already illustrated in the methodology in Table 2,
establishes the links between each specific objective and the corresponding learning
activities. This stage results in a third version of the specific objectives of module M1.
The fifth stage, Application and TransJer (Stage 5 in Figure 4), applies the four
previous stages to the other sample's modules (M2, M3, and M4) and transfers some of the
relevant results to these modules. A list of the learning activities, and a first version of the
educational contents, concrete and common objectives and specific objectives of these
modules are obtained. The categories for the specific objectives are transferred from
module M1 as well as the form of the statements used to express the concrete and common
objectives and the specific objectives.
The sixth stage, Generalization (Stage 6 in Figure 4), states the general objectives of
the program based on the specific objectives of the sample's modules. It starts by
identifying the specific objectives that are similar or common to all the sample's modules
and induces from them a draft of induced general objectives (see statements inside the
circles on Figure 5). These are then assembled into categories of synthesized objectives
(see statements inside the rectangles on Figure 5) that lead to the statement of a first version
of the general objectives of the program.
S. E1Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99 95
+
~~I
G.II.I To i n t e r p ~
sheet and to carry out an ~
erimental p . . . . dure
I +
OG.II.6 To work / experimental material and respect for how it OG.II.1 To carry out
in a group \ sl~ould be used and maintained to carry out experimental
. . . . . iments I p . . . . d. . . .
+ 1 ~ T o b e c o ~
OG ".II :7 To. develop
. [ ~ f / of the problems g . . . . . ted by waste, the " ~ +
crltlcalthlnklng r / d ~ e s ° V I 2 d e c - ° n t a m i n ~ t ~ nd-the~need I OG.III.3 Tob . . . . . . . . . .
I ( for information I of the environmentalimpact
/ i ~~x, .OG\III.3 To b . . . . . . . . . . . fthe ] of human activities
~, . . . . . .
/~./U.ll./1o acquire a N . ~ environmentalimpact of human activities I
/ critical thinking ~ ~ n~
To observe and record i
+ / v _ ..L~. . . . . . . . N an organised and systematic way
I / evidence-based ~ . . . . ' I
OG.II.10 To make ~ decisions ",~uantltatlve and qualitative data I
evidence-based decisions ] OGII.11 To integrate information about ~ /
] different options in an evidence-l~ased \'.OG •II •3 To analyze
. and /
] decision making process / interpret experimental /
~ ~ data andgraphics /
\ OGIII.7 To emphasize the adwantag . . . . d / /
~ disadvantages, the import . . . . . fth . . . . . ic /
~ ' ~ c t . . . . d the.trade-o ffs. i. . . . . id . . . . . based/./
~ making p r o ~ ~
out to be a helpful step in the procedure of stating the general objectives of the program.
They offered an efficient way of gaining an idea about how the general objectives would
look and served as a basis on which the statement of each general objective was built.
The second version of the general objectives is obtained after extrapolating, to the
program, the internal analysis of consistency (step 6.5) previously applied to the sample's
modules. This procedure, which correlates the general objectives stated in the first version
to the learning activities of the modules, via the specific objectives, has already been
illustrated in Tables 2 and 3.
Finally, the seventh and last stage, External consistency (using expert moderation)
(Stage 7 in Figure 4), considers the opinions of two groups of experts about the last stated
versions of the specific and general objectives. These experts are the users and developers
of the program. The qualification of the objectives by the experts from the two groups as
well as their comments and suggestions lead to stating the fourth and thus final version of
the specific objectives of the modules, and the third and thus last version of the general
objectives of the program.
Conclusions
References
APQUA (2003). Proyecto APQUA. Aprendizqje de los Productos Quimicos, sus Usos y
Aplicaciones [APQUAproject. Learningabout chemicals,their uses and applications]. UniversitatRovira i
Virgili.
S. E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99 97
APQUA (2004). Proyecto APQUA. Injbrme 2004 [APQUA project. Report 2004]. Tarragona.
Departamento de Ingenieria Quimica. Universitat Rovira i Virgili.
Bloom, B.S. (1979). Taxonomia de los ol)jetivos de la educaci6n. Clasificaci6n de las Metas
Educativas [Taxonomy of educational objectives. The classification of educational goals]. (Acarreta
Arnedo, I., Trans. 3rd ed.). Alcoy : Marfil.
Cronbach, L.J. (2000). Course improvement through evaluation. In G.F. Madaus, M.S. Scriven, &
D.L. Stufflebeam (Eds.), Evaluation models'. Viewpoints' on educational and human services evaluation
(13th ed., pp. 101-115). Boston: Kluwer-Nijhoff.
Eisner, E.W. (2000). Educational connoisseurship and criticism: their form and functions in
educational evaluation, In G.F. Madaus, M.S. Scriven, & D.L. Stufflebeam (Eds.). Evaluation models'.
Viewpoints' on educational and human services evaluation (13th ed., pp. 335-347). Boston: Kluwer-
Nijhoft:
E1 Boudamoussi, S., Medir, M., Jimin6z, B., Gilabert, R.M. (2001). Estudio comparativo entre el
modelo te6rico de un programa educativo de ciencias y su desarrollo en el aula [Curriculum. Compulsory
secondary education. Section of natural sciences]. VI Congreso Intemacional sobre Investigaci6n en la
Didfictica de las Ciencias: Retos de la Ensefianza de las Ciencias en el Siglo XXI. V 1, (pp. 229-230).
Barcelona.
Hashim, Y. (1999). Are instructional design elements being used in module writing? British Journal
o f Educational Technology. 30 (4), 341-358.
Medir, M., & Abell6, M. (1996). APQUA: A program on sciences' and societal issues. Proceedings
of the 2nd scientific conference on: "The Future of Science and Mathematics Teaching and the Needs of
Arab Society" (pp 500-506). Tunis: Arab Development Institute.
Micek S.S. (1979). Identifying, measuring and evaluating educational outcomes. North Central
Association Quarterly, 53 (4), 408-419.
Parlett, M., & Hamilton, D. (1989). La evaluacidn como iluminacidn [Evaluation as enlightenment].
In S.J. Gimeno & G.A. P6rez La ense~anza. Su teoriay supractica [Teaching. Its theory and practice] (pp.
450-466). Madrid: Akal.
98 S. E1 Boudamoussi et aL/ Studies' in Educational Evaluation 32 (2006) 83-99
Roegiers, X. (2000). Une p~dagogie de l'int~gration. Comp~tences et integration des acquis dans
l'enseignement [A pedagogy of integration. Competences and integration of knowledge in teaching].
Brussels: De Boeck Universit6.
Scriven, M.S. (2000). Evaluation ideologies. In G.F. Madaus, M.S. Scriven, & D.L. Stufflebeam
(Eds.). Evaluation models'. Viewpoints' on educational and human services evaluation (13th ed., pp. 229-
260). Boston: Kluwer-NijhofI:
Stake, R.E. (2000). Program evaluation, particularly responsive evaluation. In G.F. Madaus, M.S.
Scriven, & D.L. Stufflebeam (Eds.). Evaluation models'. Viewpoints' on educational and human services'
evaluation (13th ed., pp. 287-310,) Boston: Kluwer-NijhofI:
Thier, Herbert D. (1985). Societal issues and concerns: A new emphasis for science education.
Science and Education, 69 (20), 155-162.
Thier, H.D., & Daviss, B. (2001). Developing inquiry-based science materials'. Guide jbr educators'.
A project for the Lawrence Hall of Science Center for Curriculum Innovation. New York: Teachers College
Press.
Tyler, R.W. (1998). Principios bdsicos del curriculo [Basic principles of the curriculum]. Buenos
Aires: Troquel.
Tyler, R.W. (2000). A rationale for program evaluation. In G.F. Madaus, M.S. Scriven, & D.L.
Stufflebeam (Eds.). Evaluation models'. Viewpoints' on educational and human services evaluation (13th
ed., pp. 67-78). Boston: Kluwer-Nijhoff.
Tymitz-Wolf, B. (1982). Guidelines for assessing individualized education program goals and
objectives. Teaching Exceptional Children, 14 (5), 198-201.
wittig, G.R. (1992). Making use of goals and objectives for internal program evaluation. Journal oj
Education jb r Library and Injbrmatio n Science. 33 (2), 129-140.
Acknowledgements
To the memory of the late Dr. Bonifacio Jim6nez who made a great contribution to this study. The
authors would moreover like to thank the APQUA project's members, the SEPUP members, and the
teachers who answered the questionnaires and participated to the interviews. They also thank Dr. Herbert
Thief and Marlene Thier for their many and relevant suggestions to an earlier version of the paper.
The A u t h o r s
engineer from the National School of Mines and Industry at Rabat (Morocco) and gained a
Ph.D. with the European Label of the University Rovira i Virgili.
Correspondence: <seboudam@urv.net>