Learning and Instruction 16 (2006) 12e25 www.elsevier.

com/locate/learninstruc

Metacognitive macroevaluations in mathematical problem solving

Annemie Desoete a,b,*, Herbert Roeyers a
a

Department of Experimental Clinical and Health Psychology, Ghent University, Ghent, Belgium b Artevelde Bachelors, Ghent, Belgium

Abstract This paper focuses on the role of evaluation in mathematics in 749 elementary school children. The macroevaluative skills and calibration scores of high versus low mathematical problem solvers were contrasted as measures of metacognition. No relevant calibration differences were found for gender. In addition, the performances of children with mathematics learning disabilities could not be explained according to the maturational lag hypothesis. Finally, although macrometacognitive evaluation and calibration seem attractive alternatives for time-consuming on-line metacognitive assessment techniques, our data show that a global and retrospective assessment of the macroevaluation is not always enough to get the picture of mathematical problem solving in young children. 2005 Elsevier Ltd. All rights reserved.
Keywords: Metacognition; Macroevaluation; Microevaluation; Assessment; Mathematics performance; Mathematics learning disabilities; Child

1. Introduction 1.1. Metacognitive construct Metacognition has been introduced in literature to describe and explain the difculties learners meet when dealing with information processing and problem solving (Flavell, 1987). Metacognition has traditionally been differentiated into two central components, namely metacognitive knowledge and metacognitive skills. A substantial amount of data has been accumulated on prediction, planning, monitoring and evaluation or reection skills (e.g., Brown, 1987; Desoete, in press; Van Hout-Wolters, 2000). In young children, a combination of all metacognitive knowledge and skill parameters was found to explain 37% of the variance in mathematical problem solving. A combination of prediction and evaluation skills explained about 16% of the variance in mathematics. This combination was also successful to

* Corresponding author. Department of Experimental Clinical and Health Psychology, Ghent University, Henri Dunantlaan 2, B-9000 Ghent, Belgium. E-mail address: anne.desoete@Ugent.be (A. Desoete). 0959-4752/$ - see front matter 2005 Elsevier Ltd. All rights reserved. doi:10.1016/j.learninstruc.2005.12.003

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

13

differentiate children with mathematics learning disabilities from below-average performing peers and average performers from expert problem solvers (Desoete, Roeyers, & Buysse, 2001). Metacognitive prediction and evaluation skills are sometimes referred to as calibration skills. Calibration can be dened in terms of whether the predicted (or evaluated) value corresponds with the occurrence of that value on the criterion test (e.g., Lin & Zabrucky, 1998). A comparison is made of whether the prediction before a task corresponds to the actual performance on the task. Schwartz and Metcalfe (1992) differentiated micropredictive and macropredictive accuracy. The correspondence on individual problems can be dened as microprediction. The correspondence between overall expectations and proportion of correct answers can be dened as macroprediction. In macroprediction tasks children are asked in advance to look at a test on several mathematical problems without solving the problems. They then have to estimate how good they will do on such a test. When they afterwards solve the problems, the global predicted score is compared with the actual result on the test. In the measurement of microprediction, children are asked to look at each mathematical problem without solving them and give a task-dependent prediction on a 4-point rating scale, whether they will be successful in this task. Similarly, microevaluation and macroevaluation are dened, respectively, as the correspondence on individual problems and the correspondence between the proportion of correct answers and the overall expectations after the problem solving.

1.2. Metacognitive assessment Different methods to assess metacognitive knowledge and skills make study outcomes often difcult to compare (e.g., Pressley, 2000; Schwarz, 1999; Sperling, Howard, Miller, & Murphy, 2002). Prospective methods, such as selfreport questionnaires and hypothetical interviews are frequently used to assess metacognitive skills (e.g., ElshoutMohr, Meijer, van Daalen-Kapteijns, & Meeus, 2003; Pintrich & De Groot, 1990; Thorpe & Satterly, 1990). Several studies underlined the importance of these instruments (e.g., Busato, Prins, Elshout, & Hamaker, 1998). However, Veenman (2003) pointed to the limited explained variance towards learning outcomes of prospective assessment methods. Retrospective techniques, such as questionnaires, interviews and stimulated recall situations, are also being used to assess metacognitive skills (e.g., Artelt, 2000; Masui & De Corte, 1999). Several authors fruitfully used (e.g., Zimmerman & Martinez-Pons, 1990) or combined such instruments (e.g., Efklides, 2001; Elshout-Mohr et al., 2003). Moderate correlations were demonstrated between prospective and retrospective measures (Veenman, 2003). However, often retrospective verbal reports of higher order skills seem to lack accuracy (Nisbett & Wilson, 1977) and a very limited explained variance was found for such techniques (Veenman, 2003). In addition to prospective and retrospective techniques, concurrent assessment, such as think-aloud protocols (e.g., Artzt & Armour-Thomas, 1992) or a systematical observation of regulation and planning behavior (e.g., Pugalee, 2001) can take place. Protocol analyses were found to be very accurate, but time-consuming techniques to assess metacognitive skills (e.g., Pressley, 2000). Veenman (2003) found a high correlation between protocol analyses and systematical observation measures. Moreover, a good explained variance towards learning performances was found. However, the correlation between concurrent and non-concurrent (prospective or retrospective) measures was very low. Moreover an adequate level of verbal uency seems required in these kind of assessment techniques to avoid interference of the verbalization with the target task (Thorpe & Satterly, 1990; Veenman, 2003). Recently, also more indirect and multi-method techniques are being used. Often these techniques combine prospective and concurrent or concurrent and retrospective measures of metacognitive skills and/or knowledge (e.g., Dermitzaki & Efklides, 2003). For example students are asked, before and after the solution of a mathematics task, to assess the difculty of the task, the correctness of the solution (conceived or produced), the effort required and to make subjective estimations about the use of problem-solving strategies. In calibration studies, a comparison is made of whether the prediction before the tasks (calibration or comprehension paradigm) or the evaluation after a task (performance calibration or postdiction paradigm) corresponds with the actual performance on the task (Glenberg, Sanocki, Epstein, & Morris, 1987; Lin & Zabrucky, 1998). Calibration studies are therefore most related with metacognitive knowledge assessment. Tobias and Everson (2000) developed the Metacognitive Knowledge Monitoring Assessment (KMA). They compare what students think (or predict) they know or do not know and what they really know and do not know. This research design is very similar to our own research to measure micropredictions and microevaluations (De Clercq, Desoete, & Roeyers, 2000).

14

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

To conclude, several striking problems emerge in the assessment of metacognition (Artzt & Armour-Thomas, 1992). On one hand, concurrent measures are very accurate but time-consuming techniques. On the other hand, prospective and retrospective methods are less time-consuming but it is unclear whether they really reect the ongoing learning and problem-solving behavior. The present study aims to add some data into the debate on the value of calibrations and macroevaluations in young children.

1.3. The interplay between mathematics learning and metacognition in children with and without mathematics learning disabilities It is nowadays widely accepted that metacognitive knowledge and skills inuence mathematical problem solving (e.g., Borkowski, Chan, & Muthukrishna, 2000). In addition, metacognitive skills were found to be trainable and modiable with value added to mathematical problem solving (e.g., Kramarski, Mevarech, & Arami, 2000). From a developmental point of view, metacognitive knowledge precedes metacognitive skills (Flavell, 1987). Metacognitive skills were found to be maturing until adolescence (Shute, 1996). In the past decade, the effects of several mechanisms (subject, task and situation variables) have been examined on this matter (Lin & Zabrucky, 1998). Evidence has suggested that students prior domain knowledge and general mathematics prociency can inuence the self-assessments of performances (Lin & Zabrucky, 1998; Nietfeld & Schraw, 2002). In addition, high metacognitive knowledge and the cognitive style variable Need-for-Cognition was found related to enhanced posttest performance judgment accuracy and realism in condence (Jonsson & Allwood, 2003). Moreover, several motivational and socialaffective processes seem involved in postdiction performance judgments and also item difculty might have an impact on posttest performance calibration (Hunter-Blanks, Ghatala, Pressley, & Levin, 1988). The effect of these factors can be accounted for in term of specic cognitive processes they tend to activate. Nevertheless, due to the complexity of mathematical problem solving it is still an open empirical question which cognitive processes are activated by these processes, leading to accurate evaluations and calibrations in young children (Jonsson & Allwood, 2003). Several below-average mathematical problem solvers were found to lack metacognitive skillfulness or to fail to use their metacognitive skills, either because they do not know how or when to evaluate, or they do not recognize the relevance of those skills for a particular task (Baker, 1994). In addition, studies of children with mathematics learning disabilities revealed less accurate task-specic micropredictions. Furthermore, less accurate task-specic microevaluations have been found in studies conducted on children with mathematics learning disabilities (Desoete et al., 2004; Desoete & Roeyers, 2002; Lucangeli, Cornoldi, & Tellarini, 1998). However, less research data are currently available on macroevaluations, although some studies point in the direction of little or no correspondence between macroevaluations on the one hand, and actual concurrent behavior on the other hand, since often people simply do not recollect accurately what they have done (Nisbett & Wilson, 1977; Veenman, 2003). In addition, although some variables have been shown to have a signicant impact on mathematical problem solving, results of the effects of calibration have been inconsistent. Students overcondence was often illustrated but condence rates were also often barely correlated with performance scores (Lin & Zabrucky, 1998). There is also considerable research as to whether poorer skills of children with mathematics learning disabilities have to be considered as demonstrations of a delay or as a decit in those children. In the case of the delay or maturational lag hypotheses, the low performances of those children can be explained due to the lack of or immature developed (meta)cognitive skills (maturational lag hypothesis). According to this hypothesis, the skills of children with mathematics learning disabilities of the procedural type develop slower than those of their peers without disabilities (e.g., Geary, 2003). This seems especially true for procedural skills to solve simple arithmetic and word problems (Geary, 2003). Another possible explanation is the decit hypothesis, whereby metacognition is considered to be a core decit in children with learning disabilities. Children with mathematics learning disabilities of the semantic memory type would then be expected to have different or disrupted (meta)cognitive skills, not comparable to the skills of younger children (Geary, 2003). According to these hypotheses, there would be no developmental delay in terms of ability to use skills, but a more fundamental decit. The skills of children with mathematics learning disabilities would not develop slower, but fundamentally and qualitatively different from normal children in the mechanisms supporting mathematical problem solving. Such a pattern was found in the anomalous long-term memory representation of addition facts in children with mathematics learning disabilities (Geary, 2003). In this case, we cannot expect (meta)cognitive skills to develop spontaneously and therapy should focus on these decits.

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

15

1.4. Gender differences in mathematics learning For decades gender differences in mathematics attract research attention. A comparison between the research data highlights considerable overlap in mathematical performance of males and females (e.g., Cheung & Rudowicz, 2003). However, often signicant differences in mathematical performance tend to favor males (e.g., Evans, Schweingruber, & Stevenson, 2002). Boys seem to perform better than girls, especially when it comes to higher cognitive questions in application and word problems, to fast and accurate arithmetic facts retrieval from long-term memory and to visualization, estimation and visuo-spatial aspects of mathematical problem solving (e.g., Casey, Nuttall, & Pezaris, 2001; Royer, Tronsky, Chan, Jackson, & Marchant, 1999). In addition, research data often reveal that boys self-concept is higher than girls and boys often display a more positive attitude and different goal orientations towards mathematics (e.g., Casey et al., 2001; Kling, Hyde, Showers, & Buswell, 1999). However, there is evidence that gender differences tend to decrease (e.g., Eisenberg, Martin, & Fabes, 1996) or even disappear (e.g., Frost, Hyde, & Fennema, 1994; Pajares & Graham, 1999). Nevertheless, there are also studies available where males score signicantly lower on mathematical problem solving (e.g., Freeman, 2003) and seem less motivated than females (e.g., Peetsma, Roeleveld, & Stoel, 2003). 1.5. Aims of the study A great deal of research has been devoted to understand the functioning of the task-specic or micrometacognitive skills on mathematical problem solving. However, the short overview clearly shows that there is less research concerning the macrometacognition and performance calibration, especially in young children. In addition, most research is based on the behaviors of young adults (Lin & Zabrucky, 1998). In the current study, we narrow our research to three major aims. First, the present study aims to add some data on the importance of macroevaluations and performance calibrations on mathematical problem solving. The second aim of this study is to investigate whether boys differ in metacognitive macroevaluations and performance calibrations from girls. It is hypothesized that boys will have higher macroevaluations (e.g., Casey et al., 2001; Kling et al., 1999), with equal mathematical performances. Lastly, the present study aims to investigate the delay versus decit hypothesis. 2. Method 2.1. Participants Participants were second grade (n 123), third grade (n 376) and fourth grade (n 250) children attending 42 elementary schools in the Dutch-speaking part of Belgium. All children were followed during two years. The sample included only white children, 386 girls and 363 boys. All children followed regular elementary education. The socioeconomic status (SES) based on the years of education of father and mother was measured (see Table 1). Permission for children to participate in this study was obtained from their parents. To investigate the delay versus decit hypothesis, the initial sample was divided into ve mathematics performance groups, namely second grade average performers, third grade children with mathematics learning disabilities, third grade children with average mathematical problem-solving skills, above-average mathematical problem solvers in grade 3 and fourth grade children with average performances on mathematics. A combination of criteria (anamnesis,
Table 1 Description of the participants (n 749) Grade 2 M (SD) SES Father Mother Mathematics knowledge and skills Arithmetic facts 14.04 14.04 44.91 36.59 (3.65) (2.76) (24.46) (9.58) Grade 3 M (SD) 13.75 13.94 35.64 46.36 (3.33) (2.62) (25.51) (25.57) Grade 4 M (SD) 14.99 14.27 48.93 49.34 (4.16) (3.05) (28.40) (28.91) F(2,746)

0.75 0.63 20.25 10.98

SES (socio-economic status) is based on the years of education. In the Mathematics knowledge and skills test and the Arithmetic facts test, the total percentile for grade 2 was used for all second graders. For the third graders, the total percentiles for grade 3 were used. For the fourth graders, the total percentile for grade 4 was used.

16

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

testing, observation and teacher rating) was used to include a selected group of 250 children out of the initial sample of 749 children. Out of the initial sample, all children with mathematics learning disabilities in grade 3 were included in the group of third grade children with mathematics learning disabilities. All of those children had below critical cut-off scores (they scored 2SD on one or more mathematics tests) across two successive grades. In addition, only white native Dutchspeaking children without histories of extreme hyperactivity, sensory impairment, brain damage, a chronic medical condition, insufcient instruction or serious emotional or behavioral disturbance were included in this study (exclusion criterion). Moreover, each child demonstrated a severe and resistant (severeness criterion) ability-achievement, with a score of 1 or 2 out of 7 on a one-item teacher rating of mathematical competence (see Appendix A). Within this group the SES based on the years of education of father and mother was measured. Children of the control groups were matched on this parameter with the children in the group with mathematics learning disabilities (see Table 2). Three control groups were included in the contrastive analysis. The rst control group consisted of a randomly selected group of 50 average mathematical problem solvers (ages 8e9) without a diagnosis of learning disability or other problems. All of these children came out of the same school and had a comparable SES as the children with mathematics learning disabilities. They were age- and SES-matched with the children with learning disabilities based on not more than l week of difference in date of birth and not more than 2 years of difference between the level of education of both parents (see Table 2). In addition, to be accepted the participants had to (a) be in grade 3, (b) with scores between the 0.1 and 0.5SD on the mathematics component, (c) no learning problems across two successive grades and (d) ratings of 4 out of 7 on mathematics by the teacher (see Appendix A). The second control group consisted of 50 average-intelligent second grade students (ages 7e8) without a diagnosis of learning disability or other problems. All of these children were performance- and SES-matched with the children with mathematics learning disabilities. For this purpose, children with mathematics learning disabilities in grade 3 and the group of young children in grade 2 were matched on mathematics tests for grade 3. Only children in grade 2 were accepted in this study if they could be matched with a child with mathematics learning disabilities and had less than 5 points of difference in performance scores on the KRT3 (Kortrijk Arithmetic Test Grade 3, Cracco et al., 1995) and less than 5 points of difference in performance scores on the Tempotest Rekenen (De Vos, 1992) compared with

Table 2 Description of the nal sample (n 250) Average performers, grade 2 Mathematics component Matched with learning disabilities, grade 3 4 Average performers, grade 2 M (SD) SES Father Mother 14.02 (3.70) 14.02 (2.78) Learning disabilities, grade 3 2SD Average performers, grade 3 0.1/0.5 Above-average performers, grade 3 2SD Average performers, grade 4 0.1/0.5

Teacher rating

1 or 2 Learning disabilities, grade 3 M (SD) 14.39 (3.58) 14.19 (2.53) 9.94 (6.77) 13.34 (12.34)

4 Average performers, grade 3 M (SD) 13.73 (3.30) 13.97 (2.65) 42.06 (12.48) 50.52 (18.16)

6 or 7 Above-average performers, grade 3 M (SD) 13.81 (2.58) 13.66 (3.01) 73.38 (14.12) 75.96 (12.86)

4 Average performers, grade 4 M (SD) 15.29 (4.36) 14.89 (3.25) 54.08 (17.27) 50.22 (26.05) F(4,245)

2.19 1.79 188.89 90.95

Mathematics knowledge and skills Percentile 58.78 (14.66) Arithmetic facts Percentile 40.98 (8.20)

SES (socio-economic status) is based on the years of education. In the Mathematics knowledge and skills test and the Arithmetic facts test, the total percentile for grade 2 was used for the average performing second graders. For the third graders (children with learning disabilities in grade 3, average performers, above-average performers), the total percentiles for grade 3 were used. For the average performing fourth graders, the total percentile for grade 4 was used.

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

17

children with mathematics learning disabilities. In addition, the second graders had to receive a score 4 out of 7 on a teacher rating of mathematical competence (see Appendix A). In addition, only children who had no learning problems across two successive grades were included in this group. Based on these criteria, 50 children in grade 2 were matched with the children in grade 3 with specic mathematics learning disabilities. The third control group consisted of a randomly selected group of 50 average mathematical problem solvers (ages 9e10) without a diagnosis of learning disability or other problems. All of these children came out of the same school and had a comparable SES as the children with mathematics learning disabilities. In addition, to be accepted in our sample (a) the participants had to be in grade 4 and (b) scores had to be between 0.1 and 0.5SD above the mean on the mathematics component score and (c) they had to score 4 out of 7 on a teacher rating of mathematical competence (see Appendix A). Only children who had no learning problems across two successive grades were included. In addition, since academic problems can be studied on a continuum from very poor to very good mathematical problem solving, it might be interesting to investigate whether the difference between expert problem solvers in grade 3 and older children is the same as the difference between children with learning disabilities in grade 3 and average performing children in the same grade. Therefore, a control group with expert problem solvers in grade 3 was added. All of these children came out of the same school and had SES and age comparable to the children with mathematics learning disabilities. With this control group, we aim to investigate in a normal population whether children with above-average performances in the area of mathematics show subpar performance on metacognitive skills. To be accepted in our sample as above-average or expert performing children, the participants had to (a) be in grade 3, (b) obtain a score of 2SD on the mathematics component, (c) perform above-average across two successive grades and (d) score at least 6 out of 7 on a teacher rating of mathematical competence (see Appendix A). A description of the mathematics performance of the children is presented in Table 2. 2.2. Measures The Arithmetic Number Fact Test (Tempotest Rekenen, TTR) (De Vos, 1992) is a test on 200 arithmetic numberfact problems (e.g., 5 9 _). Children have to solve as many number-fact problems as possible out of 200 in 5 min. ` The test has been standardized for Flanders on 10,059 children (Ghesquiere & Ruijssenaars, 1994). We used the standardized total percentile based on national norms for all groups of children, so that we could compare the mathematics performances within and between all grades. The Kortrijk Arithmetic Test (Kortrijkse Rekentest, KRT, Cracco et al., 1995) is a 60-item Belgian mathematics test on domain-specic knowledge and skills (e.g., 129 879 _; add three tens to 61 and you have _), resulting in a percentile for mental computation (e.g., 129 879 .), a percentile for number system knowledge (e.g., add three tens to 61 and you have .) and a total percentile. The psychometric value has been demonstrated on a sample of 3246 Dutch-speaking children. A validity coefcient (correlation with school results) and reliability coefcient (Cronbachs alpha) of 0.64 and 0.91, respectively, were found. In all groups (second graders without learning disabilities, third grade children with mathematics learning disabilities, average performing third graders, above-average performing third graders, fourth graders without learning disabilities), the standardized total percentile based on Dutch norms was used. The version for grade 2 was used for second graders, the version for grade 3 was used for all third graders (third graders with learning disabilities, average performing third graders and above-average performing third graders), the version for grade 4 was used for all fourth graders. In addition, the children in grade 2 also carried out the version for grade 3 in order to make matching possible with the children with mathematics learning disabilities in grade 3. We used the standardized total percentile, percentile for mental computation and percentile for number system knowledge, based on national norms for all groups of children, so that we could compare the mathematics performances within and between all grades. The Cognitive Developmental aRithmetics test (CDR) (Desoete & Roeyers, 2001) is a 90-item test developed for the assessment of arithmetics (e.g., 5 more than 40 is ?). The number of correct answers is the performance score (e.g., 60/90 on the test). In addition, children have to gauge condence in the correctness of the given answers (e.g., I think I will obtain 70/90 on this test). The score children attribute to their work (e.g., here 70) is the macroevaluation score. The difference between the performance (e.g., here 60) and macroevaluation score (e.g., 70) is the calibration score (e.g., here 10). The CDR was specically designed for the present research line. The psychometric value has been demonstrated on a sample of 483 Dutch-speaking children in Flanders. We used the number of correct answers and not the standardized total percentile as cognitive score in Tables 3 and 5. The CDR was tested in a pilot study in order to

18

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

Table 3 Age gender differences on mathematics and metacognition Grade 2 Boys M (SD) Performance Macroevaluation Calibration 38.79c (12.61) 82.18a (14.67) 43.39c (19.19) Girls 26.28c (8.20) 74.29b (22.78) 48.02c (22.19) Grade 3 Boys M (SD) 61.98b (12.53) 80.58a (13.11) 18.34b (15.07) Girls 58.15b (11.79) 77.44b (15.24) 19.13b (15.80) Grade 4 Boys M (SD) 75.45a (9.10) 81.12a (9.85) 5.64a (10.32) Girls 76.37a (8.54) 81.35a (14.28) 4.96a (11.57) 15.45* 3.59* 1.24* F(2,743)

*p 0.05. a,b,c,d: different indexes refer to signicant between-group differences with signicance level 0.05. All children made the same CDR (Cognitive Developmental aRithmetics) test, with a subscore for performance, macroevaluation and calibration. Results can be compared in between the groups.

determine its usefulness for this age group and for its sensitivity in measuring individual differences (n 30). To examine the psychometric characteristics of the developed metacognitive parameter, students were observed and videotaped during and interviewed after the test. Protocol analyses were done on the videotaped think-aloud protocols. The children postdictions (e.g., how many correct answers do you think you gave on this test?) referred to the constructs in question (e.g., macroevaluations). In addition, Gutmanns split-half and SpearmaneBrowns coefcients were 0.70 and 0.72, respectively. Furthermore, all variables were normally distributed and testeretest correlations of 0.85 (p < 0.0005) were found on 150 of the children. The teacher questionnaire (see Appendix A), which was created for this study, is a rating scale (8-item) questionnaire for teachers on metacognitive skills (e.g., the child never (1)/always (5) knows in advance whether an exercise will be easy or difcult). Furthermore, teachers score the mathematics and reading performances as well as the intelligence of children (e.g., very low compared to peers (1)/very good compared to peers (7)). The teacher questionnaire was tested in previous studies in order to determine its construct validity. Testeretest correlations of 0.81 (p < 0.01) and interrater reliabilities varying between 0.99 and 1.00 (p < 0.01) were found (Desoete & Roeyers, 2002). 2.3. Procedure All subjects were assessed individually, outside the classroom setting, where they completed the KRT (Cracco et al., 1995), TTR (De Vos, 1992) and the CDR (Desoete & Roeyers, 2001), on two different days, for about 2 h in total. Regular teachers completed a teacher survey (see Appendix A) in the same period. The examiners, all psychologists or therapists skilled in learning disabilities, received practical and theoretical training in the assessment and interpretation of mathematics, and metacognition. The training took place two weeks before the start of the assessment. In addition, systematic, ongoing supervision and training was provided during the assessment of the rst 25 children with and without learning disabilities. The training included a review and discussion of the CDR student proles and involved one meeting during the assessment period. 3. Results Given the high intercorrelations between the percentile on the TTR (De Vos, 1992) and the percentile on the KRT Mental arithmetics (Cracco et al., 1995) (r 0.52, p < 0.0005), the percentile on the TTR and the percentile on the KRT Number system knowledge (r 0.43, p < 0.0005) and the KRT Mental arithmetics and KRT Number system knowledge percentiles (r 0.72, p < 0.0005), the internal structure of the data was analyzed with a Principal Components Analysis, to account for all the variance. This analysis was carried out to develop a small set of components empirically summarizing the correlations among the variables. With a principal axis factor analysis, allowing covariance within the data, the same factors were found and the data remained almost the same. In order to determine whether mathematics data could be combined into one or more factor components, an initial run with Principal Components extraction, without rotation, was carried out. Three components were needed to account for all the variance in our data set. This initial number of three could be reduced to one, retaining enough components for an adequate t but not so many that parsimony was lost. This number of components in our solution was based on two criteria. The rst criterion was that there was only one component with an eigenvalue higher than l (Kaizer normalization). Components 2 and 3 had an eigenvalue of 0.72 and 0.23,

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

19

respectively, and were found not as important from a variance perspective. The second criterion as to the adequacy of a one-component solution to our data set was that a one-component solution accounted for 68.35% of the common variance. The second and third factors accounted for 24.09 and 7.56% of variance, respectively. This one component means that all our sub-skills involved in mental arithmetics, number knowledge and number-fact retrieval, participate to the same general competence, probably heavily inuenced by computational skills, which we call the mathematical problem-solving competence. To investigate the relationship between the mathematical problem-solving competence, macroevaluation and calibration, Pearson correlations were computed on all subjects (n 749). A signicant relationship between mathematical problem-solving competence and macroevaluation (r 0.17, p < 0.01) and between mathematical problem-solving competence and calibration (r 0.27, p < 0.0005) was found. In addition a signicant, but negative, relationship was found between the macroevaluations and the calibrations (r 0.68, p < 0.0005). To look for age and gender related differences on mathematics, we conducted a 2 3 Univariate Analysis of Variance (ANOVA) on the total sample (n 749) with mathematics performance (on the CDR) as dependent variable and grade (grade 2, grade 3 and grade 4) and gender (boys, girls) as independent factors. Post hoc tests were performed, using an appropriate procedure (Tukey if equal variance could be assumed from the Levene test and Tamhane if equal variance could not be assumed from the Levene test) and the observed power and the effect sizes (h2) were calculated. The analysis of variance revealed a signicant main effect for age (F(2,741) 644.58, p < 0.0005; h2 0.64; Power 1.00) and for gender (F(1,741) 33.03, p < 0.0005; h2 0.04; Power 1.00) and a signicant interaction effect for age gender (F(2,741) 15.56, p < 0.0005; h2 0.04; Power 0.99). Means and standard deviations are presented in Table 3. As can be concluded from Table 3, children from grade 4 had higher scores on the CDR than children in grades 3 and 2, and children in grade 3 had higher mathematics performance scores than children in grade 2. Moreover, boys did better than girls in grade 2 mathematics performance, but no signicant gender differences were found in grades 3 and 4. To look for age and gender related differences on macroevaluation, controlling for mathematical problem-solving differences, we conducted a 2 3 Univariate Analysis of CoVariance (ANCOVA) on the total sample (n 749) with macroevaluation (on the CDR) as a dependent variable, mathematics performance (on the CDR) as covariate and grade (grade 2, grade 3 and grade 4) and gender (boys, girls) as independent factors. Post hoc tests were performed, and the observed power and the effect sizes (h2) were calculated. The ANCOVA revealed a signicant main effect for evaluation (F(6,740) 11.49, p < 0.0005; h2 0.08; Power 1.00) and mathematics performance (F(1,740) 47.84, p < 0.0005; h2 0.06; Power 1.00). In addition, signicant between-subject effects with a very small effect size were found for age (F(2,740) 9.70, p < 0.0005; h2 0.03; Power 0.98), a trend was found for gender (F(1,740) 3.21, p 0.07; h2 0.04; Power 0.43), but no signicant interaction effect for age gender (F(2,740) 0.89, p NS) was revealed. As can be concluded from Table 3, boys did better than girls in grade 2 on evaluation and mathematics performance, but no signicant gender differences were found in grade 4. To look for age and gender related differences on calibration, a 2 3 Univariate Analysis of Variance (ANOVA) was conducted on the total sample (n 749) with calibration as a dependent variable, and gender (boys and girls) and grade (grade 2, grade 3 and grade 4) as independent factors. The ANOVA was signicant (F(5,743) 115.53, p < 0.0005, h2 0.44; Power 1.00). Between-subject effects were found for age (F(2,743) 286.42, p < 0.0005; h2 0.44, Power 1.00), but not for gender (F(1,743) 1.62, p NS) or for age gender (F(2,743) 1.24, p NS). Means and standard deviations are presented in Table 3. As can be concluded from Table 3 all children overestimated their performances. No gender differences were found on this matter. Moreover, older children were found to have a more accurate calibration score than younger children. In addition, a regression analysis was performed on mathematical problem-solving competence (component score) as the outcome variables with the calibration scores and macroevaluation scores simultaneously as predictor variables. R2 was 0.24 and F(2,746) 115.69, p < 0.0005, meaning that only a small proportion of variance could be explained by the combination of calibration and evaluation scores of children (see Table 4). Calibration (F(1,747) 54.15, p < 0.0005) predicted 7% of the variance, whereas evaluation (F(1,747) 42.55, p < 0.0005) predicted 5% of the variance. To investigate the delay hypothesis, out of the initial sample of second, third and fourth graders, 250 children were selected based on a combination of selection criteria. These children were divided into ve mathematical problem-

20

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

Table 4 Prediction of mathematical problem-solving competence by calibration and evaluation scores Variable Constant Calibration Macroevaluation *p < 0.0005. Unstandardized coefcients 2.24 0.00 0.00 b 0.49 0.47 t 11.34 13.37 12.86 p * * *

solving performance groups (average performing second graders, children with learning disabilities in grade 3, average performing third graders, above-average performing third graders and average performing fourth grader). The inclusion of ve groups was important to control variables that might affect the results and to ensure that the difference found could be attributed to the delay hypothesis, rather than to other factors, such as teaching method, gender or socio-economic status (SES). A Multivariate Analysis of Variance (MANOVA) was conducted with the performance and evaluation scores as dependent variables and mathematical problem-solving competence performance group (component score) as independent variable. Post hoc analyses were conducted using the Tamhane procedure. With a power of 1.00 and a medium effect size (h2 0.60), the MANOVA revealed a signicant main effect of group (F(8,488) 93.38, p < 0.0005). Signicant between-subject effects emerged for mathematics performance (F(4,245) 298.05, p < 0.0005; h2 0.88) and for macroevaluation (F(4,245) 8.30, p < 0.0005, h2 0.12). Means and standard deviations are presented in Table 5. Post hoc follow-up analyses revealed that above-average performing children did not differ from average performing older children on mathematics performance. Therefore, also the above-average performing third graders and the one year older average performing children seem to be adequately matched. In addition, children with above-average mathematical skills and performance-matched older children with average skills also share macroevaluation skills with one another. Both groups had similar condence ratings. Furthermore, it can be concluded from Table 5 that children with mathematics learning disabilities in grade 3 did worse than age-matched children, but no differences were found between the children with learning disabilities in grade 3 and matched children in grade 2. Therefore, the children with learning disabilities in grade 3 and the younger children groups seem to be adequately matched on mathematics. However, post hoc follow-up analyses revealed that children with mathematics learning disabilities evaluated the correctness of produced solution lower than the younger children. In addition, an Univariate Analysis of Variance (ANOVA) was conducted with the calibration scores as dependent variables and mathematical problem-solving competence group (average performing second graders, children with learning disabilities in grade 3, average performing third graders, above-average performing third graders and average performing fourth grader) as independent variable. Post hoc analyses were conducted using the Tamhane procedure. With a power of 1.00 and a small effect size (h2 0.48), the ANOVA was signicant (F(4,245) 56.34, p < 0.0005). Post hoc follow-up analyses revealed that all groups of children differed from one another. Children with mathematics learning disabilities did better than performance-matched younger children but worse than agematched peers without learning disabilities. Children with mathematics learning disabilities outperformed the children in grade 2 without learning disabilities on calibration tasks. The learning disabled groups lower performance scores
Table 5 Group differences in performance, macroevaluations, and calibration scores (n 250) Average performers, grade 2 M (SD) Performance Macroevaluation Calibration 41.32a (9.46) 81.66b (11.37) 40.34e (15.61) Learning disabilities, grade 3 M (SD) 43.80a (8.07) 70.50a (21.16) 26.30d (20.00) Average performers, grade 3 M (SD) 63.44b (6.51) 79.66b (13.84) 16.22c (14.19) Above-average performers, grade 3 M (SD) 75.78c (5.19) 84.50b (4.34) 8.72b (6.05) Average performers, grade 4 M (SD) 77.96c (5.11) 81.26b (8.33) 3.28a (9.08) F(4,245)

298.05* 8.30* 56.34*

All children made the same CDR (Cognitive Developmental aRithmetics) test, with a subscore on performance, macroevaluation and calibration (macroevaluation minus performance). Results of performance, macroevaluation and calibration can be compared between the groups. *p 0.05. a,b,c,d: different indexes refer to signicant between-group differences with signicance level 0.05.

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

21

matched their condence evaluations, producing more accurate calibration scores. Means and standard deviations are presented in Table 5.

4. Discussion Since Flavell introduced the concept 20 years ago, different methods to assess metacognition have been used (Tobias & Everson, 2000). In this study we focus on macroevaluations because combined assessments at the taskspecic level have been shown to be extensively time-consuming. Moreover, we were interested in eventual gender differences and in whether children with mathematics learning disabilities had macroevaluation skills, comparable to younger children (delay hypotheses). The data from the present study are in line with earlier investigations that have documented the relationship between mathematics and micrometacognitive skills (e.g., Lucangeli et al., 1998). A small but signicant relationship between a mathematical component and performance calibration and mathematics and macroevaluation skills was found. In light of this relationship, it might be important to focus on these factors from the start in young elementary school children and in teacher education programs because postdiction might direct learners future engagement in self-regulated mathematical problem solving. However, since the overall correlations between postdiction, calibration and actual test performance are rather low, if remedial interventions are to be implemented to effectively improve performance calibration accuracy, it is essential to also focus on all the other possible criteria to predict performance (Lin & Zabrucky, 1998). In addition, the data from the present study indicate a large discrepancy between the performance calibration scores in children in grade 2 compared with grade 3 or grade 4 children. Young children were not only inaccurate in estimating their test performance, but also often wildly overestimated their mathematics performance. However, all children in this study overestimated their mathematics results. The calibration or discrepancy between mathematics performance and macroevaluation varied from 40 points for second graders to 15 points for third graders and 5 points for fourth graders. Older children had a more accurate calibration score than younger children. However, most children thought they would have a score of about 80/90 on the test, independent of their actual performances. Some children actually said in the think-aloud protocols I could not solve 10 questions, so I will have 80/90 although of the 80 solved questions there were 20 wrong answers. The pattern in these results, in line with Veenman (2003), suggests that people often do not report adequately about how they have done. Results from the present research reveal little or no correspondence between retrospective macroevaluations on one hand, and actual, concurrent mathematical problemsolving behavior on the other. Generally speaking, young children do not recollect accurately what they have done. This seems to get better once they are in grade 4. Calibration may, however, be an accurate measure for older children, but certainly not for young elementary school children. It seems appropriate to measure metacognition in young children with combined task-specic or concurrent techniques. Another implication that arises is that researchers and school practitioners should be looking into students bias towards overcondence in the rst years of elementary school. Additional research will be required to ascertain whether the tendency to make accurate and less overcondent judgments of the math capability as the years progress result in more engagement in mathematics. We also investigated the effect of gender on mathematics. Consistent with the male advantage on mathematical problem solving (e.g., Casey et al., 2001), the data from the present study indicate that boys are better mathematics problem solvers than girls in grade 2. However, when we analyze this performance further, boys no longer outperform girls in grades 3 and 4. Girls seem to start slow on mathematical problem solving, catching up the lag during elementary school. In addition, we sought to discover whether macroevaluations varied by gender. In our study, boys had higher level macroevaluations than girls in grades 2 and 3, but no differences were discovered in grade 4 children. However, girls were also biased from grade 2 till grade 4 towards overcondence, as were boys of the same age. Moreover, an objective was to reveal whether math-related calibration measures varied by gender. The data of our study indicate no gender differences on the calibration skills of young elementary school children. For our sample the condence gap between boys and girls was not in evidence in the rst grades of elementary school. Therefore, no large consistent gender related qualitative differences could be grounded based on the data of the present study. However, these ndings call for additional exploration of the contribution of the developmental aspects of gender variables. The gender differences seem to decrease as students progress through elementary school. Girls seem to get the same metacognitive prole as boys, as they grow older. It seems warranted to suggest additional research on these aspects in

22

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

order to understand more about self-evaluation and performance calibration and explain a part of development in mathematics ability in young boys and girls. In addition, we investigated the delay hypothesis. Our data are not entirely in line with the delay hypothesis. As expected, children with expert performances in grade 3 were found to have mathematics, evaluation and calibration scores comparable to those of older average performers in grade 4. These results may point in the direction of a virtual continuum from very poor to very good mathematical problem solving. Furthermore, our data revealed that children with mathematics learning disabilities and younger children were matched on mathematics but the learning disabled group had lower evaluation scores than the performance-matched children. These results could not be totally explained by the delay hypothesis, but indicate a different metacognitive prole in children with mathematics learning disabilities and performance-matched younger children. In addition, children with mathematics learning disabilities did unexpectedly better than mathematics performance-matched children on calibration tasks. This may seem inconsistent, but interviews afterwards with some of the children taught us that children with mathematics learning disabilities more often experience and are told that they make mistakes, resulting in a lower (and therefore more accurate) macroevaluation score. The children in grade 2 often had the illusion of being able to solve exercises of this kind, since they were used to solve exercises correctly. This claries this nding, which at rst glance otherwise appears strange. However, this nding also suggests that the metacognitive prole of children with mathematics learning disabilities cannot be totally interpreted according to the maturational lag or delay hypothesis. Taking into account the complex nature of mathematical problem solving, it may be useful to assess metacognitive skills in young children with mathematics learning disabilities in order to focus eventually on these factors and their role in mathematics learning and development. It may be possible that large discrepancies between postdictions and actual performance are due to the inability of some students (especially younger students) to self-generate feedback. External and guided feedback might help to improve calibration. Consistent with previous ndings on the modiability of metacognition (e.g., Desoete & Roeyers, 2002) the lower evaluations of children with mathematics learning disabilities in study support the idea that feedback acquired during mathematical problem solving helps children to make more accurate, and less overcondent judgments of math capability. The implication that arises is that teachers perhaps should stimulate children to make postdictions in order to make them better predictors of math-related outcomes. Nonetheless, research also revealed that the type of feedback given to children is not unimportant. Experimenter-provided outcome feedback was not found to inuence the capability of performance calibration (Lin & Zabrucky, 1998; Schraw, Potenza, & Nebelsick-Gullet, 1993). These results should be interpreted with care, since there are several limitations to the present study. First, the children in this study revealed to have substantially different mathematical problem-solving skills. This may bias the results because the poorer a students performance, the more likely it is that he or she will be overcondent. Nevertheless, we tried to deal with this potential bias by matching the children with mathematics learning disabilities with younger children and by matching expert problem solvers with older children. However, additional research seems indicated with larger groups of performance-matched students. Second, the results of this study should be interpreted with care since macroevaluation skills may be age-dependent and still maturing. Third, the number of other possible causes of low mathematical functioning (non-native children with eventual language problems, hyperactivity, sensory impairment, brain damage, a chronic medical condition, insufcient instruction, serious emotional or behavioral disturbance) was restricted to a minimum in this study. These restrictions, causing a limitation in the random sampling, have to be noted as limitations of this research. In addition, several cognitive, motivational and socialaffective processes seem involved in posttest performance calibration. Additional research should focus on these factors. To conclude, the assessment of macroevaluations is less time-consuming than a combined assessment at the taskspecic level. However, since a lot of children with mathematics learning disabilities differ from peers without learning disabilities on microevaluation (Desoete & Roeyers, 2002), but not on macroevaluation, it seems wise to keep measuring with combined task-specic or concurrent techniques.

Acknowledgments Several studies were supported by the Artevelde College Ghent and SIG Destelbergen, to who the authors express their thanks.

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

23

Appendix A. Teacher questionnaire

What typifies this child during the last 3 months compared with peers? How often does this behavior occur?
1 = never 2 = not often 3 = sometimes 4 = often 5 = always

Note the corresponding number in Reflecting in advance on how difficult this exercise will be P1 Controlling the work afterwards E1 Planning and working according the plan Pl1 Working slower and more precise on difficult exercises
Mo1

Knowing in advance what the difficult and easy exercises P2 Changing the plan, if the situation asks for it
Mo2

Able in advance to tell how one will work on a task Pl2 Finding mistakes in a last control and correcting these mistakes E2 How would you situate this child compared with peers? 1 = very low - 7 = very good

Note the corresponding number in Intelligence Mathematics Reading Social skills Other remarks :

24

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

References
Artelt, C. (2000). Wie pradiktiv sind retrospektive Selbstberichte uber den Gebrauch von Lernstrategien fur strategisches Lernen? Zeitschrift fur Padagogische Psychologie, 14, 72e84. Artzt, A. F., & Armour-Thomas, E. (1992). Development of a cognitiveemetacognitive framework for protocol analysis of mathematical problem solving in small groups. Cognition and Instruction, 9, 137e175. Baker, L. (1994). Fostering metacognitive development. In H. W. Reese (Ed.), Advances in child development and behaviour, Vol. 25 (pp. 201e239). San Diego: Academic Press. Borkowski, J. G., Chan, L. K. S., & Muthukrishna, N. (2000). A process-oriented model of metacognition: links between motivation and executive functioning. In G. Schraw, & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 1e41). Lincoln, NE: Buros Institute of Mental Measurements. Brown, A. (1987). Metacognition, executive control, self-regulation and other more mysterious mechanisms. In F. E. Weinert, & R. Kluwe (Eds.), Metacognition, motivation and understanding (pp. 65e116). Hillsdale, NJ: Erlbaum. Busato, V. V., Prins, F. J., Elshout, J. J., & Hamaker, C. (1998). Learning styles: a cross-sectional and longitudinal study in higher education. British Journal of Educational Psychology, 68, 427e441. Casey, M. B., Nuttall, R. L., & Pezaris, E. (2001). Spatialemechanical reasoning skills versus mathematical self-condence as mediators of gender differences on mathematics subtests using cross-national gender-based items. Journal for Research in Mathematics Education, 32, 28e57. Cheung, C. K., & Rudowicz, E. (2003). Academic outcomes of ability grouping among junior high school students in Hong Kong. Journal of Educational Research, 96, 241e254. Cracco, J., Baudonck, M., Debusschere, M., Dewulf, B., Samyn, F., & Vercaemst, V. (1995). KRT: Kortrijkse Rekentest. [Kortrijk Arithmetics Test]. Kortrijk: Revalidatiecentrum Overleie. De Clercq, A., Desoete, A., & Roeyers, H. (2000). EPA2000: a multilingual, programmable computer assessment of off-line metacognition in children with mathematical-learning disabilities. Behavior Research Methods, Instruments & Computers, 32, 304e311. Dermitzaki, I., & Efklides, A. (2003). Goal orientations and their effect on self-concept and metacognition in adolescence. In E. Besevegis, G. Georgouleas, & V. Pavlopoulos (Eds.). Psychology: The Journal of the Hellenic Psychological Society, 10, 214e227 (special issue: Issues on Psychology of Adolescence). Desoete, A. Are mathematical disabilities a special kind of metacognitive disabilities? In A. Desoete & M. Veenman (Eds.), Metacognition in mathematics education. Hauppage, NY: Nova Science, in press. Desoete, A., & Roeyers, H. (2001). Cognitive Deelhandelingen van het Rekenen, CDR [Cognitive Diagnostic of aRithmetics]. Unpublished manuscript. Ghent: RUG. Desoete, A., & Roeyers, H. (2002). Off-line metacognition. A domain-specic retardation in young children with learning disabilities? Learning Disability Quarterly, 25, 123e139. Desoete, A., Roeyers, H., & Buysse, A. (2001). Metacognition and mathematical problem solving in grade 3. Journal of Learning Disabilities, 34, 435e449. Desoete, A., Roeyers, H., & De Clercq, A. (2004). Children with mathematics learning-disabilities in Belgium. Journal of Learning Disabilities, 37, 50e61. De Vos, T. (1992). TTR. Tempotest rekenen. [Arithmetic number fact test]. Lisse: Swets & Zeitlinger. Efklides, A. (2001). Metacognitive experiences in problem solving: metacognition, motivation, and self-regulation. In A. Efklides, J. Kuhl & R. M. Sorrentino (Eds.), Trends and prospects in motivation research (pp. 297e323). Dordrecht, the Netherlands: Kluwer. Eisenberg, N., Martin, C. L., & Fabes, R. A. (1996). Gender development and gender effects. In D. C. Berliner, & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 358e396). New York: Simon & Schuster Macmillan. Elshout-Mohr, M., Meijer, J., van Daalen-Kapteijns, M., & Meeus, W. (2003). A self-report inventory for metacognition related to academic tasks. Paper presented at the 10th Conference of the European Association for Research on Learning and Instruction (EARLI) Padova, Italy, 26e30 August 2003. Evans, E. M., Schweingruber, H., & Stevenson, H. W. (2002). Gender differences in interest and knowledge acquisition: the United States, Taiwan, and Japan. Sex Roles, 47, 153e167. Flavell, J. H. (1987). Speculations about the nature and development of metacognition. In F. E. Weinert, & R. Kluwe (Eds.), Metacognition, motivation and understanding (pp. 20e29). Hillsdale, NJ: Erlbaum. Freeman, J. (2003). Gender differences in gifted achievement in Britain and the US. Gifted Child Quarterly, 47, 202e211. Frost, L. A., Hyde, J. S., & Fennema, E. (1994). Gender, mathematics performance, and mathematics-related attitudes and affect: a meta-analytic synthesis. International Journal of Educational Research, 21, 373e385. Geary, D. (2003). Learning disabilities in arithmetic: problem-solving differences and cognitive decits. In L. Swanson, K. R. Harris, & S. Graham (Eds.), Handbook of learning disabilities (pp. 199e212). New York: The Guilford Press. ` Ghesquiere, P., & Ruijssenaars, A. (1994). Vlaamse normen voor studietoetsen Rekenen en technisch lezen lager onderwijs. [Flemish standards for study evaluation of mathematics and technical reading in primary school]. Leuven: K.U.L.-C.S.B.O. Glenberg, A. M., Sanocki, T., Epstein, W., & Morris, C. (1987). Enhancing calibration of comprehension. Journal of Experimental Psychology: General, 116(2), 119e136. Hunter-Blanks, P., Ghatala, E. S., Pressley, M., & Levin, J. R. (1988). Comparison of monitoring during study and during testing on a sentencelearning task. Journal of Educational Psychology, 80, 279e283.

A. Desoete, H. Roeyers / Learning and Instruction 16 (2006) 12e25

25

Jonsson, A. C., & Allwood, C. M. (2003). Stability and variability in the realism of condence judgments over time, content domain and gender. Personality and Individual Differences, 34, 559e574. Kling, K. C., Hyde, J. S., Showers, C. J., & Buswell, B. N. (1999). Gender differences in self-esteem: a meta-analysis. Psychological Bulletin, 125, 470e500. Kramarski, B., Mevarech, Z. R., & Arami, M. (2000). The effects of metacognitive instruction on solving mathematical authentic tasks. Educational Studies in Mathematics, 49, 225e250. Lin, L. M., & Zabrucky, K. M. (1998). Calibration of comprehension: research and implications for education and instruction. Contemporary Educational Psychology, 23, 345e391. Lucangeli, D., Cornoldi, C., & Tellarini, M. (1998). Metacognition and learning disabilities in mathematics. In T. E. Scruggs, & M. A. Mastropieri (Eds.), Advances in learning and behavioural disabilities (pp. 219e244). Greenwich: JAI Press Inc. Masui, C., & De Corte, E. (1999). Enhancing learning and problem solving skills: orienting and self-judging, two powerful and trainable learning tools. Learning and Instruction, 9, 517e542. Nietfeld, J. L., & Schraw, G. (2002). The effect of knowledge and strategy training on monitoring accuracy. Journal of Educational Research, 95, 131e142. Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we know: verbal reports on mental processes. Psychological Review, 84, 231e259. Pajares, F., & Graham, L. (1999). Self-efcacy, motivation constructs, and mathematics. Performance of entering middle school students. Contemporary Educational Psychology, 24, 124e139. Peetsma, T., Roeleveld, J., & Stoel, R. (2003). Relations between pupils investment in school, self-condence, well-being and school achievement at different ages in primary education. Paper at the European Association for Research on Learning and Instruction, Padova, Italy, 26e30 August 2003. Pintrich, P. R., & De Groot, E. V. (1990). Motivational and self-regulated leaning components of classroom academic performance. Journal of Educational Psychology, 82, 33e40. Pugalee, D. K. (2001). Writing, mathematics, and metacognition: looking for connections through students work in mathematical problem solving. School Science and Mathematics, 101, 236e245. Pressley, M. (2000). Development of grounded theories of complex cognitive processing: exhaustive within- and between study analyses of thinking-aloud data. In G. Schraw, & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 262e296). Lincoln, NE: Buros Institute of Mental Measurements. Royer, J. M., Tronsky, L. N., Chan, Y., Jackson, S. J., & Marchant III, H. (1999). Math-fact retrieval as the cognitive mechanism underlying gender differences in math test performance. Contemporary Educational Psychology, 24, 181e266. Schwartz, B., & Metcalfe, J. (1992). Cue familiarity but not target retrievability enhances feeling-of-knowing judgments. Journal of Experimental Psychology: Learning, Memory and Cognition, 18, 1074e1083. Schraw, G., Potenza, M. T., & Nebelsick-Gullet, L. (1993). Constraints on the calibration of performance. Contemporary Educational Psychology, 18, 455e463. Schwarz, N. (1999). Self-reports. How questions shape the answers. American Psychologist, 54, 93e105. Shute, V. J. (1996). Learning processes and learning outcomes. In E. De Corte, & F. E. Weinert (Eds.), International encyclopedia of developmental and instructional psychology (pp. 409e418). Oxford: Elsevier Science Ltd. Sperling, R. A., Howard, B. C., Miller, L. A., & Murphy, C. (2002). Measures of childrens knowledge and regulation of cognition. Contemporary Educational Psychology, 27, 51e79. Tobias, S., & Everson, H. (2000). Assessing metacognitive knowledge monitoring. In G. Schraw, & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 147e222). Lincoln, NE: Buros Institute of Mental Measurements. Thorpe, K. J., & Satterly, D. J. H. (1990). The development and inter-relationship of metacognitive components among primary school children. Educational Psychology, 10, 5e21. Van Hout-Wolters, B. H. A. M. (2000). Assessing active self-directed learning. In R. J. Simons, J. Van der Linden, & T. Duffy (Eds.), New learning (pp. 55e83). Dordrecht/Boston/London: Kluwer. Veenman, M. V. J. (2003). The assessment of metacognitive skills: what can be learned from multi-method designs? Paper presented at the 10th Conference of the European Association for Research on Learning and Instruction (EARLI) Padova, Italy, 26e30 August 2003. Zimmerman, B. J., & Martinez-Pons, M. (1990). Student differences in self-regulated learning: relating grade, sex, and giftedness to self-efcacy and strategy use. Journal of Educational Psychology, 82, 51e59.