ICT and Academic Achievement Evaluating the effect of an ICT educational activity on student success

A three-step approach
Introduction
In his IntersTICES model, Jacques Viens proposes that anyone responsible for the integration of an ICT educational project must ensure that there is consistency between: target objectives actions implemented resources used evaluation of results

The three-step process presented here facilitates establishing this consistency: Step 1: Objectives, activities and resources Specify the objectives that will allow the goal (improving academic achievement) to be attained Specify the activities and resources put in place in a coherent manner with a view to attaining these objectives

Step 2: Formulating questions and gathering data Formulate questions that evaluate conditions and effects Develop response scales Gather data from surveys, interviews, on-site observations and productions analysis Compile results and reports in the response scales

Step 3: Observations, recommendations and dissemination Recast response scale results as statistics Formulate observations and recommendations Disseminate results and challenges
1

Step 1: Objectives, activities and resources

In order to evaluate the effect of an ICT educational activity on academic achievement, we first need to highlight the initial objectives of this use of ICT. If we posit that academic achievement is the goal of an experimental ICT project, it approaches what the Carrefour de la russite calls a support measure :
A support measure is a set of coordinated approaches that aim for a result or objective related to achievement; moreover, as with all measures, it is an approach specifically designed to solve a question or problem. Because of this definition, based on the concept of measures, what distinguishes a "support measure" is its structured nature, organized explicitly, according to results to be achieved in relation to an actual or potential obstacle to academic success. (p.12 - translation)

To assess the effect of the educational use of ICT on student achievement, we must formulate, structure, and clearly organize the objectives to be met at the time of its implementation. Objectives are specific actions put in place to achieve a goal. The cumulative and combined effects of the objectives allow us to achieve this goal. ARCs meta-synthesis of the results of experimental teaching strategies that use ICT to improve student achievement revealed that research conducted in the college system has, to date, focused on objectives related to: 1. Student grades 2. Academic motivation 3. Complex cognitive processes These three dimensions can be used as a starting point for identifying the objectives of an action involving ICT. The implementation of structured and organized actions involves strategic choices within the organizational constraints. Making the right choices is a major challenge in achieving the objectives. The analysis grid for an ICT learning activity, resulting from ARCs meta-synthesis and the work of the community of IT Representatives, helps guide and evaluate the choices made in the context of a specific project. Its use, preferably made at the time of project design, helps ensure consistency between the objectives, activities and resources, as recommended by Jacques Viens. The work highlighting the consistency between the original objectives, activities and resources involves the initiators of the original project as well as others who will be responsible, within a workgroup, to complete all evaluation operations. The expected product is a synthetic representation in the form of drawings or diagrams, design of actions (objectives), the conditions of their implementation and the expected effects on student achievement, academic motivation, complex cognitive operations, or other dimensions of student success. An explanatory text or presentation completes the summary view.
Top of document

Step 2: Formulating questions and gathering data

Once the original objectives and the conditions of their implementation have been identified, the workgroup may formulate questions related to their achievement. Clear signs of the expected effects are the indicators of goal achievement. The questions therefore serve to guide the observation of observable effects. Obviously, the effects of implemented actions can occur to varying degrees (more or less often, in a greater or lower number people, to a greater or lesser extent), so that the objectives are in effect more or less achieved. Designing scales of possible answers to the questions serves to translate the range of observations. The purpose of data collection is to answer questions and to "measure" the achievement of initial objectives based on the response scales. There are several ways to collect data; the conditions of each experiment may dictate which method is best. Among the most common methods, we can consider the following: Survey questionnaires for students, teachers or other important players Group interviews of teachers, students and, eventually, advisors and IT technicians Comments on-site, that is to say most often in the classroom and laboratory, which focus on tasks, the use of space, the definition of roles and interactions Productions by students that serve to analyze their characteristics in terms of the objectives.

The report of data collected in the response scales is a crucial step because it often requires that a judgment be made. This task must be performed by more than one person, under the principle of triangulation in data coding.
Top of document

Step 3: Observations, recommendations and dissemination

Response scales easily lend themselves to simple statistical processing (mean, standard deviation, mode). It draws out general information relevant to the questions about the achievement of initial objectives. Has the use of ICT had the desired impact? On all targets? Which target was best achieved, and which least? If they lead to clear conclusions, these initial observations can result in recommendations to improve the next experiments in educational integration of ICT. If the amount of data permits, cause and effect can be explored by statistical tests: is there a significant difference between the observed effects of this or that group, or one condition and another? Some observations may not be conclusive, lending themselves to various interpretations. The resulting recommendations may invite further investigation of possible links between certain conditions and results. The compilation of collected data, the observations made of them and the resulting recommendations are of great value for those involved in the experiment being evaluated: teachers, techno-pedagogical counsellors, technical teams and service management. Therefore it is important to share this information through a variety of media, both conventional (reports, articles) and innovative (blogs, video conferencing). The dissemination of experiment results in educational ICT integration are of interest not only to primary actors, but also to members of professional communities working in the fields of techno-pedagogy, education, training and, academic achievement.
Top of document