Running Head: CONJUGATOR EFFECTIVENESS

Final Project: Prototype Effectiveness Test Tatiana Pitstick & Setsuko Phegley California State University Monterey Bay

IST622 Assessment and Evaluation Dr. Bude Su July 30, 2013

CONJUGATOR EFFECTIVENESS Table of Contents Introduction ........................................................................................................................ 3 Methodology ....................................................................................................................... 3 Prototype ............................................................................................................................. 3 Audience................................................................................................................................ 4 Methods and Instruments ................................................................................................. 4 Results ................................................................................................................................. 5 Outcomes ............................................................................................................................ 5 Statistical Evidence ............................................................................................................ 8 Recommendations .............................................................................................................. 9 Summary ............................................................................................................................. 11 Appendix A: Participant Worksheet ................................................................................ 12 Appendix B: Data Results ................................................................................................. 14

CONJUGATOR EFFECTIVENESS Introduction The iTrain Team chose to evaluate the effectiveness of The ConjuGator, an ILM designed for learning Spanish grammar and improving student confidence levels. The ConjuGator teaches and tests the basic components of conjugating. The participants were

children aged ten to twelve enrolled in an elementary Spanish course for Middle School students. Students completed self-evaluations to indicate their degree of confidence and understanding in Spanish before and after using the ILM. The final data compiled indicates that there were overall improvements in confidence levels, suggesting a positive correlation with the ILM. Methodology Prototype. With the ConjuGator ILM, students learn basic grammar elements for conjugating Spanish verbs. Students who have basic keyboarding and mouse navigation skills can complete the training module within an estimated time of fifteen minutes while accessing the program online (or with the CD-ROM version). Users enter their names on the first slide. Their names then complete variables that personalize various components throughout the learning experience. In the first three sections (Review Launch Pad items), users study conjugating fundamentals with a progressive scaffolding of concepts. Users can access resources that are available throughout training for easy reference to these concepts and vocabulary, even during assessments. The ConjuGator introduces the basic elements of conjugation and subsequent elements build upon the previous lessons. As users advance through the ILM and complete the interactive quizzes, they can also evaluate their acquisition of linguistic concepts with the automatic feedback provided. In the culminating fourth section (The ConjuGator), students apply what they have learned to unfamiliar verb forms and vocabulary. By completing the ILM, students

CONJUGATOR EFFECTIVENESS demonstrate their ability to conjugate regular AR verbs in Spanish and thus gain linguistic confidence at the same time. Audience. The ConjuGator ILM targets a younger population, as illustrated by the brighter colors and cartoon characters. Ideal age demographics would range from eight to fourteen (upper elementary grades, from Fourth through Eighth Grades). Students do not need prior studies in Spanish: Students read the lessons in English and complete quizzes with questions in English as well (they answer, however, in Spanish). As foreign language study is a new field for young learners, the anxiety to get the right answer often inhibits production, a crucial factor in acquisition. The design of the ILM, however, is clear so that students can easily navigate through sections and advance to more complex grammatical concepts. The program simplifies grammatical concepts, breaking them

down into manageable parts. If users get confused, they can readily access resources throughout the lessons. If the audience deems completing the exercises on the ILM too easy, than the ConjuGator has succeeded in clarifying Spanish grammar sufficiently so as to boost student morale. Methods and Instruments. The researchers employed both direct observation and survey techniques to obtain the data compiled for this effectiveness analysis. The participants for this study were fourteen Middle School students aged ten to twelve possessing native Englishspeaking skills and living in central California. Prior to using the ILM, all students rated their abilities on an evaluation sheet; they scored their own levels of knowledge and confidence in Spanish acquisition. Scores ranged from 1 (high levels) to 4 (low levels). Following ILM usage, the students completed a Post-Test evaluation to evaluate again their perspectives about their abilities, employing the same score ranges.

CONJUGATOR EFFECTIVENESS To diminish test anxiety, the evaluator reassured the participants that results from the ConjuGator and from their personal evaluations would not affect their course grade (i.e., in

Elementary Spanish). The evaluation sheet also included categories to critique the ILM, thereby further distracting students from focusing too much on themselves. Using an evaluatory scale with which the students were familiar for their own report cards, the participants chose the boxes that corresponded to their personal perspectives regarding themselves and the module. The students were able to complete the prototype usage and the evaluation worksheet within one class period (fifty minutes). They also provided comments about functionality, design, and effectiveness (see the Participant Worksheet in the Appendix A). The evaluator observed the whole process from preliminary survey (Pre-Test) to culminating survey (PostTest), between which the students utilized the ConjuGator ILM. Students could not progress to other ILM sections until they had achieved 100% on each concept segment. The evaluator was present to answer questions and to take notes regarding usability. The survey portions pertaining to linguistic abilities were paramount for assessing the effectiveness of the ILM. Results Outcomes. The ConjuGator ILM included assessment items to indicate mastery of grammatical concepts (inherently the cognitive objectives), whereas the evaluation worksheets provided information regarding how well the students had improved in affective goals. The content included the following linguistic objectives the students needed fulfill: a. Translate into English a given list of Spanish pronouns, attaining 100% accuracy. b. Select the infinitive form of verbs when given a list of various verb forms, attaining 100% accuracy.

CONJUGATOR EFFECTIVENESS c. Categorize according to type (AR, ER, or IR) a list of verbs given in their infinitive forms, attaining 100% accuracy. d. Identify AR verbs from a list of various type verbs with accessible resources, attaining 100% accuracy.

e. Translate the infinitive form of a verb into English with accessible resources, attaining 100% accuracy. f. Construct the root of a Spanish AR verb when given a list of verbs with accessible resources, attaining 100% accuracy. g. Translate between Spanish and English verbs conjugated into the simple present active tense with accessible resources, attaining 100% accuracy. The student sample demonstrated mastery of the cognitive objectives (above) by obtaining 100% on every quiz. After obtaining favorable scores, students would (ideally) indicate a higher level of confidence regarding their skills after completing the ILM. Essentially, the real goal for this ILM was to boost student morale and confidence levels for studying Spanish. If students could recognize the simplicity of conjugating regular Spanish AR verbs into the present active tense after completing the module, then they could also have greater confidence to study more complex grammatical structures. Ultimately, the ConjuGator produced such results, perhaps because it had sufficiently simplified the process conjugating verbs, thereby directly contributing to greater self-confidence upon completion. These Levels of Confidence areas included the following items (see Picture #2 in the Appendix): 1.) Personal confidence in Spanish; 2.) Personal confidence in conjugating verbs;

CONJUGATOR EFFECTIVENESS 3.) Personal ability to translate pronouns; 4.) Personal ability to identify infinitives; 5.) Personal ability to identify AR verbs; 6.) Personal ability to construct verb root. Learners assessed these areas by the feedback they provided on the worksheets as well as verbal comments of approval (such as, This is so easy!). Students provided these personal assessments prior to using the ILM and then after using it. Students responded well to the ConjuGator ILM. They made positive comments about the design and about the simplicity of the content. All students eventually gained 100% on the ILMs assessment items, as they could review and re-take the quizzes until they obtained 100%. Learners were especially motivated after using the ConjuGator ILM and requested that future lessons include such learning tools. The student self-evaluations provided interesting results. The average student assessments indicated increased levels of confidence in all affective objectives. Students expressed comments of enjoyment (This is fun) and satisfaction (The colors were great and

the alligator was cool). Some comments were negative (The alligators eyebrows were scary) or critical (You should use pink), but in general the students responded positively to the ConjuGator. The data compiled clearly supports the effectiveness of the ConjuGator as a tool for increasing confidence levels for Spanish acquisition. Calculating the difference between the Pre-Test and Post-Test scores (see Pictures #1 and #2 in the Appendix B) verifies an increase in all categories pertaining to confidence levels. These calculations show that on average the students expressed greater levels of confidence after using the ConjuGator ILM. Comparing

CONJUGATOR EFFECTIVENESS

those differences visually (see Picture #3 in Appendix B) also illustrated that the Pre-Test scores were further away from the ideal mark (of a 1) than the Post-Test scores and that the differences between the two evaluations were always a positive number (a negative number would have appeared below the horizontal X-axis and would have indicated the Pre-Test score was better than the Post-Test score). Statistical Evidence. Statistical analysis provided support in evaluating the effectiveness of the results gleaned from the evaluations. First of all, the Null Hypothesis suggested that there would be no difference of or diminished levels in confidence following ILM usage (the Pre-Test scores should be less than or equal to the Post-Test scores). Secondly, the Alternative Hypothesis expected improved levels of confidence after using the ILM (the Pre-Test scores should be higher than the Post-Test scores). Thus, H0: Pre < = Post H1: Pre > Post The team then calculated the t-Test for Paired Two Sample (with Mean, t Stat, P-value, and Critical Value). The calculations showed that the Means of the Pre-Test (2.07) was higher than the Post-Test (1.56); the higher score actually meant lower levels of confidence. The calculations, therefore required the team to reject the Null Hypothesis finally because: a) The Post-Test Mean was clearly lower than the Pre-Test; b) The t-Stat (3.91) was greater than the critical value (2.02); and c) The P-value (0.0056) was less than the 0.05. The conclusions from statistical analysis suggest that the ConjuGator contributed somehow to increased levels of confidence in Spanish skills.

CONJUGATOR EFFECTIVENESS

Recommendations. After compiling the data obtained from the Pre- and Post-Test Surveys and the comments from the initial prototype session, the iTrain Team chose to report some concerns regarding the results. First of all, the purpose of an ILM is to provide instruction without interference or with preferably no guidance from the instructor. The students completed the ConjuGator program with minimal interaction with the evaluator during the ILM; the evaluator would stop to question students occasionally on their progress and to take notes if there were issues. Any conversations that took place clarified certain glitches that may need adjustment in the program design. Adjustments to the design include the following suggestions: 1) Limit the opening text entry box to less characters (perhaps twenty) as one student spent too much time typing a lengthy pseudonym and another student typed a sentence (Me llamo X). 2) Clarify in the opening instructions that the ENTER key does not function so as to advance to the next screen (or create a trigger function to allow for that factor). 3) Improve the audio quality (and possibly add music that does not distract from the learning process). 4) Clarify instruction and/or add variants to the text entry box into which the students had to spell a conjugated verb (three students included the pronoun with the correct verb and yet received INCORRECT feedback; but the instructions did not clarify that pronouns werent necessary). 5) Clarify the instructions for the Drag & Drop quizzes (two students couldnt figure out how to manipulate the objects).

CONJUGATOR EFFECTIVENESS 6) Provide more interactive games (three students suggested the program needed to be more gamey).

10

7) Limit the verb list in the open-ended conjugating section. The final section of the ILM (the actual ConjuGator game) provided an opportunity for students to conjugate new verbs. Even though they could access a glossary throughout the ILM, one student invented verbs (that is, she created a verb that doesnt exist in Spanish). While creativity is important in language learning, the student got side-tracked by the endless possibilities of this component of the ILM. Perhaps listing only five verbs and then having the students conjugate one would thereby limit answers. 9) Limit the opportunities to re-take the quizzes (so as to diminish guessing and the process of elimination to determine a correct answer). Secondly, there are concerns regarding the actual analysis in question. For example, the students participated in a classroom activity, so how much did their concern for consequences factor into their voting choices? Perhaps some students marked categories more favorably so as to gain teacher approval or to avoid negative repercussions from the instructor. Likewise, some students scored themselves with fairly similar results prior and following the ILM (especially Students B1, B3, and B13, as shown in Picture #1 in the Appendix B), indicating little if not zero improvement. How much can we attribute those scores to pre-existing judgments or poor interest? Obviously, the evaluators have no intention of psycho-analyzing the subjects, but an analyst has to consider personal factors of the subjects when administering surveys. As for the analysis itself, the iTrain Team offered the following concerns. 1) Although the grading scale corresponded to a system with which the subjects were already familiar, simplifying the marks (so that lower numbers would indicate poorer ratings) would

CONJUGATOR EFFECTIVENESS facilitate future analysis (at times it was difficult to maintain the perspective that a higher number was not desirable).

11

2) The actual grading scale should include an option for Not Applicable (four students did not access the additional resources, so their rating these resources would be superfluous). 3) Reinforcing the importance of sincerely scoring the categories may have possibly improved the validity of answers. The evaluators speculated that some students may have checked off boxes randomly or with minimal reflection. 4) Providing separate sheets for the Pre-Test and the Post-Test would minimize copying or seeking to improve previous answers, thereby skewing results. The administrator noticed, however, that the students focused on the Post-Test section of the questionnaire, possibly having forgotten from the top section (the Pre-Test) after the fifteen minute ILM lesson. Summary Statistical analysis compiled from the participant worksheets suggest that the instructional value of the ConjuGator may be potentially effective for improving student morale as well as Spanish language skills. All participants were able to demonstrate understanding of grammar components (by attaining 100% on the quizzes). This effectiveness study postulates that the ILM simplified these components sufficiently for acquisition, thereby improving personal beliefs about Spanish skills. Receiving the high scores after each lesson reinforced confidence in linguistic abilities. Besides some minor adjustments that could improve the basic functions of the program (primarily to minimize distractions), the ConjuGator is an entertaining tool for Middle School children to gain basic Spanish verb skills and the critical confidence necessary for maintaining motivational levels in their studies.

CONJUGATOR EFFECTIVENESS Appendix A: Participant Worksheet

12

CONJUGATOR EFFECTIVENESS

13

ILM: The ConjuGator by Doa Tatiana

Name: Circle the date: Circle your group:

1= Great!

June 3, 2013 5
2= Okay

6
3= Needs Work

June 4, 2013 7
4= Not Acceptable

Write an X in the box below the column to indicate your perspective of the categories. Pre-Test Survey: 1. My confidence in Spanish 2. My confidence in conjugating verbs 3. My ability to translate pronouns 4. My ability to identify infinitives 5. My ability to identify AR verbs 6. My ability to construct verb root stem 7. My retention of verb vocabulary Post-Test Survey: 7. My ability to navigate through the program 8. My ability to understand quiz instructions 9. ILMS launch menus were easily located 10. ILMS quizzes & text entry work 11. ILMs Gator-Aides helpful as resources 12. ILM display/layout 13. My equipment worked with this ILM 14. ILMs audio quality 15. ILMs color scheme 16. ILMs level of difficulty/challenge 17. ILMs duration 18. ILM reinforcing what Im learning in class 19. ILM teaching me something new 20. My confidence in Spanish 21. My confidence in conjugating verbs 22. My ability to translate pronouns 23. My ability to identify infinitives 24. My ability to identify AR verbs 25. My ability to construct verb root 26. My retention of verb vocabulary

On the back of this worksheet, please write any extra comments pertaining to the ILM.

CONJUGATOR EFFECTIVENESS Appendix B: Data Results

14

CONJUGATOR EFFECTIVENESS Picture #1: Data of Student Responses to Questions Pertaining to Confidence on Pre- & PostTest Surveys

15

NOTE: This data analysis only included the questions pertaining to the student assessments of confidence levels in Spanish skills. The colors from the Pre-Test correspond to the same colored question item in the Post-Test. Therefore, the Red Pre-Test Question #1 (Rating Confidence in Spanish) was the same as the Red Post-Test Question #20 (Rating Confidence in Spanish), the Orange to the Orange, etc. The number in white at the bottom of each column is the average (mean) for that item. These figures show that on average the students expressed greater levels of confidence in all confidence categories after using the ConjuGator ILM.

CONJUGATOR EFFECTIVENESS Picture #2: Pre-Test and Post-Test Question Items Included for Effectiveness Analysis

16

NOTE: The picture above includes 1) Itemized questions that are color-coded; 2) The scaling key used for the responding to the question items; 3) The mean results for the Pre-Test and Post-Test scores; 4) The differences between the Pre-Test and Post-Test scores (these differences indicate that the class in general expressed an increased level in all categories pertaining to confidence).

CONJUGATOR EFFECTIVENESS Picture #3: Graph with T-Test Results

17

NOTE: This picture includes 1) The t-Test for Paired Two Sample (with Mean, t Stat, P-value, and Critical Value annotated); 2) Null Hypothesis (the Pre-Test should be less than or equal to the Post-Test scores) expressing that there would be no difference of or diminished levels in confidence following ILM usage; 3) Alternative Hypothesis (the Pre-Test should be greater than the Post-Test scores) expecting improved levels of confidence after using the ILM; 4) A graph clearly illustrating that the Post-Test scores were lower than the Pre-Test scores and that the differences were always a positive number (a negative number would have indicated a Post-Test score had exceeded the Pre-Test score); 5) The conclusions that support rejecting the Null Hypothesis because a) The Post-Test Mean is clearly lower than the Pre-Test; b) The t-Stat (3.91) is greater than the critical value (2.02); and c) the P-value (0.0056) is less than the 0.05.