Running head: EVALUATION REPORT 1

Evaluation Report of Pre-Immersion Language and Culture Training Module

Cong Li

California State University, Monterey Bay

July 23, 2019

IST622 Assessment & Evaluation

Dr. Bude Su
EVALUATION REPORT 2

Table of Contents

Table of Contents ................................................................................................................ 2

Section I. Introduction ........................................................................................................ 3
Section II. Methodology ..................................................................................................... 3
Prototype ......................................................................................................................... 3
Learners .......................................................................................................................... 4
Tryout Conditions ........................................................................................................... 5
Tryout Process ................................................................................................................ 5
Pre-Test ....................................................................................................................... 5
Training ....................................................................................................................... 6
Observation ................................................................................................................. 6
Post-Test ..................................................................................................................... 6
Usability Questionnaire .............................................................................................. 7
Section III. Results .............................................................................................................. 7
Entry Conditions ............................................................................................................. 7
Instruction ....................................................................................................................... 7
Outcomes ........................................................................................................................ 8
Intended Outcomes ..................................................................................................... 8
Evaluation of Reaction ................................................................................................ 8
Evaluation of Learning ............................................................................................... 9
Evaluation of Usability ............................................................................................. 10
Recommendations ......................................................................................................... 12
Instruction Page ........................................................................................................ 12
Browser Requirement ............................................................................................... 12
Subject Matter Expert Involvement .......................................................................... 12
Instructor Feedback ................................................................................................... 13
Section IV. Summary ........................................................................................................ 13
Section V. Appendices ...................................................................................................... 14
Appendix A. Training Module...................................................................................... 14
Appendix B. Pre-test and Post-test Questions .............................................................. 14
Appendix C. Observation Checklist ............................................................................. 16
Appendix D. Usability Questionnaire ........................................................................... 16
EVALUATION REPORT 3

Section I. Introduction

The Pre-Immersion Culture and Language Training consists of four stand-alone learning

modules. The purpose of the training is to prepare Chinese learners linguistically and culturally

before they go to Taiwan for language immersion. The learning module being evaluated is titled

Historical Figures and Their Influences on Taiwan. It was developed using Adobe Captivate

software with an estimated lesson duration between 10 to 15 minutes. Two well-known historical

figures are introduced in the learning module and learners will be informed about their

contributions and the places of interest in Taiwan that are associated with them. After the

module, learners will have solid cultural background knowledge when they go to Taiwan. Five

learners participated in the evaluation of this learning module. Pre-test and post-test were

administrated, and the test scores were analyzed to evaluate the learning effectiveness. A

usability questionnaire was also distributed and collected at the end of each tryout session. The

data from the questionnaire provided useful insights for improving the design and usability of

this eLearning module.

Section II. Methodology

Prototype

This prototype is a stand-alone Chinese language and culture training module. By

studying this lesson, learners will acquire the background knowledge on two important historical

figures in Taiwan (Sun Yat-sen and Shang Kai-shek). They are likely to associate this knowledge

with their immersion experience in Taiwan.

In the first slide, basic navigation buttons are explained in English. It also includes the

instructions for users to skip certain content or use supporting materials to better understand the

narration. It gives the users more control of their own learning.

EVALUATION REPORT 4

The content includes five major slides narrated in Chinese. Each slide focuses on one

subtopic. The five subtopics are Timeline, Sun Yat-sen, Sun Yat-sen Memorial, Shang Kai-shek,

and Shang Kai-shek Residence. Incorporating elements of effective online training design,

content slides use animation, graphics, on-screen texts, and audios to keep learners engaged.

There are also two practice sections. The instruction is given in English, but the statements are in

Chinese. The first section is designed as True/False questions after the timeline slide. There are a

total of 3 questions. If the learner chooses the wrong answer, a text feedback in English will

show up. The second practice section is at the end of the learning module. It is a matching

question with two items on the left and six items on the right all in Chinese characters. Learners

have two tries and whether they get it right the first time or get it wrong twice, a narrated

feedback in Chinese is provided at the end.

The last slide of the learning module gives learners the instruction in English to close the

learning module if they are ready for the quiz. If they are not ready, they can review the whole

lesson or parts of it.

Learners

The learners for this prototype are Chinese students at the Defense Language Institute

(DLI). They are attending Chinese classes six hours a day for 64 weeks. Around week 36, they

will go to Taiwan for language immersion. At that time, their Chinese language proficiency level

ranges from limited working proficiency to professional working proficiency, according to

Interagency Language Roundtable scales. From the first 36 weeks of classroom instruction,

students learn basic Chinese vocabulary and grammar structure. They are also exposed to

Chinese culture, mainly that of mainland China. The prerequisite for this prototype is mastery of

basic Chinese vocabulary and grammar. Most students are excited to take this training because
EVALUATION REPORT 5

the content will be closely related to what they will experience in Taiwan during their language

immersion. They will also learn the background knowledge about the unique culture and

language of Taiwan. The module will be beneficial to a meaningful and effective immersion

experience.

Tryout Conditions

The tryout sessions were carried out in an available classroom after students finished

their regular classes. Since most students had left by then, the classroom was quiet without much

noise. There was only one learner per tryout session and I was the facilitator and the observer.

Learners were asked to bring their own computers and earphones to the classroom. The pre-test

and post-test were paper-based, and the learners were provided pens and pencils. For each

learner, the test scores of pre-test and post-test were paired and recorded. All the classrooms

were equipped with internet access and the learner received an email with the training link in

their email box by the time they start pre-test. When they started the training on their own

computer, they had the option to use earphones or not. There wasn’t a time limit on either the test

or the training and learners took as much time as needed.

Tryout Process

The tryout process started from recruiting the learners. I asked the students I am currently

teaching to test the prototype. After they agreed to participate, I scheduled a 30-minute face-to-

face session with each of them.

Pre-Test

In the beginning of the session, the facilitator gave a brief introduction in English on what

the learners would do. Then the facilitator administered the paper-based pre-test with 10
EVALUATION REPORT 6

multiple-choice questions regarding the main content of the learning module. All the questions

are fact-based and people with prior knowledge of Chinese history and culture can answer all of

them. Both the questions and the multiple choices are in Chinese. The pre-test is the same as the

post-test and can be found in Appendix B.

Training

Once the learners turned in their pre-test, they started the self-paced eLearning module on

their own computer. The link was already sent to their email address by that time. The learning

module uses English as instructions and Chinese for the content. Learners can choose to listen

and watch the animated slides, read the transcript, or listen and read the closed caption. They

also have the control to rewind and forward throughout the whole lesson. The practice questions

provide them a chance to check their knowledge and comprehension. They can click on the back

button to review certain slides if they need to. The training link and introduction page can be

seen in Appendix A.

Observation

While the learner was studying the learning module, the facilitator observed and used the

Observation Checklist in Appendix C to take notes. The facilitator quietly observed the learner’s

actions during the training without any interference. It was left to the learner to figure out how to

navigate through the learning module. The facilitator checked if the learner was able to navigate

through the whole lesson and noted down if they experienced any issues.

Post-Test

When they were ready to take the test, they closed their computer and the facilitator

distributed the post-test. Learners took their time to recall what they had learned in the lesson
EVALUATION REPORT 7

and answered all multiple-choice questions. The post-test has the same questions in the same

format as the pre-test. It can be seen in Appendix B.

Usability Questionnaire

The last step in the tryout process was to hand out the Usability Questionnaire. The

learners filled out the questionnaire based on their learning experience. The Usability

Questionnaire can be found in Appendix D. The facilitator would ask interview questions to

follow up the open-ended questions if needed. The submission of the questionnaire ended the

tryout process.

Section III. Results

Entry Conditions

The intended and observed entry conditions were similar. All learners were comfortable

using their own computers to access the training and navigate through the eLearning module

without any difficulty. They all had some prior knowledge on the training content at various

levels. The pre-test results showed the extent of their cultural and language proficiency about this

training topic. Their Chinese language proficiency in both listening and reading were at the

appropriate level to study this learning module.

Instruction

The observed instruction was similar to the intended instruction. All learners completed

every instructional slide as expected. They didn’t skip any practice questions either. All of them

paid attention to the audio narration. Almost all learners reviewed one or multiple content slides

before attempting the post-test. The duration for the instruction was expected to take around 12

minutes and the observed duration was between 11 minutes and 14 minutes.
EVALUATION REPORT 8

Outcomes

Intended Outcomes

The intended outcomes are increased linguistic and cultural proficiency about two

Chinese historical figures and their influences in the cognitive domain and motivation to study

more cultural background knowledge before going to Taiwan.

Cognitive Domain: The learners will increase their cultural knowledge about Sun Yat-sen

and Shang Kai-shek. They will be able to calculate the Minguo calendar and understand the

history behind it. They will also learn the two historical figures’ influences on Taiwan. Places of

interests will be introduced. Learners will be able to match different places of interest in Taipei

with the corresponding historical figures.

Affective Domain: By studying this self-paced learning module, learners will have the

autonomy to control the pace of learning, select the modality to access the content (auditory or

visual), and receive immediate feedback to check their knowledge. These features will attract the

target audience to complete the training and motivate them to seek more knowledge and transfer

learning when they go to Taiwan for language immersion.

Evaluation of Reaction

Due to the time constraint, the facilitator used open interview questions to gather learner

reaction to this learning module. Most learners liked the training and agreed that the content was

relevant and useful for overseas language immersion. All learners agreed that the small chunks

are effective in retaining information and planned to apply the knowledge in the appropriate

context.
EVALUATION REPORT 9

Evaluation of Learning

After each one-on-one session, the facilitator manually graded the pre-test and post-test.

There are 10 questions in each test and each question is worth 10 points, with a total of 100

points. The scores were then transferred to Excel and the five pairs of scores can be seen in Table

1 below. All five learners scored higher in the post-test than the pre-test as shown in Table 2.

Average score for pre-test is 54 and for post-test is 76. The learner who got the lowest score in

the post-test was less proficient in listening skill than the other four learners.

pre-test post-test
70 90
40 70
50 60
50 80
60 80
Table 1

Comparison of Pre-Test and Post-Test Scores

100
90
80
70
60
50
40
30
20
10
0
1 2 3 4 5

pre-test post-test

Table 2

The instructional designer believed that the training would improve learner knowledge

and increase their test scores. Therefore, the null hypothesis was that the learning module would

have no effect on test scores and there would be no statistical significance between the pre-test
EVALUATION REPORT 10

and post-test scores. A Paired Two-Sample for Means t-Test for dependent samples was run with

a significance level set to conventional alpha level of .05. To evaluate for statistical significance,

the one-tail values were used since the hypothesis was directional. See Table 3 for the t-test

results.

t-Test: Paired Two Sample for Means

Variable 1 Variable 2
Mean 54 76
Variance 130 130
Observations 5 5
Pearson Correlation 0.730769231
Hypothesized Mean Difference 0
df 4
t Stat -5.87974732
P(T<=t) one-tail 0.002090536
t Critical one-tail 2.131846786
P(T<=t) two-tail 0.004181072
t Critical two-tail 2.776445105
Table 3

The t Stat of 5.88 is much higher than the t critical value (one-tail) of 2.13 and the p-

value of 0.002 is much smaller than the .05 conventional alpha level. Therefore, the null

hypothesis is rejected. It can be concluded that there is a significant mean difference between the

pre-test and post-test scores. Effect size was also calculated to determine if the results are

practically significant. The pre-test scores (M=54, SD=11.40) and the post-test scores (M=76,

SD=11.40) differed significantly [t(4)=5.88, p<.05, and d=1.93]. The training was found to be a

significant factor in improving post-test scores of the learners.

Evaluation of Usability

The Usability Questionnaire as shown in Appendix D has 10 questions. 7 out 10 are 5-

point Linkert scale questions and the other 3 are open-ended questions. By assigning the highest
EVALUATION REPORT 11

score (strongly agree) as 5 and lowest score (strongly disagree) as 1, Table 4 shows the statistical

analysis of the user satisfaction for the first 7 questions regarding the interface of the learning

module.

Usability Questionnaire Result

4.8 4.8

4.6 4.6

4.4

4.2

Q1 Q2 Q3 Q4 Q5 Q6 Q7
Table 4

All users were generally satisfied with the prototype and commented in the open-ended

questions that they liked the interactivity and readily available transcripts. 100% users strongly

agreed with the statement in Q7 that the audio was clear. Q6 had the lowest rating regarding the

amount of on-screen texts. Users didn’t find the on-screen texts very useful. Q2 was about the

functionality of all navigation buttons and some users didn’t try all navigation buttons, so they

chose “Neutral” which was only worth 3 points. Users also commented that they had some

difficulty using the play bar to rewind to the exact point when reviewing the content. One user

found out that the sound didn’t work with some browser on MacBook Pro.

During the online training, the facilitator used the Observation Checklist in Appendix C

to take notes. All learners navigated through the lesson without any difficulty and all of them

completed all slides in order. 2 out of 5 learners used the “transcript” button, but none clicked on

“closed caption”. There were two reasons behind this. One was that all users wanted to focus on
EVALUATION REPORT 12

listening. The other reason was that they didn’t realize the existence of the closed caption. Most

learners didn’t read the “Instruction” page carefully because they assumed they knew what to do.

One learner had to ask for assistance because the audio was not playing in Internet Explore on

his MacBook Pro. The facilitator suggested that he should use Google Chrome.

Recommendations

Based on the evaluation result of the Pre-Immersion Language and Culture Training

Module, I would make recommendations in the following areas.

Instruction Page

Make the instruction page more visible and effective. Currently, the instruction was

written in sentences. For my target audience, they are all familiar with online navigations and

won’t bother to read instructions. They assume they know where to click and what to click. To

force them to pay attention to the navigation buttons, I need to use animations or arrows to

illustrate the functions of the buttons that I would like them to utilize.

Browser Requirement

Require the users to use Google Chrome for this training. Due to compatibility reasons,

some browsers may not function properly to show all the visual and audio content. The closed

caption wasn’t working in Internet Explore on MacBook Pro. For future training, I will require

all users to use Google Chrome for this training.

Subject Matter Expert Involvement

Ask subject matter experts to provide advice on how to improve the training content.

Though all learners improved their test scores after the training, the average post-test score was
EVALUATION REPORT 13

only 76 out of 100. Subject matter experts can provide some valuable advice on how to improve

the content of the training. The goal is to have an average score of 80 or higher. The subject

matter experts can also evaluate the validity of the test questions.

Instructor Feedback

A suggestion could be for instructors to provide test feedback at the end of the module.

After students finish the post-test, the instructors should grade the test and provide immediate

feedback to the learner. Clarify any misunderstanding or answer any questions that the students

may have. The instructor can also assign an immersion project for students to take with them to

Taiwan. This will help knowledge retention and drive for learning transfer.

Section IV. Summary

The positive evaluation results confirmed that the prototype is functional and effective to

improve learning. Both my observation and user feedback on the Usability Questionnaire

showed that the prototype is congruent with multimedia design principles and easy to use by the

target audience. The statistical analysis using paired two-sample t-test supported the research

hypothesis that the training will improve test scores of the learners. The evaluation results also

showed that some areas need improvement. To ensure all learners can use the training module

properly, a browser requirement should be stated in the beginning. A better formatted instruction

page should also be implemented in future usage. The training content and validity of test items

should be examined by several subject matter experts. I will continue to improve this prototype

and use the lessons learned in my future design projects.

EVALUATION REPORT 14

Section V. Appendices

Appendix A. Training Module

http://myspace.csumb.edu/~coli/IST526/Mandarin%20Langauge%20and%20Culture_Test%20V

ersion/

Appendix B. Pre-test and Post-test Questions

Pre-Test and Post-Test

Name:

Choose the best answer to each of the following 10 questions.

1. 中华民国是哪年成立的？
A. 1911 年
B. 1912 年
C. 1945 年
D. 1949 年

2. 关于辛亥革命，下列哪项是正确的？
A. 推翻了清朝的统治，建立了中华民国
B. 发生在 1912 年
EVALUATION REPORT 15

C. 是蒋介石领导的
D. 是抗日战争的一部分

3. 按照民国纪年，今年是民国多少年？
A. 2019 年
B. 70 年
C. 100 年
D. 108 年

4. 关于孙中山，下列哪项是错误的？
A. 也被称为国父
B. 是中华民国第一任总统
C. 在推动民主革命中起了重要的作用
D. 建立了中华人民共和国

5. 关于蒋介石，下列哪项是错误的？
A. 中正纪念堂是纪念他的
B. 参与并领导了抗日战争
C. 对中华民国的建立起了重要作用
D. 担任中华民国总统长达 27 年

6. 关于“国父纪念馆”，下列哪项是错误的？
A. 是为了纪念孙中山而建立的
B. 内有蒋介石的铜像
C. 建筑具有唐朝宫殿风格
D. 可以观看军人交接仪式

7. 关于士林官邸，下列哪项是正确的？
A. 是台湾以前的总统府
B. 是纪念孙中山的
C. 是蒋介石以前居住的地方
D. 是台北一个有名的公园

8. 国父纪念馆是纪念谁的？
A. 孙中山
B. 蒋介石
C. 毛泽东
D. 孔子

9. 中正纪念堂是来纪念谁的？
A. 孙中山
EVALUATION REPORT 16

B. 蒋介石
C. 毛泽东
D. 孔子

10.台北 101 离以下哪个景点比较近？

A. 中正纪念堂
B. 国父纪念馆
C. 总统府
D. 士林官邸

Appendix C. Observation Checklist

Check Items Y/N Notes

Does the learner follow the navigation
Does the learner have any difficulty navigating
instruction?
Does the learner get stuck on any slide?
through the module?
Does the learner encounter any technical issues?
Does the learner click on the “transcript” button?
Does the learner request any assistance to
Does the learner use closed caption button?
advance in the module?
Does the learner skip any slide by accident

without even noticing it?

Appendix D. Usability Questionnaire

Please provide your honest opinions about the interface of the eLearning module .

1. Overall, the eLearning module is easy to use.

 Strongly Agree
 Agree
 Neutral
 Disagree
 Strongly Disagree

2. The navigation buttons are all functional.

 Strongly Agree
 Agree
 Neutral
 Disagree
 Strongly Disagree
EVALUATION REPORT 17

3. The graphics are appropriate.

 Strongly Agree
 Agree
 Neutral
 Disagree
 Strongly Disagree

4. The design of the slides is consistent.

 Strongly Agree
 Agree
 Neutral
 Disagree
 Strongly Disagree

5. The interactive activities (exercise feedback) are helpful.

 Strongly Agree
 Agree
 Neutral
 Disagree
 Strongly Disagree

6. The amount of on-screen texts is appropriate.

 Strongly Agree
 Agree
 Neutral
 Disagree
 Strongly Disagree

7. The audio recording is clear.

 Strongly Agree
 Agree
 Neutral
 Disagree
 Strongly Disagree

8. What do you find most frustrating?

9. Which feature(s) do you like the most?

10. Do you have any additional comments?