Running head: INSTRUCTIONAL PROTOTYPE TEST

Testing for iLearn Instructional Prototype

Deanne Dominguez

California State University – Monterey Bay

IST622 Assessment and Evaluation

Dr. Su

June 12, 2018

 Prototype Tryout 2

Table of Contents
I. Introduction ----------------------------------------------------------------------------------------- 3
II. Methodology -------------------------------------------------------------------------------------- 3
a. Prototype ----------------------------------------------------------------------------------------- 3
b. Learners ------------------------------------------------------------------------------------------ 4
c. Tryout Process ------------------------------------------------------------------------------- 5
1) Introduction ------------------------------------------------------------------------------- 5
2) Pretest -------------------------------------------------------------------------------------- 5
3) Observation ------------------------------------------------------------------------------- 6
2) Posttest------------------------------------------------------------------------------------- 6
3) Questionnaire ----------------------------------------------------------------------------- 6
d. Tryout Conditions ------------------------------------------------------------------------------ 7
III. Results --------------------------------------------------------------------------------------------- 7
a. Entry conditions -------------------------------------------------------------------------------- 7
b. Instruction --------------------------------------------------------------------------------------- 7
c. Outcomes ---------------------------------------------------------------------------------------- 8
d. Recommendations ------------------------------------------------------------------------------ 9
IV. Summary and Conclusion ---------------------------------------------------------------------- 9
V. Appendices --------------------------------------------------------------------------------------- 10
A. Script ---------------------------------------------------------------------------------------- 10
B. Pretest --------------------------------------------------------------------------------------- 10
C. Posttest -------------------------------------------------------------------------------------- 11
D. Posttest Questionnaire -------------------------------------------------------------------- 12
E. Posttest Questionnaire Results ----------------------------------------------------------- 13
F. T-test Paired Two Sample for Means --------------------------------------------------- 14
G. Observation Checklist--------------------------------------------------------------------- 15
H. References ---------------------------------------------------------------------------------- 16
 Prototype Tryout 3

I. Introduction

This purpose of testing of this prototype is to measure its usability as it may serve as a

model for the module that will be created for the capstone project. The capstone project will be

an iLearn centered module. Because the focus of the capstone is still being discussed by its

prescribers, this prototype will only be potentially used. This prototype demonstrates the use of

modern functions in iLearn: how to sign in, post to a forum, and use a drop-box.

This trial utilizes multimedia principles to communicate the basic functions of iLearn,

providing a brief introduction to Learning Management Systems, a video demonstration, and

walkthrough. A pretest, posttest, observations during test taking, and a questionnaire, will be

used to measure usability. The results will aide in deciding if all, or any, elements support

learning. The results will be used to accordingly filter which aspects of the prototype will be

used in the module for the capstone.

II. Methodology (Be sure to describe prototype, learners, tryout conditions and process)

a. Prototype

The prototype being tested is a multimedia module. It is intended to provide an

action-by-action walkthrough of how to perform basic functions while navigating the

iLearn interface. Since its development, iLearn has been overhauled and therefore this

prototype is not conditional for reuse. However, testing the prototype’s approach to

delivering the outcome of certain procedures should still indicate whether methods are

appropriate for the capstone module. It consists of an introduction, a screencast video,

and a walkthrough-based assessment that progresses based on successful actions. The

introduction and screencast demonstration portion of the prototype is a video comprised

 Prototype Tryout 4

of simple visuals with audio designed with the intention of avoiding cognitive overload.

There is a final brief quiz that assesses learner knowledge on definitions discussed.

The module takes about 20 to 30 minutes to complete. Learners will watch the video

portion, attempt to perform the actions in the assessment portion, and attempt the quiz.

There is playback ability to aide in performing functions required in the assessment

portion. Therefore, actual completion of the module may take longer but it is projected

that it should not take longer than 30 minutes.

b. Learners

The target audience are college students who have little to no experience with iLearn

and are interested in learning how to navigate through its basic functions. As stated

previously, the prototype is rendered useless in terms of preparing someone for iLearn as

the former version of iLearn is obsolete. However, because the prototype objectives are

related to very basic functions of L.M.Ss. the module is still appropriate for individuals

seeking to familiarize with L.M.Ss. Although the prototype was developed with college-

aged students in mind, it should be noted that online courses are not limited to university

settings or a fresh out of high school age range. For example, L.M.Ss are also an integral

part of many homeschool settings, and online courses are available for an individual who

may be seeking to return to school later in life. Therefore, although the target audience is

geared toward undergraduate and graduate students, the age range is very large.

This module is targeted at high-school and above, this module is suitable for various

ages of individuals who are able to navigate the internet, basic computer functions such

as operating playback controls, fluent in English, and has someone with at least a high

school age vocabulary or someone available to explain concepts that to not match the
 Prototype Tryout 5

individual’s vocabulary. The module is also formatted with closed-captioning for an

individual with hearing specific needs.

Five users were used to selected to test the prototype. All users have at least one year

of college experience. Two of the five students had taken college level online courses

before, but none had not previously used iLearn.

c. Tryout Process

The process of evaluation was comprised of five parts; 1) verbal introduction to the

tryout 2) pretest to measure familiarity with concepts presented in objectives 3)

observations, 4) posttest to measure familiarity with iLearn navigation and objective

mastery, 5) survey questionnaire

1) Introduction

A script (appendix A) was developed to give learners a better idea of the

purpose of the tryout, a description of the format of the tryout, and the purpose of

the module. The introduction was also developed to inform learners of their

freedom to stop the testing at any point.

2) Pretest

This item was a three-question google form (appendix B). It addressed three of

the five objectives. The performance objectives that were not mentioned in the

pretest could not be measured as they were part of the demonstration in the

module and were not compatible with google form. Learners were not given the

results of their pretest so therefore; similar wording was used for both the pretest

and posttest. The pretest and part of the posttest results were used to measure

learning in a paired sample t-test.

 Prototype Tryout 6

3) Observation

This segment of the evaluation process followed ideas set forth in a structured

observation system. It was used to observe participants’ behavior while using the

module. Users were scheduled for testing in a two-week period. When contacting

the users, I explained to the the ideal type of setting would be a quiet, distraction-

free space, and all but one decided to take the module in my home. To get the

best possible observations, users were scheduled one at a time. For the one user

who decided to take the module in their home, we used Facetime as a mode of

observation and proctor. Using this method of communication did not interrupt

the trial process. As the administrator, I appreciated being slightly removed from

the process but still have the ability to focus in on the user’s behavior.

4) Posttest

The module is comprised of a demonstration with a user walkthrough to follow

that allows users to practice what was shown during the demonstration. There

were a series of slides that continue on successful actions, and gave hints for

incorrect actions. This portion of the module was recorded.

The posttest is a three-question match test (appendix C). Hints were given for

incorrect answers. This portion of the module was also recorded and used to

develop the paired sample t-test. Following the posttest, users may review the

quiz.

5) Questionnaire

Users were given a five-question survey (appendix D) that would serve in

measuring how helpful the prototype was in mastering objectives. It also was
 Prototype Tryout 7

created to measure learner’s experience, what needed to be changed, and what

features could be utilized for the prototype to be developed for the capstone. The

form was created using Google Forms and adhered to Russ-Eft’s (2009)

guidelines for successful surveys and questionnaires.

d. Tryout Conditions

In order to perform this testing, users needed access to internet and a computer as

it is not a responsive prototype. As described previously, to avoid distraction and any

type of cognitive or audio overload, were encouraged to run through the segments of the

tryout in a quiet space with headphones.

III. Results

a. Entry Conditions

For the most part my intended observations did not differ from the outcome.

Because of user’s background, I expected they would all perform with ease. Everyone I

had tested was familiar with a Mac. I did not anticipate users using my computer would

have any issues. However, when one participant was navigating from the pretest to

prototype, they stated they were unfamiliar with typical Mac gestures and asked for

assistance.

b. Instruction

My intended observations for instruction conditions differed than the outcome.

The prototype had audio glitches. There is a technical issue with my computer and

Captivate’s production process. When I would attempt to republish the prototype, the

program would crash. I will need technical support to address the issue. The module is

simple in nature, but I still intended for users to have a lot of issues. Alternatively, they
 Prototype Tryout 8

navigated through the module just fine. Two users id ask if it was the connection or the

audio and I had to explain the issue.

c. Outcomes

The intended outcome and actual outcome differed significantly. Because of the audio

factor, I had thought that it would significantly affect learner progress toward excelling in

objectives. However, according to the data (Appendix F) results show that learners made

significant increase. All users scored 100% on the posttest, which as discussed is

significantly longer than the pretest, and includes actions where the pretest is only a

definition-based pretest.

Participants’ scores were derived from crediting one point to each question that was

used for measuring learning in the pretest and the matching portion of the posttest.

Therefore, users could score a total of three points on both tests. A paired two sample t-

test was used to identify if the module attributed to an increase learning. The absolute

value of t-stat was greater than the t critical value (|t|=4 > 2.7). Therefore, from the

results, there was found to be a significant increase in learner knowledge comparing

pretest and posttest results.

The questionnaire results provided a varying degree of responses. 60% of users claimed

(Appendix E) that the module was relatively easy to navigate and 80% of users the

objectives were clear, while 20% percent somewhat disagreed in both areas. All users

shared they would recommend the module to someone interested in learning more about

iLearn. One user did not explain why they would refer this model but all indicated they

would.

There were many suggestions given for the additional feedback portion of the
 Prototype Tryout 9

questionnaire. There was a general consensus that for navigability and clarity, the module

could afford to be more simplistic visually.

d. Recommendations

Recommendations based on results are as follows:

Refine audio: The audio breaks up. The sound quality effects attention and a break up

in that audio serves as a distraction from the module.

Reconsider instructional approach and graphics: User feedback showed that features

could be even more simplistic and obvious to aid in an even easier learning experience.

Although results showed that participants were able to excel in mastery of the

objectives presented, their responses indicated objective clarity and their process of

getting through the module could be improved.

Change font color in testing : One user described the font in the testing segment was

difficult to read. It is advisable that instructions are clear so that learners understand

exactly what they are being asked to do in order to be properly evaluated.

IV. Summary and Conclusion

After analyzing results, it is found that the module significantly supported learning. Users

were generally able to complete the module with ease. However, there were areas where users

reported dissatisfaction such as in the graphic choices and overall navigation could afford more

simplistic features. Moreover, despite these potential setbacks, and despite certain audio

setbacks, users proved to navigate through the module with excellence. While this prototype

definitely needs improvement, it will serve as a framework for the capstone project.
 Prototype Tryout 10

V. Appendices

A) Script

Thank you for participating in this tryout. This tryout is intended to test the prototype not

you. If at any time you feel like you need to stop the testing before the tryout is over,

please feel free to do so. The module consists of four segments: a pretest, the module, a

posttest, and a questionnaire to evaluate your experience. I will be observing you during

this process. If at any time you need help understanding what is expected, please ask.

B) Pretest

6/12/2018 L.M.S. Pretest

L.M.S. Pretest
Thank you for doing this for me! This is a pretest for the L.M.S. module to follow. Please answer 
according to your present knowledge without looking up any answers.

1. What does the abbreviation L.M.S stand for?

2. Typically, a discussion forum is a colloborative activity that supports peer discussion.
Check all that apply.

 True
 False

3. What is a Drop Box?
Mark only one oval.

 a colloborative discussion forum activity that supports peer discussion.
 A system that extracts files from your personal device and uploads the files to the L.M.S.

Powered by
 Prototype Tryout 11

C) Posttest Matching Portion

 Prototype Tryout 12

D) Questionnaire

6/12/2018 L.M.S. posttest survey

L.M.S. posttest survey
This is a questionnaire to find out how your experience taking the module was. Feel free to be honest­ It'll 
be helpful!

1. The module was easy to navigate.
Mark only one oval.

 disagree
 somewhat disagree
 neutral
 somewhat agree
 strongly agree

2. The module objectives were clear.
Mark only one oval.

 disagree
 somewhat disagree
 neutral
 somewhat agree
 strongly agree

3. I would refer this module to someone who is interested in learning how to use iLearn.
Mark only one oval.

 T
 F

4. Please explain your answer.

5. Please provide any additional feedback that could help improve this course.
 

Powered by

https://docs.google.com/forms/d/1wk4-JpUCEsWjEk0p848GigNSOrvekUt_Rtzsmk0P18Q/edit 1/2
 Prototype Tryout 13

E) Questionnaire Results
Prototype Tryout 14
 Prototype Tryout 15

F) Pretest and Posttest T-Test: Paired Two Sample for Means

Scores
Users Prettest Posttest
Angel 2 3
Noah 2 3
Phil 3 3
Suzanne 2 3
Brittany 2 3

t-Test: Paired Two Sample for Means

Variable
Variable 1 2
Mean 2.2 3
Variance 0.2 0
Observations 5 5
Pearson Correlation #DIV/0!
Hypothesized Mean
Difference 0
df 4
t Stat -4
P(T<=t) one-tail 0.008065045
t Critical one-tail 2.131846786
P(T<=t) two-tail 0.01613009
t Critical two-tail 2.776445105

G) Observations

Checklist Yes No
Is the learner navigating with ease?
Is the learner following instructions?
Does the learner conduct recording for
each practice item?
Is the learner trying to look up answers?
 Prototype Tryout 16

References

Russ-Eft, Darlene. Evaluation in Organizations: A Systematic Approach to Enhancing Learning,

Performance, and Change (Kindle Locations 4017-4018). Basic Books. Kindle Edition.