Running head: IMACTION READINESS MODULE 1

Evaluation of the Impaction Readiness Training Module

Gwen Hansen

California State University Monterey Bay

IST 622

Dr. Bude Su

July 23, 2019

IMPACTION READINESS MODULE 2

Table of Contents

Introduction ..................................................................................................................................... 3

Methodology .................................................................................................................................... 3

Prototype ...................................................................................................................................... 3

Learners ....................................................................................................................................... 5

Expected Outcomes ..................................................................................................................... 5

Tryout Process ............................................................................................................................. 7

Tryout Conditions ........................................................................................................................ 8

Results ............................................................................................................................................. 9

Entry Conditions .......................................................................................................................... 9

Instruction .................................................................................................................................... 9

Summary of Data Collection ..................................................................................................... 10

Outcomes ................................................................................................................................... 11

Recommendations ..................................................................................................................... 13

Summary ........................................................................................................................................ 14

Appendix A Pre- and Post- Test Question................................................................................... 15

Appendix B Link to Impaction Readiness Module ..................................................................... 17

Appendix C Module Observation Check List ............................................................................. 18

Appendix D Impaction Readiness Module Post-Survey ............................................................. 19

IMPACTION READINESS MODULE 3

Introduction

This report examines pre- and post-test data for a lesson in an instructional learning

module entitled “Impaction Readiness”. The Impaction Readiness module consists of a Captivate

course designed to train on the impaction criteria and the supplemental admission process for

applicants applying to the nursing major at California State University, San Marcos (CSUSM).

Ten learners participated in the evaluation of the training module. The purpose of this report is to

determine the extent of the influence of the learning module based on pre- and post-test scores of

the learners.

Methodology

Prototype

The prototype used for this evaluation is part of a series of modules designed for my

capstone project. The prototype is an instructional learning module and consists of three lessons:

Impaction Readiness, TBSN Course Preparation, and Alternative Nursing Pathways. The lesson

covered in this evaluation is the first lesson, Impaction Readiness, and is currently the only

active lesson in the series. The lesson teaches on three elements of impaction: Defining

Impaction, Supplemental Criteria for Freshman Applicants, and Supplemental Criteria for

Transfer Applicants.

The prototype is a responsive product and designed to be accessed through a computer or

laptop. The duration of the module is approximately fifteen minutes. Prior to starting the

training, learners are given a brief description of the goal of the module to create buy-in for the

learner and increase transfer after completion of the module.

The lesson slides are narrated, and interactive learning checks appear after each element

to enhance learning transfer. There are ten quiz items embedded at the end of the module and
IMPACTION READINESS MODULE 4

presented to the learners after completion of the learning content. Responses were scored and

tracked into an excel spreadsheet. Examples of the design and feel of the prototype are displayed

in Tables 1.1 to 1.3.

Table 1.1

Table 1.2
IMPACTION READINESS MODULE 5

Table 1.3

Learners

The Impaction Readiness Training module was designed for prospective freshman and

transfer applicants and/or parents of applicants desiring to obtain a degree in the nursing major

which is impacted at CSUSM. For this tryout, learners do not need to be actual CSUSM nursing

applicants. For evaluation purposes, learners were chosen to represent a sample population of the

intended audience.

The criteria used for test user selection included learners eighteen years or older who are

familiar with basic computer usage. Additionally, criteria required learners with no prior

knowledge or familiarity with the elements of the impacted TBSN program. A total of ten

participants were recruited for the purpose of the tryout, all of whom met the criteria. The

learners comprised of two males and eight females.

Expected Outcomes

Upon completion of the instructional learning module, learners will successfully define

impaction readiness and increase their knowledge for impaction criteria for freshman applicants

as well as impaction criteria for transfer applicants. The hypothesis is that there will be an
IMPACTION READINESS MODULE 6

increase in mean score on the pre- post-test instrument. In order to measure the learning success

of the module, a comparison of means for pre-/post-test will be conducted. Additionally, it is

hypothesized that the participants will exhibit a statistically significant increase in mean scores

after completion of the learning module.

Cognitive Domain: Learners will increase their knowledge in the elements of impaction

readiness. The learning is broken into three segments to avoid cognitive overload. Learners are

required to understand the content presented and then immediately apply the attained

knowledge to interactive learning checks embedded for each segment. The module focuses on

constructing new knowledge and tests the learner on the impaction readiness elements at the

conclusion of the training.

Psychomotor Domain: Training requires learner to use navigation in order to begin the

module and to submit responses to practice and quiz questions. Additionally, the learner can

choose to move forward or return to previous slides, giving them control to move at their own

pace. Learning Checks require user to drag and drop correct answers for repeated practice and to

ensure transfer of the learning.

Affective Domain: The module begins with an introduction and learner friendly

objectives. These set clear expectations for the learner and draws attention to the important

elements of the training for the learner to focus on. Opportunities to practice are offered through

learning checks in order to keep the learner engaged with the content material. The content

material is presented with appropriate animation to increase motivation to remain engaged with

the content and to deploy learning transfer after the completion of the module. Immediate

corrective and explanatory feedback are given throughout the training.

IMPACTION READINESS MODULE 7

Tryout Process

The tryout process consists of five parts: pre-test, training, post-test, observation, and

usability survey. The pre-test, observation sheet, and survey were created with Word Documents.

The post-test was created using in Adobe Captivate. The learners’ responses were not

anonymous so that a comparison of test scores could be completed. Ten individuals were

recruited for the tryouts. The learners were required to complete the items below.

Beta Testing. During the recruitment process for participants, general information about

the learners were gathered through interviews. Learners were asked if they had any prior

knowledge regarding nursing impaction for the TBSN program at CSUSM. Learners were over

the age of 18 years and had basic computer knowledge.

Pre-Test. The learner was given a paper copy of the pre-test to complete prior to taking

the training. The pre-test was essential for evaluating the effectiveness of this training program.

The item formats were comprised of multiple choice and “Hit/Myth” questions. The questions

were designed to measure the learners’ prior knowledge of nursing impaction for the TBSN

program and specifically to test the content covered in the training. The learners were not given

the correct answers to the pre-test questions to ensure the validity of the results when compared

to the post-test results. The pre-test questions were identical to the post-test questions (see

Appendix A).

Training. The training program was an eLearning module that took fifteen minutes to

complete. The learners accessed the module on a laptop computer. The module was created as a

software simulation using Adobe Captivate. The training link is accessible in Appendix B.

Post-Test. After completing the training content, learners answered ten post-test

assessment questions embedded at the end of the module. The post-test questions were identical
IMPACTION READINESS MODULE 8

to the pre-test questions in order to evaluate the effectiveness of the training when completing a

paired analysis (see Appendix A).

Observation. Ten learners were observed through one-on-one sessions scheduled for

individual tryouts. The observation evaluations were completed over a seven-day period from

July 12th through July 18th. The time for each observation ranged from 25 to 35 minutes. All

participants were debriefed on the purpose of the training and the observation process. Prior to

beginning the session, all learners agreed to be observed and to complete the pre-test, the training

and post-test, and the usability survey. The evaluation took place sitting side by side while the

learner completed the training module. Observations were recorded during the training using an

observation checklist (see Appendix C).

Usability Survey. The Usability Survey was designed to measure the learners’ reaction

and satisfaction to the module design, the content, and the learning experience. Learners were

asked to respond to each statement using a 7-point scale ranging from “Strongly Agree” to

“Strongly Disagree”. The survey will help determine the look and feel of the module, the clarity

of the content, and the learners’ enjoyment of the module. The data collected will be conducted

to measure the effectiveness of the learning solution. The evaluation results will be used to

improve the module. See Appendix D.

Tryout Conditions

Tryout conditions were administered to mimic the same environment intended for the

targeted audience. The learners accessed the training module that was pre-loaded on a laptop

computer. The tryouts were done individually in a private office location on the campus of

CSUSM. Each learner completed the pre- test, training module, post-test, and usability survey

one time only. After completing the assessment questions, the learners reviewed their post-test
IMPACTION READINESS MODULE 9

results in the Captivate module.

Results

Entry Conditions

The entry conditions for intended and observed were congruent. There were no

unexpected entry conditions observed. All ten learners were sufficient in basic computer

knowledge and were able to access the module without problems or difficulty. The testing

process was designed to take forty-five minutes or less and each learner completed all items

within the expected timeframe.

Instruction

In general, the observed instruction was congruent to the intended instruction. All ten

learners were able to follow the training instructions and progress through the module as

intended with two exceptions. The first exception was in regard to one of the learning checks.

Three of the learners spent a longer time on trying to figure out how to ‘place’ items in the

correct sequence. Once they were instructed to ‘click and drag’ they were able to proceed

without further difficulty. Seven of the ten learners were able to complete the training without

any instruction from the facilitator.

The second exception was in regard to the navigation buttons. All learners ignored the

‘return’ and ‘continue’ navigation buttons because the Captivate module was set to automatically

proceed to the next content slide. This was a requirement for the module assignment when the

lesson was created for IST 526. None of the learners clicked the return button to review previous

content. However, this may have been because the learners did not realize they had the ability to

do so. The ten participants were able to use all the other navigation buttons to begin the module,

to submit answers to questions, and to proceed after receiving immediate feedback.

IMPACTION READINESS MODULE 10

Summary of Data Collection

The pre- and post-test test scores were obtained from the individual responses for each

assessment. The data results were manually entered into a spreadsheet in excel. Excel was used

to run statistics for the data set to determine the effectiveness of the training on learning. Table

2.1 shows individual pre- and post-test scores.

Table 2.1

IMPACTION READINESS
ID Pre-Test Post-Test
RS 4 8
TT 7 10
PH 6 10
KO 4 10
AS 3 9
MH 2 8
CV 6 9
TW 5 9
DW 7 8
SE 3 8

There were ten identical graded questions for the pre- and post-test. Mean scores were

compared to show the effectiveness of the training based on increased scores after taking the

learning module. All learners scored higher on the post-test. The pre-test mean score was 4.7.

The post-test mean scores was 8.9. Figure 1 illustrates the difference in individuals on their pre-

and post-test scores.

Figure 1. Individual Learner Pre- and Post-Test Scores

IMPACTION READINESS MODULE 11

Outcomes

The hypothesis for this study was that the learning module would result is a statistically

significant increase in the post-test score. The null hypothesis was that the training would have

no effect and no statistical difference between pre- and post-test scores would be found. The

hypothesis was directional so a paired sample t-test for one-tail results were used to evaluate for

statistical significance. Table 2.3 illustrates the paired-two sample t-test for dependent samples

run with a degree of freedom of nine.

Table 2.3

Based on the t-test results, it can be concluded that there is a statistically significant mean

difference between the pre- and post-test results. The t-stat was 8.20, which is higher than the

critical value of 1.83 and the p-value of .009 is smaller than the .05 conventional alpha level, so

the null hypothesis is rejected, and the results are found to be statistically significant. Effect size

was calculated to determine if the results are practically significant. The pre-test scores (M=4.7,

SD=1.767) and the post-test (M=8.9, SD=0.875) differed significantly [t(9)=8.20, p<.05, and

d=2.4]. The effect size of 2.4 is greater than 0.8 and therefore it shows that the training is
IMPACTION READINESS MODULE 12

practically significant. Analysis of the pre- and post-tests results indicate that learners achieved

significant learning gains and increased their knowledge of the TBSN impaction as a result of

completing the learning module.

The learners’ overall ratings for the learning module design were high. One hundred

percent of learners were satisfied overall with the module design. Ninety percent of the learners

strongly agreed that they were satisfied with the module design. The ratings included the ease of

the navigation, look and feel of the design, and the enjoyment of the learning activities (see

Figure 2.1).

Figure 2.1. Module Design

The learners’ overall ratings for the learning module content were high. One hundred

percent of learners were satisfied overall with the module design. Eighty percent of the learners

strongly agreed that the training content was effective and liked the information learned from the

module. The ratings included the clarity of the content, how the content was presented, and how

the assessment questions facilitated their learning (see Figure 2.2).

IMPACTION READINESS MODULE 13

Figure 2.2. Module Content

The learners’ overall ratings of the learning experience were high. One hundred percent

of learners indicated that they were satisfied with the learning experience. Unsolicited comments

made regarding the learning experience included “I learned” and “I like the automated

feedback”. The ratings included enjoyment of the training, the meaningfulness of the training,

and gained knowledge learned by taking the learning module (see Figure 2.3).

Figure 2.3. Learning Experience

Recommendations

The following recommendations for the Impaction Readiness learning module are

suggested to improve the usability and learning effectiveness of the training. The first is to

provide clearer instructions for the learning checks by replacing the word ‘place’ with the
IMPACTION READINESS MODULE 14

instructions “Click and Drag”.

The second is to remove the automated navigation from the Captivate module thereby

turning the controls completely over to the learner. This will enhance the ability of the learner to

move at their own pace. In addition, it will increase user control of the program functioning.

The third is to add an additional learning check to include more practice for the transfer

impaction criteria. Although every learner improved their scores on the post-test, there were two

assessment questions frequently answered incorrectly. Half of the learners (5 out of 10) missed

question number seven, and four out of ten learners missed question number nine. Adding an

additional learning check activity to the module will give the learner the opportunity to practice

the content in these specific areas thereby enhancing learning transfer.

Summary

The goal of this study was to examine the effectiveness of the Impaction Readiness

training module for increased learning by TBSN freshman and transfer applicants. The

hypothesis was that the training would improve the learners’ test scores after completing the

learning module. The evaluation of the statistical data supports the hypothesis and show a

statistically significant and practical difference between the mean scores of the pre- and post-

tests.

The results demonstrate that the Impaction Readiness module significantly contributed to

the improvement of the test scores. The observation evaluation suggests that improvements

should be made to the module to improve some of the instructions for better clarity, to remove

the automated navigation, and to add additional practice for areas of the content. The usability

survey analysis indicate that the learners were satisfied with the module design, they liked the

information presented in the module, and enjoyed the overall learning experience.
IMPACTION READINESS MODULE 15

Appendix A

Pre- and Post- Test Question

Indicate if the statement is a Hit (H) or a Myth (M)

1. ____All applicants must apply to the University and meet all deadlines to be considered for the Traditional
Bachelor of Science in Nursing (TBSN).

2. ____Due to a limited number of seats available, the impacted TBSN is difficult to get into.

3. ____Freshman applicants are rank ordered based on their college readiness and eligibility index score.

4. ____The eligibility index score for freshman is the combination of their high school grade point average
(GPA) and SAT/ACT test scores.

5. ____Transfer applicants are rank ordered for admission based on specific impaction criteria for the nursing
major.

Circle one answer

6. The maximum number of supplemental admission criteria points that transfer TBSN applicants can earn is
_____.

• 25
• 45
• 58
• 68

7. Transfer applicants must earn a minimum score of _____% on the TEAS exam in order to be considered
for the TBSN.

• 71%
• 72%
• 74%
• 75%

8. Residency is based on regional boundaries and school districts. Residency for TBSN transfer applicants is
based on the last ______CSU eligible units completed by the spring semester, prior to fall admission to the
TBSN program.

• 30
• 45
• 60
• 70
IMPACTION READINESS MODULE 16

9. Casey is a transfer applicant who has completed the eight courses required for the Pre-Nursing Core. She
passes the pre-nursing core courses with a “C” or better with an overall 2.5 Grade Point Average (GPA).
Please select the one best answer regarding Casey’s eligibility status for admission to the TBSN program.

Casey meets the minimum GPA for the Pre-Nursing Core.

Casey does not meet the minimum GPA for the Pre-Nursing Core.
Casey meets the Pre-Nursing Core requirement by completing all eight courses.
Casey does not meet the Pre-Nursing Core requirement because she needs a minimum 3.0 GPA.

10. John is a freshman applicant for the nursing major. He applied to CSUSM and meets all minimum
admission requirements. He is college ready in English and Math and meets the eligibility index score.

John will be rank-ordered based on the eligibility index score.

John will be rank-ordered based on college readiness in English and Math.
John will be rank-ordered based on college readiness and the number of applicants.
John will be rank-ordered based on the eligibility index score and college readiness.
IMPACTION READINESS MODULE 17

Appendix B

Link to Impaction Readiness Module

Click on the link to view the Adobe Captivate module in my iLearn course. The module is

located in the third section of the course.

IMPACTION READINESS MODULE 18

Appendix C

Module Observation Check List

Impaction Readiness

Checklist Yes No Navigation /slide Comments

#/Time
Can the learner begin
module without
difficulty?
Can the learner
follow the module
navigation without
difficulty?
Can the learner
follow the
instructions for all
activities?
Does the learner
ignore
functions of the
module?
Can the learner
resolve any issues on
their own?
Does the learner
require any assistance
while taking the
module?
Other
IMPACTION READINESS MODULE 19

Appendix D

Impaction Readiness Module Post-Survey

Thank you for taking time to complete the learning module. Hope you had a great
learning experience. Please take some time to answer the following questions. This survey has
three sections (Module Design, Content, and Learning Experience) and should only take 3-5
minutes to finish. Your feedback is very important to help improving this learning module.

1. Module Design
Please choose the answer which best describes your opinion of the learning module.
Strongly Agree Somewhat Neutral Somewhat Disagree Strongly
Agree Agree Disagree Disagree

It was easy to navigate the

module.
The module was well structured.
The module had a clean, simple,
and professional look.
I enjoyed the learning activities.
The graphics were well selected
and used in the module.
Overall, I am satisfied with the
module design.

2. Content
Please choose the answer which best describes your opinion of the learning content.
Strongly Agree Somewhat Neutral Somewhat Disagree Strongly
Agree Agree Disagree Disagree

The content is easy to understand.

I found the content was helpful.
The knowledge was explained
clearly in the module.
The assessment questions
facilitated my learning.
The content was presented in
appropriate ways.
Overall, I like the information I
learned from this module.
IMPACTION READINESS MODULE 20

3. Learning Experience
Please choose the answer which best describes your learning experience.
Strongly Agree Somewh Neutral Somewh Disagree Strongly
Agree at Agree at Disagree
Disagree
I enjoyed the training.
The training was meaningful and
helpful.
I understand nursing impaction
better now.
I know now how impaction
criteria is used for admission to the
TBSN.
The online training is interesting.
Overall, I had a satisfied learning
experience.