Assignment 5: Formative Evaluation Plan

by
Cybele Wu
IDT 8001, CRN 23851

Nova Southeastern University

November 13, 2016

Background
A print-based, self-instruction lesson was created in such a way
that no facilitation is needed to complete the lesson. The learners for
this lesson are ESL instructors of adult second-language learners. The
self-instructional lesson will instruct learners on the procedures for
compressing a high-definition video into We-m, VP-9 format. The
instruction should take no longer than 30 minutes to complete. It also
is available for all operating systems and is available on either a
home or work computer with full security allowances. The final
product will result in a much smaller video file that is of similar quality
for uploading to video-sharing websites.
Purpose
The purpose of this step in the process is to create a formative
evaluation plan for the print-based lesson on compressing highdefinition video into Web-m, VP-9 format. Although there are three
parts of a normal formative evaluation plan (one-on-one, small-group,
and field trial), this particular design will only include the one-on-one
phase of the evaluation plan.
Method: Design
The design for this particular part of the lesson will focus on the
one-on-one evaluation phase of the formative evaluation plan. The
Subject Matter Expert (SME) will be contacted to evaluate the
instructional materials. He will be contacted via email to review the
materials as well as the interview questions. Another in-person
interview will be scheduled to go over the instructional materials. The

interview will be recorded so that the changes to the instructional

materials can be made using the

SMEs comments and suggestions. Three learners of varying levels

of technology knowledge will be used to complete the instruction.
They will complete the self- paced print-based lesson in less than
30 minutes and make notes on the Lesson Evaluation Form.
Expert Review
Subjects:
The subject matter expert is an expert in Open Source Software.
He works in IT (Information Technology) and has worked on Linux, PC
and Mac computers. He is very familiar with open source software,
which he uses in his day-to-day operations in customizing the software
for the preferences of the client. In this project, we will be using open
source software to compress the video to Web-m, VP-9 format, which
my SME has been helpful in providing the command prompts for.
Instruments:
The Subject Matter Expert will be emailed the draft of the
instructional materials with the evaluation protocol (Appendix B)
prior to implementation. Open-ended questions will be asked. This
feedback will help to determine what changes in quality or
effectiveness is needed in the instructional materials prior to
implementation.
Procedure:
The Subject Matter Expert has been contacted to consult on
different aspects of this project. In this phase of the project, the SME
will be contacted via email to review the instructional materials. In the

email (Appendix C), will be the Subject Matter Expert Protocol

(Appendix B), along with the instructional materials,

learner checklist (Appendix G) and lesson evaluation form (Appendix

H). The SME will have one week to review the materials. Once he has
reviewed the materials, a face-to-face interview will be conducted to
go over the open-ended questions (Appendix D) to assess the design
of the lesson. In this session, we will also address specific areas of
concern. This feedback will be used to make necessary changes prior
to learners having access to the instructional materials.
First Learner Review
Subjects
The first learner is a fellow classmate who will be playing the
part of a novice learner. This learner has used basic video recording
software and Internet browsers but is not completely comfortable with
basic computer functions, including finding and saving files. This
learner will represent the below average level of technological abilities
concerning the target audience. The learner will complete the printbased instruction on their fully accessible computers.
Instruments
The instructional designer will use the evaluation protocol
(Appendix E) to describe the evaluation process to the learner. The
learner checklist (Appendix G) is used to determine if all steps in the
proper order have been completed for the lesson. It will be given to the
learner along with all instructional materials. Finally, after the lesson
has been completed, the learner will use the learner evaluation form
(Appendix H) to indicate what changes (if any) should be made
clearer in the lesson content, goals, and objectives.

Procedure
This learner will read the learner evaluation protocol (Appendix
E) prior to going through the print-based procedure, as laid out in the
lesson. They will complete the observer checklist (Appendix G) to
ensure they have completed all steps in order. Once they have
completed the checklist, they will complete the lesson evaluation form
(Appendix H) to determine if all goals and objectives were completed.
They will make a note of what aspects should be reassessed.
Second Learner Review
Subjects
The second learner is a learner with intermediate technical
experience. This learner is a fellow classmate who will be playing the
part of an intermediate learner. This learner is comfortable basic video
recording software, as well as basic computer functions. This learner
has played around with command prompts but is not completely
comfortable with them. This learner will represent the intermediate
level of technological abilities concerning the target audience. The
learner will complete the print-based instruction on their fully
accessible computers Instruments
The instructional designer will use the evaluation protocol to
describe the evaluation process to the learner. The learner checklist
(Appendix G) is used to determine if all steps in the proper order have
been completed for the lesson. It will be given to the learner along with
all instructional materials. Finally, after the lesson has been completed,
the learner will use the lesson evaluation form (Appendix H)

to indicate what changes (if any) should be made clearer in the lesson
content, goals, and objectives.
Procedure
This learner will read the learner evaluation protocol (Appendix
E) prior to going through the print-based procedure, as laid out in the
lesson. They will complete the learner checklist (Appendix G) to
ensure they have completed all steps in order. Once they have
completed the checklist, they will complete the lesson evaluation form
(Appendix H) to determine if all goals and objectives were completed.
They will make note of what aspects should be reassessed.

References
Dick, W., Carey, L., & Carey, J. O. (2015). The systematic design of instruction (8th
ed.). Upper Saddle River, NJ: Pearson.

Appendix
A Overall
Plan
Phase
Exper
t
Revie
ws
One-toOne
Evaluatio
ns

Evaluator
Subject
Matter
Expert
Learner
1:
Beginner
Learner 2:
Intermedia
te

Date
Target
st
December 1 Goals,
Objectives,
&
Lesson
Due by
Content,
December
5th
Assessme
nt
December
Lesson
5th
Content,
Assessme
nt

Instrument
s
Interview
Observer
Checklist
, Lesson
Evaluatio
n
Observer
Checklist
, Lesson
Evaluatio
n

APPENDIX
B
SUBJECT MATTER EXPERT FORMATIVE EVALUATION
PROTOCOL
The Subject Matter Expert will receive the directions as well as the instructional
materials via email. The Subject Matter Expert will participate in the formative
evaluation phase of the design process. He will be given these instructions, along with the
instructional materials. The SME will review the instructional materials prior to meeting
with the instructional designer. At the meeting, the SME will be asked questions to
determine whether the instructional goal: using pre-recorded high-definition video,
students will compress the video in Web-M, VP-9 format, on a computer with full
security allowances, while maintaining similar quality produced in a smaller size and
upload time is met through the instructional materials. The questions will be recorded to
have a record of what changes should be made. The feedback from the SME from this
interview will determine what changes, if any, should be made to make the content,
goals, & objectives. The design of the instruction will be changed based on the feedback
from this interview, prior to giving learners access to the materials.

APPENDIX C
SUBJECT EVALUATION LETTER:
Dear Subject Matter Expert:
Thank you for participating in the project. You have been chosen because of your
expert knowledge in video compression software. The goal of this project is to instruct
higher-Ed instructors of English as a Second Language on how to use the compression
software with a high-definition video so that it is compressed into web-m, VP-9 format,
and maintaining similar quality in a smaller file size that can be uploaded for sharing.
Please review the enclosed instructional materials along with the protocol. We will me
st

meeting on December 1 to go over the instructional materials, where you will be asked a
series of questions to determine the quality and focus of the design materials. Through
these questions, we will determine what changes, if any, should be made in the design of
the instructional materials.
I appreciate the time and knowledge you have contributed to this project
Cybele Wu
Instructional Designer

APPENDIX D
SUBJECT MATTER EXPERT INTERVIEW
QUESTIONS
1. The target audience is ESL higher-Ed instructors. Do you think that the objectives
in the materials are applicable for this target audience? Should they be adjusted? If
so, how?
2. Are the skills that are needed to perform this lesson identified? Are there any
skills that should be added or removed?
3. Do the steps for the instructional materials make sense? If not, how should they
be organized?
4. Is the instructional material fitting for the target audience? If not, what aspect of
the materials should be changed to fit the target audience?
5. Is the vocabulary in the instructional materials clear and easy to understand? If
not, how would your reword them?
6. Are the images present in the instructional materials easy to understand for
learners to complete the lesson? If not, what images should be included or
removed to strengthen lesson?
7. Is the learner checklist form and lesson evaluation form applicable for assessing
the learners performance of the lesson? If not, how would you change the
assessment so that it better assesses the learning outcome?
8. Do you have any comments, questions, or suggestions for either the design or the
instructional materials?

APPENDIX E
LEARNER FORMATIVE EVALUATION
PROTOCOL
After the Subject Matter Expert (SME) has reviewed the instructional materials,
the learners will be sent the learner evaluation protocol, instructional materials, checklist,
and lesson evaluation form.
Each learner will complete all steps in order of the print-based lesson. They will
complete the checklist to ensure they have completed all steps in the instructional
materials. Once the lesson has been completed, they will complete the lesson evaluation
form. The information on this form will help the instructional designer better redesign the
instruction so that all instructional goals and objectives are met.
APPENDIX F
LEARNER EVALUATION
LETTER
Dear
Learner:
Thank you for participating in this project. For this project, you will be
playing the part of an ESL (English as a Second Language) Higher-Ed
instructor. The goal of this lesson is to use open source compression
software to compress videos to Web-M, VP-9 format to maintain
similar quality in a smaller file size for faster uploading speed.
In this project, you will first read the learner protocol, use the
checklist to ensure all items are completed in order. Finally, you will
complete the lesson evaluation. The purpose is to provide feedback
on the lesson design and format to ensure that areas in question are
redesigned so that the goals and objectives are met.
I have enclosed the instructional materials, learner protocol,
checklist, and lesson evaluation. Please complete all sections and

send back to me by December 5 th. Thank you again for your

participation.
Cybele Wu
Instructional Designer

APPENDIX G
LEARNER
CHECKLIST
Checklist for Entry Skills
1.1.1 Learner is able to move around web page in 30 seconds or less.
1.1.1 Learner is able to move their mouse over an image to bring up options
1.6.1 Learner is able to install software correctly.
2.1.1 Learner is able to record a video with their cellphone.
2.3.1 Learner is able to copy files to their computer.
2.3.2 Learner is able to name files on the computer.
3.1.1 Learner is able to use command prompts to locate files.
3.2.1 Learner is able to find where file has been saved.
3.3.1 Learner is able to save file in a different format.
4.3.1 Learner is able to post on social media sites (Facebook, YouTube, Vimeo,
etc.,).for Subordinate Skills for Installing Video Transcoding Software
Checklist
1.1 Learner is able to open a browser (Safari, Internet Explorer, or Chrome)
http://www.ffmpeg.org/download.html.
1.3 Learner is able to access the "Get Packages" section of the web page.
1.4 Learner is able to hover the mouse over the appropriate image for
their operating system.
1.5 Learner has been successful in downloading software.
1.6 Learner has installed software successfully.
1.7 Learner has rebooted computer.
Checklist for Subordinate Skills for Recording 1-minute video (2.0)
(*Note if Video is Pre- recorded go to 3.0)
2.1 Learner has recorded a 1-minute video.
2.2 Learner has connected cellphone to computer via USB cable.
2.3 Learner has copied video recorded onto their computer.
Checklist for Subordinate Skills for Transcoding Video into Web-m, VP9
(3.0) has been successful in bringing up command prompt or terminal
3.1 Learner
format
3.2 Learner has located the file saved using the command prompt.
window
3.3 Learner has saved file as Web-m, VP9 format using the command prompt.
Checklist for Subordinate Skills for Uploading Transcoded video onto
a video-sharing website (4.0)
4.1 Learner has accessed video-sharing website.
4.2 Learner has located the file in Web-m format.
4.3 Learner has successfully uploaded video file in Web-m, VP9 format

APPENDIX H
LESSON EVALUATION
FORM
Yes No Comments
1. The instructional materials met
the criteria of the objectives for
this lesson.
2. The directions are clear and
easy to understand.
3. The images included in the
instructional material were
helpful in finishing the lesson.
4. The steps in the instructional
material were given in an
obvious order.
5. The layout of the
instructional materials had
an appealing
6. The
instructionlook.
provided was
adequate to compress highdefinition video to similar
quality in a smaller file size in
Web-M, VP-9 format for
uploading
and sharing.
7. The
instructional
materials,
including the text and images,
motivated me throughout the
lesson.
8. The
lesson can be completed in
less than thirty minutes.
9. I feel my knowledge is
proficient enough to complete
this lesson again using the
10.instructional
I will use thematerials.
instructional
materials again to compress
high-definition videos to WebM, VP-9 format for sharing and
uploading.