MANSCEN RAPID DEVELOPMENT SUITE (MRDS)

Evaluation of the MRDS Resident Training Course

by LaTrelle Walker

Submitted to MANSCEN G3 Directorate of Training May 2010

TABLE OF CONTENTS
ABSTRACT ................................................................................... 3 INTRODUCTION ........................................................................... 3 PROGRAM DESCRIPTION .......................................................... 4 EVALUATION METHOD ............................................................... 5 RESULTS ...................................................................................... 7 DISCUSSION ................................................................................ 8 PROJECT COST .......................................................................... 8 APPENDIX A ................................................................................. 9

ABSTRACT
As the emphasis to use Distributed Learning (dL) products continues to grow, so does the demand for personnel to possess the knowledge, skills, and abilities to produce them. This demand grows exponentially and continues to gain momentum. When combined, these two ideas form an objective to rapidly develop functional dL production personnel. Currently, dL development training consists of a one-day resident course environment. The students/employees are trained on the utilization of a local software platform called the MANSCEN Rapid Development Suite (MRDS). The MRDS serves as a production tool to assist entry-level staff members in becoming fully functional dL production personnel. The purpose of this evaluation is to determine the feasibility of continuing this course in resident form and to modify the current lesson plan to enhance the efficiency and effectiveness of its delivery. This will improve the learning outcomes of the personnel trained and provide an enduring reach back capability to existing staff. As a direct result of this evaluation, alternative solutions will be identified that could directly affect the design, delivery, longevity, and future evolution of this course.

INTRODUCTION
The MRDS is used as a dL courseware development tool. It is designed to standardize the user interface and content layout for courseware development, reducing design and development time. It also separates content from the user interface, making it easier to perform updates to existing dL products. Developers work with lesson page and exam question templates in the user interface. No design or programming knowledge is required due to the template-driven interface. The infrastructure of the MRDS is comprised of Microsoft Access databases. The databases are used to store content and media files for storyboard and exam designs. The system is comprised of two separate databases: one for the lesson, and one for the exam. The databases contain tables that form the lesson or exam storyboard. As previously mentioned, the purpose of this evaluation is to identify alternative solutions to the design and delivery of the MRDS resident course. The goal is to improve upon the current curriculum, increasing the efficiency and effectiveness of the course materials while simultaneously improving learner outcomes. The evaluation will consist of a comprehensive review of the lesson plan, an audit of the resident course, and a solicitation of information from former students. The results of the evaluation will help determine if the MRDS is antiquated technology. The concept may be enduring; however, the user interface may need to be redesigned. A

redesigned user interface could expedite training of new personnel and provide the software platform, needed technology upgrades, and integration capabilities that align with industry standards and organizational directives. Additionally, it will help establish if the resident course is still necessary or if an online version would be more cost effective.

PROGRAM DESCRIPTION
The target audience for MRDS training is personnel who have a direct responsibility for dL production. Production includes creating new courseware, editing/updating current courseware, converting resident materials to dL courseware, and contracting oversights of dL courseware that have been outsourced. The MRDS resident course is designed to provide personnel with the minimal amount of training necessary to become independent and fully functional dL production staff members. The course includes a compact disk (CD) containing MRDS software, student handouts, and 5 hours of resident instruction. Program Objectives The goal of MRDS training is to rapidly train staff members with minimal training requirements, enabling personnel to produce Shareable Content Object Reference Model (SCORM) conformant dL courseware. Course Objectives: TLO: Create a Shareable Content Object (SCO) file. ELO: Familiarize students in the MRDS process. ELO: Download MRDS files. ELO: Create a new Database Editor for Enhanced Learning (DEEL). ELO: Publish the Shared Content Object. ELO: Create a new database Exam Editor for Enhanced Learning (EEEL). ELO: Publish a lesson. ELO: Review the Comment Viewer for Enhanced Learning (CVEL). Terminal Leaning Objective = TLO Program Components The MRDS consists of databases that contain content and media descriptor files for lesson and exam storyboard designs. Each database includes the StoryboarderTable. The StoryboarderTable is the main lesson table that consists of records that correspond with single pages or screens of the storyboard. Enabling Learning Objective = ELO

QuestionTable. The QuestionTable is the main exam table that consists of records that correspond with exam questions. Each record is comprised of fields which hold the information describing what is to be displayed (text and media) on the page.

The Overview and Process components of the class includes Creating records (pages). Deleting records. Editing records. The MRDS creates and maintains a storyboard in an established database (dB) structure. Each record consists of fields containing information that describes to the data that will be displayed on the page. Editing the StoryboarderTable is accomplished by using the DEEL. The Publisher for Enhanced Learning (PEL) combines storyboard content and media within the template. The PEL accommodates publishing in three different SCORM versions (CD, SCORM 1.2, or SCORM 2004) that are based on the user's selection criteria. Designers/developers use the CVEL to view comments, make modifications in the DEEL or EEEL, and republish the SCO with the PEL or PEEL.

EVALUATION METHOD
Participants The evaluation team will consist of an instructional designer, program evaluator, programmer/software specialist and instructor. Each individuals normal duties should closely align with the following duty descriptions. Team Member 1 - Designer Educational and training expert, providing professional education and training advice and consultation services for the accomplishment of organizational objectives in the systematic analysis, design, development, implementation and evaluation of virtual, constructive and live advanced training simulations. Provides technical supervision and guidance to an integrated staff of instructors, subject matter experts, training specialists, training technicians, simulation analysts, database analysts and hardware managers and repairers. Advice relates directly to the phases of a systems approach to training (SAT) and its application to training simulation design and development, distributive simulations and other applications of high technology to training. Develops and implements long term, Modeling and Simulation Master Plan and training strategies.

Team Member 2 - Evaluator Provides solutions for operational deficiencies in training development areas by experiments with emerging technology and emerging concepts. Applies professional knowledge, concepts and principles common to statistics, computer science, operations research, artificial intelligence and related sciences to the development, study, analysis, modification, improvement and documentation of training and simulations. Evaluates complex research problems relating to communications, hardware/software, operating systems, computer graphics, and virtual reality systems as they relate to live, virtual, and constructive simulations. Team Member 3 - Programmer Develops and maintains the Web Page hierarchies: Effects logical organization and presentation of information. Available guidelines are general in nature and frequently change; they consist of existing and emerging standards, (i.e., World Wide Web, communications standards, and HTML). Due to the constantly changing nature, they require research, interpretation, cross-reference, and in some instances deconfliction to determine their applicability to a specific assignment. Incumbent must evaluate technical ramifications using of HTML, XML, JavaScript, CGI, ASP and other selected tools that apply to web page design Team Member 4 - Instructor Researches, develops, maintains, and updates material for courses typically covering upper-division undergraduate equivalent level courses, in advanced technical digital systems. Resolves problems with source materials, subject matter and/or students. Determines the need for and initiates changes/updates in course content. Participates substantially in original course development or course modification. Given the aggressive timeline for completion the team members were selected based on their currency and level of experience in the field. It was determined to be critical to have not only the educational background but also the application and work experience to support it. Therefore individuals were recruited based on three factors; an advanced degree (graduate or higher) in their chosen field, current job position and years of experience in the field, and a sample of work. Procedures On day one the evaluation team had a meeting with corresponding team members from the organization. At the meeting all the course materials were made available so the team could immediately begin the analysis of the software, student handouts and presentation materials, the team was also provided with a list of students who had formerly attended the course and that have a direct responsibility for dL production. It was pre-arranged for these personnel to be available for questioning when the team arrived. The instructor (who happened to be the designer of the software as well) was made available to the team for the duration of the evaluation, except for the

time that would be spent teaching the MRDS course on day two. Day two of the evaluation was spent auditing the course delivery and continuing analysis of course materials. Day three was focused on identifying any needed changes as well as appropriate solutions and integration strategies for those changes. Data Sources Data was collected and analyzed on the following factors throughout the duration of the evaluation. The design of the software and user interface that is known as the MRDS, this software was compared to many mainstream industry tools that are currently at the leading edge of technology development. The techniques used to deliver the content of the course itself, with particular emphasis on is there a better way to do this and the student handouts/course materials were also analyzed for currency and compliance with industry best practices.

RESULTS
The concept of rapidly training personnel to be functional dL developers in a short time frame is not only a necessity but it is an enduring concept, and in this day and age it is also an achievable goal. However the overall design of the MRDS is antiquated and is in need of an upgrade. The user interface is uncommon to todays user and is built on database technology that is rarely used in mainstream industry. The user interface needs to have a more up-to-date look and feel, such as a browser look with more graphical symbols that require less reading and more recognition for the user. A more intuitive platform would also be beneficial as the platform could then accelerate or decelerate depending on the users ability. The delivery of software application skills is best taught in an online environment where the user can watch (a tutorial) and do as they are being instructed, this too provides a reach back capability and a self paced environment for each individual learner. This course, if revised will enhance the current workforce and streamline operations across the organization. In further attempt to justify revising this course two surveys were conducted; one for supervisors to help determine what they want, need, and are currently working on as well as the tools they currently use to produce their particular products. The second survey was targeted at users, to assess individual skill levels and knowledge in the topical areas. The results of these surveys have been included in the final proposed plan for changes to training.

DISCUSSION
The purpose of this evaluation was to determine if the MRDS resident course is aligned with what industry considers best practices. Special emphasis was placed of the lesson plans, delivery techniques and the user interface. An analysis was conducted to also determine if changes were made to the program could a reach back capability be added into the solution. This study reveals the potential that can be gained through the proper training of dL development personnel. The simple integration of an effective authoring program that already exists, but is in need of some re-designing as well as some technical upgrades and a new means of delivery, can be used to develop the SCORM conformant dL products your organization desires. Again the concept of a tool that provides rapid equipping of personnel to be fully functional dL developers is a noble and enduring cause. This can be evidenced in many fields in mainstream industry. Software companies have recognized this and are tailoring products towards the non-user more often than not. Likewise a tool such as the MRDS that has a target audience of non-designers should have a more non-designer user interface as opposed to one designed for a computer programmer.

PROJECT COST
Personnel Amount per day Amount per day + Expenses

Team Member 1 Team Member 2 Team Member 3 Team Member 4

$500.00 x 3 $500.00 x 3 $500.00 x 3 Outside funding source N/A

$2252.00 $2252.00 $2252.00

Administrative Sub total

$300 $6756.00

Total number of days = 3

Total cost: $ 7056.00

APPENDIX A
Survey one - (for supervisors)

Check the appropriate response 1. Create/optimize multimedia products for delivery on the web I have a current need for this skill I anticipate a future need for this skill I do not need this skill 2. Create new SCORM conformant courses/modules I have a current need for this skill I anticipate a future need for this skill I do not need this skill 8. 3. Edit existing SCORM conformant courses/modules I have a current need for this skill I anticipate a future need for this skill I do not need this skill 9. 4. Upload SCORM conformant packages into an Learning Management System I have a current need for this skill I anticipate a future need for this skill I do not need this skill 5. Troubleshoot existing SCORM conformant products I have a current need for this skill I anticipate a future need for this skill 7. I do not need this skill Storyboarding I have a current need for this skill I anticipate a future need for this skill I do not need this skill dL course design I have a current need for this skill I anticipate a future need for this skill I do not need this skill dL course development I have a current need for this skill I anticipate a future need for this skill I do not need this skill

6.

Knowledge and discernment of what is and is not acceptable concerning contract deliverables I have a current need for this skill I anticipate a future need for this skill I do not need this skill

10. Converting resident products to dL I have a current need for this skill I anticipate a future need for this skill

I do not need this skill

Please provide us with a little information about your organization and responsibilities. (Circle your response) 1. How many training developers work in your organization, Directorate, School? (Please do not include contractors.) 1-5 5-10 More than 10 None

2. How many training development projects are completed by contractors in any given month? 1-5 5-10 More than 10 None

3. How many SCORM modules does your organization create/generate in any given month? 1-5 5-10 More than 10 None

4. How many SCORM compliant course modules does your organization troubleshoot or edit in any given month? 1-5 5-10 More than 10 None

5. How many distance learning courses does your organization currently manage and edit?

10

1-5

5-10

More than 10

None

Summary: The findings of the survey reveal that the subordinate agencies and their training development staff are currently undermanned. The amount of work that is expected is unmanageable evidenced by the amount of work that has been outsourced to contractors for completion. It is obvious that it will be expected that dL production personnel have a high knowledge and experience in dL production operations.

11

Survey two - (For Users)

Check the appropriate response (some questions may have more than one response) 1. What is your knowledge of computers? None Basic Intermediate Advanced What operating system(s) are you comfortable with? (check all that apply) None Windows Linux/Unix MAC What software applications are you proficient with (check all that apply) Photoshop or Fireworks Dreamweaver Flash MRDS What is your level of knowledge with MRDS None Basic Intermediate Advanced How many Learning Management Systems are you proficient with 0-1 2-3 3-4 5+ What is your level of programming knowledge None Basic Intermediate Advanced 7. What scripting languages are you proficient with (check all that apply) HTML/XHTML CSS JavaScript XML Action Script VB Script What is you level of knowledge with SCORM None Basic Intermediate Advanced What is your level of knowledge in database theory None Basic Intermediate Advanced

2.

8.

3.

9.

4.

5.

10. What is your skill level in instructional design None Basic Intermediate Advanced 11. What is knowledge level of synchronous vs. asynchronous delivery None Basic Intermediate Advanced 12. What is your degree in Education Instructional Design

6.

12

Summary: Users are dL production personnel therefore it is no surprise that after reviewing the findings the overall determination is that the base skill level of the entire user group is very low. The majority of them have little to no programming experience and their web development knowledge is limited to basic HTML.

13