How We Rate Interactive Media: About the

Ratings, and CTR's Software Evaluation

Instrument
Here's the evaluation instrument that we use to do our full-length reviews
(below).
Ratings Key
If you're looking for an easy to print form to use with test families (especially
Rating good for video games, here's the Serious Games Testers Evaluation form, used by
Spread the Mediatech foundation. It is a PDF. Please keep in mind that these forms are
copyrighted. If you use them, please attribute the source, and don't modify them
The without a proper reference. If you do use them and have a suggestion, please feel
Instrument free to contact the author, Warren Buckleitner (warren at childrenssoftware.com).

The six categories help you keep in mind what it takes to make a quality
children's software title. This instrument favors software that is easy to use, child
Read the controlled, has solid educational content, is engaging and fun, is designed with
article: features you'd expect to see, and is worth the money.
The State Of
Children's To be safe, we recommend staying with a rating above 4 stars. CTR editors
Software typically recommend programs that receive a 4.3 star rating or better. You can
Evaluation-- easily search for these titles in the Children's Software Finder (just enter 4.2 or
Yesterday, greater for the rating search field.
Today And
In The
21stCentury Some Interesting Facts About the Ratings
The ratings are assigned by educators (CTR reviewers) who have been "trained"
on the use of the instrument below. The word trained means that they have
achieved inter-rater reliability when rating the same title independently. In cases
when two reviewers come up with different ratings, a third reviewer is consulted,
along with additional child testing until all raters "can sleep with" the rating. The
system is not perfect; but it attempts to be the "least worst" rating system. In
assigning ratings, the reviewers consider feedback from test schools and families.

Key: 1=dud, 2=poor, 3=fair, 4=good, 5=excellent

CTR's Checklist for Quality
To get a program's 1-5 star rating, you need to do some simple math; a process
that is automated for reviewers using the instrument, but is defined in detail here,
for replication purposes. Add up the points in each category (always = 1
point, some extent = .5 points, never = 0 points, and N.A. = Not Averaged) and
then divide by the number of items in the category. This number can then be
converted to a 0 to 5 point scale. It is important to match the instrument with the
type of software. In other words, you can't rate a program low in "Educational
Value" if it is designed primarily as a game. Even though That's where the "N.A."
category comes in. Finally, it is very important to consider the date that the
review was written (in the Children's Software Finder, all you can do is check the
copyright date, which is a searchable field). Remember that a highly rated
program in 1993 might be equal to a poorly rated program in the context of
current day software and hardware.For a more in-depth discussion of the art and
craft of software evaluation, consult the article by CTR editor Warren
Buckleitner: The State Of Children's Software Evaluation-- Yesterday, Today
And In The 21st Century.
TEACHERS & LIBRARIANS: PERMISSION TO COPY THIS PAGE IS GRANTED

Children's Software Evaluation Instrument

© 1998 Children's Technology Review

I. Ease of Use (Can a child can use it with minimal help?)

A
S. E. N. N.A. S. E. N N.A.
1
__ __ __ __ Skills needed to operate the program are in range of the child
2
__ __ __ __ Children can use the program independently after the first use
3
__ __ __ __ Accessing key menus is straightforward
4
__ __ __ __ Reading ability is not prerequisite to using the program
5
__ __ __ __ Graphics make sense to the intended user
6
__ __ __ __ Printing routines are simple
7
__ __ __ __ It is easy to get in or out of any activity at any point
8
9 __ __ __ __ Getting to the first menu is quick and easy
10 __ __ __ __ Controls are responsive to the touch
11 __ __ __ __ Written materials are helpful
12 __ __ __ __ Instructions can be reviewed on the screen, if necessary
13 __ __ __ __ Children know if they make a mistake
14 __ __ __ __ Icons are large and easy to select with a moving cursor
__ __ __ __ Installation procedure is straightforward and easy to do

II. Childproof (Is it designed with "child-reality" in mind?)

1 __ __ __ __ Survives the "pound on the keyboard" test

2 __ __ __ __ Offers quick, clear, obvious response to a child?s action
3 __ __ __ __ The child has control over the rate of display
4 __ __ __ __ The child has control over exiting at any time
5 __ __ __ __ The child has control over the order of the display
6 __ __ __ __ Title screen sequence is brief or can be bypassed
7 __ __ __ __ When a child holds a key down, only one input is sent to the computer
8 __ __ __ __ Files not intended for children are safe
9 __ __ __ __ Children know when they've made a mistake
10 __ __ __ __ This program would operate smoothly in a home or classroom setting

III. Educational (What can a child learn from this program?)

1 __ __ __ __ Offers a good presentation of one or more content areas

2 __ __ __ __ Graphics do not detract from the program's educational intentions
3 __ __ __ __ Feedback employs meaningful graphic and sound capabilities
4 __ __ __ __ Speech is used
5 __ __ __ __ The presentation is novel with each use
6 __ __ __ __ Good challenge range (this program will grow with the child)
7 __ __ __ __ Feedback reinforces content (embedded reinforcements are used)
8 __ __ __ __ Program elements match direct experiences
9 __ __ __ __ Content is free from gender bias
10 __ __ __ __ Content is free from ethnic bias
11 __ __ __ __ A child's ideas can be incorporated into the program
12 __ __ __ __ The program comes with strategies to extend the learning
13 __ __ __ __ There is a sufficient amount of content

IV. Entertaining (Is this program fun to use?)

1 __ __ __ __ The program is enjoyable to use

2 __ __ __ __ Graphics are meaningful and enjoyed by children
3 __ __ __ __ This program is appealing to a wide audience
4 __ __ __ __ Children return to this program time after time
5 __ __ __ __ Random generation techniques are employed in the design
6 __ __ __ __ Speech and sounds are meaningful to children
7 __ __ __ __ Challenge is fluid, or a child can select own level.
8 __ __ __ __ The program is responsive to a child's actions
9 __ __ __ __ The theme of the program is meaningful to children

V. Design Features (How "smart" is this program?)

1 __ __ __ __ The program has speech capacity

2 __ __ __ __ Has printing capacity
3 __ __ __ __ Keeps records of child's work
4 __ __ __ __ "Branches" automatically: challenge level is fluid
5 __ __ __ __ A child's ideas can be incorporated into the program.
 6 __ __ __ __ Sound can be toggled or adjusted
7 __ __ __ __ Feedback is customized in some way to the individual child
8 __ __ __ __ Program keeps a history of the child's use over a period of time
9 __ __ __ __ Teacher/parent options are easy to find and use

VI. Value (How much does it cost vs. what it does? Is it worth it?)
Considering the factors rated above, and the average retail price of software, rate this program's
relative value considering the current software market. Consider also any extra hardware
attachments required to get full potential of the programming, e.g., a sound card, CD-ROM, etc.

Poor Good
__ __ __ __ __ __ __ __ __ __
1 2 3 4 5 6 7 8 9 10

Don't forget that this form is generic! To use it properly, you have to look at a lot
of similarly designed products, and that the "NA" field is particularly powerful in
the overall score.

Home | Children's Software Finder (TM) | Write A Review

Subscribe to Children's Technology Review| About Children's Technology Review |

Submit a Product

Contact us

Disclaimer and Copyright © 2007 Active Learning Associates, Inc. All rights reserved.

This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative

Works 3.0 United States License.