TEACHER.

TV DETAILED DESIGN,
FORMATIVE AND SUMMATIVE EVALUATION

Fostering creative thinking

skills in first-year students of
the graphic design program
at Cgep du Vieux-Montral
by milie Ren-Vronneau
ETEC651
Nadia Naffi
Concordia University

TABLE OF
CONTENTS
3 Rationale for this intervention
4 Objectives
4 Actions to be taken prior to production
4 Defining guidelines
5 Production schedule
6 Homepage
6 Footer
7 Registration process
8 Channel landing page
9 Broadcast front-end

COVER SOURCE Nordisk familjebok (1876, 1904 or 1923)

10 Broadcast back-end
11 Conducting the formative and summative evaluations
12 Formative evaluation - Usability and technical review
13 Formative evaluation - Content and editorial review
14 Summative evaluation - Performer engagement
15 Summative evaluation - Individual performance
15 Summative evaluation - Performance transfer
15 Summative evaluation - Organizational performance
16 References

Rationale for
this intervention
This intervention aims to establish an innovative and multi-institutional platform where
students and teachers can gather in an online
informal learning setting.

As viewers tune in the broadcasters channel, they access a chatroom where all can discuss what is happening and ask questions. The
broadcaster reads the chatroom every once in a
while and can discuss with the viewers.
This kind of informal and relaxed setting lends
itself to highly situated questions. While it can
still take place within educational institutions,
informal learning covers learning that takes
place outside of the required curricula of formal learning. Depending on the intention and
awareness of the learner, it can take the form
of self-directed learning, incidental learning or
socialization. (Schugurensky, 2000)
Intentionality

Awareness

the graphic design classroom

Form
Self-directed

yes

yes

For a long time, graphic design was a trade

which was taught through apprenticeships.
Nowadays taught in college, classes have a limited attendance of 25 students and most of the
time spent teaching consists of giving feedback
and coaching on unique projects.
In the current setting, there is very little
time for the teacher to show students how they
would go about solving a problem or creating
something in their daily practice. Moreover,
the graphic design program at Cgep du VieuxMontral does not include an internship in
the field. These internships can be difficult to
access and even when found, employers are
busy and may not have time to take proper care
of the student.
Therefore, the students have limited opportunities to observe expert behaviors which
could inspire their own practice.

Incidental

no

yes

Socialization

no

no

new opportunities
for informal learning

The speed of todays Internet connections

enables people to communicate with each
other through live online video.
In the past years, the gaming community has
seen the rise of video streaming platforms such
as Twitch.tv, Livestream and Ustream. These
platforms allow people who are playing, working, or experimenting to share and broadcast
their experience online, as if they were on
television.

SOURCEAdapted from Schugurensky, (2000)

The platform supports all three types of

informal learning. As a self-directed learner,
a viewer may ask the broadcaster a specific
question and add a timestamp to the provided
answer for future retrieval. Learning incidentally, the viewer may benefit from someone
elses question, or simply from information
provided by the broadcaster. In terms of socialization, the viewer may simply be interacting
with other viewers and broadcasters, becoming aware that they have learned something at
a later time.
timestamps

Marsick and Watkins (2001) claim that

informal learning can be improved through
facilitation and increased awareness from
the learner. The timestamp feature serves to
facilitate and increase that awareness. It also
allows the platform administrators to assess
the type of content which viewers value most
and makes the platform different from other
existing alternatives.
Timestamps allow viewers to keep track of
moments they value during broadcasts and
gather a collection they are able to revisit
as needed to inform their own thinking or
practices.

Objectives
This intervention caters to all levels of performers and mainly addresses the following
performance objectives:
performance objectives
for students

Given information about creativity and creative

thinking as well as proper models of behavior,
performers apply techniques to leverage their
creative skills.
performance objectives
for teachers

Given information about creativity, creative

thinking and reflexive teaching, the performers
explain their own creative processes by modelling creative behaviors to the students, facilitating the students acquisition of key creative
personal characteristics.
performance objectives
for all performers

Given a positive environment where creative

behaviors are valued, performers interact in
playful, cooperative and trusting ways to facilitate creative outcomes and processes.

Actions to be taken prior

to production
In order to get this initiative started, funding
will need to be secured from the Programme
daide la recherch sur lenseignement et lapprentissage (parea), offered by the Ministre
de lducation, Enseignement suprieur et
Recherche. If the first iteration is successful,
more sponsors will be involved to grow the project to other institutions.
A committee made of pedagogical advisors
and teachers interested in innovative teaching practices and technology will be formed to
administer the platform.
In order to stay focused on our goal of passing tacit knowledge from expert to novice,
only teachers will be allowed to broadcast. Any
teacher interested in broadcasting can simply
sign up on the site using their institutional
email. At the beginning of the school year,
teachers who have manifested interest will be
met at their institution to receive background
information and training for the intervention, discuss ideas for presentations, and draw
inspiration from each other.
For the success of this intervention, it is crucial that all performers believe in the idea and
see the value that informal learning can bring
to students first.
As additional institutions join the initiative,
they will be expected to advertise the platform
through their institutional publications and
occasional advertisements.

Defining guidelines

Production schedule

Because of the innovative nature of this intervention, some training content will be required.
Concerns are also expected to arise in terms of
managing the community, therefore best practices need to be defined.

Production of the platform will undergo the

following steps. Some of these steps may overlap in order to speed up the production process.
The time from drafting the platform to launch
at a first institution is estimated to one year. If
the first initiative is successful, other institutions will be invited to join and a mobile version
will also be produced.

video and audio production training

A video production expert will be required

to create a training which will aim at ensuring basic proficiency in shooting video and
recording audio when using the platform, as
well as establishing best practices for a variety
of broadcasting conditions.
terms and privacy guidelines

Interactions between all participants should be

respectful. While each broadcaster is allowed
to define specific rules, no bullying or harassement should be tolerated.
Although the learning context is informal,
broadcasters need to be aware that they are
revealing their world to anyone registered to
the site and that their privacy is not protected.
For example, teachers should not engage in
illegal behaviors such as consuming illicit substances or revealing confidential information.
Since this type of intervention is uncommon,
a social media expert, as well as a usability
expert from Ustream will be asked to define
appropriate and unappropriate behaviors based
on their experience. These guidelines will be
used to define the terms for the platform.

platform development

Review of current wireframes

and production of any missing

wireframes with usability expert

Integration of wireframes

and development of the platform


Pilot study, usability and load testing

Review of platform according

to pilot study and load testing


Integration of graphic design to platform

Quality control and debugging

Official launch

strategy and communications


Branding and graphic design of platform

Advertisement strategy for enlisting

broadcasters and viewers


Identification of potential broadcasters

and viewers for early adoption


Focus group on branding, graphic design

and advertisement strategy


Review of branding, graphic design

and advertisement strategy according

to focus group

Advertisement campaign


Official launch

LOGO
Home /

Browse

Search by institution, teacher or interest...

Currently live
Julie Sigouin
Logo sketches

Lambert Ouellet
Air Chamber chair

Daniel Lvesque

Experimenting with Bootstrap

Lambert Ouellet Consectetur adipisicing elit, sed do eiusmod tempor incididunt

ut labore et dolore magna aliqua. Ut enim ad minim veniam.

Tune in!

Jose Dallaire
Asian Bookbinding

Upcoming events
March 3rd 2015 10 pm

March 11th 2015, 9 am

March 20th 2015, 10 pm

April 2nd 2015, 10 am to 12 pm

Luc Girouard

milie Ren-Vronneau

Christian Blanger

Franois Martin

Cgep du Vieux-Montral
Drone photography

About

Help

Cgep du Vieux-Montral
Letterpress printing

Terms and Privacy

FAQ

Collge Marie-Victorin
Hand lettering

Dawson College
Weekend Sessions

Contact Us

Footer
The footer contains some of the standard links
for websites.
about

The about page gives more information on

the rationale of this intervention as well as
some background information about informal
learning.
help

The help section contains documentation to

guide users through the use of the sites features, as well as a few video tutorials to help
broadcasters get started.

Homepage

The homepage presents featured channels

based on popularity and trends as well as
upcoming events. Once the students log into
their account, these elements are adapted to
better represent the interests of the students
as defined in their user profile. Users also have
the possibility to search directly for what they
are looking for by using the search field.
Users can navigate events and channels
through different means: they can pick among
the choices suggested by the platform on the
homepage, they can search for a specific institution, teacher or topic in the search form, or
they can browse through a list organized with
the same categories.
As they add events or subscribe to different
channels or topics, users get email notifications
which bring them directly to the events they
want to participate in. Notifications are sent a
day in advance, as well as when the broadcast
goes live.Users can return to their account settings by clicking on their avatar in the top right
of the screen.

terms and privacy

This page contains the terms of service and

privacy policy for the platform.
faq

The FAQ (Frequently asked questions) page

regroups questions that users may have and
that are not answered by the help file or that
do not fit its format in one knowledge base.
An example could be, "How can I become a
moderator?"
contact us

The contact form is used to provide means to

reach the administrators of the site.

LOGO
Home / Your account

Browse

Your Profile

Search by institution, teacher or interest...

Search...

ID 7271646

Bookmarked moments

Name Camille Lehoux

February 15th 2015

Department Graphisme

 0:45:31

Institution Cgep du Vieux-Montral

Nickname

Camillou

About you

Tell us more about you...

Cool technique!
Use for series of
circus posters

Upload picture

Logo sketching
February 2th 2015
 1:01:35
Nice explanation

Your channels

Manage your subscribed channels

Your interests

Web design

Printing

Graphic design

Photography

T-shirt printing
December 18th 2014
 0:20:36
Funny! Trying to
combine things
that are opposite

Save changes

Brainstorming for chair

About

Help

Terms and Privacy

FAQ

...

Contact Us

Registration process

When first signing up for the service, students

pick their institution and department from a
dropdown menu and input their student ID.
This creates a temporary password that is then
sent to their institutional emails. They can log
in using their student ID and password to view
their profile and change their settings.

In order to get personalized recommendations,

students can choose among a variety of interests which are linked to the taxonomy used in
the site. While they can subscribe and unsubscribe from specific channel pages, a users
subscriptions can also be managed from the
account page.

profile

timestamps

Since informal learning should be a fairly

comfortable context where students feel free
to ask questions, they will be allowed to pick
any nickname or avatar they wish to use. Their
account remains linked to their student ID in
cases where there would be a necessity to identify them.

Timestamps collected during the different

broadcasts are stored in the users profiles. The
event and time of the marking are stored and
students can add a short description. A search
engine assists the students in finding those
moments again..

LOGO
Home / Cgep du Vieux-Montral / Lambert Ouellet

Browse

Lambert Ouellet

Subscribed

Search by institution, teacher or interest...

Past events
February 20th 2015
 2:45:31
 86

 graphic design,

logo, ideas

Logo sketching
February 4th 2015
 1:33:31
 117

 printing, t-shirt

silkscreen

98

OFF-AIR

140


T-shirt printing

About Lambert Ouellet

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin nisi orci, vulputate sed ligula quis, posuere
condimentum ipsum. Donec pulvinar fringilla lobortis. Integer non tristique nisl. Morbi vestibulum sed
lectus in eleifend. In aliquam consectetur metus, sit amet varius elit sagittis ac. Mauris tincidunt eros in
sapien sodales euismod. Proin magna ligula, dapibus ut blandit vitae, lobortis eu lacus. Fusce lobortis
consectetur urna sed volutpat. Integer at ex ut orci volutpat dictum id eget libero. Donec sed iaculis est,
ac dictum arcu. Nulla sollicitudin felis sit amet ante sodales ultricies. Cras eu urna a nibh venenatis
malesuada fringilla non nulla. Pellentesque eget nibh at justo vulputate malesuada in vel velit.

December 20th 2014

Quisque volutpat, purus sed pretium imperdiet.

Christmas cards printing

 2:01:25
 15

 printing, christmas

silkscreen

About

Help

Terms and Privacy

FAQ

Contact Us

Channel landing page

The channel pages are homepages for specific
broadcasters. If the broadcaster is on-air, the
viewer can tune in directly to the event. If the
broadcaster is off-air, an image that represents
the channel is shown instead. Broadcasters can
enter custom text to describe the objectives of
their channels or rules that they want to establish. An archive of past events is found in the
sidebar.

...

LOGO
Home / Cgep du Vieux-Montral / Lambert Ouellet

Browse

Air chamber chair -- Sunday, March 29th, 2015

Search by institution, teacher or interest...

Lambert Ouellet

Subscribed

Viewer1: What kind of wood are you using?

Viewer8: earlier he said its cedar wood
Viewer2: are they bike air chambers?
Viewer4: @Viewer6 did u study at UQAM?
Viewer6: @Viiewer4 Im doing the BA right now
Viewer4: @Viewer6 cool
Viewer8: are you bringing it to class next week? 
Viewer6: but were on strike for now
Viewer12: Lambert! Long time no see! What are
you up to this evening?
Timestamp
45:31

About

Help

Terms and Privacy

FAQ

14

Contact Us

Broadcast front-end
As the teacher broadcasts, viewers tune in
using the front-end interface and ask questions
or comment. The teacher scours the chatroom
window every now and then to answer questions verbally. Viewers can attempt to answer
others questions if the broadcaster delays in
providing answers. This allows to foster a helpful environment where everyone can benefit
from each others knowledge.
As time goes by, the broadcaster may notice
that some viewers are regulars and would
make good community moderators. When
viewers are promoted to moderate a channel,
they retain this right unless it is removed from
them by the broadcaster. Moderators have the
ability to warn or exclude other viewers from
the stream.
addressing potential issues

The wrench and flag buttons allow to to warn

the broadcaster that audio or video are not
working properly so they can modify their settings or to signal inappropriate behavior to the
platform administrators.
Every flag received by a broadcaster will be
reviewed by the administrators of the platform. Depending on the seriousness of the

inappropriate behavior, sanctions can range

from a warning to being prevented from
broadcasting.
Broadcasters can flag users who misbehave
in order to deliver a first warning. For every
other subsequent flag, the viewer gets locked
out of the channel, and then the platform itself,
for an increasing amount of time. After a number of warnings, the user gets banned from the
platform permanently. A banned account can
only be readmitted to the platform on the condition that the banned user reaches an agreement with the administrators.
adding a timestamp

The add timestamp button allows viewers to

create a bookmark of what is happening in the
video. When clicked, the user can adjust the
time of stamping and add a short description.
These collected moments are stored into the
users profile page.

LOGO

Start broadcasting

Home / Cgep du Vieux-Montral / Lambert Ouellet

Browse

Search by institution, teacher or interest...

Edit broadcast title

OFF-AIR

Tags for this broadcast

Share your broadcast with your followers:

Enter your custom message here.

+
45:31

Share

Broadcast back-end

Start broadcast
Choose video source

4:3 16:9

ON

Choose PIP source

OFF

Quality
Choose audio source

OFF

Volume

About

Help

Terms and Privacy

FAQ

Contact Us

LOGO

Start broadcasting

Home / Cgep du Vieux-Montral / Lambert Ouellet

Browse

Search by institution, teacher or interest...

Air chamber chair -- Sunday, March 29th, 2015

OFF-AIR

Viewer1: What kind of wood are you using?

Viewer8: earlier he said its cedar wood
Promote
to moderator
Viewer2: are they
bike air chambers?
Viewer4: Viewer6 did u study at UQAM?
View user profile
Viewer6: Viiewer4 Im doing the BA right now
Flag for inappropriate behavior
Viewer4: Viewer6 cool
Viewer8: are you bringing it to class next week? 
Viewer6: but were on strike for now
Viewer12: Lambert! Long time no see! What are
you up to this evening?
45:31

Start broadcast

Private

Choose video source

4:3 16:9

ON

Choose PIP source

OFF

Quality
Choose audio source

OFF

Volume

About

10

Help

Terms and Privacy

FAQ

Contact Us

In order to share and control their broadcast,

teachers have access to the back-end interface.
Once they start broadcasting, the chatroom
takes the place of sharing options on the right
of the screen.
This interface allows them to edit the details
for their event and share it with their subscribers and social media.
The broadcaster back-end is used to set up
and troubleshoot issues during broadcasting.
Before broadcasting, the teacher can fill in the
details for the title, as well as relevant tags.
Using the settings at the bottom of the screen,
the broadcaster chooses what video and audio
sources they wish to use as well as the quality
and volume.
Depending on the activity undertaken, the
teacher may want to use a camera or capture
what is happening on their screen and add a
picture-in-picture view of themselves.
There are no set ways to interact with viewers other than following the minimal guidelines
set by the platform in terms of ethics and privacy. As such, interaction styles may differ from
one broadcaster to the next.

Conducting the formative

and summative evaluations

SOURCENordisk familjebok (1876, 1904 or 1923)

The following pages define the formative and

summative evaluations for this intervention.
participants

Depending on which aspect of the platform

is being evaluated, administrators of the platform, teachers from relevant disciplines, as
well as previously consulted experts in video
and audio production, usability and social
media will participate.
Viewers will also be solicited through questionnaires. For usability testing, five viewers of
different abilities will test the platform. While
this may seem a small number of participants,
involving five testers is usually sufficient to
uncover 85% of usability issues. (Nielsen, 2009)

11

evaluation instruments
and frequency of evaluation

The evaluation instruments need to offer some

flexibility as creativity is a complex phenomenon. Interviews will mostly be used when
consulting with broadcasters, as they are fewer
in number, while viewers will fill out questionnaires with open-ended questions.
The metrics provided by the platform will
also allow to further investigate the behaviors
of all performers.
Evaluations will normally be conducted on
the yearly basis but multiple evaluations will be
conducted when first establishing the platform
as we want to ensure that a core of performers
create a sustainable online culture.

Source

Tool

Data collected

When

Rationale

Questionnaire - Likert-scale questions

The site meets standard guidelines for accessibility

12

Questionnaire
and interview

Links are working

The site is well-designed on desktop, tablet and mobile iterations
The site is functional on current browsers
The site is functional on current browsers
For any item not meeting at least 3 points out of 5, further information
will be gathered through interviews in order to explore and resolve usability issues.
Questionnaire - Open-ended questions
Are there any usability issues with the platform? Explain.

Before
implementation,
Assessing usability
after the first delivery
of the intervention
and on a yearly basis

Are the technical guidelines relevant, user-friendly and complete?

Website - Metrics
Performers are using the features of the site (tags, sharing, timestamps, etc.)
Amount of problems reported through the system, as well as types of problems
Amount of problems reported, as well as types of problems

Usability experts

Which feature of the platform do you find most useful? Explain.

Metrics

Usability and technical review

Formative Evaluation

The navigation is user-friendly

of different abilities

Broadcasting is a user-friendly process

Broadcasters, all consulting experts,

and a sample of five viewers

Signing up is a user-friendly process

Before
implementation,
Assessing usability
after the first delivery
of the intervention
and on a yearly basis

 Is the level and tone of the content appropriate for college students?

Source

Are there spelling or grammatical errors in the content?

Social media and

usability experts

Are the guidelines for privacy protection relevant, and complete?

Before
implementation

Broadcasters

Interview - Open ended questions

Are the guidelines for appropriate behaviors and moderations
relevant, and complete?

When

Three, six, and nine

months after
implementation,
and on a yearly basis

Asssess the content

and interaction quality
of the intervention

Social media and

usability experts

Tool
Interview

Data collected

Three, six, and nine

months after
implementation,
and on a yearly basis

Asssess the content

and interaction quality
of the intervention

Rationale

Asssess the content

quality in order to implement
the intervention

Interview and metrics

Interview - Open ended questions

What kind of value do you think this type of online informal learning
could bring to the program?
Are the broadcasters discussing material relevant to the curriculum?
Are the viewers asking questions relevant to the curriculum?
Are the interactions respectful and age appropriate?
Are the questions asked in the chatroom addressed by the broadcaster?
Website - Metrics
Quantity and quality of questions in chatroom
Interview - Open ended questions

Interviews and metrics

Content and editorial review

Formative Evaluation

Using a sample of broadcasts, one yearly broadcast per teacher:

Are the suggested channels relevant?

Is the taxonomy relevant and consistent?
Are some questions asked often enough to move them in the FAQ?
Should some questions in the FAQ be integrated to the help documentation?
Website - Metrics
Quantity and reasons of flagged behaviors for broadcasters
Quantity and reasons of flagged behaviors for viewers
Quantity of clicks on suggested channels

13

Source

Interview - Open ended questions

Do you have any concerns with regards to delivering an online event? Explain.

Interview - Open ended questions

Do you feel this intervention has had
an impact on your interactions
with students? In what way?

Website - Metrics
Quantity and length of events delivered

Broadcasters

To what extent do you envision yourself engaging in these events again?

Rationale

Before
implementation,
after experiencing
some examples

To what extent do you envision yourself engaging in these events? Explain.

Interview - Open ended questions

Have you encountered any non-technical issues during delivery?

When

Variety and relevance of topics covered

Immediately
after first delivery

Assessing perceptions
towards the intervention.

Yearly maintenance
evaluation

Chatroom sentiment analysis

Quantity of events viewed by students
Quantity of moderators
Level of activity in chatroom

Questionnaire - Open ended questions

How often do you envision participating in activities like this?

Before
implementation,
and immediately
after first delivery

What did you enjoy most about this activity?

What did you enjoy least about this activity?

Questionnaire - Open ended questions

Which events have you preferred over the past year? Why?
Do you feel this intervention has had an impact on your interactions
with teachers and other students? In what way?

Viewers

Tool
Interview
Interview
Interview and metrics
Questionnaire
Questionnaire

Summative Evaluation

Performer engagement

14

Data collected

Assessing perceptions
towards the intervention.

Yearly maintenance
evaluation

Source
Viewers

Three, six, and nine

months after
implementation,
and on a yearly basis

Assessing results in meeting

performance objectives.

Broadcasters

Three, six, and nine

months after
implementation,
and on a yearly basis

Assessing results in meeting

performance objectives.

Broadcasters

Have the events youve watched had any impact on your habits, thinking or techniques?

When

Three, six, and nine

months after
implementation,
and on a yearly basis

Assessing results in meeting

performance objectives.

Institution administration

Tool
Questionnaire
Interview
and metrics

Questionnaire - Open ended questions

What have you learned from the events that you have participated in
which could make you a more creative person?

On a yearly basis

Assessing results in meeting

performance objectives.

What are the reasons for which you usually timestamp a video?
Interview - Open-ended question
Has the broadcasting of events improved your ability to voice your creative processes?
Website - Metrics
Amount and content of moments timestamped by students for a specific teacher

Metrics

Website - Metrics
Students describe their timestamped moments in way that could suggest
they will be relevant to their performance and projects.

Attendance - Metrics
Attendance to different exhibits of the graphic design department,
especially the graduation exhibit.

Metrics

Individual performance
Performance transfer

Summative Evaluation

Organizational performance

15

Data collected

Employment rates - Metrics

Percentage of students hired in graphic design jobs or admitted
to graphic design undergraduate programs pre- and post-intervention.
Admission rates - Metrics
Quantity and quality of new students requesting admission to the program.

Rationale

References
Marsick, V. J., Watkins, K. E. (2001). Informal and
incidental learning. New directions for adult and
continuing education, 2001(89), 2534.
Nielsen, J. (2000, March 19). Why You Only Need
to Test with 5 Users. Retrieved April 14, 2015, from
http://www.nngroup.com/articles/why-you-onlyneed-to-test-with-5-users/
Schugurensky, D. (2000). The forms of informal
learning: Towards a conceptualization of the
field. Retrieved April 14, 2015, from https://tspace.
library.utoronto.ca/bitstream/1807/2733/2/
19formsofinformal.pdf.

16

17