YouTubeRatingGuidelines

1.
2.
3.

4.

5.

6.
7.

8.

Version1.1
January8,2016

Introduction
Important Notes
Types of YouTube Search Results
3.1. Video Results
3.2. Channel Results
3.3. Playlist Results
3.4. Movie Results
3.5. Pay-Restricted Results
Assigning Utility Rating
4.1. Utility Rating for Video Results
4.1.1. Video Result Utility Rating Examples
4.2. Utility Rating for Channel Results
4.2.1. Channel Result Utility Rating Examples
4.3. Utility Rating for Playlist Results
4.3.1. Playlist Result Utility Rating Examples
4.4. Utility Rating for Movie Results
4.4.1. Movie Result Utility Rating Examples
4.5. Utility Rating for Pay-Restricted Results
4.5.1. Pay-Restricted Result Utility Rating Examples
Assigning Quality Rating
5.1. Quality Rating for Video Results
5.1.1. Video Result Quality Rating Examples
5.2. Quality Rating for Playlist Results
5.2.1. Playlist Result Quality Rating Examples
5.3. Quality Rating for Channel Results
5.3.1. Channel Result Quality Rating Examples
5.4. Quality Rating for Pay-Restricted Movies and Videos
5.4.1. Channel Result Quality Rating Examples
Assigning Flags to YouTube Result Blocks
6.1. Assigning flags to video results
6.2. Assigning flags to non-video results
Reporting Results with Duplicate Landing Pages
7.1. Pre-Identified Duplicates
7.2. Rater-Identified Duplicates
7.3. Reporting Duplicate Results
Assigning an Overall Side by Side Score (if requested)

1
ProprietaryandConfidential2016

1.Introduction
The goal of YouTube search is to provide users with relevant, engagingvideos.Thedata derivedfromyour rating
tasks is used to inform YouTube search quality engineers about what can be done to achieve this goal. These
guidelinesaredesignedtohelpmakeyourresponsesonratingtasksasusefulaspossible.
InYouTube Block Utility tasks, you will be givenaquerythatwasissuedon
www.youtube.comandoneortwosetsof
YouTuberesultblockstoevaluate.
In
all
YouTubeBlockUtilitytasks,youwill:

1. AssignaUtilityRatingtoeachYouTuberesultblockusingthegivenUtilityRatingslider.
2. AssignaQualityRatingtoeachYouTuberesultblockusingthegivenQualityRatingslider.
3. AssignFlagstoYouTuberesultblocks,whenappropriate.
4. ReportDuplicateResultBlocks(resultsthathavethesamelandingpage).

In
some YouTube Block Utility tasks, you will also be asked toassign anOverallSide bySiderating.Youwillbe
requiredtoleaveacommentifyouthinkonesideisbetterthantheotherside.
These guidelines provide information about the different types of results provided byYouTubeand how to evaluate
them. Please note that YouTube continually makeschangestotheappearance of pagesonthesite,sosomeofthe
screenshotsintheseguidelinesmaynotexactlymatchwhatyouseeonthesite.

2.ImportantNotes

Resultblocksaresometimesslow toload
: Beforeassigninganyratings,pleasescanthetaskpagetomakesure
thatallresult blocks have loaded. Pleasenote thatresultblocks may be slow to loadand/orthat the task page may
need reloading if there are empty result blocks. If the result blocks fail to loadafterrefreshingafew times,please
releasethetaskandsendanotetoyouremployer.

Sometimes a side willnot haveanyresultblocks

: If onesidedisplays themessageThissidedidnot generate
any results, please do not release the task or send a notetoyouremployer. Pleaseassign utility ratings toeach
resultblockusingyourbestjudgment.PleasealsoassignanoverallSxSrating,ifthetaskasksforone.

Sometimesthetaskwillnotbedividedintotwosides
:Youwillbegivenjustonesetofresultblockstoevaluate.

Donot refreshthetask pageafterassigning ratings

: Ifyourefreshthepageafterassigningratings,youwilllose
themandtheywillhavetobeenteredagain.

u=hkpjohpj

2
ProprietaryandConfidential2016

3.TypesofYouTubeSearchResults

Most YouTube search results are videos, channels,or playlists. These guidelinesprovide guidance onrating these
typesofresults.

ResultType

Examples

Video

https://www.youtube.com/watch?v=uBWrpVrazzA

http://www.youtube.com/watch?v=W795W63n7mA
https://www.youtube.com/user/PewDiePie

http://www.youtube.com/user/whitehouse

Channel

https://www.youtube.com/channel/UCE80FOXpJydkkMoBYoJdEg
(topic channel)
http://www.youtube.com/watch?v=e3i6LSW_Kyg&list=UUYxRlFDqcWM4y7FfpiAN3KQ

http://www.youtube.com/watch?v=JGPQcGu2Cso&list=UUGm3CO6LPcNY7HIuyE0Rew

https://www.youtube.com/watch?v=EVBsypHzF3U&list=RDEVBsypHzF3U
(mix)

Playlist

3
ProprietaryandConfidential2016

ThisishowYouTuberesultblocksfortheseresulttypesweredisplayedatthetimetheseguidelineswerewritten:

ResultType
ResultBlock

VideoResult

ChannelResult

NoticetheCHANNEL
label

PlaylistResult

Noticethethumbnail
difference.User
curatedplaylistresult
andYouTube
autogeneratedmix
playlistresultcouldlook
differently.

YouTubeprovidesother types ofsearchresults,suchas movieresults,artistwatchcards,andshowwatchcards,and

you may occasionally be asked to rate these other result types. Please use yourjudgment andassign reasonable
ratingsbasedontheutilityofthelandingpagetothequery.

ThisishowYouTuberesultblocksfortheseresulttypesweredisplayedatthetimetheseguidelineswerewritten:

ResultType

ResultBlock

MovieResult

4
ProprietaryandConfidential2016

ArtistWatchCard

ShowWatchCard

5
ProprietaryandConfidential2016

3.1.VideoResults

Video results are the most common typeofYouTube searchresult.

Hereis alinktothevideothatisdisplayedwhen
theuserclicksonthetitle
10CutestCatMoments
ofthisvideosearchresult:

Thevideoresultpagelookslikethisatthetimetheseguidelineswerewritten:

6
ProprietaryandConfidential2016

3.2.ChannelResults

Achannel is acollection ofYouTubewebpages. Here are somesignsthatusuallyindicatethataparticular YouTube

webpageispartofachannel:

A channel specific banner at the top the page, followed by the channeltitle, aSubscribe button,andthe
numberofsubscribers.

Navigationaltabs,suchasHome,Videos,Popular,Channels,Discussions,About,etc.

Prominentdisplayofthewordchannelonthepage.

AsearchboxtitledSearchChannel.

Allchannelshaveat leastoneof thesesigns.Withalittleexperienceyouwillsoonbeableto recognizechannelsata

glance.

Here are some examples. Please visit these channel pages by clicking the screenshots or the titles above the
screenshots.

ChannelExample1:
TaylorSwiftChannel

ChannelExample2:
KitesurfChannel

Examples 3 and4 below areknown as

Topic
channels. Topicchannelsarecreated automaticallybyYouTube, and
thetopicchannelnamesusuallystartwithahashtag(#).
7
ProprietaryandConfidential2016

Example3:
TaylorSwiftTopicChannel

Example4:
MinecraftTopicChannel

Whenyouclickontheabouttab,youllseetopicchannelsareautogeneratedbyYouTube.

8
ProprietaryandConfidential2016

9
ProprietaryandConfidential2016

3.3.PlaylistResults

Aplaylist isa listofvideos arranged in alinear order. Thearrangement of the videos is controlledby theregistered
YouTubeuser whoowns andcreates theplaylist,or by theautomaticmixalgorithmfromYouTube.Underthedefault
playlistsettings, videosintheplaylistwillplay sequentiallywithnofurtheractionfromusers. Usersmayclickvideosin
theplaylistiftheywishtowatchvideosinadifferentorder.

ThisishowYouTubePlaylistresultblocksweredisplayedatthetimetheseguidelineswerewritten:

The playlist result page looks similar to a video page, with an additional list of videos on the right sideabove the
relatedvideospanel.

10
ProprietaryandConfidential2016

11
ProprietaryandConfidential2016

3.4.MovieResults

Some movies on YouTube are free for users. Others require payment and users would notbe able toviewthem
without paying. When assigninga utilityrating to suchmovies, pleaseassumethatthemovieplaysasadvertised. If
the title of the movie is a good fit for the query, it should get a good utility rating. You will learn more aboutthe
YouTubevideoratingscaleinSection4.

Hereareexamplesoffreemoviesandmoviesthatrequirepayment.

MovieResult
ResultBlock
Type

FreeMovieResult

PaidMovieResult

Noticethedollarsign
bluelabelandthe
$4.99fee

12
ProprietaryandConfidential2016

3.5.PayRestrictedResults

PayrestrictedresultsareYouTubesearchresultsthatcannotbeviewedwithoutfirstmakingapayment.These
resultshaveapriceiconsomewhereintheresultblockwhichlookslikethis:

Someofthe
movie
and
video
resultsarepayrestricted.Uponclickingonapayrestrictedresult,thepreviewtrailer
mightstarttoplay.
Pleasedonotconfusesuchpreviewtrailerswithactualmovies!

Sometimes,thevideoresultdirectstoaYouTubepagewithavideoplayerinformingusersthatpaymentisrequired
beforethevideocanbeviewed.

13
ProprietaryandConfidential2016

14
ProprietaryandConfidential2016

4.AssigningUtilityRating

Understandingthequeryanduserintent

As with othertypesofrating,youshouldfamiliarizeyourselfwiththepossibleinterpretations ofthequeryanddevelop

an understandingoftheusers intent.Ifthemeaningofthe queryisnot perfectlyclear,dosomeresearch:lookat the
YouTube search results in the task and/ortry issuingthequery on
google.com or
wikipedia.com
.Youmay also be
able to find definitions of unfamiliar terms at
urbandictionary.com
. Try to imaginethevariousreasons a usermight
haveissuedthequeryonYouTubeandthekindofvideosthattheuserexpectstosee.

EvaluatingYouTubeVideoResultBlocks

The same ratingscalewillbe usedwhenevaluatingalltypesofYouTuberesultshowever,youwillbegivenindividual

instructionsforratingthesefourtypes:videos,channels,playlists,andmovies.

4.1.UtilityRatingforVideoResults

The
usefulness ofa YouTube
videoresult isdeterminedbythevideoresult(themainvideoonthelandingpage)and
bytheusers intent.Iftheuserhasaparticularresult (video)inmind,thenthatresultisveryuseful.Resultsthathave
nothingtodowiththeusersintentarenotrelevantandthereforenotuseful.

YouwillassignautilityratingtoYouTubevideoresultsaccordingtothefollowingscale:

Rating

Description

Very
Useful

Usethisratingwhenthevideoisnotonlyclearlyrelevanttothequery,butisavideothatausertypingthequery
wouldexpecttoseeandbepleasedtosee.

Useful

Usethisratingwhenthevideohasaclearconnectiontothequery,andisaresultthatsomeuserswouldbe
pleasedtoseebutwouldnotappealtoaswideanaudienceasa
VeryUseful
video.

Marginally
Useful

Usethisratingwhenthevideohasaconnectiontothequery,butitwouldnotappealtothevastmajorityofusers
whoissuethequery.

Thisratingisalsoappropriateforvideoresultsthatserveaminorinterpretationofthequery.

Useless

Usethisrating:

WhenthevideohasnoclearconnectiontothequeryandwouldevoketheresponseIhavenoidea
whytheyareshowingmethis!inthevastmajorityofusers.
Thevideodoesnotplay.Youwillalsocheckthe
DidNotLoad
flag.
Thevideoisunratablebecauseitisinaforeignlanguageandyouareunabletodetermineutility.You
willalsocheckthe
ForeignLanguage
flag.

Thisratingalsoapplieswhenyoucannotdeterminetheutilityofthevideowithintheallottedtime.

IMPORTANT:
Do
NOT
usethisratingwhenthevideorequirespaymenttoplay.Somemovieresultsrequire
paymenttoplay.Formoviesthatdonotplaywithoutpayment,pleaseassumethatthemovieiswhatitclaimsto
be.

Important: Youare not required to assignthe

Useless
ratingtoforeignlanguagevideos. Ifyoubelievethataforeign
languagevideomayhavesomeutility,youmayselectanotherrating.Herearetwopossiblescenarios:
15
ProprietaryandConfidential2016

The video doesnt need language to be appreciated

. For the query [skateboarding], English (US),
this
Chinesevideo
mayhavesomeutilityforEnglishusers.

Theuserislooking for a foreignlanguage video

.For the query[Comola Flor by Selena],English (US)
this Spanish videoisexactlywhat theuserislooking for. Infact,thisvideois
VeryUsefuleventhoughit isin
aforeignlanguage.

4.1.1.VideoResultUtilityRatingExamples

Query

URL

Rating

Description

[babypanda
sneeze]

http://www.youtube.com/watch?
v=FzRH3iTQPrk

VeryUseful

Videopagewithavideoofababypanda
sneezing

[zidaneheadbutt]

http://www.youtube.com/watch?
v=zAjWi663kXc

VeryUseful

VideopagewithavideoofZinedineZidane
givingaheadbutttoanItaliansoccerplayer
duringthe2006FIFAWorldCupmatch.

[pug]

http://www.youtube.com/watch?
v=Njo28CN5XuY

Useful

Videopagewithavideoofatalkingpugonthe
MontelWilliamsTVShow.

[elf]

http://www.youtube.com/watch?
v=9jyCfRHumHU

Useful

VideopagewithavideoclipofthemovieThe
Elf.

[snakesonaplane]

http://www.youtube.com/watch?
v=J2IZkwgXl8M

MarginallyUseful

VideopagewithavideooftheUniversityof
SouthernCaliforniaMarchingBandplayinga
songfromthemovieSnakesonaPlane.

[StarWars]

http://www.youtube.com/watch?
v=hVrIyEu6h_E

MarginallyUseful

VideopagewithavideoofGroceryStore
Warswithanimatedfruitsandvegetables
playingcharactersfromthemovie,StarWars

[hotdog]

https://www.youtube.com/watch
?v=lzQ8GDBA8Is

Useless

Videopagewithavideowitharecordingof
ElvisPresleysingingHoundDog.

[oscarceremony]

https://www.youtube.com/watch
?v=AvpKwMmZA8

Useless

VideopagewithavideoofaJapanese
animationwithamaincharacternamedOscar.

[modernfamily]

http://www.youtube.com/watch?
v=59twO1fJwtQ

Useless

VideopagewithvideofromtheTVshowThe
View,notModernFamily

16
ProprietaryandConfidential2016

4.2.UtilityRatingforChannelResults

The first stepin evaluating a

channelresultis todeterminewhetherthe query ischannelseeking(navigational)or
nonchannelseeking(nonnavigational). In otherwords,youshouldtrytodeterminewhetherthequery islookingfor
a particular channel.Note: the term navigationalis used here in the context of YouTubechannel results, not Web
results.

Channelseekingqueries
.WeassumethattheuserislookingforaYouTubechannelwhen:

a.

The word channel or official is contained in the query. Here are some examples: [madonna
official], [taylor swift channel], [sesame street official channel], [elton john vevo]. Note: Vevo is a
musicwebsite,andchannelsofVevoaffiliatedartistsaredisplayedonYouTube.
b. The queryis anunusual word or name whichis only associated with a registered YouTube channel
user.Herearesomeexamples:[CJTClol],[kjgophish],[jimmykimmellive].

Nonchannelseeking queries
. We will assume that the user is
not
looking for a particular YouTube
channelresultforallotherqueries.

Userswhovisitchannelpagesaretypicallyinterestedineithergoingtoaparticularchannelorfindingvideosona
particulartopic.

A particular channel
. A user may visit Lady Gagas channel to see if Lady Gaga has posted any recent
videos or toseeif LadyGaga orherfans have posted new messages. Similarcontent maybeavailable on
otherchannels,buttheuserisinterestedinthisparticularchannel.

Videosona particular topic

. Auser mayvisit a particularchannel to find agoodcollectionofvideosona
particular topic. Thechannel ownerand thespecificchannel may notbeimportanttotheuseritisthetopic
that is of interest. For example, a user interested in snowboarding may visit a channel devoted to
snowboardingtoperusethecollectionofvideosonthatchannel.

It isimportanttounderstandthese twocasesbecausetherating you assigntochannelresultsshoulddependonthe

userintent.

Youwillassignautilityratingtochannelresultsfor
channelseekingqueries
accordingtothefollowingscale:

Rating

Description

VeryUseful

Usethisratingwhenyouarecertainthattheuserislookingfortheparticularchannel.Forexample:
[madonnachannel]

Useful

Usethisratingwhenyoubelieveitis
likelyorpossible
butnotcertainthattheuserislookingforthe
particularchannel.Forexample,thequeryhasmultipleinterpretationsandthisinterpretationislikelyor
possible,butnotdominant.

MarginallyUseful

Pleasedo
not
usethisratingwhenthequeryischannelseeking.Thechannelwilleitherbethetargetof
thequery(
VeryUseful
),oneofseverallikelytargetsofthequery(
Useful
),orthewrongchannel
(
Useless
).

Useless

Usethisratingwhenyoubelievetheuserislookingforadifferentchannel.

For
nonchannelseeking queries
, please watch parts of the three videos on the channel that seem to bemost
relevant to the query. Based on your impression of these three videos, please assign a rating according to the
followingscale:
17
ProprietaryandConfidential2016

Rating

Description

VeryUseful

Pleasedonotusethisratingwhenthequeryisnonchannelseeking.

Useful

Usethisratingwhenthefollowingconditionsaretrue:

Theratioofgoodtobadvideosisveryhigh.
VeryUseful
and
Useful
videosareconsidered
goodvideos.
MarginallyUseful
and
Useless
videosareconsideredbadvideos.
Mostofthebestvideosonthechannelareeasytofindonthechannellandingpagethatis,the
pagethatloadswhenyouclickthetitleofthechannelresult.

MarginallyUseful

Usethisratingifoneormoreoftheaboveconditionsissomewhatlacking.Forexample,theratioof
goodtobadvideosisevenlowerand/ormanyofthebestvideosarehardtofind.

Useless

Usethisratingwhenmostofthevideosarebadand/orthegoodvideosaredifficulttofind,i.e.,thegood
videosarenotfoundonthelandingpage.Forexample,ifyouhavetogotoanotherpageorifyouhave
tousethesearchboxonthechannelpagetofindgoodvideos,thegoodvideosaredifficulttofind.

4.2.1.ChannelResultUtilityRatingExamples

Query

Channel
Seeking
?

Result

Targetof
theQuery?

Rating

Explanation

[rihanna
channel]

Yes

Rihannas
Channel

Yes

VeryUseful

Thischannelisthetargetofthequery,so
itis
VeryUseful
.

[rihanna
channel]

Yes

RihannaStay
ft.MikkyEkko

No,nota
channel
result

Useful

ThisisaspecificvideofromRihanna's
channel,butsincethequeryischannel
seeking,thebestvideoresultsare
Useful
.

[ladagagavevo]

Yes

LadyGagas
VevoChannel

Yes

VeryUseful

Thechannelisthetargetofthequery,soit
is
VeryUseful
.

[ladygaga]

No

LadyGagas
VevoChannel

Nota
channelsee
kingquery

Useful

Thischannelis
Useful
becauseitis
relatedtotheartist,theratioof
goodtobadvideosishigh,andthegood
videosareeasytofind.

[chrisbrown
official]

Yes

Madonnas
Channel

No

Useless

Eventhoughthisisahighqualitychannel,
itisnotthechanneltheuserisseeking.

No

LadyGagas
VevoChannel

Nota
channelsee
kingquery

Useful

Theuserseeksaspecificvideowhichis
easilyfound(atthiswriting)onthechannel
landingpage.Othervideosonthechannel
are
Useful
tothequery.

Marginally
Useful

Eventhoughthechannelhasahighratio
ofgoodtobadvideos,BadRomance
(thevideospecifiedinthequery)isnoton
thelandingpagemakingithardtofind.
Thischannelistherefore
Marginally
Useful
.

[tilithappensTo
you]

[badromance]

No

LadyGagas
VevoChannel

Nota
channelsee
kingquery

18
ProprietaryandConfidential2016

Query

[skateboarding
twinsofficial]

[puehsetwins
channel]

[puehsetwins
channel]

[CJTClol]

[stmarys
collegeofficial]

[Yosemite]

[dogs]

Channel
Seeking
?

Result

Yes

ThePuehse
Twins'
Skateboarding
Channel

Targetof
theQuery?

Rating

Explanation

Yes

VeryUseful

ThePuehsetwinsareapairofparticularly
wellknownskateboardingtwins.Even
withouttheirnameinthequery,research
showsthattheyarethetargetofthequery.
Thisistheirofficialskateboardingchannel.

Yes

ThePuehse
Twins'
Skateboarding
Channel

Yes

VeryUseful

ThePuehsetwinsarewellknownfortheir
skateboarding.Thischannelisthetarget
ofthequery.

Yes

ThePuehse
TwinsTennis
Channel

Likelyor
possible,
butnot
certain

Useful

ThePuehsetwinshavetakenuptennis.
Thisistheirtennischannel.Thisisnotthe
dominantinterpretationofthequery,butit
issomewhatlikely.

Yes

VeryUseful

Eventhoughthequerydoesntcontain
channelorofficial,itisnavigational.
Therearenoreasonableinterpretationsfor
thequeryotherthanthisYouTube
channel.

StMarys
Collegeof
Maryland
Channel

Likelyor
possible,
butnot
certain

Likelyor
possible,
butnot
certain

Useful

Useful

Thequeryisnavigational,butwecanttell
whichSt.Maryschanneltheuseris
lookingfor.Channelsforbothcollegesare
Useful
.

Yosemite
NationalPark
Topic
Channel

Nota
channelsee
kingquery

Useful

This
topic
channelis
Useful
becauseitis
relatedtothequery,theratioof
goodtobadvideosishigh,andthegood
videosareeasytofind.

Marginally
Useful

Eventhoughthevideosarehighquality,
theyhavejustamarginalconnectiontothe
query.Thevastmajorityofuserswho
issuethequeryarelookingforvideosof
householddogs,notcoyotesthatlivein
thewild.

Yes

CJTClols
Channel

St.Marys
Collegeof
California
Channel
Yes

No

No

Coyote
Topic
Channel

Nota
channelsee
kingquery

Please note
: Even though topic channels are created differently,they are ratedin the sameway asother channel
results.

19
ProprietaryandConfidential2016

4.3.UtilityRatingforPlaylistResults

Whenevaluatingplaylistresults,youarenotrequiredtowatchallofthevideosintheplaylist.

You will form anopinion of the videos in the playlistbywatching

partsof thefirstthreevideosthatplaywhenyou
clickonthetitleorthumbnailintheratingtask.Pleasebaseyourratingonthesethreevideos:

Thevideothatplayswhenyouclickthetitleorthumbnailintheratingtask,and
Thefollowingtwovideosthatappearintheplaylistrighthandsidefeeder.

Pleaseignoretheremainingvideosintheplaylist.

It is possible that one or more of the three videos you should watch doesnt load or fails to play.If thishappens,
pleasedothefollowing:

Flagtheplaylistas
DidNotLoad
toindicatethereisaproblemwithatleastoneofthevideos.
Watchtheothervideosandassignaratingaccordingtotheguidelinesbelow.

Important:Pleasedo
not
watchany
extra
videostocompensateforvideosthatdonotload.

EvaluatingPlaylistResults

The first step in evaluating a playlist result is to determine whether the query is playlistseeking or
nonplaylistseeking.

Playlistseekingqueries
.When the queryincludeswords likeplaylist,songs,top hits, orvideos,we
will assume theuseris looking foraplaylistresult.Notice thatsongs,top hits,andvideosarepluralized,
whichindicatestheuserislookingformorethanonesongorvideo.

Nonplaylistseekingqueries
.Wewillassumethattheuseris
not
lookingforaplaylistresultwhenthe
abovewords(orsimilartypesofwords)arenotincludedinthequery.

20
ProprietaryandConfidential2016

Youwillassignautilityratingtoplaylistresultsfor
playlistseekingqueries
accordingtothefollowingscale:

Rating

Description

VeryUseful

Usethisratingwhenallormostofthethreevideosyouwatchare
VeryUseful
forthequery.

Useful

Usethisratingwhenallormostofthethreevideosyouwatchare
Useful
forthequery.

MarginallyUseful

Usethisratingwhenallormostofthethreevideosyouwatchareonly
MarginallyUseful
forthequery.

Useless

Usethisratingwhenallormostofthethreevideosyouwatchare
Useless
tothequeryandmostusers
wouldwonderwhytheyarebeingshowntheplaylistresult.

Youwillassignautilityratingtoplaylistresultsfor
nonplaylistseekingqueries
accordingtothefollowingscale:

Rating

Description

VeryUseful

Pleasedo
not
usethisratingwhenthequeryisnonplaylistseeking.

Useful

Usethisratingwhenallormostofthethreevideosyouwatchare
VeryUseful
forthequery.

MarginallyUseful

Usethisratingwhenallormostofthethreevideosyouwatchare
Useful
forthequery.

Useless

Usethisratingwhenallormostofthethreevideosare
Useless
forthequery.

Please note that the rating standards are high when the query is nonplaylistseeking. To be rated
Useful
,allor most of the
three individual videos you watch must be
Very Useful
. To be
Marginally Useful
, all or most of the three videos you watch
mustbe
Useful.

21
ProprietaryandConfidential2016

4.3.1.PlaylistResultUtilityRatingExamples

Query

[Madonna
playlist]

[Madonna
songs]

[Madonna]

[Madonna
songs]

[Madonna]

[sesame
street
songs]

Playlist
Seeking?

Playlist
Result

Description

Yes

Madonna
YouTube
Playlist
Result1

ThemainvideoisMadonnas
LikeaVirginvideo.The
nexttwovideosareher
ExpressYourselfand
Eroticavideos.

Yes

Madonna
YouTube
Playlist
Result1

No

Madonna
YouTube
Playlist
Result1

Yes

Madonna
YouTube
Playlist
Result2

No

Madonna
YouTube
Playlist
Result2

Yes

Sesame
Street
Songs
Playlist
Result

Rating

Explanation

Very
Useful

Sincethequeryhasplaylistinit,
weknowitisplaylistseeking.The
mainvideoandthenexttwovideos
areMadonnasongs.Mostusers
wouldconsiderthesevideos
Very
Useful
or
Useful
.

ThemainvideoisMadonnas
LikeaVirginvideo.The
nexttwovideosareher
ExpressYourselfand
Eroticavideos.

Very
Useful

Sincethequeryhassongsinit,we
knowitisplaylistseeking.Themain
videoandthenexttwovideosare
Madonnasongs.Mostuserswould
considerthesevideos
VeryUseful
or
Useful
.

ThemainvideoisMadonnas
LikeaVirginvideo.The
nexttwovideosareher
ExpressYourselfand
Eroticavideos.

Useful

Fornonplaylistseekingqueries,
Useful
isthehighestratingthatcan
beassigned.

Useless

Eventhoughthequeryis
playlistseekingandtheresultisa
playlist,thisparticularplaylistresult
is
Useless
forthequery.Userintent
istowatchvideosofMadonnas
songs,notwatchinterviews,
documentaries,etc.

Marginally
Useful

Mostuserswouldbelookingfor
musicvideosandnotinterviews,
documentaries,etc.Allofthese
videosare
MarginallyUseful
forthe
userintent,andbecausethequeryis
notplaylistseeking,theratingis
demotedonelevel.

Very
Useful

Sincethequeryhassongsinit,we
knowitisplaylistseeking.Themain
videoandthenexttwovideosare
songsfromthechildrensshow
SesameStreet.Mostuserswould
considerthesevideos
VeryUseful
or
Useful
.

Themainvideoisavideoof
aninterviewMadonnadidon
aTVshow.Thesecond
videoisapromotional
documentaryofW.E.,a
moviedirectedbyMadonna.
Thethirdvideoisatrailerfor
theW.E.movie.
Themainvideoisavideoof
aninterviewMadonnadidon
aTVshow.Thesecond
videoisapromotional
documentaryofW.E.,a
moviedirectedbyMadonna.
Thethirdvideoisatrailerfor
theW.E.movie.
The
mainvideo
andthenext
twovideosaremusicvideos
fromthechildrensshow
SesameStreet.

22
ProprietaryandConfidential2016

[sesame
street
]

No

Sesame
Street
Songs
Playlist
Result

The
mainvideo
andthenext
twovideosaremusicvideos
fromthechildrensshow
SesameStreet.

Useful

Fornonplaylistseekingqueries,
Useful
isthehighestratingthatcan
beassigned.

4.4.UtilityRatingforMovieResults

The first stepin evaluating a

movie resultis to determinewhat the queryis looking for. Insomecases,the queryis
looking for a particularmovie. Inothercases,thequeryislooking foracelebrityorasubject. Youmusttryto imagine
userintentwhenassigningaratingtomovieresults.

Youwillassignautilityratingtomovieresultsaccordingtothefollowingscale:

Rating

Description

VeryUseful

Usethisratingwhenyouarecertainthattheuserislookingforthemovie.

Useful

Usethisratingwhenthemoviehasaclearconnectiontothequery,andisaresultthatsomeusers
wouldbepleasedtoseebutwouldnotappealtoaswideanaudienceasa
VeryUseful
movie.

MarginallyUseful

Usethisratingwhenthemoviehasaconnectiontothequery,butitwouldnotappealtothevastmajority
ofuserswhoissuethequery.

Thisratingisalsoappropriateformovieresultsthatserveaminorinterpretationofthequery.

Useless

Usethisrating:

WhenthemoviehasnoclearconnectiontothequeryandwouldevoketheresponseIhaveno
ideawhytheyareshowingmethis!inthevastmajorityofusers.
Themoviedoesnotplay.Youwillalsocheckthe
DidNotLoadplay
flag.
Thevideoisunratablebecauseitisinaforeignlanguageandyouareunabletodetermine
utility.Youwillalsocheckthe
ForeignLanguage
flag.

Thisratingalsoapplieswhenyoucannotdeterminetheutilityofthemoviewithintheallottedtime.

IMPORTANT:
Do
NOT
usethisratingwhenthemovierequirespaymenttoplay.Formoviesthatdonot
playwithoutpayment,pleaseassumethatthemovieiswhatitclaimstobe.

Important: Youare not required to assignthe

Useless
ratingtoforeignlanguagemovies. Ifyoubelievethat aforeign
languagemoviemayhavesomeutility,youmayselectanotherrating.Herearetwopossiblescenarios:

The user is looking for a foreign language movie

. For the query [Die Legende von Paul und Paula],
English (US)
this German movie is exactly what the user is looking for. In fact,thismovie is
VeryUseful
eventhoughitisinaforeignlanguage.

23
ProprietaryandConfidential2016

4.4.1.MovieResultUtilityRatingExamples

Query

[fletch]

URL

http://www.youtube.com/
movie?v=ssId2r7F1I

[gonein60
seconds]

http://www.youtube.com/
movie?v=vt1AFkNitcU&
ob=av1n&feature=mv_sr

[police]

http://www.youtube.com/
movie?v=QuHMBFTBD5
s

Rating

Description

VeryUseful

Thismoviematchesthequery.Itiswhattheuseris
lookingfor.

Thismovierequirespayment.Althoughwecant
watchthemoviewithoutpaying,wewillassumethat
themovieiswhatitclaimstobe.

VeryUseful

Thismoviematchesthequery.Itiswhattheuseris
lookingfor.

Thismovierequirespayment.Althoughwecant
watchthemoviewithoutpaying,wewillassumethat
themovieiswhatitclaimstobe.

Marginally
Useful

Thismoviehasaconnectiontothequery,butitwould
notappealtothevastmajorityofuserswhoissuethe
query.Mostuserswhoissuethisqueryarenot
lookingforamovietheywanttowatchvideosofthe
bandThePolice.

4.5.UtilityRatingforPayRestrictedResults

If the result requires payment, you are notrequiredtopayand arenotrequiredtowatch. Westillaskthatyouratethe

block,even without the abilityto view the video.Usethetitle,snippet,and thumbnailtogive yourbestguess.
Donot
assign the "DidNotLoad"flagtopayrestrictedresults.
(The
DidNotLoadflagsignalstousthattheremaybe
aproblemwiththeresult,butpayrestrictedresultstypicallyfunctionasweintendthemtofunction.)

4.5.1.PayRestrictedResultUtilityRatingExamples

Query

[adventuretime]

PayRestricted
ResultBlock

Rating

Useful

Description

TheuserseeksepisodesorclipsfromtheAdventureTimeTVshow.Thevideouploader
(CartoonNetworkEps)seemsliketheofficialownerofthisTVshowandthisseemslikeanepisodefromthe
show.Manyuserswouldfindthisresultappealingifitwerefree,soitshouldberated
Useful
.

Query

[adventuretimejiggler]

24
ProprietaryandConfidential2016

PayRestricted
ResultBlock

Rating

VeryUseful

Description

TheuserseeksaparticularepisodeofthisTVshow.Thisisexactlywhattheuserislookingfor.

Query

[adventuretimejiggler]

PayRestricted
ResultBlock

Rating

MarginallyUseful

Description

TheuserislookingforaparticularepisodeofthisTVshow.Thisisthewrongepisode,soitwillhavelimited
appeal.

Query

[adventuretimejigglerfree]

PayRestricted
MovieResult
Block

Rating

Useless
to
MarginallyUseful

Description

TheuserseeksaparticularepisodeofthisTVshowthatcanbeviewedwithoutcharge.This
payrestrictedresulthaslimitedappeal.

25
ProprietaryandConfidential2016

5.AssigningQualityRating

The goalofYouTube Quality rating istoevaluatehowwell a YouTube page achieves its purpose. Unlikemostweb
pages,mostYouTubepagesonlyhaveoneortwomainpurposes:
1. toprovideavideothatuserswillwanttowatch
2. tohelpusersdecidewhatvideotowatchnext

To assignaYouTubequality ratingyouwillneedtoknowthe purpose ofthepage. Hereare the mainpurposesfor

variousresulttypes.

Result
type

Purpose

Video

1,
Section 5.1 for
ratingguidelines

Example

Playlist

1&2,
Section 5.2 for
ratingguidelines

Channel

2,
Section 5.3 for
ratingguidelines

Payrestricted
videos
and
movies

1,
Section 5.4 for
ratingguidelines

5.1.QualityRatingforVideoResults

Ifthevideodoesnotloadorplayforyou,assigntheLowestpossiblequalityrating.

Guidingprinciplewhenevaluatingvideoquality:didcontentrequireeffortandskilltoproduce?

YoucannotbeanexpertjudgeofcontentqualityforeverynicheinterestonYouTube,butyoucanprobablymakea
guessabouthowmucheffortorskillwasrequiredtoproduceavideoorcollectionofvideos.Effortandskillisauseful
indicatorofcontentqualitysorelyonthisheavilywhenassigningQualityratings.
26
ProprietaryandConfidential2016


NeutralorWorthlessIndicatorsofquality(
thesesimplecountscanbeaggregatedbymachines.Ifwewantedto
countthesethingswewouldaskamachine,notyou!):
ViewCount
SubscriberCount
Likes/Dislikecountorratio

Negative
indicatorsofeffortandskillinclude:
Copiedcontent

Itiseasytouploadsomeoneelsesvideo(oramovieorTVshow
youdidntmake).Ifthechannelonwhichthevideowasuploaded
doesnotseemlikeanofficialorauthoritativesourceforthe
content,itisprobablycopiedcontent.NOTE:parodiesinvolving
subtleorcleveraudio/videoeditingDOtakeefforttoproduceand
donotdeserveaharshpenaltyiftheyseemwelldone.

Lowquality,blurry,grainyor

N/A

scratchyaudioorvideo
Distractingvideoannotations

N/A

Uselessvideometadata

Forexample,keywordstuffingorlinkstospammywebsites.

Positive
indicatorsofeffortandskillinclude:
Highproductionvalue

Videosthatshowevidenceofpostprocessing,videoediting,or
soundmixing.Forexample,multiplecamerashotsinasingle
sceneorimpressivespecialvideooraudioeffects.

Highquality,crispandclear
audioandvideo

N/A

Astrongnarrativeorplotstory
developmentmultiplescenes

N/A

Effortfrommultiplepeopleto
produce.

Forexample,ifalargecastofactorsactivelyparticipatesinthe
video.

Useofstageprops.

Didthevideouploaderbuild,buy,orfindsomethingimpressive,
expensiveorrare?Compare
thisvideo
ofarocketmadeoutof
woodenmatchesand
thisvideo
showingtestingofa
machinemilledsteelrocketnozzleinarocketlab:thepropsinthe
lattervideoarehardertomake.

Usefulvideometadata.

Forexample,lyricstosongsorlinkstorelatedcontent.NOTE:not
allvideometadataisuseful.Forexample,keywordstuffingand
linkstospammysitesarenothelpful.

27
ProprietaryandConfidential2016

Qualityindicatorsnotrelatedtoeffortandskillinclude:
Authority,expertise,orfame

Videosfromanofficialsource(liketheofficialWhiteHouse
channel)oranauthority(liketheMayoClinic)orafamoussource
(likeapopstar)areoftenperceivedashighquality.

Misleadingvideothumbnail
imagesorvideotitles

Theseareindicatorsoflowqualityvideos.Usersdonotwantto
betrickedormisled.Ifthevideothumbnailandtitleseemto
advertisesomethingthevideodoesnotprovide,userswillnotbe
satisfied.

Strongsentimentdirected
towardscontentinuser
comments

Iftherearealotofcommentsaboutcontentthatexpressstrong
feelings,positiveornegative,thismaybeevidencethatthe
contentwasengagingandgeneratedanemotionalresponse,
whichmaybeasignofhighquality.However,commentsdirected
towardsothercommentersortoanythingotherthanthecontent,
nomatterhowvehement,donotprovidemuchevidenceabout
contentquality.

Uniqueaccess

Videoofaninterestingrecentnewsworthyevent,forexample
videoofadisasterorprotest,canhavehighqualityifitistheonly
videofootageofitskind.Forexample,
thisvideo
showsaplane
crash(warning:fatalaccidentfootage).Thisfootageisdifficultto
obtain,solowaudioandvideoqualitycanbesomewhatexcused.

Yourgutfeeling

Youmayhavewatchedalotinternetvideos,aswellasTVshows
andmovies.Basedonyourpersonalexperience,doyouthink
thisvideowillappealtousers?Ifyouhaveastrongopinionyou
canallowittoinfluenceyourrating,especiallyifthevideois
relatedtoyourinterests.

5.1.1.VideoResultQualityRatingExamples

Example1
resultblock

28
ProprietaryandConfidential2016

landingpage

URL

https://www.youtube.com/watch?v=Lm1fya3Qx4s

content

No useful content: just static images with no relation to the title.

quality
factors

The video required little effort to produce.

Result title and thumbnail are misleading/deceptive.
29
ProprietaryandConfidential2016

quality rating

Audio/image quality are low because there is no audio and

images are static.
Video description is not useful. The click here link provided in
the description is deceptive.
The video content is not provided by an authority or expert.

Lowest

Example2
result
block

30
ProprietaryandConfidential2016

landing
page

URL

https://www.youtube.com/watch?v=KIdTKE5g6bc

content

Man remarks that a soccer game was well-played.

quality
factors

The video was not made with care or effort.

Result title and thumbnail are misleading/deceptive.
Image/audio quality is very low.
Video description is distracting and unhelpful.
The video content does not come from an authority or expert.

31
ProprietaryandConfidential2016

quality
rating

Lowest

Example3
result
block

landing
page

URL

https://www.youtube.com/watch?v=3vE2WABErEM
32
ProprietaryandConfidential2016

content
quality
factors

A compilation of short amusing video clips.

quality
rating

The video is a compilation of videos from other users. Selecting

these videos and editing them into this single video probably
required some time and effort.
Result title is okay but thumbnail is misleading and deceptive. It
suggests that the video has racy content but there is very little
racy content in the video.
Image/audio quality is fair. The annotation overlays/frames
seem distracting and unnecessary.
Video description contains some links and info of moderate value
and some keyword stuffing near the bottom.
The video content is not produced by an authority or expert.

Low

Example4
result
block

33
ProprietaryandConfidential2016

landing
page

URL

https://www.youtube.com/watch?v=1sv8JGEdYw

content

Clips of women and girls failing at various tasks.

quality
factors

quality
rating

The video is a compilation of videos from other users. Selecting

these videos and editing them into this single video probably
required some time and effort.
Result title and thumbnail are deceptive: they advertise racy
content but the video is not very racy.
Image and audio quality is poor.
Description consists mainly of keyword stuffing.
The video content is not produced by an authority or expert.

LowLowest
34
ProprietaryandConfidential2016


Example5
result
block

landing
page

URL

https://www.youtube.com/watch?v=VuAo_YpZk8A

content

Clips of people failing at various tasks.

35
ProprietaryandConfidential2016

quality
factors

quality
rating

The video is a compilation of videos from other users. Selecting

these videos and editing them into this single video probably
required some time and effort.
Title and thumbnail are mostly accurate. The thumbnail image
scene is not present in the video but it is suggestive of the video
content.
Image and audio quality is mostly good.
Description consists mainly of keyword stuffing but the three links
it provides may be of some use.
The author has a website devoted to funny fail videos, with
content stretching back to 2009. We may regard him or her as a
minor authority on fail videos.

MediumorLow+

Example6
result
block

36
ProprietaryandConfidential2016

landing
page

URL

https://www.youtube.com/watch?v=5Cod_wgM7n0

content

A fan talks about an episode of a cartoon.

quality
factors

quality
rating

Audio is corrupted by white noise in several places in the video.

The image quality is not very good.
The author did not make this video with effort and care, otherwise
he would have fixed the audio issues.
Result title suggests that the video is an actual episode of a cartoon
rather than a review of the cartoon. The thumbnail is accurate.
Video description is essentially empty.
The video content is not from an expert or authority.

Lowest

Example7

37
ProprietaryandConfidential2016

result
block

landing
page

URL

ThevideowasremovedfromYouTubeduetocopyrightviolations.

content

Acompleteepisodeofacartoonshrunktofitinsideaframeorborder.

quality
factors

Thisisdirectlycopiedcontentitrequiredlittleefforttoproduce.
Resulttitleandthumbnailareaccurate.
Imagequalityismarredbythedistractingframeorborder.
Thedescriptionhaskeywordstuffing.
Theuserwhouploadedthevideoisnotanauthorityorexpert.

quality
rating

Lowest

Example8
38
ProprietaryandConfidential2016

result
block

landing
page

URL

https://www.youtube.com/watch?v=i6anvsoeOGw

content

A short clip from a TV show.

quality
factors

This is copied content. However, the user who uploaded the

video selected this particular scene for a particular purpose (i.e.
to provide a preview) rather than uploading the entire TV show
episode, so it does not deserve the lowest quality rating just
because it is unoriginal.
Result title and thumbnail are accurate.
Image and audio quality are poor.
The description is brief but accurate. It does not improve or
harm quality.
The video is provided by a devoted fan on a channel containing
content related only to this TV show. The user who uploaded the

39
ProprietaryandConfidential2016

video may be generously regarded as a minor authority or

expert.
quality
rating

Low

Medium

Example9
result
block

landing
page

URL

https://www.youtube.com/watch?v=FRTf3UXCpiE

content

The corporate anthem of Fujitsu Ltd. with lyrics slideshow.

quality
factors

This seems like copied content. However, finding this obscure old
song and adding a lyrics slideshow requires some effort and care.
40
ProprietaryandConfidential2016

quality
rating

Result title and thumbnail are accurate.

Image quality is fine, but the audio is a little scratchy.
The description includes helpful information about the singer.
Little information about the user who uploaded the video is
available. Theres no reason to think that the user who uploaded
the video is an authority or expert.

Medium

High

Example10
result
block

41
ProprietaryandConfidential2016

landing
page

URL

https://www.youtube.com/watch?v=dys8KUnwGGg

content

A moaning Ahhh noise set to jarring music and disturbing images.

quality
factors

The author seems to have animated this video and made the
soundtrack on his own. This required effort and care.
Result title and thumbnail are accurate.
Image and audio quality are good.
42
ProprietaryandConfidential2016

quality
rating

The video description contains links and facts that may be

helpful.
The author is an authority or expert for this video because he has
built a brand around this type of content, including a channel of
similar videos and an online store. His Twitter feed says I write
songs that drive you insane.

High

Highest

Example11
result
block

landing
page

URL

https://www.youtube.com/watch?v=61G8iXGuSUg

content

Software tutorial

43
ProprietaryandConfidential2016

Quality
factors

quality
rating

The video required care and effort to produce. The author

demonstrates his expertise about the software and used
artistic/design skills in the video as well.
Video title and thumbnail image are accurate.
Image quality is fine. Audio could be a bit clearer or louder.
Description contains links that may be helpful.
Because the author has a YouTube channel and a separate
website devoted to this type of software tutorial he can be
regarded as a minor authority or expert on this type of content.

High

Example12
result
block

44
ProprietaryandConfidential2016

landing
page

URL

https://www.youtube.com/watch?v=LMsVIbc_48

content

Trip to the esthetician role playing video in

ASMR
style

Quality
factors

Thevideoseemstohaverequiredeffortandcaretoproduce.Itwas
doneinasingle40minuteshot.Theauthorpaidcarefulattentionto
lightingandaudionoteespeciallytheseparateleftandrightaudio
tracks.
Resulttitleandthumbnailareaccurate.
Imageandaudioqualityareexcellent.
Videodescriptionisextensive.

45
ProprietaryandConfidential2016

quality
rating

Theauthorisanauthorityorexpertinthisareabecauseshemakes
manyofthesevideos,andaccordingtoherwebpage,peopledonate
moneytosupporthertodothis.

High

Highest

Example13
result
block

46
ProprietaryandConfidential2016

landing
page

URL

https://www.youtube.com/watch?v=T_vGAPvjOc8

content

Man narrates a video game adventure

quality
factors

The video required some planning, effort and editing to create

and narrate.
Result title and thumbnail are accurate.
Image and video quality are excellent.
Description contains links that may be helpful.
The author is an authority or expert in this area because he has
built a brand (including an online store) and a following around
this type of video game adventure video.

47
ProprietaryandConfidential2016

quality
rating

High

Highest

Example14
result
block

landing
page

URL

https://www.youtube.com/watch?v=3a2qoyONVA

content

Music video

quality
factors

The video required care and effort to create. It seems to have

required filming at multiple locations and lots of editing and audio
processing.
Result title and thumbnail describe the video well.
Image and audio quality are good.
The description is sparse but accurate.
48
ProprietaryandConfidential2016


quality
rating

RobynVEVO, the channel that hosts this video, is the primary

authority on YouTube for music videos from the artist Robyn.

Highest

Example15
result
block

landing
page

URL

https://www.youtube.com/watch?v=V2MBiS1kc_0

content

Instructions on how to perform a neurological examination.

quality
factors

This seems to be a reliable instructional video produced by a

medical professional. It requires a unique skill -- knowledge of
neurological examination techniques -- to produce.

49
ProprietaryandConfidential2016

quality
rating

The instructor in the video is a board-certified nurse practitioner

with a doctorate in nursing practice. She is an expert in this
area.
Result title and thumbnail are accurate.
Video audio and image quality are good.
Video description is sparse but accurate.

Highest

5.2.QualityRatingforPlaylistResults

Thepurposeofaplaylististoprovidetheuserwithsomethingtowatchrightawayandagoodcollection
ofvideostowatchnext.Ideally,thequalityratingforaplaylistwouldbetheaverageofthequalityratings
forthevideositcontainsplusorminusanadjustmentforthecohesiveness,variety,order,size,and
playabilityofthecollectionconsideredasawhole.Thisidealisimpossibletoachievebecauseyouwill
nothavetimetoevaluatethequalityofallthevideosinaplaylist.Whenevaluatingaplaylist,pleasedo
thefollowingtwothings,thenmakeyourbestguessaboutwhattheidealqualityratingwouldbe.

1. SkimthefirstthreeplaylistvideostogetanideaaboutVideoQuality.Animagebelowshows
wheretofindthefirstthreevideosoftheplaylist.
2. Visittheplaylistlistviewandskimtitlesandthumbnails.Thefirstscreenshotbelowshowswhere
thelinktolistviewcanbefound,andthesecondimageshowswhatlistviewlookslike.Lookat
thelistviewandaskyourselfiftheplaylistseemstocontainagoodcollectionofvideos,keeping
inmindthefollowingpoints:
a. cohesiveness
isnecessary:videosinagoodplaylistmusthaveanobviouscommon
themeortopiclikeamazingsoccergoalsor15thcenturylutemusicorbirdsof
prey,andthetitleoftheplaylistshoulddescribethetopic.
b. variety
helpsthequalityofaplaylist,solongastheaddedvarietydoesnotcause
departuresfromthethemeortopic.
c. order
isimportant:ifthereisarightorderforvideos(likeEpisode1,Episode2,Episode
3ofaTVshow,orActs1,2,3ofaplay)thenplaylistsshouldputvideosintherightorder.
d. size
helpsquality:longerplaylistsaregenerallybetterthanshorterplaylists.However,
lengthislessimportantthancohesiveness,variety,andorder.
e. videosshouldbeplayable
:sometimesplaylistscontainvideosthatnolongerplay,or
arepersonalizedandplayonlyforaselectgroupofpeopledesignatedbythepersonwho
uploadedthevideo.Theseareobviousinplaylistlistviewseethesecondimagebelow
forexample.

50
ProprietaryandConfidential2016

PlaylistListview:displaysallthevideosintheplaylist:

51
ProprietaryandConfidential2016

5.2.1.PlaylistResultQualityRatingExamples
Example1
result
block

52
ProprietaryandConfidential2016

playlistlist
page

URL

https://www.youtube.com/playlist?list=PLJvQXRgtxlulYgB1tznRR0h8biN2Q
CiM7

content

Footage from Metallica concerts from 2014.

quality
factors

Audio quality is generally good, which is important for

music-related videos.
There is high production value: multiple shots and evidence of
video post-processing.
The playlist and videos come from the official Metallica YouTube
channel.
Unique access: videos show backstage footage of the band, which
would would be difficult for most people to obtain.
The playlist is organized around an obvious theme: Metallica
concerts in 2014.
Videos are arranged in an obvious order: date of concert.

53
ProprietaryandConfidential2016

quality
rating

Highest

Example2
result
block

landing
page

URL

https://www.youtube.com/playlist?list=PLoip94oFJ4XuhyreC_ckBvihCya2C
mLaB

content

List of bad or funny auditions, typically a few minutes long, from various
TV-audition shows.

quality
factors

Many of the videos lack audio-visual crispness.

Many of these are uploaded by non-official uploaders (like
shaunuk21) and are thus copied content, but some are
apparently from authoritative sources (Britains Got Talent,
The X Factor UK).
The collection as a whole is not organized around a tight theme:
there are singing audition clips and music audition clips from
several years.
The collection is not particularly comprehensive.

54
ProprietaryandConfidential2016

quality
rating

Medium

Example3
result
block

landing
page

URL

https://www.youtube.com/playlist?list=PLC944E2DDE82E9904

content

Eight music videos from various country music artists.

quality
factors

quality
rating

Some of the videos have low quality due to issues around

audio-visual crispness (like the first video, a cell-phone video of a
live concert), or copied content.
The bigger issue is that the playlist content has little to do with
the playlist title: the title indicates that the list has to do with a
particular song by a particular artist, but most of the videos in
the list have little or nothing to do with that song.

Low

Example4
55
ProprietaryandConfidential2016

result
block

landing
page

URL

https://www.youtube.com/playlist?list=PLF9D86C3DCFEDF1EE

content

Official Lady Gaga music videos alternating with Loading Video spam
videos (see screenshot).

quality
factors

There are several copies of a video titled Loading Video . This short
video has no content and seems intended to raise video view count
statistics for the uploader who curated this playlist: from the typical
users perspective it is spam, with no value.

quality
rating

Lowest

5.3.QualityRatingforChannelResults

Channelsshowcontentuploadedbyoneparticularvideouploader,whichmaybeanindividuallike
JennaMarbles
oranentitylike
SonyPicturesEntertainmentGroup
.Toassignaqualityratingtothe
channel,clickontheuploadedvideoslink(illustratedbelow),pickanythreeuploadedvideosonthe
56
ProprietaryandConfidential2016

channelpageandskimthemtogetanideaaboutthequalityofthevideosuploadedtothechannel.
Thenmakeaguessabouttheaveragequalityoftheratingsonthechannel,andassignthatforaquality
rating.

5.3.1.ChannelResultQualityRatingExamples

Example1
result
block

57
ProprietaryandConfidential2016

landing
page

URL

https://www.youtube.com/user/NationalGeographic/videos

content

VideosfromNationalGeographic

quality
factors

Thisistheofficialchannelforthiswellknownsourceofeducational,
scientificanddocumentarycontent.Videoshavehighproductionquality,
involvemanypeopleandprops,andclearlytookagreatdealofeffortto
produce.Audiovisualqualityisquitegood.

quality
rating

Highest

Example2
result
block

58
ProprietaryandConfidential2016

landing
page

URL

https://www.youtube.com/user/FranzLisztFerentz/videos

content

Classicalmusicrecordings

quality
factors

Therecordingshavegoodaudioquality.
Thevideosseemtobecopiedcontent.

quality
rating

Medium

Example3
result
block

59
ProprietaryandConfidential2016

landing
page

URL

URLnotgiven:thiscontentisinviolationofYouTubetermsofserviceand
willberemoved.

content

Fulllengthmovies

quality
factors

quality
rating

Thesefulllengthmovieshavelowaudiovisualquality,many
distractingvideoannotations,keywordstuffinginvideometadata,
andtheyarecopiedcontent.
Thechannelhasuploadedonlythesethreevideos.
Thechannelnamesuggeststhatallvideoswillbehorrormovies,but
thisisnotthecase.

Lowest

5.4.QualityRatingforPayRestrictedMoviesandVideos

Alwaysassignthe
highestquality
ratingtopayrestrictedmoviesandvideos.

5.4.1.ChannelResultQualityRatingExamples
Example
result
block

60
ProprietaryandConfidential2016

landing
page

URL

https://www.youtube.com/watch?v=sIbhaVkWDes

content

Pay-restricted cartoon episode.

quality
factors

Please assume that pay-restricted content is highest-quality.

quality
rating

Highest

61
ProprietaryandConfidential2016

6.AssigningFlagstoYouTubeResultBlocks

Youwillassignthefollowingflagsasneeded:
Porn/Offensive,ForeignLanguage,DidNotLoad.

6.1.Assigningflagstovideoresults
Porn/Offensive
:Usethisflagwhenthecontentofthevideoresultisoffensiveorshockingbecauseofsexrelated
content,violence,badlanguage,etc.

Foreign Language:
Use this flag for videos that would be regarded as foreign by most users in your locale. In
general,the
Useless ratingis appropriate for foreign videosunlessyouthinkthevideohassome utilitydespitebeing
foreign.Inthatcase,pleaseassignaratingfromtheratingscale.

DidNotLoad
:Usethisflagwhenthevideodoesnotplayforanyreason,e.g.takendown,markedasprivate,
prohibitedforthelocale,etc.
DoNOT
usethisflagforpaidresults,whichtypicallyfunctionasintendedafterpayment.

6.2.Assigningflagstononvideoresults
Ifaresultpagecontainsmultiplevideos(suchasplaylists,channels),assigntheappropriateflagwhensomeofthe
firstfewvideosonthepageareporn/offensive,foreignlanguageordidnotload.

7.ReportingResultswithDuplicateLandingPages

Inaddition to assigning utilityratingstoeachdisplay blockresult, wewouldlikeyoutotell uswhether tworesultsare

duplicates(dupes).

7.1.PreIdentifiedDuplicates
Someduplicateresults are automaticallydetectedand will bepreidentifiedforyou.Theywillbehighlightedbyagrey
orgreenbackground,andannotatedbyeither"
Sameas
..."textrightbelowtheflagsandabovetheresultblock.

Thisiswhatapreidentified
Sameas
resultblocklooklike:

Sameas
preidentifiedduplicateresult

62
ProprietaryandConfidential2016

Pleasenote
:

Results annotated by
Same as
may
not be given different ratings/flags/comments. Theirutility ratings,
pagequalityratings,flags,andcommentswillbeautomaticallytransferredtoeachother.
Youcannotuncheckdupesthathavebeenautomaticallydetectedandpreidentified.

7.2.RaterIdentifiedDuplicates
Wewouldlikeyourhelpidentifyingduplicateresultsthathavenotbeenautomaticallydetected.
Encountering a list of identical or very similar results on the YouTube search results page is generally a bad
experience for users. Use your judgment when applying thisdefinition: twovideoresults areduplicates ifthey have
similar or identical content and the user who issuedthequery would not find the second video at all engagingafter
having enjoyed the first to its fullest extent. A
video result should
NOT be marked as a duplicate of a
playlist or
channel
result.

7.3.ReportingDuplicateResults
Whenyounotice thattheresultsintwoor moredisplayblocksareduplicates,pleaseclickonthe
ReportDupebutton
ofone oftheresults.Thecurrentresult(theresultyouclickedon)willbehighlightedbyathicksolidredborderand the
button's namewillchange to
Select Dupes
.Youcanthen check results thatareduplicatesofthecurrentresult,and
thecheckedresultswillbehighlightedbyareddottedborder.
The duplicate results that you checked will be annotated by "
Dupe of..." text right below the flags and abovethe
display block. Afterselecting all dupes, please clickthered
SelectDupesbuttontoreturntothe normalratingmode.
The button's name will changeback to
ReportDupe
,andyou willbe ableto report other setsofdupes(ifthereare
any). If you changeyour mind,youcanalways uncheck a result. Please notethat rateridentifiedduplicateresults
annotatedby"
Dupeof..."
maybegivendifferentutilityratings/flags/comments.
Please
identify dupes both withinthesameside and across sides
.Even forcrosssideresults,youshouldstill
askyourselfthequestion"Wouldyouwanttoseebothresultsiftheywerereturnedbythesamesearchengine?"

Herearesomeexamples:

Dupes
QUERY:[PrincessMononokeTrailer]
URL1:

http://www.youtube.com/watch?v=oZ4pxURy5I

63
ProprietaryandConfidential2016

URL2:
http://www.youtube.com/watch?v=pkWWWKKA8jY
Reason
:Thesetwovideoresultsareduplicatesbecausetheyaremovietrailersthatdifferonlyinaspectratio.Even
thoughtheywereuploadedbydifferentYouTubeusersandtheirruntimesareslightlydifferent,theyareduplicates.
UserswouldnotwanttoseebothresultsreturnedbyYouTube.

NotDupes
QUERY:[Porunacabeza]
URL1:
http://www.youtube.com/watch?v=ZbyIUd6tfp0
URL2:
h
ttp://www.youtube.com/watch?v=F2zTd_YwTvo
Reason
:ThesevideosshowactorsdancingthetangotoPorunaCabezafromtwodifferentmovies(TrueLiesand
ScentofaWoman).Eventhoughthesongisthesame,thevideosareverydifferent.Userscouldbenefitby
YouTubereturningbothvideosforthequery.

8.AssigninganOverallSidebySideScore(ifrequested)
Whenyouaredoneassigning ratings andflagsandflaggingpageswithduplicate landingpages,pleasedecidewhich
sideisbetteraccordingtothe
SxSGuidelines
.

Reminder
:YouwillonlybeaskedtoassignanOverallSidebySidescorein
some
BlockUtilitytasks.

64
ProprietaryandConfidential2016