12/3/2016

APictureisWorth32.33Words:ImportanceofAnalyzingImagesonOnlineSocialMedia|PreCog'sBlog

APictureisWorth32.33Words:Importanceof
AnalyzingImagesonOnlineSocialMedia
PostedonAugust1,2016byPK
Doyourememberthelasttimeyourushedorsawanyonerushtogetanautographofafamouspersonality?No,
right?Becausethosedaysarelonggone.Todaysgenerationbelievesintakingaselfieinstead.Andwhynot,digital
mediaisforever,oratleast,itcaneasilyoutliveapieceofpaperwithanautograph!Thereisanexplosionofdata
thatisgeneratedontheOnlineSocialMedia(OSM),wesee422,340tweetsonTwitter,3.3millionupdateson
Facebook,55,555picturesuploadedonInstagrameverysecond[1].Intherecentpast,withtheupdates,large
fractionofitisimages/picturesoneanalysisshowsthat1.8billionphotosaresharedonFacebook,Instagram,
Flickr,Snapchat,andWhatsAppeveryday[2].Itisalsofoundthatupdateswithimagesincreasetheengagementof
theposts,like[3]shows18%moreclicks,89%morefavorites,and150%moreretweetswhenthetweethasan
imagecomparedtoonlytextupdates.Anotherarticlereports93%ofthemostengagingpostsonFacebookhavean
image[4].ResearchersarealsostudyingwhatmakesanimagepopularonnetworkslikeFlickr[5].
Inlastfewyearstherehavebeenmanyacademicpapers,technologiesinrealworldalllookingatthisgrowthof
contentandanalyzingthemweseemostofthemanalyzingonlythetextualpartofthecontent.Hereisanon
comprehensivelistofpublicationsinsomeofthetoptierconferencesinthisspaceallofthesepaperslookat
contentgeneratedinEnglish[620].Someresearchersarealsolookingatstudyingthesentimentandtextual
characteristicsofnonEnglishcontentonOSM[2127].Languagesinclude,Farsi,andHindi.
IhavebeencuriousforalittlewhilenowaboutnontextualcontentonOSMsomeofmyrecentinteresthasbeento
lookatimagesandvideosonOSM.IrecentlyhadmystudentSonalGoelinvestigateimagesonOSM,she
completedherMastersthesisImageSearchforImprovedLawandOrder:Search,Analyse,Predictimagespread
onTwitterwhereshepredictedtheviralityofimagesonOSMusingtweetsfrommultipleevents.PrateekDewan,my
Ph.D.studentandIhavebeenplayingaroundthebroadertopicofimagesandOSM.Webelievethattheinferences
thatwedrawfromtextualanalysiscanbedifferentfromtheanalysisdonewithimagesfromthesameposts.For
example,textualanalysisdoneinHurricaneSandy[28]andBostonMarathon[29]couldhaveclassifiedtheposts
withimages(alongwithtext)tobelegitimate,whereas,ifweanalyzetheimagesitselfitmaybefake.Belowisafake
imagewhichwentviralduringSandy,buttextualanalysisforthepostswiththeseimagescouldhaveleanedtowards
crediblecontent.

http://precog.iiitd.edu.in/blog/2016/08/imagesononlinesocialmedia/

1/5

12/3/2016

APictureisWorth32.33Words:ImportanceofAnalyzingImagesonOnlineSocialMedia|PreCog'sBlog

SentimentanalysisoftheOSMcontentisusedtomakedecisionsonthepulseofcitizens,customers,etc.
Sometimesthesentimentofthetextualcontentisverydifferentfromtheimagespostedwiththetext.Belowimage
waspostedwiththecontentThankyouPiersMorganforspeakingtruth.#PrayForParis#MuslimsStandWithParis
[30]Textanalysiswillgivepositive/neutralsentiment,whilethecontentfromtheimageattachedwiththepostis
negative.Wefoundotherexamplestosubstantiatethispoint,postbeingnegativeandimagebeingmorepositive[33,
34]andpostbeingpositiveandimagebeingmorenegative[35].

Justtotestourhypothesisofhowmuchinformationisspreadthroughimages,weanalyzedsomeeventsforwhich
wehavebeencollectingdata.Belowisthetablewhichshowsdatafor9eventsconsistentlyweseethatonaverage
about2025%ofthecontenthasonlyimageswithouttext.Inmostoftheanalysisthatisdonenowwithtextual
contentwillmissthisinformation.Inoneoftheeventthatweareanalyzingnow,wewereabletoextracttextfrom
8,200imagestheseimageswerepostedonOSMwithnotext.Tounderstandtheamountoftextthatareshared
throughimages,wegotimagesannotatedandusingTesseractOCR[31],wewereabletoget1,030,471wordsfrom
31,869images.
ColumnwithtextreferstothenumberofpostscontainingthemessagefieldasreturnedbytheGraphAPI.This
fieldcontainsthestatus/textmessagepostedbytheuser.Thewithimagecolumnrepresentsthenumberofposts
wherethetypeofpostisphoto.Facebookautomaticallydeterminesthistypewhileauseriscomposingapost.
ThisfieldisassignedtoALLposts,andcantakeuponeofthefollowingvalues:link,status,photo,video,offer[32].
Thismakescolumntextandimageanintersectionofprevioustwocolumns.Similarly,imageandnotextisa
http://precog.iiitd.edu.in/blog/2016/08/imagesononlinesocialmedia/

2/5

12/3/2016

APictureisWorth32.33Words:ImportanceofAnalyzingImagesonOnlineSocialMedia|PreCog'sBlog

subsetofcolumnwithimage,andtextandnoimageisasubsetofcolumnwithtext.Allvaluesinthetablein
parenthesisispercentagevalue.

Event

Total

Postswith

Postswith

Postswithtext

Postswithimage

Postswithtextand

posts

text

image

andimage

andnotext

noimage

22,820

6,868(30)

10,192(44)

538(2)

9,654(42)

6,330(28)

20,960

17,217(82)

7,463(36)

5,756(27)

1,707(8)

11,416(54)

67,453

28,030(42)

12,386(18)

1,553(2)

10,833(16)

26,477(39)

Eurocup2016

109,189

77,355(71)

61,119(56)

40,518(37)

20,601(19)

36,837(34)

Wimbeldon2015

111,417

80,469(72)

52,756(47)

37,862(34)

14,894(13)

42,607(38)

Parisattacks2015

131,548

78,803(60)

75,277(57)

32,861(25)

41,416(32)

45,942(35)

MalasiyanMH17

22,490

5,270(23)

2,947(13)

316(1)

2,631(12)

4,954(22)

IPL8cricket2015

48,329

31,526(65)

19,116(40)

9,251(19)

9,865(20)

22,275(46)

Gazaunrest2015

31,537

10,142(46)

6,157(20)

1,716(5)

4,441(14)

8,426(27)

AirAsiaflight
missing2014
Cricketworldcup
2015
Ebolaoutbreak
2014

crash2014

GiventhisgrowthofimagesandpicturesonOSM,andlessworkdoneontopicsrelatedtoOSM&images,thereisa
greatscopeforcontributinginthisdomain.TherearefullfledgedanddedicatedtraditionalconferenceslikeIEEE
InternationalConferenceonComputerVision,InternationalConferenceonMachineLearning(ICML),andIEEE
ConferenceonComputerVisionandPatternRecognition(CVPR)whichlookatimages.Thereneedssome
knowledgetransferfromtheseclassicdomainstoOSM.Itmayalsobethecasethat,inthepast,imageanalysiswas
notasadvancedasitisnow,so,advancementsinimageanalysis,includingneuralnetworksnowmakesitpossible
todosomereallycoolimageanalysiswhichcouldhavebeendifficultorimpossibletodoitearlier.Giventhelarge
amountofdataonOSM,andwithadvancedimageanalysistechniques,weshouldbeabletoanswersomevery
excitingresearchquestions.
SomespecifictopicsandproblemsthatIthinkthatwillbeinterestinginthisspaceofOSMandimages(thesearejust
myrandomthoughtsandtheyarenoncomprehensive):
Spreadofuntrustworthy/MisinformationonOSMthroughimages
Leakageofpersonalinformationlikecurrentlocation,etc.throughimagesonOSM
LeakageofsensitiveinformationlikeDOB,gender,etc.throughimagesonOSM
IfyouareinterestedinkeepingupdatedaboutouractivitiesatPrecog,youcanvisitourwebsiteorourFacbeook
pageIfyouhaveanysuggestionsorideastoexploreinthisdirection,feelfreetowritetome.
Acknowledgements:IthankmybrilliantstudentsPrateekDewan,NiharikaSachdeva,IndiraSen,KushagraSingh,
MeghaArora,HemankLamba,andVarunBharadhwajforhelpingwithputtingtogetherthesethoughts/some
numbers/analysisinthispost.ThankstoallmembersofPrecoggroupwheretheideaofstudyingimagesandtrying
itoutfromdifferentperspectivesstarted.
References
1.http://www.smartinsights.com/internetmarketingstatistics/happensonline60seconds/
2.http://www.businessinsider.com/werenowpostingastaggering18billionphotostosocialmediaeveryday
20145
3.http://www.adweek.com/socialtimes/twitterimagesstudy/493206
4.http://www.socialbakers.com/blog/1749photosmakeup93ofthemostengagingpostsonfacebook
5.https://people.csail.mit.edu/khosla/papers/www2014_khosla.pdf
http://precog.iiitd.edu.in/blog/2016/08/imagesononlinesocialmedia/

3/5

12/3/2016

APictureisWorth32.33Words:ImportanceofAnalyzingImagesonOnlineSocialMedia|PreCog'sBlog

6.PollyannaGonalves,MatheusArajo,FabrcioBenevenuto,andMeeyoungCha.2013.Comparingand
combiningsentimentanalysismethods.InProceedingsofthefirstACMconferenceonOnlinesocialnetworks
(COSN13).ACM,NewYork,NY,USA,2738.DOI=http://dx.doi.org/10.1145/2512938.2512951
7.TomerSimon,AvishayGoldberg,LimorAharonsonDaniel,DmitryLeykin,BruriaAdini.TwitterintheCrossFire
TheUseofSocialMediaintheWestgateMallTerrorAttackinKenya,PlosOne.
8.SarithaSK,DevshriroyD(2013)SemanticOrientationofSentimentAnalysisonSocialMedia.International
JournalofComputers&Technology11(4)24012409.
9.MunmunDeChoudhury,ScottCounts,andEricHorvitz.2013.PredictingPostpartumChangesinEmotionand
BehaviorviaSocialMedia.InProc.CHI13
10.MunmunDeChoudhury,ScottCounts,EricJHorvitz,andAaronHoff.2014.characterizingandpredicting
postpartumdepressionfromsharedfacebookdata.InProc.CSCW14.ACM,626638.
11.MunmunDeChoudhury,AndresMonroyHernandez,andGloriaMark.2014.NarcoEmotions:Affectand
DesensitizationinSocialMediaduringtheMexicanDrugWar.InProc.CHI14.ACM.
12.SatarupaGuha,TanmoyChakraborty,SamikDatta,MohitKumar,VasudevaVarma.TweetGrep:Weakly
SupervisedJointRetrievalandSentimentAnalysisofTopicalTweets.IntheproceedingsofICWSM2016.
13.SoroushVosoughi,DebRoy.ASemiAutomaticMethodforEfficientDetectionofStoriesonSocialMedia.Inthe
proceedingsofICWSM2016.
14.DavidAlvarezMelis,MartinSaveski.TopicModelinginTwitter:AggregatingTweetsbyConversations.Inthe
proceedingsofICWSM2016.
15.TimAlthoff,CristianDanescuNiculescuMizil,DanJurafsky.HowtoAskforaFavor:ACaseStudyonthe
SuccessofAltruisticRequests.IntheproceedingsofICWSM2014.
16.EfthymiosKouloumpis,TheresaWilson&JohannaMoore2011.TwitterSentimentAnalysis:TheGoodtheBad
andtheOMG!(ICWSM11)
17.AlexanderPakandPatrickParoubek2010.TwitterasaCorpusforSentimentAnalysisandOpinionMining.In
LREC,vol.10,pp.13201326.
18.AliakseiSeveryn,andAlessandroMoschitti.Twittersentimentanalysiswithdeepconvolutionalneural
networks.Proceedingsofthe38thInternationalACMSIGIRConferenceonResearchandDevelopmentin
InformationRetrieval.ACM,2015.
19.CceroNogueiradosSantos,andMairaGatti.DeepConvolutionalNeuralNetworksforSentimentAnalysisof
ShortTexts.COLING.2014.
20.DuyuTang,FuruWei,NanYang,MingZhou,TingLiu,andBingQin.LearningSentimentSpecificWord
EmbeddingforTwitterSentimentClassification.InACL(1),pp.15551565.2014.
21.1.Vaziripour,Elham,ChristopheGiraudCarrier,andDanielZappala.AnalyzingthePoliticalSentimentof
TweetsinFarsi.TenthInternationalAAAIConferenceonWebandSocialMedia.2016.
22.2.Peng,Nanyun,YimingWang,andMarkDredze.LearningPolylingualTopicModelsfromCodeSwitched
SocialMediaDocuments.ACL(2).2014.
23.3.Weerkamp,Wouter,SimonCarter,andManosTsagkias.Howpeopleusetwitterindifferentlanguages.
(2011):12.
24.4.Volkova,Svitlana,TheresaWilson,andDavidYarowsky.ExploringDemographicLanguageVariationsto
ImproveMultilingualSentimentAnalysisinSocialMedia.EMNLP.2013.

http://precog.iiitd.edu.in/blog/2016/08/imagesononlinesocialmedia/

4/5

12/3/2016

APictureisWorth32.33Words:ImportanceofAnalyzingImagesonOnlineSocialMedia|PreCog'sBlog

25.AnupamJamatia,BjornGambck,andAmitavaDas.2015.PartofSpeechTaggingforCodeMixedEnglish
HindiTwitterandFacebookChatMessages.ProceedingsofRecentAdvancesinNaturalLanguage
Processing,page239.
26.SujanKumarSaha,ParthaSarathiGhosh,SudeshnaSarkar,andPabitraMitra.2008.NamedEntity
RecognitioninHindiusingMaximumEntropyandTransliteration.ResearchjournalonComputerScienceand
ComputerEngineeringwithApplications,pp.3341.
27.AyushKumar,SarahKohail,AsifEkbal,andChrisBiemann.2015.IITTUDA:Systemforsentimentanalysisin
indianlanguagesusinglexicalacquisition.MiningIntelligenceandKnowledgeExploration,pages684693.
28.Gupta,A.,Lamba,H.,Kumaraguru,P.,andJoshi,A.FakingSandy:CharacterizingandIdentifyingFake
ImagesonTwitterduringHurricaneSandy.2ndInternationalWorkshoponPrivacyandSecurityinOnlineSocial
Media(PSOSM),inconjunctionwiththe22thInternationalWorldWideWebConference(WWW)(2013).
29.Gupta,A.,Lamba,H.,andKumaraguru,P.$1.00perRT#BostonMarathon#PrayForBoston:AnalyzingFake
ContentonTwitter.IEEEAPWGeCrimeResearchSummit(eCRS),2013.
30.https://www.facebook.com/americanmuslims1/photos/a.809524959106862.1073741828.527806667278694/990645217661501/?
type=1&theater
31.https://github.com/tesseractocr/tesseract
32.https://developers.facebook.com/docs/graphapi/reference/v2.7/post#read
33.https://www.facebook.com/ChristianChronicle/photos/a.83579936833.99565.11127431833/10153806013491834/?
type=3&theater
34.https://www.facebook.com/roberta.metsola/photos/a.406836966100205.1073741826.406824526101449/839065439544020/?
type=3&theater
35.https://www.facebook.com/516601545154233/photos/a.519535361527518.1073741828.516601545154233/574613042686416/?
type=3&theater
Like 37

Share

Tweet

Share

ThisentrywaspostedinResearchbyPK.Bookmarkthepermalink
[http://precog.iiitd.edu.in/blog/2016/08/imagesononlinesocialmedia/].

AboutPK
AssociateProfessor@IIITDelhihttp://precog.iiitd.edu.in
ViewallpostsbyPK

http://precog.iiitd.edu.in/blog/2016/08/imagesononlinesocialmedia/

5/5