SearchEngines

InformationRetrievalinPractice

AllslidesAddisonWesley,2008

SearchEngineArchitecture
Asoftwarearchitectureconsistsofsoftware components,theinterfacesprovidedbythose components,andtherelationshipsbetween them
describesasystemataparticularlevelofabstraction

Architectureofasearchenginedeterminedby2 requirements
effectiveness(qualityofresults)andefficiency (responsetimeandthroughput)

IndexingProcess

IndexingProcess
Textacquisition
identifiesandstoresdocumentsforindexing

Texttransformation
transformsdocumentsintoindextermsor features

Indexcreation
takesindextermsandcreatesdatastructures (indexes)tosupportfastsearching

QueryProcess

QueryProcess
Userinteraction
supportscreationandrefinementofquery,display ofresults

Ranking
usesqueryandindexestogeneraterankedlistof documents

Evaluation
monitorsandmeasureseffectivenessand efficiency(primarilyoffline)

Details:TextAcquisition
Crawler
Identifiesandacquiresdocumentsforsearch engine Manytypes web,enterprise,desktop Webcrawlersfollowlinks tofinddocuments
Mustefficientlyfindhugenumbersofwebpages (coverage)andkeepthemuptodate(freshness) Singlesitecrawlersforsitesearch Topicalor focusedcrawlersforvertical search

Document crawlersforenterpriseanddesktop search

Followlinksandscandirectories

TextAcquisition
Feeds
Realtimestreamsofdocuments
e.g.,webfeedsfornews,blogs,video,radio,tv

RSSiscommonstandard
RSSreadercanprovidenewXMLdocumentstosearch engine

Conversion
Convertvarietyofdocumentsintoaconsistenttext plusmetadataformat
e.g.HTML,XML,Word,PDF,etc.XML

Converttextencodingfordifferentlanguages
UsingaUnicodestandardlikeUTF8

TextAcquisition
Documentdatastore
Storestext,metadata,andotherrelatedcontent fordocuments
Metadataisinformationaboutdocumentsuchastype andcreationdate Othercontentincludeslinks,anchortext

Providesfastaccesstodocumentcontentsfor searchenginecomponents
e.g.resultlistgeneration

Coulduserelationaldatabasesystem
Moretypically,asimpler,moreefficientstoragesystem isusedduetohugenumbersofdocuments

TextTransformation
Parser
Processingthesequenceoftexttokensinthe documenttorecognizestructuralelements
e.g.,titles,links,headings,etc.

Tokenizer recognizeswordsinthetext
mustconsiderissueslikecapitalization,hyphens, apostrophes,nonalphacharacters,separators

MarkuplanguagessuchasHTML,XMLoftenusedto specifystructure
Tags usedtospecifydocumentelements
E.g.,<h2>Overview</h2>

Documentparserusessyntax ofmarkuplanguage(orother formatting)toidentifystructure

TextTransformation
Stopping
Removecommonwords
e.g.,and,or,the,in

Someimpactonefficiencyandeffectiveness Canbeaproblemforsomequeries

Stemming
Groupwordsderivedfromacommonstem
e.g.,computer,computers,computing,compute

Usuallyeffective,butnotforallqueries Benefitsvaryfordifferentlanguages

TextTransformation
LinkAnalysis
Makesuseoflinks andanchortextinwebpages Linkanalysisidentifiespopularity andcommunity information
e.g.,PageRank

Anchortextcansignificantlyenhancethe representationofpagespointedtobylinks Significantimpactonwebsearch

Lessimportanceinotherapplications

TextTransformation
InformationExtraction
Identifyclassesofindextermsthatareimportant forsomeapplications e.g.,namedentityrecognizersidentifyclasses suchaspeople, locations, companies, dates, etc.

Classifier
Identifiesclassrelatedmetadatafordocuments
i.e.,assignslabelstodocuments e.g.,topics,readinglevels,sentiment,genre

Usedependsonapplication

IndexCreation
DocumentStatistics
Gatherscountsandpositionsofwordsandother features Usedinrankingalgorithm

Weighting
Computesweightsforindexterms Usedinrankingalgorithm e.g.,tf.idf weight
Combinationoftermfrequencyindocumentand inversedocumentfrequencyinthecollection

IndexCreation
Inversion
Coreofindexingprocess Convertsdocumentterminformationtoterm documentforindexing
Difficultforverylargenumbersofdocuments

Formatofinvertedfileisdesignedforfastquery processing
Mustalsohandleupdates Compressionusedforefficiency

IndexCreation
IndexDistribution
Distributesindexesacrossmultiplecomputers and/ormultiplesites Essentialforfastqueryprocessingwithlarge numbersofdocuments Manyvariations
Documentdistribution,termdistribution,replication

P2P anddistributedIR involvesearchacross multiplesites

UserInteraction
Queryinput
Providesinterfaceandparserforquerylanguage Mostwebqueriesareverysimple,other applicationsmayuseforms Querylanguageusedtodescribemorecomplex queriesandresultsofquerytransformation
e.g.,Booleanqueries,IndriandGalago querylanguages similartoSQLlanguageusedindatabaseapplications IRquerylanguagesalsoallowcontentandstructure specifications,butfocusoncontent

UserInteraction
Querytransformation
Improvesinitialquery,bothbeforeandafterinitial search Includestexttransformationtechniquesusedfor documents Spellcheckingandquerysuggestion provide alternativestooriginalquery Queryexpansionandrelevancefeedback modify theoriginalquerywithadditionalterms

UserInteraction
Resultsoutput
Constructsthedisplayofrankeddocumentsfora query Generatessnippets toshowhowqueriesmatch documents Highlights importantwordsandpassages Retrievesappropriateadvertising inmany applications Mayprovideclustering andothervisualization tools

Ranking
Scoring
Calculatesscoresfordocumentsusingaranking algorithm Corecomponentofsearchengine Basicformofscoreis qi di
qi anddi arequeryanddocumenttermweightsfor termi

Manyvariationsofrankingalgorithmsand retrievalmodels

Ranking
Performanceoptimization
Designingrankingalgorithmsforefficient processing
Termatatimevs.documentatatime processing Safe vs.unsafe optimizations

Distribution
Processingqueriesinadistributedenvironment Querybrokerdistributesqueriesandassembles results Caching isaformofdistributedsearching

Evaluation
Logging
Logginguserqueriesandinteractioniscrucialfor improvingsearcheffectivenessandefficiency Querylogsandclickthrough datausedforquery suggestion,spellchecking,querycaching,ranking, advertisingsearch,andothercomponents

Rankinganalysis
Measuringandtuningrankingeffectiveness

Performanceanalysis
Measuringandtuningsystemefficiency

HowDoesItReally Work?
Thiscourseexplainsthesecomponentsofa searchengineinmoredetail Oftenmanypossibleapproachesandtechniques foragivencomponent
Focusisonthemostimportantalternatives i.e.,explainasmallnumberofapproachesindetail ratherthanmanyapproaches Importancebasedonresearchresultsandusein actualsearchengines Alternativesdescribedinreferences