CPSC 490

Backtracking: Other Techniques

Bidirecti o n a l Searc h
Bidirectionalsearchisaneattrickthatcanoftentakeanexponentialalgorithmandmakeitrunon problems that are twice the size of those it could previously solve. This may sound like a small improvement,butconsidera 2n algorithmthatworksfornupto25.Ifbidirectionalsearchcan beapplied,andncanbeincreasedto50,thenwehavejustmadeanimprovementinrunningtimeby afactorofseveralmillion! Let'sstartwithanexample.Youprobablyknowtheclassic"15puzzle".Itisa4x4gridwith15squares slidingup,down,leftandrightonit.Eachsquarehastheanumberbetween1and15onit,andthe goalistoarrangethesquaresinincreasingorderwiththeholeappearinginthelowerrightcorner. Here is a nice, short description and a few random facts about the 15 puzzle: http://mathworld.wolfram.com/15Puzzle.html. Suppose that we are given a scrambled puzzle and askedtofindtheshortestsequenceofmovesthatsolvesit. Thisisjustagraphsearchproblem.Eachpuzzleconfigurationisavertex,andithasupto4edges comingoutofitbecausethereareatmost4moves available.Theproblemisthereare16!vertices, soasimpleBFSwillnotwork,butlet'sconsideritanyway.Westartatthegivenconfigurationand exploreitsneighboursinthegraph,markingthemasvisitedandaddingthemtotheBFSqueue,as usual.Weshouldalsostorethemovethatweusedtoconstructeachnewconfigurationsothatwecan recoverthemovesequencelater. Thefirstquestionhereis:"Howdowerepresentaconfiguration?"Wecouldusea4x4arrayora vectorofvectorsofintegers,orastring.Aneatapproachistonoticethatwehave16squares,each containing an integer in the range 0 though 15 (0 representing the hole). So each cell can be representedbya4bitnumberfor atotalof4*16=64bits.Thisfitsnicelyintoa64bitinteger(long longtypeingcc/g++).Sincewewillbe generatinglotsoftheseconfigurations,usingthiscompact representationisaverygoodidea,especiallybecausewearegoingtoneedamaporahashtableof somesorttostorethedepthandpredecessorofeachnodeduringBFS. NowwehaveaBFSalgorithmonagraphofsize16factorial.WecouldterminateourBFSassoonas wereachthetargetconfigurationandavoidexploringthewholegraph.Thiswillworkforquiteafew probleminstances.Unfortunately,theintheworstcase,thenumberofmovesrequiredtosolvethe puzzleis80,andtheBFSsolutionwillcrawltoahaltafterabout20orsomoves.Althoughsolvingthe puzzleinthegeneralcaseisverydifficult,wecanatleastimproveoursolutiontoworkoninstances thatrequireatmost40moves. Thisiswherebidirectionalsearchcomesin.Weknowtwothingstheinitial,scrambledconfiguration (callitS)andthefinal,solvedone(let'scallitT).Untilnow,wehavebeensearchingfromStoT,but theproblemissymmetricwemightaswellhavebeensearchingfromTtoS.Infact,thisinitselfisa betterideabecauseTisfixed,andSisdifferentineveryinstanceoftheproblem,sowecoulddojust onecompleteBFSbackwardsfromTandthenimmediatelysolveanyprobleminstancebytracingthe path fromStoT.Thiswouldwork ifwehadenoughmemory tostore all16!configurationsand enoughCPUtimetocomputethem. AbetterideaistoruntwoBFSprocessesonefromSforwardandonefromTbackward.Bothwill growBFStreeswithconfigurationsasthevertices.Ifweeverhitthesamevertexfrombothsides,then wearedone.Thisvertex(callitv)isanintersectionpointbetweentheBFStreeofSandtheBFStree ofT,sowecantracethepathfromStovandappendtoitthepathfromvtoT,andwegetashortest pathfromStoTouranswer.Ifwecanmanagetogettothedepthof19ineachtree,thenthetotal pathlengthwillbe38almosttwiceasgoodaswithasingleBFS,butweneedtwicetheamountof memoryforit. BidirectionalsearchworksinmanysituationswherewearesearchingforapathofsomesortfromX

CPSC 490

Backtracking: Other Techniques

toY,andeachmoveonthepathisreversibletoallowforabackwardssearch.Toimplementthetwo searches,wecouldeitheralternatebetweenthem,makingonemoveatatime(popfromtheBFS queueandexploreneighbours),orwecouldrunonesearchtoafixeddepth,andthenruntheother search.

Iterativ e Deepe n i n g
We have seen backtracking with branchandbound that looked like a DFS. We have also seen bidirectionalsearchwithBFS.Thelatterrequiredalotofmemoryitisproportionaltothenumberof visitedvertices.Fortunately,eachvertexonlyhasatmost4neighbours,manyofthemrepeated,sowe couldgettoadepthofabout20beforerunningintomemoryproblems.Whatifeachvertexhas10 neighbours?TheneachlevelintheBFStreewouldbe10timeslargerthanthepreviouslevel,andwe wouldstarthavingmemoryproblemsonlevel7or8."Forgetting"DFSfromlasttime,ontheother hand,requiresverylittlememory;itsmemoryusageisproportionaltothedepthofthecurrentpath. IterativedeepeningallowsustosimulateBFSwithmultiplerunsofDFSatthecostofdoublingthe runningtime,butinmostcasestheextracostisinsignificant.Themainideaistoaddadepthlimit totheDFSandmaketherecursivefunctionreturnimmediatelyoncethelimitisreached.Wefirstrun itwithdepthlimit0.Thiswillonlyvisitthestartingvertex,S.Thenwerunitagainwithdepthlimit1. This time DFS will visit S and all of its immediate neighbours. We continue this way, each time increasingthedepthlimitby1untilwefindthedestination. Considerthefollowingproblem.Anadditionchainforagivennumber,K,isanincreasingsequenceof integersthatstartswith1andendswithK.Eachmemberofthe sequence,exceptforthefirstoneis thesumoftwooftheprevious,notnecessarilydistinctmembers.Forexample,(1,2,3,4,7)isan additionchainfor7because1+1=2,1+2=3,1+3=4and3+4=7.Theproblemistofindtheshortest additionchainforagivenK.(1,2,3,4,7)isashortestadditionchainfor7,butitisnotunique.(1,2, 4,6,7)works,too. Additionchainscanbeusedinpublickeycryptography.ComputingtheshortestadditionchainofKis afamousprobleminnumbertheory,andthereareseveralheuristicsthatcanbeused.Wewilllookat howiterativedeepeningcanhelphere. Thesimplestbacktrackingsolutionistostartwith1andsuccessivelyaddoneintegertothesequence, tryingallpossiblevaluesforthenewinteger.However,thisapproachtendstogeneratelongchainsof theform(1,2,3,4,...).WhatwewouldideallylikeistorunBFSfromtheinitialsequence(1)that wouldgiveustheshortestpath(shortestadditionchain)toK.However,thiswouldrequirekeeping lotsofpartiallybuiltadditionchainsinaqueue,eatinguptoomuchmemory.Thereisabetterway. SupposethattheoptimaladditionchainforKhaslengthR.Thenifwebuildonlythoseadditions chains that are no longer than R, we will find the answer. Unfortunately, we do not know R in advance,butwecantryalllengthsincrementally.First,wegeneratealladditionchainsoflength1 (there'sonlyoneofthose).Thenwegenerateallchainsoflength2(only1again).Thenfindallof length3,etc.Thisisdonebyaddingamaximumrecursiondepthvariabletothebacktrackingfunction. Oncethedepthlimitisreached,wecuttherecursionbranchandmoveontoadifferentbranch. Aftertryingonemaximum depth,wethrowawayeverythingandstartcleansearchingtoahigher depth.Thistechniqueworksquitewellforthisproblem.Straightforwardbacktrackingwillnotbeable tohandleKvalueshigherthan30or40.Iterativedeepeningincreasesthelimittoover1000. Itmayseemlikeawasteofcomputationtimetothrowawayallpreviouscomputationandstartanew

CPSC 490

Backtracking: Other Techniques

withthenextdepthlimit,butwearenotlosingthatmuchtime.Supposethatwehaveanexponential backtracking algorithm that takes 2n steps to explore every configuration up to depth n. Then runningittoalldepthsuptonwillrequire 2 021...2n=2 n1 stepsonlytwiceasmanyasfor depthnalone.Ifthebranchingfactorislargerthan2,thenthedifferenceinrunningtimeiseven smaller.And,ofcourse,thebenefitofiterativedeepeningisthatwesavealotofmemorycompareto plainBFSsearch.

Split and Merge

Afamousproblemsolvingstrategyis"divideandconquer"breakdownaproblemintosubproblems, thencombinetheirresultsintoasolutionofthebiggerproblem.Whenaproblemcanbesolvedthis way,thealgorithmisusuallyfastandefficient,becausethe"division"stepgeneratesalinearnumber of problem instances. While it's unlikely that we canuse divide and conquer on an NPcomplete problem,itsideaisstillquiteuseful.SplitandMergeisaclevertrickthatusesthisideaandmakesan algorithmabletosolveproblemstwicethesizeofthosethatitcouldsolvepreviously. Considerafamous NPcompleteproblem the Partition Problem. Thepartition problem asks, if givenasetofpositiveintegers,canwedividethesetintotwodisjointsubsetssothatthesumofone subsetequalsthesumoftheother.Forexample,giventhesetS={1,2,4,7,9,13},wecanbreakit intoS1={1,4,13}and S2={2,7,9},andweget1+4+13=18=2+7+9. Thereareothersolutions, too,forthisset,butgivenanarbitraryset,canwedeterminewhetherthereissuchadivision?(We ignorethetrivialcasewhenthetotalsumofthegivensetisodd.) Let'slookatthestraightforwardbacktrackingsolutiontothisproblem.Althoughwecanstillmodel thisproblemasagraphsearchproblem,thegraphbecomesverymessyandinconvenienttouse.We canusethestructureoftheproblemtoouradvantage.Theproblemstatementasksthatwedividethe setintotwodisjointsubsets,soeachnumberinthesetcanbeeitherinthefirstsubsetornotinit. Hence,thealgorithmisquitesimplewerecurseoneachnumberinsomeorder,andforeachnumber wefirstuseitandrecurse,andthenbacktracktonotusethenumberandrecurseagain.Forasetof n size n, this gives a 2 algorithm. Implementing this algorithm is a fun and rewarding experience,asitillustratessomeofthemostfundamentalconceptsofrecursivesearch. Thisis,however,quiteslow. Whenngetstoabout29,therunningtimeofthealgorithmhits10 seconds(worstcase),andavalueofn=60wouldbeunbearable.Wecandobetterthough,byusing theSplitandMergetrickmentionedabove.Thismethodfirstsplitstheproblemintotwoparts,runs theexponentialalgorithmoneachpartindividually,andtriestocombinetheresultsfrombothsidesto solvethemainproblem.Forthepartitionproblem,wewouldfirstsplitthesetintotwopartsofthe same(ornearlythesame)sizearbitrarily.Notethatthissplithasnothingtodowiththetwodisjiont subsets;thesplitsimplydividesthesizeby2sothattheexponentialalgorithmcanhandleeachpart. Now,tocombinetheresultsfromthetwoparts,weneedtoknowallpossiblesumsfromthefirstand secondpart,sothatwecancheckwhetherthereisasumfromthefirstpartandanotherfromthe secondpartthatadduptohalfthetotalsum.Whatthismeansisthat,whenwerecurseonthefirst partandgettothelastinteger,westoretheaccumulatedsuminaSTLset.Now,whenwerecurseon thesecondpartandgettothelastinteger,wecheckintheSTLsetforacorrespondingsumthatgives usasolution.Ineffect,wearemergingthesumsfromthetwopartstogethertoformonesolution. Howdoessplitandmergealgorithm affecttheruntimeandmemoryofouralgorithm? First,the n/ 2 memoryusageishuge,asforasetofsizen,wewouldhave 2 numberofsumsforeachpart,

CPSC 490

Backtracking: Other Techniques

sotheSTLsethasexponentialsize.Apracticallimitfortoday'scomputersisaround256MB,which means n/2 can be as big as 2426 (depends on implementation). The time complexity is quite interesting. Forthefirstrecursion,wehaveanalgorithmthatinserts O 2n/ 2 sumsintotheset. Eachinsertiontakeslogarithmictime,sothetoaltimeis

n O 2n/2 log 2n/ 2 =O 2n/ 2 . 2

Forthesecondrecursionwedon'tinsertanything,butwesearchwithintheSTLsetthesamenumber oftimes,andasearchalsotakeslogarithmictime,sothetimeforthesecondpartisthesameasthe firstpart.Thistotalsuptoa O n 2n/ 2 algorithm.Hence,atalargecostonmemory,wemanage toturnanalgorithmintoonethatsolvesprobleminstancesthatisalmosttwicethesize. Withthis algorithm,itispossibletosolvetheproblemwithnasbigas50(bearingmemorylimit). Onemayasknow,whycouldn'twesplitthesmallerpartsevenmore,andsolvetheirsubproblems, likeinDivideandConquer?Theansweristhatwhenwesplit,thequestionthatweaskonthesmaller partsisdifferentfromthequestionthatweaskontheoriginalset.Forthebiggerproblem,wewantto knowwhetherthereissomepartitionoftheset;butforeachsmallerpart,wewanttoknowevery possiblesumthatthepartcanmake. Wecanmergetheresultsonlybecausetheresultswegather fromthesmallerpartsarecompatibleinsolvingthelargerproblem. Tosplitthesmallerpartseven n/ 2 more,wewouldhavetofindallsumsofthepartintimelessthan 2 ,butthisistheoretically impossible(ingeneral)astheoutputisofthesamesize.(AlthoughlateroninDynamicProgramming we'll see how some restrictions on the size of the integers in the input set can improve this part significantly). The discerning reader will have probably noticed that Split and Merge is much like bidirectional search. Indeed, split and merge is only a special case of bidirectional search. We mention it particularlybecausesomeproblemscannotbeeasilytransformedintoagraphsearchproblem,andare morecombinatorialinnature. Intheseproblems(likethepartitionproblem),wegetto choose at eachpointsomevariablestodeterminethenextpointinthesearch. Ingeneral,theseproblemsare moresusceptibletoattackbysplitandmerge,becausethetrickitselfiscombinatorialinnature.