7/2/2015

LinuxDevCenter.com:ImprovingNetworkReliabilitywithKeepalived

PublishedonLinuxDevCenter(http://www.linuxdevcenter.com/)
Seethisifyou'rehavingtroubleprintingcodeexamples

ImprovingNetworkReliabilitywithKeepalived
byPhilipHollenback
09/01/2005
Redundancyisoneofthekeywaysyoucanincreasethereliabilityofyournetwork.Astheconceptof
RAID(redundantarraysofinexpensivedisks)hasshown,itcanbemuchmorecosteffectivetogroup
anumberofinexpensivecomponentstogetherthantospendmuchmoremoneyononehighpriced
item.Youcanapplythesameideatoyournetwork:insteadofinvestinginoneveryexpensive
proprietaryrouter,whynotinstallseveralredundantLinuxroutersmadeoutofcommoditypartsand
freesoftware?ThisarticleshowshoweasyitistodojustthatwithKeepalivedonLinux.

Concepts
Theproblemwithroutingisthatmostclientcomputersdoitinthesimplestwaypossible:byusinga
defaultroute.Anynetworktrafficnotdestinedforthelocalnetworkgoeshappilyontothegateway
router,assumingthatitknowshowtosenditalongappropriately.
Thismakesthegatewayasinglepointoffailureforyournetwork.Ifitgoesdown,noneofyourclient
machinescancommunicatewiththeoutsideworld.
OneanswertothisistohaveyourclientmachinesrunaroutingprotocolsuchasRIPorOSPF.
Generallythisisn'tdone,duetotheincreasedcomplexityandoverhead.Thustheonlypracticalway
tomakeroutingmorerobustistofooltheclientsintothinkingtheyarealwayscommunicatingwith
onegatewayrouter.YoucanuseVRRP(VirtualRouterRedundancyProtocol)todothisthe
KeepalivedprogramprovidesoneimplementationofVRRPforLinux.
VRRPisanIETFprotocolthatallowstwoormorerouterstoactasonevirtualrouter.Accordingto
theVRRPspecification,therouterspresentavirtualIPaddress(VIP)thatcorrespondstoavirtual
MACaddress(VMAC).EachrouterhasarealhardwareandIPaddress.Initiallythemasterrouter
handlesthevirtualIPandMACaddresses.Ifthemasterrouterfails,thebackupthentakesoverthe
virtualaddresses.Themasterandbackuproutersmonitoreachotherwithregularmulticast
advertisements,atadefaultrateofonepersecond.
BecauseLinuxdoesnotcurrentlysupportVMACs,Keepalived
implementsonlyVIPs.Inpractice,thisworksfineonmodern
networks,althoughyoushouldbeawarethatthiscancause
problemsforolderhardwarethatdoesnotsupportgratuitousARP
requests.

RelatedReading

Innormaloperation,thebackupVRRPservermonitorscontinually
bylisteningformulticastadvertisementsfromthemaster.Ifthe
masterdisappears,thebackupsendsagratuitousARPmessageout
onthenetwork,whichsays,ineffect,"Iownthehardwareaddress
thatthemasterpreviouslyowned."Thiscausesallothersystemson
http://www.linuxdevcenter.com/lpt/a/6162

1/6

7/2/2015

LinuxDevCenter.com:ImprovingNetworkReliabilitywithKeepalived

thenetworktostartusingthebackupVRRPserverastheir
gateway.Thiscontinuesuntilthemasterserverreappears.Thekey
pointhereisthatnoreconfigurationisnecessaryontheclient
machinesitallhappensontheservers.
Whilethisarticlecoversthesimplecaseofonemasterandone
backupserver,inrealitytherecanbemultiplebackupserversfor
increasedreliability.TheVRRPprotocolworksonanelection
process:afailingmastercausesanelectiontohappen,andthe
highestprioritybackuptakesover.Ifthatbackupfails,thenext
takesover,andsoon.
Keepalivedisn'ttheonlyVRRPimplementationforLinuxfor
example,thereisalsoaVRRPdproject.However,itdoesn'tseem
tobeactivelymaintainedthelastsoftwarereleaseforthatproject
wasin2002.ThereisalsoanIETFoverviewoftheotherVRRP
implementations.

InstallingtheSoftware

LinuxinaNutshell
ByEllenSiever,Aaron
Weber,StephenFiggins,
RobertLove,Arnold
Robbins

KeepalivedmaycomepreinstalledonyourLinuxserver,depending
onwhichdistributionyouuse.(Itdoesn'tcomewithFedora,atleast
uptoFC3.)TherehavebeenmanyupdatestoKeepalivedinthepastfewmonths,soyouareprobably
bestoffdownloadingthelatestversiondirectlyfromtheKeepalivedwebsite.
KeepalivedinstallsandbuildsinthestandardUnixway:justunpackthetarballandfollowthe
instructionsintheinstallfile.MakesureyouputtheKeepalivedinitfileinyourinitdirectoryfor
example,/etc/rc.d/rc3.d/S99keepalived.
RememberthatyoualsoneedtoinstallKeepalivedonboththemasterandbackuprouters.The
installationisthesameonallserversexceptfordifferencesintheconfigurationfile.

Configuration
TheKeepalivedconfigurationusesasinglefile,/etc/keepalived/keepalived.conf.Thisfilecanbe
intimidating,becausetherearemanyconfigurationoptionsandthedocumentationisabitscattered.
Keepalivedincludesseveralotherhealthcheckmechanisms,andthedocumentationfocuseson
configurationssuchaswebserverfarms.Luckily,youcanignoremostofthoseconfigurationoptions
ifyouarejustconfiguringVRRP.Oneofmymaingoalsinwritingthisarticleistomakepeople
awarethatKeepalivedworksperfectlyfineasjustaVRRPserver.
Assumethatyourmasterrouterisataddress192.168.1.253andyourbackupisat192.168.1.254.
Traditionally,thegatewayonanetworkisonthe.1address,sosettheVRRPvirtualaddressto
192.168.1.1.Thatwayifyourexistingclientconfigurationsuseadefaultgatewayon192.168.1.1,you
won'thavetochangetheconfigurationoneachclientmachine.
Editkeepalived.confonthemastersothatitcontainsjustthefollowing:
vrrp_instanceVI_1{
stateMASTER
interfaceeth0
virtual_router_id1
priority100
authentication{
auth_typePASS
http://www.linuxdevcenter.com/lpt/a/6162

2/6

7/2/2015

LinuxDevCenter.com:ImprovingNetworkReliabilitywithKeepalived

auth_pass<password>
}
virtual_ipaddress{
192.168.1.1/24brd192.168.1.255deveth0
}
}

ThisfirstdefinesanewVRRPinstanceandcallsitVI_1.IfyouwanttorunVRRPonmultiple
interfacesonarouter,giveeachoneadifferentinstancename.TypicalnamesareVI_1,VI_2,andso
on,butyoucannamethemanythingyouwant.
ThenextlinedefinesthestateVRRPwillbeinwhenKeepalivedstarts.Becausethisisthemaster,
VRRPshouldstartinthemasterstatesothatitwillcontrolthevirtualIPaddress.
TheinterfacelinedefineswhichnetworkinterfacethisVRRPinstancewilloperateon,sotypically
thisiseth0oreth1orsomethingsimilar.KeepalivedworksjustfinewithVLANs,soyoucanuse
VLANaddressessuchaseth0.2aswell.

Thevirtual_router_idvaluematchesupmasterandbackupVRRPservers.Allserversinaparticular
VRRPgroup(onemasterandoneormorebackupservers)shouldhavethesamevrrp_router_id.
VRRPusesanelectionmechanismtodeterminewhoisthemasterinaVRRPgroup,andthehighest
prioritywins.Themastershouldhaveapriorityatleast50higherthanthatofanyofthebackup
servers,becausetheprioritycontributestothefailovertime.Inthisexample,ifyousetthemaster
priorityto100,noneofthebackupserversshouldhaveapriorityhigherthan50.Remember,Isaid
thestatesettingcontrolsthestateinwhichtheVRRPserverstartsup.Immediatelyafterstartup,all
VRRProutersinthesamegroup(onthesamenetworkandwiththesamevirtual_router_id)will
holdanelection.Theserverwiththehighestprioritywinsandbecomesmaster,evenifthatmachine
hasalowerprioritythanasystemthatstartedinthemasterstate.
TheVRRPspecificationdescribesseveralauthenticationmechanisms.Obviouslythereshouldbe
somewayfortheVRRPserverstocommunicatesecurely,becausearoguesystemcouldcreatea
denialofserviceattackonyournetworkbyoverridingyourrealVRRPmasterserver.Keepalived
supportsboththepasswordandIPSECAuthenticationHeaderauthenticationmethods,butpassword
authenticationiseasierfornormaluseduetosomeimplementationproblemswithIPSECAH
authentication.ThusinmyexampleIsettheauth_typetopassword(plainpassword)andIspecifya
passwordontheauth_passline.Notethatthisisaplaintextpasswordthatgoesoverthenetworkvery
often(atleastoncepersecondatthestandardadvertisementinterval),sothisisreallynotastrong
securitymeasure.
Finally,thefilesetsthevirtualrouteraddressinthevirtual_ipaddresssection.AsImentioned
earlier,itprobablymakessensetosetyourvirtualIPaddress(VIP)towhateveryourexistinggateway
wasusing,tominimizeclientconfigurationchanges.Youalsohavetospecifythedevicethisaddress
ison.Thisshouldmatchthevalueoftheinterfacesettingabove.
Youcanspecifymultipleaddressesinthevirtual_ipaddresssection.ThisisusefulifyourVRRP
serverisonseveraldifferentVLANs.Inthatcase,eachVIPgoesonaseparatelineandthedevice
entrycorrespondstotheVLANofthevirtualIPaddress.
That'sanentireminimalmasterkeepalived.conffile.Seethekeepalived.conf(5)manpageforthe
other,optionalsettings.
Thebackupserverkeepalived.confisalmostidentical.First,changethestatesettingtoBACKUP,asyou
wantthisservertocomeupinthebackupstate.Then,changetheprioritytoalowernumberthanthat
http://www.linuxdevcenter.com/lpt/a/6162

3/6

7/2/2015

LinuxDevCenter.com:ImprovingNetworkReliabilitywithKeepalived

ofthemaster.Rememberthatitshouldbeatleast50percentlowerthanthemasterpriority,so50isa
goodchoiceinthiscase.Everythingelseintheconfigurationfile(including,mostimportantly,the
auth_pass)shouldhavethesame.

TimetoStartKeepalived
NowthatyouhaveKeepalivedconfiguredonboththemasterandbackupservers,startVRRPby
runningtheKeepalivedinitscriptthatcameintheKeepalivedsourcetarball(assumingyouareona
RedHatorFedorasystemadjustforotherdistros):
#/sbin/servicekeepalivedstart

Dothisonboththemasterandslaveservers.Then,checkthesyslog(/var/log/messages)oneach
machine.YoushouldseemessagesindicatingthatKeepalivedhasstartedinmodeMASTERonthe
masterserverandmodeBACKUPontheslave.Howcanyoutellifthemasterserverisansweringon
thevirtualIPaddress?Thebestwaytocheckthisiswiththeipcommand.Run:
#ipaddrshow

onthemaster.Assumingthatyou'rerunningKeepalivedoneth0,youshouldseesomethinglikethis:
2:eth0:<BROADCAST,MULTICAST,UP>mtu1500qdiscnoqueue
link/ether00:e0:81:2b:aa:b5brdff:ff:ff:ff:ff:ff
inet192.168.1.253/24brd192.168.1.255scopeglobaleth0
inet192.168.1.1/24brd192.168.1.255scopeglobalsecondaryeth0

Theoutputontheslaveshouldbe:
2:eth0:<BROADCAST,MULTICAST,UP>mtu1500qdiscnoqueue
link/ether00:e0:81:2b:aa:c3brdff:ff:ff:ff:ff:ff
inet192.168.1.254/24brd192.168.1.255scopeglobaleth0

Thisshowsthatthemasterisansweringbothonits192.168.1.253addressandthevirtualIPof
192.168.1.1.Thisinformationisnotavailableinthetraditionalifconfigcommandoutput,sothisisa
goodreasontobitethebulletandstartusingtheipcommandtoviewandorchangeyournetwork
settingsifyouhaven'talreadybeenusingit.

Testing
TestingKeepalivedisstraightforward:unplugthemasterfromthenetworkandseeiftheslavetakes
over.YoucanseetheKeepalivedstatechangesinsyslog,howeveryouwon'tseeanythinginthe
mastersyslogwhenyoudisconnectitfromthenetwork.Themasteractuallydoesnoticethatthe
backuphasdisappearedandittransferstoafaultstatebecausethemasteralsolistensformulticast
advertisementsfromthebackup.Inpractice,themasterdoesn'tdoanythinginthefaultstateexcept
waittohearfromthebackup.
Thebackupismorechatty.Itssyslogwillcontainmessagessuchas:
Keepalived_vrrp:VRRP_Instance(VI_1)TransitiontoMASTERSTATE
Keepalived_vrrp:VRRP_Instance(VI_1)EnteringMASTERSTATE
Keepalived_vrrp:VRRP_Instance(VI_1)settingprotocolVIPs.
Keepalived_vrrp:VRRP_Instance(VI_1)SendinggratuitousARPoneth0

Thisshouldbeprettyeasytounderstand:thebackuplosttrackofthemaster,soitdecidedtobecome
master,takeovertheVIP,andsendagratuitousARPtonotifytheclients.Withthesample
configuration,thiswilltake3.6secondsatmost.
http://www.linuxdevcenter.com/lpt/a/6162

4/6

7/2/2015

LinuxDevCenter.com:ImprovingNetworkReliabilitywithKeepalived

Oncethistransitionoccurs,thebackupisnowintheMASTERstateandcontrolsthevirtualIP
address.Thiscontinuesuntilthemastercomesback(well,oruntilthebackupservergoesdown).
VerifythatthebackupisinchargeoftheVIPbyrunningipaddrshowonthebackupandverifying
thattheVIPisthere(asintheprevioussection).
Whenthemastercomesback,youwillseethisinthebackupserversyslog:
Keepalived_vrrp:VRRP_Instance(VI_1)Receivedhigherprioadvert
Keepalived_vrrp:VRRP_Instance(VI_1)EnteringBACKUPSTATE
Keepalived_vrrp:VRRP_Instance(VI_1)removingprotocolVIPs

Again,thisisprettyeasytofollow.Thebackupheardfromanother,moreimportantVRRPserver(the
master,becausethat'stheonlyotheroneintheexample),soitwenttotheBACKUPstateanddeletedthe
virtualIPaddresses.

Atthesametime,whenthemastercomesbackontothenetwork,itnoticesthebackupandforcesthe
backuptogiveuptheVIP:
Keepalived_vrrp:VRRP_Instance(VI_1)Receivedlowerprioadvert,
forcingnewelection
Keepalived_vrrp:VRRP_Instance(VI_1)SendinggratuitousARPoneth0

Atthispoint,themasterisbackincharge.NowyouknowthatyourKeepalivedsetupisworking.

FailoverTimeouts
Whyisthemaximumfailovertimeintheexample3.6seconds?Thiscomesfromtheadvertisement
intervalandtheskewtime.Thedefaultadvertisementintervalis1second(configurablein
keepalived.conf).Theskewtimehelpstokeepeveryonefromtryingtotransitionatonce.Itisa
numberbetween0and1,basedontheformula
(256priority)/256

AsdefinedintheRFC,thebackupmustreceiveanadvertisementfromthemasterevery
(3*advert_int)+skew_time

seconds.Ifitdoesn'thearanythingfromthemaster,ittakesover.Withabackuprouterpriorityof100
(asintheexample),thefailoverwillhappenatmost3.6secondsafterthemastergoesdown.

ClosingThoughts
Keepalivedprovidesarichsetoftoolsforservermonitoring.Forourpurposesofincreasingrouter
redundancy,themostinterestingoneisVRRP.TakeacoupleofLinuxrouters,addKeepalivedwith
VRRP,andyouhaveamuchmoreredundantconfiguration.
Ofcourse,itisimportanttonotethatthisisnotacompletesolution.Considerthestandardoffice
setupofoneT1connectedtoonerouter.Evenifyousetaabackuprouter,youdon'thavefull
protection:iftheonerouterwiththeT1goesdown,yourclientswilllosealltheirconnectivity.Any
completeredundancysolutionmustalsoconsiderexternalnetworklinks,notjustinternalVRRP
routers.TheIBMRedpaperonVRRPhassomegoodinformationondesigninganetworkwithrobust
upstreamrouting.
Inthepast,somepeoplehavehesitatedtoconsiderusingKeepalivedforjustaVRRPsetup,asthey
http://www.linuxdevcenter.com/lpt/a/6162

5/6

7/2/2015

LinuxDevCenter.com:ImprovingNetworkReliabilitywithKeepalived

perceiveKeepalivedasalargeandcomplexsystem.Icanassureyou,basedonmyexperience,that
thisisn'tthecase.Ifyouarerunningapoolofsystemssuchaswebservers,youshouldcheckoutthe
otherfeaturesKeepalivedhastooffer.However,ifyoujustwanttoaddrouterredundancytoyour
Linuxnetwork,VRRPviaKeepalivedisjusttheticket.
SpecialthankstoKeepaliveddeveloperAlexandreCassenforreviewingthisarticleandproviding
valuablefeedbackandcorrections.
PhilipHollenbackisasystemadministratoratafinancialfirminManhattan.Whenhe'snot
upgradingLinuxserversorskateboarding,Philspendshistimeupdatinghiswebsite,
www.hollenback.net.
ReturntotheLinuxDevCenter.
Copyright2009O'ReillyMedia,Inc.

http://www.linuxdevcenter.com/lpt/a/6162

6/6