3/31/2017 AnIllustratedGuidetoIPsec

SteveFriedl'sUnixwiz.netTechTips
AnIllustratedGuidetoIPsec

IPsecisasuiteofprotocolsforsecuringnetworkconnections,butthedetailsandmanyvariationsquicklybecomeoverwhelming.Thisisparticularlythecasewhentryingtointeroperatebetween
disparatesystems,causingmorethanoneengineertojustmindlesslyturntheknobswhenattemptingtobringupanewconnection.

TableofContents ThisTechTipmeanstogivebottomupcoverageofthelowlevelprotocolsusedinanIPv4context(weprovidenocoverage
ofIPv6).Thisisnotadeploymentguideorbestpracticesdocumentwe'relookingatitstrictlyattheprotocollevelonup,
1.Somanyflavors... ratherthanfromthebigpictureondown.
2.TheIPDatagram
NOTE:originallythiswastobeapairofpapers,withthesecondcoveringKeyExchangeandthelike,butitappearsthatthis
3.AH:Authenticationonly
wasnotmeanttobe.Sorry.
4.ESP:EncapsulatingSecurityPayload

5.BuildingarealVPN
6.OtherMatters

Somanyflavors...
OneofthefirstthingsthatonenoticeswhentryingtosetupIPsecisthattherearesomanyknobsandsettings:evenapairofentirelystandardsconformingimplementationssportsabewildering
numberofwaystoimpedeasuccessfulconnection.It'sjustanastonishinglycomplexsuiteofprotocols.
OnecauseofthecomplexityisthatIPsecprovidesmechanism,notpolicy:ratherthandefinesuchandsuchencryptionalgorithmoracertainauthenticationfunction,itprovidesaframeworkthatallows
animplementationtoprovidenearlyanythingthatbothendsagreeupon.
Inthissection,we'lltouchonsomeoftheitemsintheformofaglossary,withacompareandcontrasttoshowwhichtermsrelatetowhichotherterms.Thisisnotevenremotelycomplete.

AHversusESP
"AuthenticationHeader"(AH)and"EncapsulatingSecurityPayload"(ESP)arethetwomainwirelevelprotocolsusedbyIPsec,andtheyauthenticate(AH)andencrypt+authenticate(ESP)thedata
flowingoverthatconnection.Theyaretypicallyusedindependently,thoughit'spossible(butuncommon)tousethembothtogether.
TunnelmodeversusTransportmode
TransportModeprovidesasecureconnectionbetweentwoendpointsasitencapsulatesIP'spayload,whileTunnelModeencapsulatestheentireIPpackettoprovideavirtual"securehop"between
twogateways.ThelatterisusedtoformatraditionalVPN,wherethetunnelgenerallycreatesasecuretunnelacrossanuntrustedInternet.
MD5versusSHA1versusDESversus3DESversusAESversusblahblahblah
SettingupanIPsecconnectioninvolvesallkindsofcryptochoices,butthisissimplifiedsubstantiallybythefactthatanygivenconnectioncanuseatmosttwoor(rarely)threeatatime.
AuthenticationcalculatesanIntegrityCheckValue(ICV)overthepacket'scontents,andit'susuallybuiltontopofacryptographichashsuchasMD5orSHA1.Itincorporatesasecretkeyknownto
bothends,andthisallowstherecipienttocomputetheICVinthesameway.Iftherecipientgetsthesamevalue,thesenderhaseffectivelyauthenticateditself(relyingonthepropertythat
cryptographichashescan'tpracticallybereversed).AHalwaysprovidesauthentication,andESPdoessooptionally.
Encryptionusesasecretkeytoencryptthedatabeforetransmission,andthishidestheactualcontentsofthepacketfromeavesdroppers.Therearequiteafewchoicesforalgorithmshere,withDES,3DES,Blowfish
andAESbeingcommon.Othersarepossibletoo.
IKEversusmanualkeys
Sincebothsidesoftheconversationneedtoknowthesecretvaluesusedinhashingorencryption,thereisthequestionofjusthowthisdataisexchanged.Manualkeysrequiremanualentryofthesecretvaluesonboth
ends,presumablyconveyedbysomeoutofbandmechanism,andIKE(InternetKeyExchange)isasophisticatedmechanismfordoingthisonline.
Mainmodeversusaggressivemode
ThesemodescontrolanefficiencyversussecuritytradeoffduringinitialIKEkeyexchange."Mainmode"requiressixpacketsbackandforth,butaffordscompletesecurityduringtheestablishmentofanIPsec
connection,whileAggressivemodeuseshalftheexchangesprovidingabitlesssecuritybecausesomeinformationistransmittedincleartext.

We'llcertainlyfacemoreoptionsasweunwrapIPsec.

TheIPDatagram
Sincewe'relookingatIPsecfromthebottomup,wemustfirsttakeabriefdetourtorevisittheIPHeaderitself,whichcarriesallofthetrafficwe'llbeconsidering.Notethatwearenottryingtoprovidecomprehensive
coveragetotheIPheaderthereareotherexcellentresourcesforthat(thebestbeingTCP/IPIllustrated,vol1).

ver
Thisistheversionoftheprotocol,whichisnow4=IPv4

http://www.unixwiz.net/techtips/iguideipsec.html 1/13
3/31/2017 AnIllustratedGuidetoIPsec

hlen
IPHeaderlength,asafourbitnumberof32bitwordsrangingfrom0..15.AstandardIPv4headerisalways20byteslong(5words),andIPOptionsifanyare
indicatedbyalargerhlenfielduptoatmost60bytes.Thisheaderlengthneverincludesthesizeofpayloadorotherheadersthatfollow.
TOS
TypeofService
Thisfieldisabitmaskthatgivessomecluesastothetypeofservicethisdatagramshouldreceive:optimizeforbandwidth?Latency?Lowcost?Reliability?
pktlen
Overallpacketlengthinbytes,upto65535.Thiscountincludesthebytesoftheheader,sothissuggeststhatthemaximumsizeofanypayloadisatleast20bytesless.The
vastmajorityofIPdatagramsaremuch,muchsmaller.
ID
TheIDfieldisusedtoassociaterelatedpacketsthathavebeenfragmented(largepacketsbrokenupintosmallerones).
flgs
Thesearesmallflagsthatmainlycontrolfragmentation:onemarksthepacketasineligibleforfragmentation,andtheothersaysthatmorefragmentsfollow.
fragoffset
Whenapacketisfragmented,thisshowswhereintheoverall"virtual"packetthisfragmentbelongs.
TTL
ThisistheTimetoLive,andisdecrementedbyeachrouterthatpassesthispacket.Whenthevaluereacheszero,itsuggestssomekindofroutingloop,soit'sdiscardedto
preventitfromrunningaroundtheInternetforever.
proto
Thisrepresentstheprotocolcarriedwithinthispacket,andit'sgoingtobecentraltomostofourdiscussions.ThoughthedatagramitselfisIP,italwaysencapsulatesasubsidiaryprotocol(TCP,UDP,ICMP,etc.see
thechartbelow)within.Itcanbethoughtofasgivingthetypeoftheheaderthatfollows.
headercksum
ThisholdsachecksumoftheentireIPheader,andit'sdesignedtodetecterrorsintransit.Thisisnotacryptographicchecksum,anditdoesn'tcoveranypartofthedatagramthatfollowtheIPheader.
srcIPaddress
The32bitsourceIPaddress,whichtherecipientusestoreplytothisdatagram.Generallyspeaking,it'spossibletospooftheseaddresses(i.e.,lieaboutwherethedatagramiscomingfrom).
dstIPaddress
The32bitdestinationIPaddress,whichiswherethepacketisintendedtoarrive.
IPOptions
TheseareanoptionalpartoftheIPheaderthatcontainsapplicationspecificinformation,thoughtheyarenotcommonlyusedforroutinetraffic.ThepresenceofIPoptionsisindicatedbyahlengreaterthan5,and
they(ifpresent)areincludedintheheaderchecksum.
Payload
EachprotocoltypeimpliesitsownformatforwhatfollowstheIPheader,andwe'veusedTCPherejusttoshowanexample.

TheseprotocodesaredefinedbyIANAtheInternetAssignedNumbersAuthorityandtherearemanymorethanwouldeverbeusedbyanysingleinstallation,butmostwillringabellwithanetworksavvytechnician.
TheserepresentativetypesaretakenfromtheIANAwebsitelistingprotocols:
SomeIPprotocolcodes

Protocol
ProtocolDescription
code

1 ICMPInternetControlMessageProtocol

2 IGMPInternetGroupManagementProtocol

4 IPwithinIP(akindofencapsulation)

6 TCPTransmissionControlProtocol

17 UDPUserDatagramProtocol

41 IPv6nextgenerationTCP/IP

47 GREGenericRouterEncapsulation(usedbyPPTP)
http://www.unixwiz.net/techtips/iguideipsec.html 2/13
3/31/2017 AnIllustratedGuidetoIPsec

Protocol
ProtocolDescription
code

50 IPsec:ESPEncapsulatingSecurityPayload

51 IPsec:AHAuthenticationHeader

We'llbestudyingthelasttwoindetail.

AH:AuthenticationOnly
AHisusedtoauthenticatebutnotencryptIPtraffic,andthisservesthetreblepurposeofensuringthatwe'rereallytalkingtowhowethinkweare,detectingalterationofdatawhileintransit,and(optionally)toguard
againstreplaybyattackerswhocapturedatafromthewireandattempttoreinjectthatdatabackontothewireatalaterdate.
AuthenticationisperformedbycomputingacryptographichashbasedmessageauthenticationcodeovernearlyallthefieldsoftheIPpacket(excludingthosewhichmightbemodifiedintransit,suchasTTLortheheader
checksum),andstoresthisinanewlyaddedAHheaderandsenttotheotherend.
ThisAHheadercontainsjustfiveinterestingfields,andit'sinjectedbetweentheoriginalIPheaderandthepayload.We'lltouchoneachofthefieldshere,thoughtheirutilitymaynotbe
fullyapparentuntilweseehowthey'reusedinthelargerpicture.

nexthdr
Thisidentifiestheprotocoltypeofthefollowingpayload,andit'stheoriginalpackettypebeingencapsulated:thisishowtheIPsecheader(s)arelinkedtogether.
AHlen
Thisdefinesthelength,in32bitwords,ofthewholeAHheader,minustwowords(this"minustwowords"provisospringsfromtheformatofIPv6'sRFC1883ExtensionHeaders,of
whichAHisone).
Reserved
Thisfieldisreservedforfutureuseandmustbezero.
SecurityParametersIndex
Thisisanopaque32bitidentifierthathelpstherecipientselectwhichofpossiblymanyongoingconversationsthispacketapplies.EachAHprotectedconnectionimpliesahashalgorithm(MD5,SHA1,etc.),somekind
ofsecretdata,andahostofotherparameters.TheSPIcanbethoughtofasanindexintoatableofthesesettings,allowingforeasyassociationofpacketwithparameter.
SequenceNumber
Thisisamonotonicallyincreasingidentifierthat'susedtoassistinantireplayprotection.Thisvalueisincludedintheauthenticationdata,somodifications(intentionalorotherwise)aredetected.
AuthenticationData
ThisistheIntegrityCheckValuecalculatedovertheentirepacketincludingmostoftheheadersTherecipientrecomputesthesamehashMismatchedvaluesmarkthepacketaseitherdamagedintransit,ornot
havingthepropersecretkey.Thesearediscarded.

TransportMode
TheeasiestmodetounderstandisTransportMode,whichisusedtoprotectanendtoendconversationbetweentwohosts.Thisprotectioniseitherauthenticationorencryption(orboth),butitisnotatunnelingprotocol.It
hasnothingtodowithatraditionalVPN:it'ssimplyasecuredIPconnection.

http://www.unixwiz.net/techtips/iguideipsec.html 3/13
3/31/2017 AnIllustratedGuidetoIPsec

InAHTransportMode,theIPpacketismodifiedonlyslightlytoincludethenewAHheaderbetweentheIPheaderandtheprotocolpayload(TCP,UDP,etc.),andthereisashufflingoftheprotocolcodethatlinksthevarious
headerstogether.
ThisprotocolshufflingisrequiredtoallowtheoriginalIPpackettobereconstitutedattheotherend:aftertheIPsecheadershavebeenvalidateduponreceipt,they'restrippedoff,andtheoriginalprotocoltype(TCP,UDP,
etc.)isstoredbackintheIPheader.We'llseethischainofnext headerfieldsagainandagainasweexamineIPsec.
Whenthepacketarrivesatitsdestinationandpassestheauthenticationcheck,theAHheaderisremovedandtheProto=AHfieldintheIPheaderisreplacedwiththesaved"NextProtocol".ThisputstheIPdatagramback
toitsoriginalstate,anditcanbedeliveredtothewaitingprocess.

TunnelMode
TunnelModeformsthemorefamiliarVPNfunctionality,whereentireIPpacketsareencapsulatedinsideanotheranddeliveredtothedestination.
LikeTransportmode,thepacketissealedwithanIntegrityCheckValuetoauthenticatethesenderandtopreventmodificationintransit.ButunlikeTransportmode,itencapsulatesthefullIPheaderaswellasthepayload,
andthisallowsthesourceanddestinationaddressestobedifferentfromthoseoftheencompassingpacket:Thisallowsformationofatunnel.

http://www.unixwiz.net/techtips/iguideipsec.html 4/13
3/31/2017 AnIllustratedGuidetoIPsec

WhenaTunnelmodepacketarrivesatitsdestination,itgoesthroughthesameauthenticationcheckasanyAHtypepacket,andthosepassingthecheckhavetheirentireIPandAHheadersstrippedoff.Thiseffectively
reconstitutestheoriginalIPdatagram,whichistheninjectedintotheusualroutingprocess.
MostimplementationstreattheTunnelmodeendpointasavirtualnetworkinterfacejustlikeanEthernetinterfaceorlocalhostandthetrafficenteringorleavingitissubjecttoalltheordinaryroutingdecisions.
Thereconstitutedpacketcouldbedeliveredtothelocalmachineorroutedelsewhere(accordingtothedestinationIPaddressfoundintheencapsulatedpacket),thoughinanycaseisnolongersubjecttotheprotectionsof
IPsec.Atthispoint,it'sjustaregularIPdatagram.
ThoughTransportmodeisusedstrictlytosecureanendtoendconnectionbetweentwocomputers,Tunnelmodeismoretypicallyusedbetweengateways(routers,firewalls,orstandaloneVPNdevices)toprovideaVirtual
PrivateNetwork(VPN).

TransportorTunnel?
Curiously,thereisnoexplicit"Mode"fieldinIPsec:whatdistinguishesTransportmodefromTunnelmodeisthenextheaderfieldintheAHheader.
WhenthenextheadervalueisIP,itmeansthatthispacketencapsulatesanentireIPdatagram(includingtheindependentsourceanddestinationIPaddressesthatallowseparateroutingafterdeencapsulation).Thisis
Tunnelmode.
Anyothervalue(TCP,UDP,ICMP,etc.)meansthatit'sTransportmodeandissecuringanendpointtoendpointconnection.
ThetopleveloftheIPdatagramisstructuredthesamewayregardlessofmode,andintermediaterouterstreatallflavorsIPsec/AHtrafficidenticallywithoutdeeperinspection.
We'llnotethatahostasopposedtoagatewayisrequiredtosupportbothTransportandTunnelmodes,butwhencreatingahosttohostconnection,itseemsalittlesuperfluoustouseTunnelmode.
Furthermore,agateway(router,firewall,etc.)isonlyrequiredtosupportTunnelmode,thoughsupportingTransportmodeisusefulonlywhencreatinganendpointtothegatewayitself,asinthecaseofnetwork
managementfunctions.

http://www.unixwiz.net/techtips/iguideipsec.html 5/13
3/31/2017 AnIllustratedGuidetoIPsec

AuthenticationAlgorithms
AHcarriesanIntegrityCheckValueintheAuthenticationDataportionoftheheader,andit'stypically(butnotalways)builtontopofstandardcryptographichashalgorithmssuchasMD5orSHA1.
Ratherthanuseastraightchecksum,whichwouldprovidenorealsecurityagainstintentionaltampering,itusesaHashedMessageAuthenticationCode(HMAC)whichincorporatesasecretvaluewhilecreatingtheICV.
Thoughanattackercaneasilyrecomputeahash,withoutthesecretvaluehewon'tbeabletorecreatetheproperICV.
HMACisdescribedbyRFC2104,andthisillustrationshowshowthemessagedataandsecretcontributetothefinalIntegrityCheckValue:

We'llnotethatIPsec/AHdoesn'tdefinewhattheauthenticationfunctionmustbe,itinsteadprovidesaframeworkwhichallowsanyreasonableimplementationagreedtobybothendstouse.It'spossibletouseother
authenticationfunctions,suchasadigitalsignatureoranencryptionfunctionaslongasbothsidesprovideforit.
http://www.unixwiz.net/techtips/iguideipsec.html 6/13
3/31/2017 AnIllustratedGuidetoIPsec

AHandNATNotGonnaHappen
ThoughAHprovidesverystrongprotectionofapacket'scontentsbecauseitcoverseverythingthatcanbepossiblyconsideredimmutable,thisprotectioncomesatacost:AHisincompatiblewithNAT(NetworkAddress
Translation).
NATisusedtomaparangeofprivateaddresses(say,192.168.1.X)toandfroma(usually)smallersetofpublicaddress,therebyreducingthedemandforroutable,publicIPspace.In
thisprocess,theIPheaderisactuallymodifiedontheflybytheNATdevicetochangethesourceand/ordestinationIPaddress.
WhentheappropriatesourceorheaderIPaddressischanged,itforcesarecalculationoftheheaderchecksum.Thishastobedoneanyway,becausetheNATdevicetypicallyservesas
one"hop"inthepathfromsourcetodestination,andthisrequiresthedecrementoftheTTL(TimeToLive)field.
BecausetheTTLandheaderchecksumfieldsarealwaysmodifiedinflight,AHknowstoexcludesthemfromcoverage,butthisdoesnotapplytotheIPaddresses.Theseareincluded
intheIntegrityCheckValue,andanymodificationwillcausethechecktofailwhenverifiedbytherecipient.BecausetheICVincorporatesasecretkeywhichisunknownby
intermediateparties,theNATrouterisnotabletorecomputetheICV.
ThissamedifficultyalsoappliestoPAT(PortAddressTranslation),whichmapsmultipleprivateIPaddressesintoasingleexternalIPaddress.NotonlyaretheIPaddressesmodified
onthefly,buttheUDPandTCPportnumbers(andsometimeseventopayload).ThisrequiresmuchmoreintelligenceonthepartoftheNATdevice,andmoreextensivemodifications
tothewholeIPdatagram.
Forthisreason,AHwhetherinTunnelorTransportmodeisentirelyincompatiblewithNAT,anditmayonlybeemployedwhenthesourceanddestinationnetworksarereachable
withouttranslation.
We'llnotethatthisparticulardifficultydoesn'tapplytoESP,asitsauthenticationandencryptiondonotincorporatetheIPheaderbeingmodifiedbyNAT.Nevertheless,NATdoes
imposesomechallengesevenonESP.
NATtranslatesIPaddressesontheflybutithastokeeptrackofwhichconnectionsareflowingthroughitsothatrepliescanbeproperlyassociatedwithsources.WhenusingTCPor
UDP,thisiscommonlydonewithportnumbers(whetherrewrittenontheflyornot),butIPsecprovidesnohooktoallowthis.
AtfirstonemightsuspecttheSPI,whichappearstobeausefulidentifier,butbecausetheSPIisdifferentinbothdirections,theNATdevicehasnowaytoassociatethereturning
packetwiththeoutgoingconnection.
AddressingthisrequiresspecialfacilitiesforNATtraversal,somethingnotcoveredinthispaper.

ESPEncapsulatingSecurityPayload
AddingencryptionmakesESPabitmorecomplicatedbecausetheencapsulationsurroundsthepayloadratherthanprecedesitaswithAH:ESPincludesheaderandtrailerfieldsto
supporttheencryptionandoptionalauthentication.ItalsoprovidesTunnelandTransportmodeswhichareusedinbynowfamiliarways.
TheIPsecRFCsdon'tinsistuponanyparticularencryptionalgorithms,butwefindDES,tripleDES,AES,andBlowfishincommonusetoshieldthepayloadfrompryingeyes.The
algorithmusedforaparticularconnectionisspecifiedbytheSecurityAssociation(coveredinalatersection),andthisSAincludesnotonlythealgorithm,butthekeyused.
UnlikeAH,whichprovidesasmallheaderbeforethepayload,ESPsurroundsthepayloadit'sprotecting.TheSecurityParametersIndexandSequenceNumberservethesame
purposeasinAH,butwefindpadding,thenextheader,andtheoptionalAuthenticationDataattheend,intheESPTrailer.
It'spossibletouseESPwithoutanyactualencryption(touseaNULLalgorithm),whichnonethelessstructuresthepacketthesameway.Thisprovidesnoconfidentiality,andit
onlymakessenseifcombinedwithESPauthentication.It'spointlesstouseESPwithouteitherencryptionorauthentication(unlessoneissimplydoingprotocoltesting).
Paddingisprovidedtoallowblockorientedencryptionalgorithmsroomformultiplesoftheirblocksize,andthelengthofthatpaddingisprovidedinthepad lenfield.Thenext
hdrfieldgivesthetype(IP,TCP,UDP,etc.)ofthepayloadintheusualway,thoughitcanbethoughtofaspointing"backwards"intothepacketratherthanforwardaswe'veseen
inAH.
Inadditiontoencryption,ESPcanalsooptionallyprovideauthentication,withthesameHMACasfoundinAH.UnlikeAH,however,thisauthenticationisonlyfortheESPheader
andencryptedpayload:itdoesnotcoverthefullIPpacket.Surprisingly,thisdoesnotsubstantiallyweakenthesecurityoftheauthentication,butitdoesprovidesomeimportant
benefits.
WhenanoutsiderexaminesanIPpacketcontainingESPdata,it'sessentiallyimpossibletomakeanyrealguessesaboutwhat'sinsidesavefortheusualdatafoundintheIPheader(particularlythesourceanddestinationIP
addresses).Theattackerwillcertainlyknowthatit'sESPdatathat'salsointheheaderbutthetypeofthepayloadisencryptedwiththepayload.
EventhepresenceorabsenceofAuthenticationDatacan'tbedeterminedbylookingatthepacketitself(thisdeterminationismadebyusingtheSecurityParametersIndextoreferencethepresharedsetofparametersand
algorithmsforthisconnection).
However,itshouldbenotedthatsometimestheenvelopeprovideshintsthatthepayloaddoesnot.WithmorepeoplesendingVoIPinsideESPovertheinternet,theQoStaggingsareintheoutsideheaderandisfairly
obviouswhattrafficisVoIPsignaling(IPprecedence3)andwhatisRTPtraffic(IPprecedence5).It'snotasurething,butitmightbeenoughofacluetomatterinsomecircumstances.

http://www.unixwiz.net/techtips/iguideipsec.html 7/13
3/31/2017 AnIllustratedGuidetoIPsec

ESPinTransportMode
AswithAH,TransportModeencapsulatesjustthedatagram'spayloadandisdesignedstrictlyforhosttohostcommunications.TheoriginalIPheaderisleftinplace(exceptfortheshuffledProtocolfield),anditmeansthat
amongotherthingsthesourceanddestinationIPaddressesareunchanged.

ESPinTunnelMode
OurfinallookofstandaloneESPisinTunnelmode,whichencapsulatesanentireIPdatagraminsidetheencryptedshell:

http://www.unixwiz.net/techtips/iguideipsec.html 8/13
3/31/2017 AnIllustratedGuidetoIPsec

ProvidinganencryptedTunnelModeconnectionisgettingveryclosetothetraditionalVPNthatspringstomindwhenmostofusthinkaboutIPsec,butwehavetoaddauthenticationofonetypeoranothertocompletethe
picture:thisiscoveredinthefollowingsection.
UnlikeAH,whereanonlookercaneasilytellwhethertrafficisinTunnelorTransportmode,thisinformationisunavailablehere:thefactthatthisisTunnelmode(vianext=IP)ispartoftheencryptedpayload,andissimply
notvisibletooneunabletodecryptthepacket.

Puttingitalltogether:BuildingarealVPN
WithcoverageoftheAuthenticatingHeaderandEncapsulatingSecurityPayloadcomplete,we'rereadytoenablebothencryptionandauthenticationtobuildarealVPN.ThewholepurposeofaVirtualPrivateNetworkisto
jointwotrustednetworksacrossanuntrustedintermediatenetwork,asisbystringingaverylongEthernetcablebetweenthetwo.Thisiscommonlyusedtoconnectbranchofficeswithcompanyheadquarters,allowingall
userstosharesensitiveresourceswithoutfearofinterception.

http://www.unixwiz.net/techtips/iguideipsec.html 9/13
3/31/2017 AnIllustratedGuidetoIPsec

Clearly,asecureVPNrequiresbothauthenticationandencryption.WeknowthatESPistheonlywaytoprovideencryption,butESPandAHbothcanprovideauthentication:whichonedoweuse?
TheobvioussolutionofwrappingESPinsideofAHistechnicallypossible,butinpracticeisnotcommonlyusedbecauseofAH'slimitationswithrespecttoNetworkAddressTranslation.ByusingAH+ESP,thistunnelcould
neversuccessfullytraverseaNAT'eddevice.
Instead,ESP+AuthisusedinTunnelmodetofullyencapsulatethetrafficonitswayacrossanuntrustednetwork,protectedbybothencryptionandauthenticationinthesamething.
TrafficprotectedinthismanneryieldsnearlynousefulinformationtoaninterlopersaveforthefactthatthetwositesareconnectedbyaVPN.Thisinformationmighthelpanattackerunderstandtrustrelationships,but
nothingabouttheactualtrafficitselfisrevealed.EventhetypeofencapsulatedprotocolTCP,UDP,orICMPishiddenfromoutsiders.
What'sparticularlyniceaboutthismodeofoperationisthattheenduserhostsgenerallyknownothingabouttheVPNorothersecuritymeasuresinplace.SinceaVPNimplementedbyagatewaydevicetreatstheVPNas
yetanotherinterface,trafficdestinedfortheotherendisroutednormally.
Thispacketinapacketcanactuallybenestedyetmorelevels:HostAandHostBcanestablishtheirownauthenticatedconnection(viaAH),andhavethisroutedovertheVPN.ThiswouldputanAHinnerpacketinsidean
enclosingESP+Authpacket.
Updateit'simportanttouseauthenticationevenifencryptionisused,becauseencryptonlyimplementationsaresubjecttoeffectiveattackasdescribedinthepaperCryptographyinTheoryandPractice:TheCaseof
EncryptioninIPsecseetheResourcessectionformoreinformation.

http://www.unixwiz.net/techtips/iguideipsec.html 10/13
3/31/2017 AnIllustratedGuidetoIPsec

TouchingonOtherMatters
IPsecisaverycomplexsuiteofprotocols,andthisTechTipcannotpossiblygiveproperjusticetomorethanasmallpartofit.Inthissectionwe'llmentionafewareasthatbegformorecoverage.

SecurityAssociationsandtheSPI
Itseemsselfevidentthatiftwoendpointsorgatewaysaregoingtoestablishasecureconnection,somekindofsharedsecretisrequiredtoseedtheauthenticationfunctionand/orkeytheencryptionalgorithm.Thematter
ofjusthowthesearesecretsareestablishedisasubstantialtopictobeaddressedelsewhere,andforthepurposesofthisdiscussionweshalljustassumethatthekeyshavemagicallylandedwheretheybelong.
WhenanIPsecdatagrameitherAHorESParrivesataninterface,justhowdoestheinterfaceknowwhichsetofparameters(key,algorithm,andpolicies)touse?Anyhostcouldhavemanyongoingconversations,each
withadifferentsetofkeysandalgorithms,andsomethingmustbeabletodirectthisprocessing.
ThisisspecifiedbytheSecurityAssociation(SA),acollectionofconnectionspecificparameters,andeachpartnercanhaveoneormoreSecurity
Associations.Whenadatagramarrives,threepiecesofdataareusedtolocatethecorrectSAinsidetheSecurityAssociationsDatabase(SADB):
Unsure!
It'sbeenpointedoutthattheSADBonlyusestheprotocoltype
PartnerIPaddress andSPItoselectanentry,notthepartnerIPaddresswesimply
IPsecProtocol(ESPorAH) don'tknow.
SecurityParametersIndex Thismightdependonwhethertheassociationisconfiguredwith
mainmodeoraggressivemode,butwewelcomeclarifications.
InmanywaysthistriplecanbelikenedtoanIPsocket,whichisuniquelydenotedbytheremoteIPaddress,protocol,andportnumber.

http://www.unixwiz.net/techtips/iguideipsec.html 11/13
3/31/2017 AnIllustratedGuidetoIPsec

SecurityAssociationsareoneway,soatwowayconnection(thetypicalcase)requiresatleasttwo.Furthermore,eachprotocol(ESP/AH)hasitsownSAineachdirection,soafullAH+ESPVPNrequiresfourSecurity
Associations.TheseareallkeptintheSecurityAssociationsDatabase.
AtremendousamountofinformationiskeptintheSADB,andwecanonlytouchonafewofthem:

AH:authenticationalgorithm
AH:authenticationsecret
ESP:encryptionalgorithm
ESP:encryptionsecretkey
ESP:authenticationenabledyes/no
Manykeyexchangeparameters
Routingrestrictions
IPfilteringpolicy

SomeimplementationsmaintaintheSPD(SecurityPolicyDatabase)withcommandlinetools,otherswithaGUI,whileothersprovideawebbasedinterfaceoverthenetwork.Theamountofdetailmaintainedbyany
particularimplementationdependsonthefacilitiesoffered,aswellaswhetherit'soperatinginHostorGatewaymode(orboth).

KeyManagement
Finally,webrieflyvisittheverycomplexmatterofkeymanagement.Thisareaincludesseveralprotocolsandmanyoptions,andthebulkofthiswillbecoveredinafuturepaper.Thissectionisnecessarilyhighly
incomplete.
IPsecwouldbenearlyuselesswithoutthecryptographicfacilitiesofauthenticationandencryption,andtheserequiretheuseofsecretkeysknowntotheparticipantsbutnottoanyoneelse.
Themostobviousandstraightforwardwaytoestablishthesesecretsisviamanualconfiguration:onepartygeneratesasetofsecrets,andconveysthemtoallthepartners.Allpartiesinstallthesesecretsintheir
appropriateSecurityAssociationsintheSPD.
Butthisprocessdoesnotscalewell,norisitalwaysterriblysecure:themereactofconveyingthesecretstoanothersite'sSPDmaywellexposethemintransit.Inalargerinstallationwithmanydevicesusingthesame
presharedkey,compromiseofthatkeymakesforaverydisruptiveredeploymentofnewkeys.
IKEInternetKeyExchangeexiststoallowtwoendpointstoproperlysetuptheirSecurityAssociations,includingthesecretstobeused.IKEusestheISAKMP(InternetSecurityAssociationKeyManagementProtocol)as
aframeworktosupportestablishmentofasecurityassociationcompatiblewithbothends.
Multiplekeyexchangeprotocolsthemselvesaresupported,withOakleybeingthemostwidelyused.We'llnotethatIPseckeyexchangetypicallytakesplaceoverport500/udp.

Resources
TheInternethasagreatmanyresourcessurroundingIPsec,somebetterthanothers.Thestartingpoint,ofcourse,isalwayswiththeRFCs(RequestsforComment)thatformtheInternetstandardsdefiningtheprotocols.
Thesearethemainreferenceworksuponwhichallotherdocumentationincludingthisoneisbased.
Update:InDecember2005,awholenewsetofRFCswasissuedbyIETF,andthe43xxserieslargelyobsoletedthe24xxseries.WeincludedreferencestoalltheRFCs(oldandnew)below,thoughthisdocumenthasnot
reallybeenupdatedforthenewones.

RFC2401SecurityArchitectureforIPsec obsolete
RFC4301SecurityArchitectureforIPsec newDec2005
ThisistheoverviewoftheentireIPsecprotocolsuitefromthepointofviewoftheRFCs.This,andtheDocumentationRoadmap(RFC2411)aregoodplacestostart.
RFC2402AH:AuthenticationHeader obsolete
RFC4302AH:AuthenticationHeader newDec2005
ThisdefinestheformatoftheIPsecAuthenticationHeader,inbothTunnelandTransportmodes.
RFC2403UseofHMACMD596withinESPandAH
RFC2404UseofHMACSHA196withinESPandAH
ThesetwoRFCsdefineauthenticationalgorithmsusedinAHandESP:MD5andSHA1arebothcryptographichashes,andtheyarepartofaHashedMessageAuthenticationCode.AHalwaysperformsauthentication,
whileESPdoessooptionally.
RFC2104HMAC:KeyedHashingforMessageAuthentication
ThisRFCdefinestheauthenticationalgorithmthatusesacryptographichashalongwithasecrettoverifytheintegrityandauthenticityofamessage.It'snotwrittentobepartofIPsec,butit'sreferencedinRFC2403
andRFC2404.
RFC2405TheESPDESCBCCipherAlgorithmWithExplicitIV
ThisdefinestheuseofDES(theDataEncryptionStandard)asaconfidentialityalgorithminthecontextofESP.
RFC2406ESP:EncapsulatingSecurityPayload obsolete
RFC4303ESP:EncapsulatingSecurityPayload newDec2005
ESPistheencryptingcompaniontoAH,anditaffordsconfidentialitytothecontentsofitspayload.ESPbyitselfdoesnotdefineanyparticularencryptionalgorithmsbutprovidesaframeworkforthem.
http://www.unixwiz.net/techtips/iguideipsec.html 12/13
3/31/2017 AnIllustratedGuidetoIPsec

RFC2407TheInternetIPSecurityDomainofInterpretationforISAKMP
ThisRFCdescribestheuseofISAKMPInternetSecurityAssociationandKeyManagementProtocolinthecontextofIPsec.It'saframeworkforkeyexchangeatthestartofaconversation,anditsuseobviatesthe
poorpracticeofusingmanualkeys.
RFC2408InternetSecurityAssociationandKeyManagementProtocol(ISAKMP)
HandinhandwithRFC2407,thisRFCdivesintomuchmoredetailontheISAKMPprotocolusedtosupportkeyexchange(thoughitdoesn'tdefinethekeyexchangeprotocolsthemselves).
RFC2409TheInternetKeyExchange(IKE)Protocol obsolete
RFC4306TheInternetKeyExchange(IKE)Protocol newDec2005
ThoughISAKMPprovidesaframeworkforkeyexchange,itdoesn'tdefinetheprotocolsthemselves:thisRFCdoesthat.IKEincludesinitialauthentication,aswellasOakleykeyexchange.
RFC2410TheNULLEncryptionAlgorithmandItsUseWithIPsec
IPsec'sESPprotocolperformsencryptionofpayloadusingoneofseveralavailablealgorithms,butaNULLencryptionalgorithmistypicallymadeavailablefortesting.Ofcourse,thisprovidesnoconfidentialityforthe
"protected"data,butitmaybeusefulfordevelopersorthoseattemptingtounderstandIPsecbysniffingthewire.ThisRFCiswrittenhumorouslyandcouldhavebeen(butwasnot)writtenonApril1.
RFC2411IPSecurityDocumentRoadmap
ThisRFCprovidesanlayoutofthevariousIPsecrelatedRFCs,aswellasprovidesaframeworkfornewRFCsofparticulartypes("authenticationalgorithms","encryptionalgorithms").It'sagoodstartingpoint.
RFC2412TheOAKLEYKeyDeterminationProtocol
OAKLEYformspartofIKE(InternetKeyExchange),anditprovidesaservicewheretwoauthenticatedpartiescanagreeonthesecretsrequiredforIPseccommunications.
AnIllustratedGuidetoCryptographicHashesUnixwiz.netTechTip
AnintroductorypaperontheuseofcryptographichashessuchasMD5orSHA1,whichareusedinAH'sHMACforauthentication.
IPsecTechnicalReferencebyMicrosoft
ThisprovidesinformationonMicrosoft'simplementationofIPsecintheWindowsServer2003product,includingagreatdealaboutthelargerinfrastructurerequiredtosupportIPsecintheenterprise.
TCP/IPIllustrated,Volume1,byW.RichardStevens.
ThisistheclassictextbookontheTCP/IPprotocol,coveringdowntothepacketheaderinexquisitedetail:Thisisanextraordinaryresource.
ACryptographicEvaluationofIPsec,byBruceSchneierandNielsFerguson.
AninterestingpaperonthesecurityofIPsec,whosemainpointisthatIPsecisfartoocomplextoeverreallybesecure(somethingwhichhascrossedourmindsaswell).Amongtheirproposalsaretoeliminateboth
TransportModeandAH:ESPinTunnelmodecanprovideallthissamefunctionality.
RFC3884UseofIPsecTransportModeforDynamicRouting
IncontrasttotheSchneierpaper,it'salsobeensuggestedthatTransportModeistheonlyonethat'sstrictlyrequiredtoaccomplisheverything,andRFC3884showsawayofprovidingtunnelmode.It'sbeensuggested
tousthatthismakessomeimplementationissuesmucheasier,thoughwe'venotreallyinvestigatedanyofit.
CryptographyinTheoryandPractice:TheCaseofEncryptioninIPsecPaterson&Yau
ThisveryinterestingpaperdiscussessomeofthedangersofencryptedbutnotauthenticatedIPsecconnections,witheffectiveattacksonrealsystems(includingtheLinuxkernel'simplementationofIPsec).It'savery
cleverpaper.
ProtocoloIPsecAlexandruIonutGrama
ThispaperwastranslatedintoSpanish!

N.B.WearenotIPsecexperts,andthoughwe'vespentagreatdealoftimeresearchingthesematters,wemayhavesomedetailswrong.Feedbackandcorrectionsareverymuchwelcome.We'reparticularlygratefulfor
theextensivetechnicalfeedbackprovidedbyIPsecarchitectsatSunandMicrosoft.
TheseoriginalfigureswereproducedbytheauthorusingAdobeIllustrator.
Firstpublished:20050824

Home StephenJ.Friedl SoftwareConsultant OrangeCounty,CAUSA

http://www.unixwiz.net/techtips/iguideipsec.html 13/13