Escolar Documentos
Profissional Documentos
Cultura Documentos
ProtocolsandToolsforProject3
(version1.0)
Overview
ThisdocumentprovidessupplementaryinformationforbothProject3aand3b.Toaccomplishtheproject,
youwillneedtounderstandthepacketformatsofvariousprotocols,sothatyourfirewallcandecode
packetsandapplyfirewallrulestothem.Also,youwillneedtousevariousnetworktestingtoolsto
generatenetworktrafficandtoverifythebehaviorofyourfirewallbytappingnetworkinterfaces(int
andext).
Thisdocumentisalsointendedtoprovidesomedetailsonnetworkprotocolsconceptuallycoveredinthe
courselectures.
Notethatthisdocumentonlyincludesabriefintroductiontotheprotocolsandnetworktestingtools.For
morecomprehensiveanddetailedinformation,youshouldrefertotheRFCstandardsand(wo)manpages
(yes,youcanusewomancommandinsteadofmanintheVM).Alsonotethatwedonotguaranteethe
correctnessofprotocoldescriptionsprovidedinthisdocument.Ifthestandards(thespecdocument
includesreferences)conflictwiththisdocument,trusttheformer.
Endianness
Formoredetails:
http://en.wikipedia.org/wiki/Endianness
http://docs.python.org/2/library/struct.html
Whenacomputerstoresortransmitsmultibytedata,itmayuseoneoftwoapproachesfororganizingthe
bytes.Oneistoplacebytesindecreasingorderofsignificance(i.e.,MSBfirst).Thisiscalledbig
endian.Forexample,thenumber1234567890is0x499602d2inhex.Inbigendiansystems,thenumber
willbestoredasfollows:
address
a+1
a+2
a+3
data
0x49
0x96
0x02
0xd2
Theotheroneistoplacebytesinincreasingorderofsignificance(littleendian).
address
a+1
a+2
a+3
data
0xd2
0x02
0x96
0x49
Mostnetworkprotocolsarebasedonbigendian.Ontheotherhand,yourlaptop/desktop(whichlikelyuses
thex86architecture)useslittleendian.Sometimesthetermsnetworkorder(bigendian)andhost
order(littleendian,inx86)areused.Wheneveryoudecodeorencodemultibytedatafromnetwork
packets,youneedtoconvertitsendiannessbeforeuse.YouwillfindthefollowingPythonfunctions
useful.
socket.htons(0x1234)==0x3412
#2Bhostorderintegernetworkorderinteger
socket.ntohs(0x3412)==0x1234
#2Bnetworkorderintegerhostorderinteger
socket.htonl(0x12345678)==0x78563412
#4Bhostorderintegernetworkorderinteger
socket.ntohl(0x78563412)==0x12345678
#4Bnetworkorderintegerhostorderinteger
chr(0x12)==\x12
#1Binteger1Bstring
struct.pack(!B,0x12)==\x12
#1Binteger1Bstring
struct.pack(!H,0x1234)==\x12\x34#2Binteger2Bbigendianstring
struct.pack(!L,0x12345678)==\x12\x34\x56\x78 #4Binteger2Bbigendianstring
ord(\x12)==0x12
#1Bstring1Binteger
struct.unpack(!B,\x12)==(0x12,)
#1Bstring1Binteger
struct.unpack(!H,\x12\x34)==(0x1234,) #2Bbigendianstring2Binteger
struct.unpack(!L,\x12\x34\x56\x78)==(0x12345678,)
#4Bbigendianstring4Binteger
#Youcandecodemultiplefieldsatonce.
2
struct.unpack(!HH,\x12\x34\x56\x78)==(0x1234,0x5678)
TolearnmoreaboutPythonsstructmodule,seethelinkatthetopofthissection.
Protocols
IPv4Header
(http://nmap.org/book/images/hdr/MJBIPHeader800x576.png)
Inthefirewall,youwillonlyseeIPv4packetstheprovidedbasecodewillhandoverIPv4packetstothe
FirewallclassandblindlypassallnonIPv4packets.
HeaderLength
TheHeaderLengthfieldcontainsthelengthoftheIPheader,dividedby4,sincethelengthoftheIP
headerisalwaysamultipleof4bytes.TheminimumIPv4headersizeis20bytes,unlessithasIPoptions.
Ingeneral,mostpacketsdonotcarryIPoptions,sothevalueofthisfieldwillusuallybe5.Ifyouseea
3
packetwithasmallerheaderlengththan5,youshoulddropthepacket.
Notethatatransportlayerheader(TCP/UDP/ICMP)mayhaveavariableoffsetinthepktstringinthe
handle_packet()method,dependingonthelengthoftheIPheader.Forexample,ifthevalueoftheIP
headerlengthfieldis7andthepacketisTCP,theTCPheaderwillbeginatthe28thbyteofpkt.
Sincethisfieldisonly4bitswide,youwillneedtousesomebitoperations.
TotalLength
ThisfieldindicateshowlongtheIPpacketis,includingtheIPheaderitself.Sincethepktstringhasthe
wholedataoftheIPpacket,thevalueofthetotallengthfieldmustbeequaltolen(pkt).Ifnot,youmay
wanttodropthepacket(butnotrequiredbytheprojectspec).
Identification,FragmentFlags/Offset
ThesefieldsareforIPfragmentation.Sinceyouwillnotseeanyfragmentedpacketsforthisproject,you
canignorethosefields.
TTLandHeaderChecksum
ForProject3a,youarenotrequiredchecktheTTLandchecksumofreceivedpackets.ForProject3b,
youwillneedtocraftIPv4packetsfromscratch,whichmeansthatthechecksumvalueshouldbe
correctlycalculatedandfilledinthefield.NotethatIPchecksumonlyappliestotheheaderbytes.ForIP
checksumcalculation,refertothefollowingdocuments:
http://en.wikipedia.org/wiki/IPv4_header_checksum
http://www.thegeekstuff.com/2012/05/ipheaderchecksum/
Source/DestinationAddresses
IPv4addressesarerepresentedasa4bytebinarydata.Forexample,123.45.67.89isstoredas
\x7b\x2d\x43\x59inhex.ToconvertanIPv4addressstringintoa4bytebinarydata,youcanuse
socket.inet_aton().Toconverta4bytebinarydataintoanIPv4addressstring,use
socket.inet_ntoa().Refertothebypass.pyfileforanexample.
TCPHeader
(http://nmap.org/book/images/hdr/MJBTCPHeader800x564.png)
Source/DestinationPorts
ForProject3a,youonlyneedtoconsiderthesetwofields.
Yourfirewallshouldexamineexternalports.Forincomingpackets(fromtheoutsidenetworktothe
VM),thesourceportfieldcontainstheexternalport.Foroutgoingpackets(fromtheVMtotheoutside
network),thedestinationportfieldcontainstheexternalport.Donotignoreendianness,sincetheseare
2bytefields.
Sequence/AcknowledgementNumber
Asdiscussedinthelecture,eachTCPpacketcarriesitssequencenumber(inbytes,notinpackets).
RecallthatTCPisafullduplexprotocol.Foreachdirectionofaconnection,aseparatesequencenumber
isused.TheSequenceNumberfieldcontainsthesequencenumberofthefirstbyteoftheTCPsegment
data.
5
WhentheACKflagisset(itisusuallysetallthetime,exceptfortheveryfirstSYNpacketofTCP
handshake),theAcknowledgementNumberfieldcontainsthecumulativeacksequencenumber.For
example,iftheacksequencenumberisX,itmeansthereceiversuccessfullyreceiveduptoX1thbyte
andexpectssequencenumberXforthenextdata.
PacketswithaSYNorFINflagincreasesthesequencenumberby1.Lookatthefollowingexample
(supposethattheinitialsequencenumbersare1000and2000).
Formoredetailsaboutsequenceandacknowledgementnumbers,readthisarticle:
http://packetlife.net/blog/2010/jun/7/understandingtcpsequenceacknowledgmentnumbers/
Offset
ThisisverysimilartotheHeaderLengthfieldintheIPv4header.ItspecifiesthelengthoftheTCP
headerinbytes,dividedby4.SincetheminimumTCPheaderlengthis20bytes,thevalueshouldnotbe
lessthan5(20B).ItistheoffsetoftheTCPpayload,beginningfromtheTCPheader.
Checksum
ForProject3a,youarenotrequiredtocheckthechecksumvalueofpackets.ForProject3b,youneedto
understandhowtocalculatetheTCPchecksum.
UnlikeIPv4checksum,whichonlycoverstheIPheader,TCPchecksumcalculationismorecomplex.Itis
calculatedwithaTCPpseudoheaderandthepayloaddata.
6
Wikipediahasadetaileddescriptionofthepseudoheader.
http://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_checksum_for_IPv4
http://www.tcpipguide.com/free/t_TCPChecksumCalculationandtheTCPPseudoHeader.htm
UDPHeader
http://nmap.org/book/images/hdr/MJBUDPHeader800x264.png
UDPheaderissimplerthanTCP,sinceitisdesignedasabasicwrapperforrawIPpackets.Theoffsets
ofSourcePortandDestinationPortarethesameasinTCP.
ICMPHeader
http://nmap.org/book/images/hdr/MJBICMPHeader800x392.png
TheIPprotocoldefinesthedataplaneoftheInternetIPpacketscarrydataamongendhostsandrouters.
7
ICMPisaswissarmyknifeprotocolthatisspeciallydesignedtosupplementthefunctionalityofIP(e.g.,
diagnosticaanderrorreporting).
LikeTCPandUDP,ICMPisimplementedontopofIP,thustheICMPheaderbeginsattheendofthe
IPv4header.TheformatofaICMPpacketgreatlyvariesaccordingtoitstype.ForProject3a,youwill
needtoonlyexaminethe1byteTypefield.
DNSPackets
DNScanbeimplementedonbothTCPandUDP,butmostimplementationsprimarilyuseUDP.Forthe
project,weonlyconsiderUDPbasedDNSpacketswithdestinationport53.Allcommunicationsinsideof
thedomainnamesystemprotocolarecarriedinasingleformatcalledamessage.Thetoplevelformatof
messageisdividedinto5sections(someofwhichareemptyincertaincases)shownbelow:
+---------------------+
|
Header
|
+---------------------+
|
Question
|
+---------------------+
|
Answer
|
+---------------------+
|
Authority
|
+---------------------+
|
Additional
|
+---------------------+
RRstandsforresourcerecord.Inthisproject,weonlycareaboutRRrecordswithA(IPv4)orAAAA
(IPv6)type.WhiletheVMisconfiguredtodisableIPv6,theDNSresolverlibrarymaystillgenerates
AAAAtypequeriesifanAtypequeryfails.
Header
Theheadercontainsthefollowingfields:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|
ID
|
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|QR| Opcode |AA|TC|RD|RA| zero | RCODE |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|
QDCOUNT
|
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|
ANCOUNT
|
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|
NSCOUNT
|
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|
ARCOUNT
|
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
Longstoryshort,thisis16bits(twobytes)by6rows,foratotalof12bytes.Fortheproject,youshould
examinetheQDCOUNTfield,whichspecifiesthenumberofquestionentriesintheQUESTIONsection.
ThespecdocumentsstatesthatweonlyconsiderDNSmessageswithQDCOUNT==1.
QUESTIONSectionFormat
Thequestionsectionisusedtocarrythe"question"inmostqueries,i.e.,theparametersthatdefinewhatis
beingasked.ThesectioncontainsQDCOUNT(usually1)entries,eachofthefollowingformat:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|
|
/
QNAME
/
/
/
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|
QTYPE
|
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|
QCLASS
|
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
where:
QNAME:adomainnamerepresentedasasequenceoflabels,whereeachlabelconsistsofa
lengthbytefollowedbythatnumberofbytes.Thedomainnameterminateswiththezerolength
byte.Notethatthisfieldmaybeanoddnumberofbytesnopaddingisused,sothefollowingtwo
fieldsmaynotbe16bitaligned.
QTYPE:atwobytecodewhichspeciesthetypeofthequery.Thevaluesforthiseldincludeall
codesvalidforaTYPEfield,togetherwithsomemoregeneralcodeswhichcanmatchmorethan
onetypeofRR.
QCLASSatwobytecodethatspeciestheclassofthequery.Forexample,theQCLASSfieldis
IN(1)fortheInternet.
AnexampleofwhatQNAMEwilllooklike:
03 77 77 77 06 67 6f 6f 67 6c 65 03 63 6f 6d 00
w w w
g o o g l e
c o m
YouwillprimarilybeinterestedinArecords,whichmaphostnametoIPv4address(QTYPE==1),and
AAAArecords,whichmaphostnametoIPv6address(QTYPE==28).
NetworkTestingTools
9
tcpdump/Wireshark
Asyougoaboutdevelopingyourfirewallyoumightfinditusefultoobservethepacketsarrivingatthe
networkinterface.Amongstotherthingsobservingpacketdataishelpfulfordebugging(youcantryand
determinepropertiesofpacketsnotbeingprocessedcorrectly),makingsurethatyourfirewallisactually
beingtestedandjustdeterminingthekindsofpacketsgeneratedbyavarietyofapplications.
Packetsniffersarecommonlyusedtoaccomplishthesetasks,yourVMhastwooftheseinstalled:
Wiresharkandtcpdump.Wiresharkisgraphical,whiletcpdumpisacommandlinetool.Bothare
capableoffilteringpacketsandarealmostequallypowerful.Webrieflydescribebothbelowandpointtoa
fewsourcesofinformationonline.Westronglyencourageyoutolookthroughtutorialandother
documentation.
NormallyThebothtoolsrequirerootprivilegestorun,sincepacketscontainsecuritycriticalinformation.
However,intheprovidedVM,youdontneedtodosudoeverytimetorunthem(wedidsomething
specialforyou).
Whenyouruntcpdump/Wiresharkonyourownmachine,youwillseealotofpackets,sincethereare
manybackgroundapplicationsthatconnecttotheInternet.IntheVM,wedisabledmostsuchbackground
applications,soyouwillseemuchfewernoisepackets.
tcpdump
tcpdumpisacommandlinepacketsniffer,whichprintsoutadescriptionofpacketsgoingthrougha
networkinterface.Bydefaulttcpdumpsdescriptionofapacketisdependentupontheprotocol,forTCP
packetsitwillprintadescriptionlike:src>dst:flagsdataseqnoackwindowurgentoptions.Asan
example,considerthefollowingoutput:
Whatdoestheniintvvport53mean?
-n:Normallytcpdumpwilltrytoconvertnumericaddressesintohumanfriendlystrings(e.g.,IP
address8.8.8.8googlepublicdnsa.google.com,portnumber53DNS,etc.).The-n
optionpreventsthisbehavior.
10
-i int:specifiestheinterfacetomonitor.
Forthisproject,youareinterestedinintandext.
-vv:specifiestheentirepayloadshouldbedecoded.
port 53:specifiesanoptionalfilter,inthiscasestatingthatweonlywanttocapturepacketswith
TCPorUDPsourceordestinationport53.
Formoreexamplesoffilterexpressions,tryman pcap-filterintheVMorreferto
this:http://wiki.wireshark.org/CaptureFilters
Whilesuchinterpretedrecordscanhelpdeterminethekindofpacketsbeingsent,itisoftenusefultojust
seerawpacketdata.Thiscanbeaccomplishedusingthe-Xflagwhichprintsrawbytesinhexandascii,
sidebyside,forinstance:
$ tcpdump -n -i int -X
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on int, link-type EN10MB (Ethernet), capture size 65535 bytes
17:40:14.501879 IP 10.0.2.15.46031 > 8.8.8.8.53: 17322+ [1au] A? www.berkeley.edu. (45)
0x0000: 4500 0049 7a1d 0000 4011 e468 0a00 020f E..Iz...@..h....
0x0010: 0808 0808 b3cf 0035 0035 ad8c 43aa 0120 .......5.5..C...
0x0020: 0001 0000 0000 0001 0377 7777 0862 6572 .........www.ber
0x0030: 6b65 6c65 7903 6564 7500 0001 0001 0000 keley.edu.......
0x0040: 2910 0000 0000 0000 00
)........
tcpdumphasseveralotherusefuloptions.Werecommendreadingthroughthemanpages(man
tcpdumpintheVM)orlookingat
http://www.thegeekstuff.com/2010/08/tcpdumpcommandexamples/
http://www.danielmiessler.com/study/tcpdump/
andGooglingaroundformoreinformation.
Wireshark
Wiresharkprovidesagraphicalinterfaceforcapturingpacketssimilartowhatisallowedbytcpdump.
YoucanstartWiresharkbyrunningwireshark &fromthecommandline,orbyclickingthe
followingiconbelowthedesktopscreen:
Tip:YoumaywanttolaunchtwoinstancesofWireshark,tomonitorbothintandextinterfacesatthe
sametime.Thiscanbeusefultoverifyyourfirewallsbehavior.
Oncestarted,thewindowwilllooklikethis:
11
Chooseanetworkinterface(extorintonthelistbox),andclickStart.Wiresharkwillstartcapturing
packetsontheinterfaceanddisplaythem:
Thescreenshowsthreepanels.Thetoponeisthelistofcapturedpackets.Ifyouareseeingtoomany
12
packets,youcanapplyadisplayfiltertoshowinterestedpacketsonly.ThesyntaxofWiresharkdisplay
filterisdifferentfromthatoftcpdump.Refertothislink:http://wiki.wireshark.org/DisplayFilters
Clickonapacketyouwanttoinspect.Onthebottomleftpanel,youwillseethedetailedinformationof
thedecodedpacket,foreachnetworklayer(Intheaboveexample,Ethernet,IP,UDP,andDNS).When
youclickononeoftheentriesinthepanel,thebottomrightpanelwillshowhowtheselectedpartofthe
packetcorrespondstothepacketbinarydata.Intheabovescreenshot,itrepresentshow
www.berkeley.eduisrepresentedintheQNAMEfield.
OneofthemostusefulfeatureofWiresharkisthatitcanreconstructawholeTCPstreamfrom
individualTCPpackets.YouwillheavilyrelyonthisfeatureforProject3b.RightclickonaTCPpacket,
andchooseFollowTCPStream.ThedisplayfilterwillbeautomaticallysetfortheTCPconnection,and
thefollowingpopupwindowwillappear.Theexamplebelowisfromthepacketswithwget
www.berkeley.edu
tcpdumpandWiresharkaresomeoftheultimatetoolsthateveryprogrammershouldknowhowtouse.
TherearemanyvideotutorialsonYouTube.Trythemout!
13
nslookup/dig
BothnslookupanddigperformsDNSlookups,soyoucanutilizethemtogenerateDNSquerypacketsfor
testingDNSrulematching.nslookupisdeprecated,butitisstillwidelyused.Here,webrieflyintroduce
howtousedig.
YoucanoptionallyspecifyaDNSresolver(e.g,@8.8.4.4).Ifunspecified,thedefaultDNSserverofthe
system(8.8.8.8intheprovidedVM)willbeused.ThedestinationIPaddressoftheDNSpacketwillbe
thatoftheDNSresolver.Thequeryisthenameoftheresourcerecordtobelookedup.Sinceboth
nslookupanddiggenerateaDNSqueryforArecords,thequeryshouldbeadomainname.
IN
;; ANSWER SECTION:
www.berkeley.edu. 154
www.w3.berkeley.edu.145
IN
IN
CNAME www.w3.berkeley.edu.
A
169.229.216.200
Fromtheresult,youseethatwww.berkeley.eduisjustanaliasofwww.w3.berkeley.edu(the
CNAMErecord),anditsIPaddressis169.229.216.200(theArecord).
digsupportsavarietyofoptions.Youmayfindthefollowingtwooptionsuseful.
-t AAAA:AskforanAAAA(IPv6address)record,insteadofA.
+trace:Makeiterativequeries,insteadofrecursivequeries.
wget/curl
14
wgetisusedfornoninteractivedownloadingoffilesviaHTTPorFTP.Themostbasicwayofusing
wgetisjust"wget http://foo.com/bar/baz"whichwilljustdownloadthefiletothecurrent
directory.Anicefeatureisthatadownloadbarwillbeshowntoportraytheprogress,aswellasspeed
andpredictedtimeuntilcompletion.wgetisveryusefulforBellsandWhistles2and3ofProject3a.
ForProject3b,wgetcanbeagoodalternativetoFirefoxforgeneratingHTTPtesttraffic,becauseofits
streamlinedbehavior(e.g.,youdontneedtoemptythelocalcacheofFirefoxeverytime).
Variousoptionsyoucanputonthecommandlineare:
-O [output file]:Thisallowsspecifyingtheoutputlocation.Notethatthisisnot-o
(lowercase),whichiswritingdebugmessagestoalogfile(andlikelynotwhatyouwanttodo).
-p:Downloadrecursively.Thisallowsdownloadinganentirepage(e.g.,theHTMLfileandits
embeddedimages)insteadofjustasinglefile.
-nd:Donotcreatedirectoryhierarchieswhendownloadingrecursively.
-nc:Donotclobber.Thisistopreserveanypreviousinstancesofthesamefile.Anewcopy,in
thatcase,willbenamedfilename.N,whereNistheNthcopyofthesamefile.
-c:Continue.Thisistocontinuedownloadingapartiallydownloadedfile.
curlisasimilartowgetinitspurpose,butitsupportsmorevariousprotocols.IfyouareaMacuser,you
maybemorefamiliarwithcurlthanwget,asitisinstalledbydefault.Fordifferencesbetweencurland
wget,readthisarticle:http://daniel.haxx.se/docs/curlvswget.html
nc
nc(shortfornetcat)isacommandlinetoolthatcanbeusedforvarioussocketoperations.Youwillfind
thistoolveryusefultogenerateTCP/UDPpacketsforProject3aItcanopenTCPconnections,send
UDPpackets,listenonarbitraryTCPandUDPports,[...][manpage].
BydefaultncwilluseTCPasitstransportprotocol.Thebasicusageisnc [destination][port],where
thedestinationcanbeeitheranIPaddressoradomainname.ncwillinitiateaTCPconnectiontothe
specifieddestination/port,whichtriggersTCP3wayhandshake.Onceconnected,whatyoutype(via
standardinput)willbetransferredtothedestinationviaTCPpackets,andtheresponsewillbedisplayed
onthescreen(viastandardoutput).YoucanspecifytheuflagtouseUDP,insteadofTCP.
Donotplaywiththeportscanningfunctionofnc.Youmaygetintotroublefordoingthis(network
abuse).
15