Intro to x64 Reversing

SummerCon 2011 - NYC Jon Larimer

email: jlarimer@gmail.com twitter: @shydemeanor

%e&ore we 'egin...

(his )resentation assumes you can re!erse x*6 code +ou might learn something e!en i& you can,tso don,t lea!e I& I go to &ast- yell at me .ind a mista/e- I drin/ THERE

ILL !E " #$I%&

I& you answer wrong- you drin/

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

1genda

Intro 2 3istory o& x64 (he x64 4lat&orm 5icroso&t x64 1%I Sys6 x64 1%I (ools &or re!ersing x64

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

x64 re!ersing challenges

I& you,re used to re!ersing 02 'it x*6 code- x64 can 'e con&using at &irst 7asy )arts

Instructions are mostly the same as you,re used to (here are a &ew more registers

3ard )arts

Calling con!ention is totally di&&erent 8e'ugging o)timi9ed code can 'e tric/y
Intro to x64 e!ersing " #on $arimer 4

SummerCon 2011

;ame sou)<

158

%S8 " am,64 $inux /ernel " x'6-64 >CC " am,64 8e'ian2?'untu " am,64 .edora2SuS7 " x'6-64 Solaris " am,64

x'6-64 "()64 I"-*2e E(64T Inte+ 64 x64

Intro to x64

Intel

=racle25icroso&t

Note. I"-64 is Itanium/ N0T x'6-x64&

SummerCon 2011

e!ersing " #on $arimer

3istory o& x64

1111 " 158 announces x*6"64 2000 " 158 releases s)ecs 2001 " .irst x*6"64 $inux /ernel a!aila'le 200* " .irst 15864 =)erton released 2004 " Intel announces I1"02e27564(- releases &irst x64 @eon )rocessor 2002 " x64 !ersions o& Aindows @4 and Ser!er 2000 released 2001 " 5ac =S 10.6 BSnow $eo)ardC includes x64 /ernel 2001 " Aindows Ser!er 200* 2 only a!aila'le in x64 !ersion

2010 " :0D o& Aindows E installs running the x64 !ersion 2011 " 40D o& Steam users in 1)ril 2011 3A sur!ey use AinE x64
Intro to x64 e!ersing " #on $arimer 6

SummerCon 2011

(he x64 4lat&orm

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

Ahat is x64F

7xtension to 02 'it x*6 " x64 Glong modeH

Can address u) to 64 'its B16E!C o& !irtual memoryI Can address u) to :2 'its B43!C o& )hysical memoryII

64 'it general )ur)ose registers " RAX- RBX- ...

* new >4 registers BR8"R15C * new 12* 'it @55 registers BXMM8"XMM15C

;ew 64 'it instructions: cdqe- lodsq- stosq- etc 1'ility to re&erence data relati!e to instruction )ointer B ripC

I $imited 'y )rocessor im)lementation- most only su))ort 4* 'its now... II Intel currently su))orts 40 'its o& )hysical memory
SummerCon 2011 Intro to x64 e!ersing " #on $arimer *

$ong mode

64 'it &lat BlinearC addressing

Segment 'ase is always 0 exce)t &or FS and GS Stac/ BSSC- Code BCSC- 8ata BDSC always in the same segment

8e&ault address si9e is 64 'its 8e&ault o)erand si9e is 02 'its

64 'it o)erands BRAX- RBX- ...C are s)eci&ied with G 7@ )re&ixH in the o)code encoding

64 'it instruction )ointer BRIPC 64 'it stac/ )ointer BRSPC

Intro to x64 e!ersing " #on $arimer J

SummerCon 2011

Canonical addresses
%it 60

%it 0

Current im)lementations only su))ort 4* 'it linear addresses Canonical &orm means most signi&icant 'it o& address is extended to 'it 60

0xFFFFFFFFFFFFFFFF
Canonical 3igh 4art

0xFFFF800000000000 0xFFFF7FFFFFFFFFFF
;on"canonical 1ddress ange

%its 0"4E are the address- 'its 4*"60 are the same as 'it 4E

Aindows uses high addresses &or /ernel- low addresses &or user mode ;on"canonical address access results in K>4

0x0000800000000000 0x00007FFFFFFFFFFF
Canonical $ow 4art

0x0000000000000000
4* 'it canonical address ranges

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

10

x64 registers

02 'it registers extended to 64 'its

eax rax ebx rbx esp rsp

* additional 64 'it registers

r8- r - r10- ... r15

* additional 12* 'it @55 BSS7C registers

x!!8- x!! - ... x!!15 ?sed &or !ector and &loating )oint arithmetic
Intro to x64 e!ersing " #on $arimer 11

SummerCon 2011

Intel2158 16@

16@ is ",van4e, 5e4tor e6tension 1dds * 2:6 'it registers

"!!0""!!7

$ow 12* 'its o& 16@ registers o!erla) with @55 BSS7C registers

x!!0"x!!7

1lso a &ew new instructions .irst C4?s with 16@ were the Intel Sandy %ridge )rocessors released L1 2011
$5% 1$8 #MM0 XMM0 0

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

12

x64
%' RAX RBX RCX RDX RBP RSI RDI RSP R8 R R10 R11 R1$ R1' R1( R15

egisters
'1 &AX &BX &CX &DX &BP &SI &DI &SP 0 RIP RF)AGS %' '1 &IP &F)AGS 0

N0TE. To7 8a+9 o9 R:L";S is reserve,/ a+<a=s 0

> ne< in x64

Intro to x64 e!ersing " #on $arimer 10

SummerCon 2011

egister o)eration in x64 mode

%' '1 RAX -ero.exte/ded /ot0!odi1ied /ot0!odi1ied A* &AX AX A) 15 7 0

%'

'1 R8 -ero.exte/ded /ot0!odi1ied /ot0!odi1ied

15

R8D R8+
R8B,R8)

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

14

4=4 L?IM K1<

3ow many 'its is R DF 3ow many 'its is RSPF 3ow many 'its is R1$+F 3ow many 'its is R10BF 3ow many 'its is R1%F

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

1:

4=4 L?IM K1<

3ow many 'its is R DF *2 3ow many 'its is RSPF 3ow many 'its is R1$+F 3ow many 'its is R10BF 3ow many 'its is R1%F

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

16

4=4 L?IM K1<

3ow many 'its is R DF *2 3ow many 'its is RSPF 64 3ow many 'its is R1$+F 3ow many 'its is R10BF 3ow many 'its is R1%F

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

1E

4=4 L?IM K1<

3ow many 'its is R DF *2 3ow many 'its is RSPF 64 3ow many 'its is R1$+F 16 3ow many 'its is R10BF 3ow many 'its is R1%F

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

1*

4=4 L?IM K1<

3ow many 'its is R DF *2 3ow many 'its is RSPF 64 3ow many 'its is R1$+F 16 3ow many 'its is R10BF ' 3ow many 'its is R1%F

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

1J

4=4 L?IM K1<

3ow many 'its is R DF *2 3ow many 'its is RSPF 64 3ow many 'its is R1$+F 16 3ow many 'its is R10BF ' 3ow many 'its is R1%F Not a register???

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

20

4=4 L?IM K2<

Ahat,s in RAX a&ter each instructionF 00M230RAX401111111111111111500 0 00I6C0A)000000000000000000000 00I6C0AX 00I6C0&AX

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

21

4=4 L?IM K2<

Ahat,s in RAX a&ter each instructionF 00M230RAX40111111111111111150 RAX0700x1111111111111111 00I6C0A)000000000000000000000 00I6C0AX 00I6C0&AX

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

22

4=4 L?IM K2<

Ahat,s in RAX a&ter each instructionF 00M230RAX40111111111111111150 RAX0700x1111111111111111 00I6C0A)000000000000000000000 RAX0700x111111111111111$ 00I6C0AX 00I6C0&AX

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

20

4=4 L?IM K2<

Ahat,s in RAX a&ter each instructionF 00M230RAX40111111111111111150 RAX0700x1111111111111111 00I6C0A)000000000000000000000 RAX0700x111111111111111$ 00I6C0AX RAX0700x111111111111111' 00I6C0&AX

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

24

4=4 L?IM K2<

Ahat,s in RAX a&ter each instructionF 00M230RAX40111111111111111150 RAX0700x1111111111111111 00I6C0A)000000000000000000000 RAX0700x111111111111111$ 00I6C0AX RAX0700x111111111111111' 00I6C0&AX RAX0700x000000001111111(

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

2:

64 'it instructions

CD8& Con!ert dou'leword to Nuadword Bsign"extend &AX into RAXC CMPS8 Com)are Nword at RSI with Nword at RDI CMPXC*G1%B Com)are RDX:RAX with m128 )2DS8 $oad Nword at address RSI into RAX M23S8 5o!e Nword &rom address RSI to RDI M239X 9ero"extend dou'leword to Nuadword S:2S8 Store RAX at address RDI S#SCA))0.ast system call- re)lacement &or0S#S&6:&R S#SR&:0.ast system call- re)lacement &or S#S&XI:
Intro to x64 e!ersing " #on $arimer 26

SummerCon 2011

I4"relati!e addressing

Instruction")ointer"relati!e o)erands only used &or jum)s2'ranches in x*6

Can,t access &IP register ex)licitly in instructions

Can 'e used &or data access in x64 now:

!o;0rax40q<ord0ptr0=rip>0x1000?

.aster loading o& )osition"inde)endent code

Aindows: .ewer 'ase relocations in 47 &iles $inux: ;o >=( )ointer setu) in &unction )rologue ;o )re"lin/ing and no )er&ormance hit &or 1S$ on x64

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

2E

I4"relati!e addressing
I81 has G7x)licit I4 addressingH mode in analysis o)tions so you can see when rip"relati!e addresses are used:

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

2*

1))lication %inary Inter&ace

(he 1%I descri'es how to call &unctions

4assing )arameters eturn !alue Stac/ &rame 7xce)tions

GCalling con!entionH (here are two widely used x64 1%Is:

5icroso&t,s x64 1%I BAindowsC Sys6 x64 1%I B$inux- %S8- 5acC
Intro to x64 e!ersing " #on $arimer 2J

SummerCon 2011

5icroso&t x64 1%I

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

00

5icroso&t x64 1%I

(here,s only one calling con!ention Bno cdecl2stdcall2&astcallC Calling con!ention modeled a&ter &astcall

.irst 4 )arameters )assed in registers- rest on stac/ eturn in RAX or XMM0

Some registers are considered !olatile across &unction calls- some are not

1 &unction needs to sa!e non"!olatile registers i& it uses them

Intro to x64 e!ersing " #on $arimer 01

SummerCon 2011

5S x64 1%I: 4arameters O

eturn

.irst &our )arameters )assed in registers

RCX- RDX- R8- R &or integers XMM0- XMM1- XMM$- XMM' &or &loats

.or !aria'le arguments B!arargsC- &loating )oint !alues are stored in the &loating )oint and integer registers< i.e.- 4arameter 2 is always RDX or XMM1 1ny )arameter P * 'ytes )assed 'y re&erence Bno s)littingC

1:1 corres)ondence 'etween )arameters and registers

1dditional )arameters on stac/ eturn !alue in RAX or XMM0

XMM0 used &or &loats- dou'les- and 12* 'it ty)es B__m128C

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

02

5S x64 1%I: str@ct )arameters

I& a str@ct can 'e )ac/ed into * 'ytesit,s )assed in a register

=r on the stac/ i& it,s the :thQ argument

1ll str@cts o!er * 'ytes are )assed 'y re&erence Caller allocates s)ace and co)ies the str@ct 'e&ore )assing to the callee

(his is to a!oid )ro'lems with the callee modi&ying the caller,s co)y
Intro to x64 e!ersing " #on $arimer 00

SummerCon 2011

5S x64 1%I: 4arameters

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

04

5S x64 1%I: 4arams exam)le

pri/t1ABCi0C10Ci0Ci0C1DrD/B40140$E040.(40%0405E5FG

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

0:

5S x64 1%I: str@ct )aram exam)le

In this exam)le- the structure is )assed 'y re&erence- 'ut a new co)y is created on the stac/ &or the called &unction

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

06

4=4 L?IM K0

Ahat registers are used &or the &irst &our integer )arameters o& a &unctionF (rue2.alse: I& a structure has two 64 'it !alues- it can 'e )assed to a &unction s)lit across two registers Bi.e.- r8 and r C

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

0E

4=4 L?IM K0

Ahat registers are used &or the &irst &our integer )arameters o& a &unctionF
&CX- &DX- R8- R

(rue2.alse: I& a structure has two 64 'it !alues- it can 'e )assed to a &unction s)lit across two registers Bi.e.- r8 and r C

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

0*

4=4 L?IM K0

Ahat registers are used &or the &irst &our integer )arameters o& a &unctionF
&CX- &DX- R8- R

(rue2.alse: I& a structure has two 64 'it !alues- it can 'e )assed to a &unction s)lit across two registers Bi.e.- r8 and r C

:"LSE&

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

0J

5S x64 1%I: 6olatile registers

Some registers are !olatile and can 'e destroyed 'y &unctions

RAX- RCX- RDX- R8- R - R10- R11 +ou can,t rely on them 'eing the same a&ter calling a &unction Bthe com)iler might 'e a'le to...C

Some registers are non"!olatile and must 'e sa!ed 'y &unctions that use them

RBX- RBP- RDI- RSI- R1$- R1'- R1(- R15 +ou can rely on them 'eing the same a&ter calling a &unction 1 &unction that needs these registers must sa!e them to the stac/ and )o) them o&& 'e&ore returning
Intro to x64 e!ersing " #on $arimer 40

SummerCon 2011

5S x64 1%I: (he stac/

.unction )rologue needs to allocate stac/ s)ace &or sa!ed registers- local !aria'les- arguments to callees 4arameters are always at 'ottom o& stac/- right a'o!e return address

(here,s always s)ace &or 4 )arameters- e!en i& they,re not used Bhome s)aceC

Stac/ is always 16 'yte aligned

(his means address ends in 9ero hex 7xce)t within )rologue ?nless the &unction doesn,t call any other &unctions

1ll memory 'eyond RSP is !olatile Bcould 'e used 'y the =S or a de'uggerC ;o &rame )ointer Bi.e.- no !o;0rbp40esp in )rologueC unless stac/ is dynamically allocated BallocaC
Intro to x64 e!ersing " #on $arimer 41

SummerCon 2011

5S x64 1%I: Stac/ home s)ace

Caller,s )rologue allocates stac/ s)ace &or arguments to callees .or non"lea& &unctions- s)ace &or &our arguments is always allocated B4 I * 'ytes R 02 R 0x20C

s@b0esp400x$0

See) in mind that a&ter this instruction- stac/ needs to 'e aligned on 16 'yte 'oundary Bend in 0 hexC

So you,ll usually see s@b0esp400x$8 instead

In de'ug code- the callee usually )uts the register )arameters there in the )rologue In o)timi9ed- code- all 'ets are o&&- callee can do whate!er it wants
Intro to x64 e!ersing " #on $arimer 42

SummerCon 2011

5S x64 1%I: Stac/ diagram

1008 1010 1018 10$0 10$8

SummerCon 2011

ret@r/0addr rcx rdx r8 r

ArspF Arsp>08F05I/sta/ce Arsp>10F05Pre;I/sta/ce Arsp>18F0lpC!d)i/e Arsp>$0F0/S5o<C!d

Intro to x64

e!ersing " #on $arimer

40

0FD0 0FD8 0F&0 0F&8 0FF0 0FF8 1000 1010 1018 10$0 10$8

ecx05o!e edx05o!e r805o!e r 05o!e lp:ext III III rcx rdx r8 r

ArspF00005o!e0space Arsp>08F05o!e0space Arsp>10F05o!e0space Arsp>18F05o!e0space Arsp>$0F0lpC!d)i/e Arsp>$8F0III Arsp>'0F0III Arsp>(0F05I/sta/ce Arsp>(8F05Pre;I/sta/ce Arsp>50F0lpC!d)i/e Arsp>58F0/S5o<C!d

1008 ret@r/0addr

Arsp>'8F0Aret@r/0to0Ht!ai/CR:EEF

5S x64 1%I: Stac/ diagram

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

4:

5S x64 1%I: Stac/ diagram

0FD8 0F&0 0FD8 0F&0 0F&8 0FF0 0FF8 1000 1008 1010 1018 10$0 10$8
SummerCon 2011

ret@r/0addr ecx edx r8 r lp:ext III III ret@r/0addr rcx rdx r8 r

ArspF00005o!e0space Arsp>08F0arJHa Arsp>10F0arJHb Arsp>18F0arJHc Arsp>$0F0arJHd Arsp>$8F0lpC!d)i/e Arsp>'0F0III Arsp>'8F0III Arsp>(0F0Aret@r/0to0Ht!ai/CR:EEF Arsp>(8F05I/sta/ce Arsp>50F05Pre;I/sta/ce Arsp>58F0lpC!d)i/e Arsp>%0F0/S5o<C!d

Intro to x64

e!ersing " #on $arimer

46

5S x64 1%I: Stac/ 7xam)le K2

=)timi9ed code ;ote that the +i/Mai/ )arameters are not sa!ed in their home s)ace 1lso note that 0x2* 'ytes o& stac/ s)ace are still reser!ed &or the )arameters to 5essage%ox1

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

4E

System 6 x64 1%I

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

4*

System 6 x64 1%I

?sed 'y $inux- %S8- 5ac- others (otally di&&erent than 5S x64 1%I

1lso totally di&&erent than >CC,s x*6 $inux 1%I

Calling con!ention uses many registers:

6 registers &or integer arguments * registers &or &loat2dou'le arguments

Some registers considered !olatile and can change across &unction calls- others must 'e sa!ed 'y the callee
Intro to x64 e!ersing " #on $arimer 4J

SummerCon 2011

Sys6 1%I: 4arameters

.irst a!aila'le register &or the )arameter ty)e is used 6 registers &or integer )arameters

RDI- RSI- RDX- RCX- R8- R

' registers &or &loat2dou'le2!ector )arameters

XMM0"XMM7

;o o!erla)- so you could ha!e 14 )arameters stored in registers str@ct )arams can 'e s)lit 'etween registers 7!erything else is on the stac/ RAX holds num'er o& !ector registers BXMMxC
Intro to x64 e!ersing " #on $arimer :0

SummerCon 2011

Sys6 1%I: 4arameter seNuence

7xam)les< i/t01@/c1Ai/t0a401loat0b40i/t0cF

rax01@/c1Ardi40x!!040rsiF

1loat01@/c$A1loat0a40i/t0b401loat0cF

x!!001@/c$Ax!!040rdi40x!!1F

1loat01@/c'A1loat0a40i/t0b40i/t0cF

x!!001@/c'Ax!!040rdi40rsiF

;otice anything interesting a'out 1@/c1 and 1@/c'F

Intro to x64 e!ersing " #on $arimer :1

SummerCon 2011

Sys6 1%I: 4arameter exam)le K1

pri/t1ABCi0Ci0C10Ci0C10CiD/B40140$40'E040(405E040%FG

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

:2

Sys6 1%I: 4arameter exam)le K2

t"pede10str@ct0K i/t0a40bG do@ble0dG L0str@ctpar!G str@ctpar!0sG i/t0e40140J40540i40M40NG lo/J0do@ble0ldG do@ble0!40/G HH!$5%0"G
RDIO0e RSIO01 RDXO0sEa4sEb RCXO0J R8O05 R O0i XMM0O0sEd XMM1O0! #MM$O0" XMM'O0/ =RSP>0?O0ld =RSP>1%?O0M =RSP>$(?O0N

exter/0;oid01@/c0Ai/t0e40i/t0140str@ctpar!0s40 0000i/t0J40i/t0540lo/J0do@ble0ld40do@ble0!4 0000HH!$5%0"40do@ble0/40i/t0i40i/t0M40i/t0NFG 1@/c0Ae40140s40J40540ld40!40"40/40i40M40NFG

B(his exam)le is &rom the Sys6 x64 1%I s)ecsC SummerCon 2011 Intro to x64 e!ersing " #on $arimer :0

Sys6 1%I: (he stac/

;othing new here- exce)t changes due to 64 'it )lat&orm 1ligned on 16 'yte 'oundaries >CC still uses

%4 as a &rame )ointer 'y de&ault

;o reNuired home s)ace li/e 5S,s 1%I

Sometimes )arameters are sa!ed on the stac/ It,s in local !aria'les and not 'ehind the return address

.unctions can use stac/ s)ace u) to RSP>$5%

%eyond that is the RE) %0NE

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

:4

x64

e!ersing (ools

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

::

(ools &or x64

e!ersing: I81

(ools &or x64

e!ersing: Aind'g

(ools &or x64 e!ersing: 6isual 8ux8e'ugger

(ools &or x64

e!ersing: ed'

=ther re!ersing tools &or x64

8ynamic instrumentation

4I; 8ynamo I=

6irtual machines

%=C3S L75?

(hat thing @msuiche is wor/ing on !d'2!trace

3ow to get 'etter at re!ersing

(a/e a 'inary- any 'inary- 'ut smaller is )ro'a'ly easier e!erse it all

;ame e!ery &unction- )arameter- and !aria'le Comment almost e!ery line o& assem'ly 8o this without running it- unless you a'solutely ha!e to

+ou,ll 'e a )ro in no time< 1lso- read the 00: ol& olles inter!iew in 3I(%

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

61

x64

e&erences<

x64 architecture
Intel 1rchitecture So&tware 8e!elo)ment 5anuals: htt):22www.intel.com2)roducts2)rocessor2manuals2 158 1rchitecture 4rogrammer,s 5anuals: htt):22de!elo)er.amd.com2documentation2guides2)ages2de&ault.as)x

5S x64 1%I

x64 So&tware Con!entions: htt):22msdn.microso&t.com2en"us2li'rary2E/cdt6&yD2*6S.*0D2J.as)x @64 8ee) 8i!e: htt):22www.codemachine.com2articleTx64dee)di!e.html

Sys6 x64 1%I

System 6 1))lication %inary Inter&ace: htt):22www.x*6"64.org2documentation2a'i.)d&

Intro to x64 e!ersing " #on $arimer 62

SummerCon 2011

LuestionsF

Contact in&o:

7"mail: jlarimer@gmail.com (witter: @shydemeanor eddit: r0swell

SummerCon 2011

Intro to x64

e!ersing " #on $arimer

60