India TAC Training

2006 Cisco Systems, Inc. All rights reserved.

Course acronym vx.x#-1

Troubleshooting IPSec VPN

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-2

Troubleshooting

Show commands Debug commands

Common Issues/Errors

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-3

Troubleshooting - Show Commands

IPSEC depends on successful policy negotiation. While IPSEC peers are negotiating IKE and IPSEC parameters, if the policies do not match the negotiations will result in failure. We can troubleshoot IKE & IPSEC by the following show commands:

show crypto isakmp sa (PIX / ASA and IOS routers) show crypto ipsec sa (PIX / ASA and IOS routers)
From the show commands we can determine if the SAs are in the right state, and if ISAKMP went through fine and now the IPSec traffic is being Encrypted/Decrypted between the two IPSec endpoints.

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-4

Troubleshooting - Debug Commands

IPSEC depends on successful policy negotiation. While IPSEC peers are negotiating IKE and IPSEC parameters, if the policies do not match the negotiations will result in failure. We can troubleshoot IPSEC by the following commands:

debug crypto ipsec debug crypto isakmp

From the debug error messages we can determine what part of the negotiation is failing and correct the appropriate parameter.

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-5

IPSEC Common Issues

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-6

IPSEC Common Issues

NAT with IPSec
Firewalling and IPSec MTU Issues Loss of Connectivity of IPSec Peers Routing

Interoperability Troubleshooting

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-7

Bypassing NAT Entries in ASA

access-list no_nat permit ip 10.1.1.0 255.255.255.0 10.1.2.0 255.255.255.0 Access-List bypassnat Defines Interesting Traffic to bypass NAT for VPN NAT 0 Command Bypasses NAT for the Pkts Destined over the IPSec Tunnel nat (inside) 0 access-list no_nat interface GigabitEthernet0/0 nameif outside security-level 0 ip address 209.165.202.129 255.255.255.0 interface GigabitEthernet0/1 nameif inside security-level 100 ip address 10.1.1.1 255.255.255.0

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-8

NAT in the Middle of an IPSec Tunnel

VPN Client
NAT

VPN Gateway
Internet

VPN Client
IPSec pass-through: ISAKMP cookie and ESP SPI are used to build translation table ASA(config)# fixup protocol esp-ike

IPSec Nat Transparency (NAT-T): UDP 500 UDP 4500 ASA(config)# isakmp nat-traversal <natkeepalive> IPSec over TCP: TCP 10000 ASA(config) isakmp ipsec-over-tcp port 10000

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-9

IPSEC Common Issues

NAT with IPSec
Firewalling and IPSec MTU Issues Loss of Connectivity of IPSec Peers Routing

Interoperability Troubleshooting

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-10

Firewall in the Middle

Internet
Router A Private Public Router B Private

One way block UDP port 500 (ISAKMP)

show crypto isakmp sa: MM_NO_STATE

ping from R A, R B has debug, ping from R B , R A has no debug

One way block ESP (IP protocol type 50)

show crypto isakmp sa: QM_IDLE R A has encryption no decryption, R B has decryption and encryption

UDP port 4500 (NAT-T)

VPN client tunnel is up, VPN client statistics transparent tunnel inactive

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-11

IPSEC Common Issues

NAT with IPSec
Firewalling and IPSec MTU Issues Loss of Connectivity of IPSec Peers Routing

Interoperability Troubleshooting

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-12

IPSec MTU Issue

Internet Internet

a. Original Packet b. IPSec Transport Mode 36 bytes c. IPSec Tunnel Mode 20+36=56 bytes a
IP Hdr 1 IP hdr 1 IP hdr 2 ESP hdr ESP hdr IP Hdr 1 TCP hdr TCP hdr TCP hdr Data Data Data

b
c
2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-13

IPSec and Path MTU Discovery

10.1.1.2 MTU 1500 e1/1 172.16.172.10/28 e1/0 MTU 1500 MTU 1400 IPSec Tunnel 172.16.172.20/28 MTU 1500 Path 1500 Media 1500 MTU 1500 10.1.2.2

Path 1500 Media 1500 1500 DF=1 ICMP Type3 Code 4 (1444) 1444 DF=1

ICMP: dst (10.1.2.2) frag. needed and DF set unreachable sent to 10.1.1.2 (debug ip icmp output)

1500 DF copied ICMP (1400) IPSec SPI copied

ICMP: dst (172.16.172.20) frag. needed and DF set unreachable rcv from 172.16.172.11 Adjust path MTU on corresponding IPSec SA
path mtu 1400, media mtu 1500 current outbound spi: EB84DC85

ICMP Type3 Code 4

(1344)
1344 DF=1
2006-7 Cisco Systems, Inc. All rights reserved.

1400

1400

1344
ASA/PIX 8.02-14

How to manually determine Path MTU

Ping from client PC:
ping www.cisco.com -l 1400 -f Pinging www.cisco.com [198.133.219.25] with 1400 bytes of data: Reply from 198.133.219.25: bytes=1400 time=168ms TTL=120 ping www.cisco.com -l 1500 -f Pinging www.cisco.com [198.133.219.25] with 1500 bytes of data: Packet needs to be fragmented but DF set.

Ping from the router:

sv3-6#ping ip Target IP address: 198.133.219.25 Repeat count [5]: 1 Datagram size [100]: 1400 Extended commands [n]: y Source address or interface: FastEthernet0/0 Set DF bit in IP header? [no]: yes Sweep min size [36]: 1400 Sweep max size [18024]: 1500 Sweep interval [1]: 10

!!!!......

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-15

MTU Issues Work Around: Adjusting IP MTU & TCP MSS

ASA/PIX: mtu outside 1492 sysopt connection tcpmss 1392

IP Fragmentation and PMTUD

http://www.cisco.com/en/US/tech/tk827/tk369/technologies_white_pa per09186a00800d6979.shtml Adjusting IP MTU, TCP MSS, and PMTUD on Windows and Sun Systems http://www.cisco.com/en/US/tech/tk870/tk877/tk880/technologies_tec h_note09186a008011a218.shtml

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-16

IPSEC Common Issues

NAT with IPSec
Firewalling and IPSec MTU Issues Loss of Connectivity of IPSec Peers Routing

Interoperability Troubleshooting

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-17

Loss of Connectivity of IPSec Peers

Internet
IPSec SA SPI Peer Local_id Remote_id Transform IPSec SA SPI Peer Local_id Remote_id Transform

ESP SPI=0xB1D1EA3F

00:01:33: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSec packet has invalid spi for destaddr=172.16.172.28, prot=50, spi=0xB1D1EA3F(-1311643073)
2006-7 Cisco Systems, Inc. All rights reserved. ASA/PIX 8.02-18

Loss of Connectivity of IPSec Peers

DPD Message (R-U-There) DPD Message (R-U-There ACK)

Dead Peer Detection

crypto isakmp keepalive <# of sec. between keepalive> <# of sec. between retries if keepalive fails>

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-19

IPSEC Common Issues

NAT with IPSec
Firewalling and IPSec MTU Issues Loss of Connectivity of IPSec Peers Routing

Interoperability Troubleshooting

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-20

show crypto ipsec sa

I sent encrypted packets, and got nothing back from remote host

IPSec

ASA1(config)# sh crypto ipsec sa

#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0 #pkts decaps: 32906, #pkts decrypt: 32906, #pkts verify: 32906

I sent decrypted packets, and got nothing from the local host

ASA2(config)# sh crypto ipsec sa

#pkts encaps: 32829, #pkts encrypt: 32829, #pkts digest: 32829 #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-21

IPSEC Common Issues

NAT with IPSec
Firewalling and IPSec MTU Issues Loss of Connectivity of IPSec Peers Routing

Interoperability Troubleshooting

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-22

Interoperability Tips
Start with configuring the two ends side by side with exact matching policies
Phase I Parameters
IKE authentication method Hash algorithm DH group

Phase II Parameters
IPSec mode (tunnel or transport) Encryption algorithm Authentication algorithm PFS group

ISAKMP SA lifetime
Encryption algorithm Turn off vendor specific features: Mode config, Xauth, IKE keepalive Matching pre-shared secret

IPSec SA Lifetime
Interesting traffic definition

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-23

Other Issues - Errors

IKE Policy mismatch

Pre-shared key mismatch

Access-list mismatch IPSec policy mismatch

IKE Pool misconfigured

IPSec peer misconfigured Additional Considerations

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-24

IKE Policy mismatch

If there is a mismatch or if there are no common ISAKMP policies then the following error will be seen.The solution is to configure a common ISAKMP policy on both peers.

ISAKMP (0): atts are not acceptable. Next payload is 0 ISAKMP (0): no offers accepted! ISAKMP (0): SA not acceptable!

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-25

Pre-shared key mismatch

If the pre-shared keys on both the peers do not match then the following error will be seen. 1d00H:%CRPTO-4-IKMP_BAD_MESSAGE: IKE message from 172.16.172.34 failed its sanity check or is malformed which will result in : %CRYPTO-6-IKMP_MODE_FAILURE: Processing of Main Mode failed with peer at172.16.172.34

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-26

Access-list mismatch
If the access-lists on the peer IPSEC devices do not match that is if they are not mirror images of each other then the following error will occur : IPSEC(validate_transform_proposal): proxy identities not supported ISAKMP: IPSec policy invalidated proposal It is also important to note that the word any should not be used in the access-list .

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-27

IPSec policy mismatch

If the IPSEC transform-set policies do not match , then the following error will be seen. Both the peer should have identical IPSEC transform-set policies. ISAKMP: SA life duration (VPI) of 0x0 0x46 0x50 0x0 ISAKMP: authenticator is HMAC-MD5 IPSEC(validate_proposal): transform propos al (prot 3, trans 2, hmac_alg 1) not supported

ISAKMP (0): atts not acceptable. Next payload is 0 ISAKMP (0): SA not acceptable!

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-28

IKE Pool misconfigured

If the PIX is configured for IKE mode-config and the pool is misconfigured then the following error will be seen: IPSEC(key_engine_delete_sas): delete all SAs shared with 171.69.89.116 return status is IKMP_NO_ERR_NO_TRANS04101: ISAKMP: Failed to allocate address for client from pool

ISADB: reaper checking SA 0x80e02638, conn_id = 0 DELETE IT!

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-29

IPSec peer misconfigured

If the IPSEC peer is misconfigured under the crypto map , then the following error message will be seen
1d00h: ISAKMP: No cert, and no keys (public or preshared) with remote peer 172.167.172.33 1d00h: ISAKMP (0:1): purging SA

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-30

Additional Considerations - Split tunneling

We need to use split tunneling when using the Unity client if we want to simultaneously have a IPSEC tunnel to the PIX and also INTERNET connection. vpngroup vpn3000 split-tunnel 160 access-list 160 permit ip 192.168.2.0 255.255.255.0 30.1.1.0 55.255.255.0 Here the IPSEC tunnel will be only established between the source destination specified by the access-list.

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-31

Additional Considerations IPSec Multiple peers

If there are multiple peers to a PIX , make sure that the match address access-lists for each of the peers are mutually exclusive from the match address access list for the other peers If this is not done, the PIX will choose the wrong crypto map to try and establish a tunnel with one of the peers

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-32

Additional Considerations IPSec from behind low-end firewalls

Issues With IPSec/ESP or IPSec/UDP, two VPN users to SAME IPSec VPN server 2nd user may be disallowed 2nd user may cause disconnection of 1st user Solutions Multiples ISAKMP sessions Vary source port [NOT UDP 500] and keep track Based on SPI [Keep UDP 500/500]

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-33

Additional Considerations DES - 3DES issue

When using SSH, if the pix has only DES key enabled and SSH client is 3DES then the following error will occur pix520-1(config)# 315011: SSH session from 171.69.89.116 on interface outside for user "" disconnected by SSH server, reason: "Invalid cipher type" (0x06)315011: SSH session from 171.69.89.116 on interface outside for user "" disconnected by SSH server, reason: "Invalid cipher type" (0x06) We can also use the sh ssh sessions to view the current ssh connections

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-34

Q&A

2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-35

 2006-7 Cisco Systems, Inc. All rights reserved.

ASA/PIX 8.02-36