IPSec VPN:

How does it really work?

Yasushi Kono
(ComputerLinks Frankfurt)

Before the Agenda

My intention of this presentation:
I know that many people do not have any clue
what s happening while establishing Security
Associations prior to create VPN tunnels.
This topic is quite complex. I want you to get
started into this topic taking away a bit of its
complexity.

Agenda
Introductory Information on IPSec VPN
Why Diffie-Hellman Algorithm?
IKE SA in Main Mode
IPSec SA in Quick Mode
Some Troubleshooting Tips

Agenda
Introductory Information on IPSec VPN
Why Diffie-Hellman Algorithm?
IKE SA in Main Mode
IPSec SA in Quick Mode
Some Troubleshooting Tips

Introductory Information on IPSec VPN

Before establishing a Site-to-Site VPN

connection, both gateways must agree upon
parameters for encrypting communication.
This negotiation process is divided into two phases:
Phase 1 and Phase 2.
2
In Phase 1 a IKE Security Association (IKE SA) will be created. Those
parameters are therefore:

Before establishing a Site-to-Site VPN

connection, both gateways must agree upon
parameters for encrypting communication.
This negotiation process is divided into two phases:
Phase 1 and Phase 2.
2
In Phase 1 a IKE Security Association (IKE SA) will be created. Those
parameters are therefore:
1.) Encryption Algorithm (3DES, AES-128, AES-256,) to ensure
privacy

Before establishing a Site-to-Site VPN

connection, both gateways must agree upon
parameters for encrypting communication.
This negotiation process is divided into two phases:
Phase 1 and Phase 2.
2
In Phase 1 a IKE Security Association (IKE SA) will be created. Those
parameters are therefore:
1.) Encryption Algorithm (3DES, AES-128, AES-256,) to ensure
privacy
2.) Hash Algorithm (SHA-1 or MD5) to ensure data integrity

Before establishing a Site-to-Site VPN

connection, both gateways must agree upon
parameters for encrypting communication.
This negotiation process is divided into two phases:
Phase 1 and Phase 2.
2
In Phase 1 a IKE Security Association (IKE SA) will be created. Those
parameters are therefore:
1.) Encryption Algorithm (3DES, AES-128, AES-256,) to ensure
privacy
2.) Hash Algorithm (SHA-1 or MD5) to ensure data integrity
3.) Diffie-Hellman Group (1, 2, 5, or 14)

Before establishing a Site-to-Site VPN connection,

both gateways must agree upon parameters for
encrypting communication.
This negotiation process is divided into two phases:
Phase 1 and Phase 2.
2
In Phase 1 a IKE Security Association (IKE SA) will be created. Those
parameters are therefore:
1.) Encryption Algorithm (3DES, AES-128, AES-256,) to ensure
privacy
2.) Hash Algorithm (SHA-1 or MD5) to ensure data integrity
3.) Diffie-Hellman Group (1, 2, 5, or 14)
4.) Method of mutual authentication (Preshared Key or Certificate)

You have the choice between two different

modes in Phase 1:
Main Mode
Aggressive Mode
Main Mode consists of a 6 packet negotiation
In Aggressive Mode only three packets are
exchanged between both VPN gateways.

Before beginning to analyze the

information exchanged by the
gateways, lets have a look at the DiffieHellman Algorithm:
Why is Diffie-Hellman necessary?

Agenda
Introductory Information on IPSec VPN
Why Diffie-Hellman Algorithm?
IKE SA in Main Mode
IPSec SA in Quick Mode
Some Troubleshooting Tips

Usually, payloads are encrypted

symmetrically by means of symmetric
encryption algorithms, like 3DES or AES.
The problem with symmetric encryption is
exchanging the keys over the Internet
while preventing them from falling into
the wrong hands.
One answer is asymmetric encryption.

So, when asymmetric encryption is

addressing problems with symmetric
encryption, why is latter technology still
necessary?

The answer is:

Performance!

The Diffie-Hellman Algorithm is not an

Encryption Algorithm rather than a Key
Agreement Protocol.

To understand the Diffie-Hellman Algorithm,

you have to have basic knowledge on the
Modulo operation.
The Modulo operation finds the remainder
of division of one number by another.
Given two numbers, a and b,
n = a mod b
is the remainder on division of a by b.

Examples:
30 mod 16 = 14
114 mod 100 = 14
8 mod 3 = 2
Isnt that simple?

Diffie-Hellman:
Initiator takes a prime number p and an
integer a with 1 < a < p and a secret
integer x with:
X = ax mod p
X, a, p: public parameters,
x: secret parameter.

Initiator sends X, a, and p to the Responder.

The Responder takes a secret integer y and
computes:
Y = ay mod p
The number Y, which is public will be sent
back to the Initiator.

Initiator and Responder are computing their

common encryption key:
Kx = Yx mod p

= (ay mod p)x mod p

= (ax mod p)y mod p

= Xy mod p = Ky

Hence Kx = Ky

Now, that we have the basic

understanding of the DiffieHellman Algorithm, we can now
fully understand IKE SA in Main
Mode.

Agenda
Introductory Information on IPSec VPN
Why Diffie-Hellman Algorithm?
IKE SA in Main Mode
IPSec SA in Quick Mode
Some Troubleshooting Tips

As all of you might already know,

IKE SA in Main Mode consists of 6 packets

First of all, the Initiator sends the IKE SA

Parameters to be negotiated upon to the
Responder:
IKE SA: 3DES or AES-128?
SHA-1 or MD5?
DH-Group 5 or 14?
Certificate or Preshared Key?

Initiator
Packet Number 1:

Responder

The Responder sends back the parameters

to be used in common to the Initiator:
IKE SA:

AES-128!
SHA-1!
DH-Group 14!
Preshared Key!

Initiator
Packet Number 1:

Packet Number 2:

Responder

Then, the Initiator sends the public DiffieHellman Parameters and a random
number, which is called Nonce:

Initiator
Packet Number 3:

Responder

Then, the Responder sends back its own

public Diffie-Hellman Parameters and its
own random number (Nonce):

Initiator
Packet Number 3:

Packet Number 4:

Responder

Both parties know from each other, which

parameters and encryption key to use in
common.
The Initiator then
builds the hash of the Preshared Key
encrypts the Nonce of the Responder and
the Hash of the Preshared Key

Initiator
Packet Number 5:

Packet Number 6:

Responder

What happens next?

Each of the Gateways are receiving their
own Nonces encrypted by other parties.
The next step is to decrypt the encrypted
Nonces to verify the identity of the
communicating gateways.

After receiving the sixth

packet, you will get the
following message in
SmartView Tracker:
IKE SA: Main Mode completion

Agenda
Introductory Information on IPSec VPN
Why Diffie-Hellman Algorithm?
IKE SA in Main Mode
IPSec SA in Quick Mode
Some Troubleshooting Tips

Why is Quick Mode

necessary?
To establish the IPSec SA!

Quick Mode to establish an

IPSec SA consists of 3
packets.

You have to negotiate upon:

The Encryption algorithm

The Hash Algorithm
The IPSec Protocol (ESP, AH)
If PFS is to be used or not
(if yes, the DH-Group is to be
determined again)

One word regarding IPSec

Protocols:
Authentication Header (AH)
provides:
Data Integrity Checking
Replay Protection

Encapsulating Security Payload

(ESP) provides:

Payload Encryption
Data Integrity Checking
Replay Protection

Why is it necessary to
agree upon the
Encryption Algorithm
Hash Algorithm
Eventually DH-Group
again?

Because the parameters

negotiated there is for
applying to the payload of
the packets and not to the
identity of the other gateway!

The 3 Packets in Quick Mode

The Message you will get after succesful

IPSec SA Negotiation is:

IKE SA: Quick Mode completion

Agenda
Introductory Information on IPSec VPN
Why Diffie-Hellman Algorithm?
IKE SA in Main Mode
IPSec SA in Quick Mode
Some Troubleshooting Tips

Some Troubleshooting Methods:

Mostly, you will get the right information with
SmartView Tracker
There, the most frequent error messages are:
IKE SA: No proposal chosen
Encryption failure: No valid SA
INVALID_ID_INFORMATION

Some Troubleshooting Methods:

Received Notification from Peer: Malformed Payload

Any solution?

One standard method for Debugging

IKE/IPSec is
vpn debug ikeon
to generate the output file ike.elg.

But, without any knowledge of the theory

of IPSec, is it useful to analyze the ike.elg
file?

For more troubleshooting IKE/IPSec in a Check

Point Environment, attend the next
presentation:

305: Troubleshooting in the

Check Point Environment
Part II
By Tobias Lachmann

Any Questions?

Thanks a lot for

your attention!
Should you have questions:
yk@computerlinks.de