Inside IKEv2: bringing up an IPsec tunnel one exchange at a time

A site-to-site tunnel that won't come up is one of those tickets where it's easy to start changing settings at random. Swap the encryption, re-type the pre-shared key, flip PFS on and off, and hope. I find it much calmer to treat IPsec as a short series of steps that each either worked or didn't. If you know which step you're on, the logs tell you what to fix next.

This post brings up an imaginary tunnel between two gateways, GW-A and GW-B, one exchange at a time. At each step I'll note what success looks like and what failure looks like. The references are the standards themselves: RFC 4301 (the IPsec architecture), RFC 7296 (IKEv2) and RFC 4303 (ESP). For configuration advice I lean on NIST's Guide to IPsec VPNs, SP 800-77 Rev. 1 (June 2020). It's long, but its troubleshooting chapter is very practical.

For the lab, GW-A protects the subnet 198.51.100.0/24 and GW-B protects 192.0.2.0/24. These are the example ranges RFC 7296 uses in section 2.9, and that's where we'll need them.

Step 0: the model in your head (SPD, SAD, PAD)

Before any packet moves, it helps to know what the gateway is actually deciding. RFC 4301 describes three databases:

• SPD, the Security Policy Database. Every packet that crosses the IPsec boundary matches an SPD entry, and each entry has one of three actions: PROTECT, BYPASS or DISCARD (RFC 4301, section 4.4.1). "Traffic from 198.51.100.0/24 to 192.0.2.0/24: PROTECT" is the policy behind our tunnel.
• SAD, the Security Association Database. Each entry holds the parameters of one Security Association: keys, algorithms, sequence counters, lifetimes (section 4.4.2).
• PAD, the Peer Authorization Database. It links the SPD to IKE. It says which peers may talk to us, how each one authenticates, and which identities and traffic each peer is allowed to claim (section 4.4.3).

One detail is easy to forget: an SA is simplex. It protects traffic in one direction only, so two-way traffic needs a pair of SAs, and IKE creates them in pairs (RFC 4301, section 4.1). When a vendor's "show" command lists two SPIs for one tunnel, that's why: one inbound, one outbound.

What IKE does, then, is fill the SAD. A packet matches a PROTECT entry in the SPD, no SA exists yet, and IKE's job is to create one (RFC 7296, section 2.9).

Step 1: IKE_SA_INIT, two messages in the clear

Every IKEv2 conversation starts with two exchanges, IKE_SA_INIT and IKE_AUTH. RFC 7296 notes they were "known in IKEv1 as Phase 1", and that they normally take four messages in total (section 1.2). Every IKE exchange is a request and a response.

 ┌─────────┐                                        ┌─────────┐ 
 │ GW-A  ● │         IKEv2  ·  site-to-site         │ GW-B  ● │ 
 └─────────┘                                        └─────────┘ 
      │                                                  │      
      │──────────── IKE_SA_INIT  SAi1 KEi Ni ───────────►│      
      │◄─────────── IKE_SA_INIT  SAr1 KEr Nr ────────────│      
      │                                                  │      
      │ ┌┄ IKE SA  SK{…} ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┐ │      
      │═══════ IKE_AUTH SK{IDi AUTH SAi2 TSi TSr} ══════►│      
      │◄══════ IKE_AUTH SK{IDr AUTH SAr2 TSi TSr} ═══════│      
      │ ┆                                              ┆ │      
      │ ┆  ┌┄ CHILD SA pair (ESP) ┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┐  ┆ │      
      │ ┆  ┆                                        ┆  ┆ │      
      │ ┆  ┆                  data                  ┆  ┆ │      
      │ ┆  ┆                                        ┆  ┆ │      
      │ ┆  └┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┘  ┆ │      
      │ └┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┘ │      
 ─ in the clear   ═ inside the IKE SA                · ● ·      
 UDP 500 · 4500 if NAT is seen          rekey timer ·     ·     
                                                     · · ·      

The first pair negotiates algorithms, exchanges nonces and does a Diffie-Hellman exchange:

GW-A  HDR, SAi1, KEi, Ni           -->
      <--  HDR, SAr1, KEr, Nr, [CERTREQ]   GW-B

SAi1 is GW-A's list of acceptable crypto suites for the IKE SA. KEi is its Diffie-Hellman public value and Ni is its nonce. GW-B picks one suite, sends that choice back in SAr1, and completes the Diffie-Hellman exchange with KEr. After this, each side can compute a value called SKEYSEED. From it they derive SK_e and SK_a (encryption and integrity keys, one set per direction) and SK_d, which seeds the keys for Child SAs later (RFC 7296, section 1.2).

There's one wrinkle. GW-A has to send its Diffie-Hellman value before it knows which group GW-B will pick, so it guesses. If it guesses wrong, GW-B replies with an INVALID_KE_PAYLOAD notify naming the group it wants, and GW-A must retry with the full proposal list again (section 1.2). One extra round trip in the logs is normal. A loop of them is not.

What failure looks like here:

• No response at all. Usually reachability: IKE must be sent to and from UDP port 500 or 4500 (section 2.23), so check the path, ACLs and any upstream firewall.
• NO_PROPOSAL_CHOSEN. None of the offered suites was acceptable (section 3.10.1). Compare encryption, PRF, integrity and DH group on both ends, field by field.
• A COOKIE notify. This is a DoS defence, not an error. A responder under load can ask the initiator to prove it can receive packets at its claimed address before the responder commits any state (section 2.6).

Don't expect a packet capture to help much, even here. NIST says plainly that tcpdump is "not very useful" for debugging IKE: you can only inspect the first IKE_SA_INIT exchange, which shows little more than the DH groups, and everything after it is encrypted. Turn on the IKE debug logs instead, and turn them off again when you're done (SP 800-77r1, section 6.3).

Step 2: IKE_AUTH, proving who you are

The second pair travels inside the new IKE SA. The SK{…} notation means "encrypted and integrity-protected with that direction's SK_e and SK_a":

GW-A  HDR, SK {IDi, [CERT,] [CERTREQ,] [IDr,] AUTH, SAi2, TSi, TSr}  -->
      <--  HDR, SK {IDr, [CERT,] AUTH, SAr2, TSi, TSr}                 GW-B

Each side asserts an identity (IDi, IDr) and proves it with the AUTH payload. The proof uses either a signature with a certificate's private key or a shared secret. The AUTH payload also protects the integrity of the earlier IKE_SA_INIT messages, so a tampered negotiation is caught here. With a shared secret, the names in the ID payload must correspond to the key used (RFC 7296, section 1.2).

Notice the second half of each message: SAi2, TSi, TSr. IKE_AUTH doesn't only authenticate. It also negotiates the first Child SA, so a working IKE_AUTH gives you a usable tunnel in four messages.

What failure looks like here:

• AUTHENTICATION_FAILED. The IKE SA is not created (section 1.2). The usual suspects are a PSK mismatch, an ID type or value that doesn't match what the peer expects (IP address vs FQDN vs certificate DN), or a certificate chain the peer doesn't trust.
• The IKE SA comes up but the Child SA doesn't. RFC 7296 lists notify types that fail the Child SA without tearing down the IKE SA, including NO_PROPOSAL_CHOSEN and TS_UNACCEPTABLE (section 1.2). If your gateway shows phase 1 "up" and phase 2 "down", this is the case. Look at the Child SA proposal (SAi2) and the traffic selectors.
• Big IKE_AUTH messages that disappear. With certificates, IKE_AUTH messages can reach several kilobytes. When a message is bigger than the path MTU it gets fragmented at the IP layer, and some NAT devices and ISPs drop IP fragments. IKEv2 message fragmentation (RFC 7383, section 1.1) was written for exactly this case. It's worth enabling on both ends if your platform supports it.
• The responder logs a "successful but very short" connection. NIST points out that the responder sets up the Child SA before the initiator has checked the responder's AUTH. If the initiator then rejects it, all it can do is delete the IKE SA straight away, which looks odd from the responder's side (SP 800-77r1, section 6.4). When you see this, read the initiator's logs.

A word on pre-shared keys. NIST notes that PSK authentication is often deployed because it's the easiest to configure, and that it's most commonly used for gateway-to-gateway tunnels. It recommends a PSK that is "pseudorandomly created and has at least 128 bits of entropy", warns that short dictionary-word PSKs are open to offline dictionary attacks, and strongly discourages group PSKs shared by many peers (SP 800-77r1, section 3.3.4). It goes further later on: public-key authentication is preferred over PSKs, PSKs fit gateway-to-gateway tunnels that cross an administrative boundary, and within one administrative domain they should not be used at all (section 7.2.2).

Step 3: traffic selectors, the part that causes the most head-scratching

TSi and TSr are how the two gateways agree on which traffic the Child SA carries. Each traffic selector is an address range, a port range and an IP protocol ID. In our lab, GW-A would propose TSi = 198.51.100.0–198.51.100.255 and TSr = 192.0.2.0–192.0.2.255, and if GW-B agrees it sends the same values back (RFC 7296, section 2.9).

IKEv2 lets the responder narrow the proposal to a subset that its own policy allows. If nothing in the proposal is acceptable, it answers TS_UNACCEPTABLE (section 2.9). Narrowing is helpful, but it can also hide a mismatch: the tunnel comes up, yet some subnets silently don't pass because the SA only covers part of what one side expected.

Vendors don't all call these "traffic selectors". Palo Alto Networks calls them proxy IDs. Its documentation says that with IKEv1, PAN-OS supports only exact proxy-ID matches, while IKEv2 supports traffic-selector narrowing. It also says that each proxy ID counts as a VPN tunnel against the firewall's tunnel capacity, and that if traffic is NATed you should use the post-NAT networks (Palo Alto Networks, Proxy ID for IPSec VPN). Whatever the name, write the selectors for both ends down side by side, as source/destination pairs from each gateway's point of view. Mirror-image mistakes are easy to spot on paper and hard to spot in two different GUIs.

Step 4: ESP, what actually goes on the wire

With the Child SA in place, user traffic is carried by ESP. In tunnel mode, the gateway takes the whole original packet, wraps it, and puts a new IP header in front.

                 ESP tunnel mode, layer by layer                
                                                                
  GW-A: encapsulate ▸                      ◂ GW-B: decapsulate  
                                                                
                                                                
                                                                
                      [IP][TCP][  data  ]                       
                    host→host                                   
                                                                
                                                                
                                                                
 ▒ = ciphertext                                                 
                                                                
                                                                
 GW-A ─■────────═════════───────────────────────────────══ GW-B 
         on the wire: outer header, SPI and seq readable        

RFC 4303 is precise about which parts get which protection. This is its Table 1, for the case of separate encryption and integrity algorithms (section 2); combined-mode ciphers such as AES-GCM have their own table there:

Three consequences matter in practice:

1. The outer header isn't protected by ESP. In transport mode, ESP protects only the next-layer protocol, not the IP header in front of it (RFC 4301, section 4.1). In tunnel mode, everything you care about is inside, and the outer header just carries gateway-to-gateway addresses.
2. The SPI and sequence number are readable but can't be altered undetected. That's what lets a receiver find the right SA and run anti-replay. Anti-replay is possible only when integrity is selected, and with 32-bit sequence numbers, receivers must support a window of at least 32 packets, with 64 as the recommended default (RFC 4303, sections 1 and 3.4.3).
3. The outer IP header says protocol 50. The header in front of ESP carries protocol value 50 (RFC 4303, section 2). If a NAT sits between the peers and both support NAT traversal, IKEv2 detects it during IKE_SA_INIT, and both sides must then wrap ESP in UDP, moving to port 4500 (RFC 7296, section 2.23; the format is in RFC 3948). So a firewall in the path needs to allow UDP 500, UDP 4500 and, when there's no NAT, IP protocol 50.

What about transport mode? RFC 4301 says that, apart from two narrow exceptions, whenever either end of an SA is a security gateway, the SA must be tunnel mode (section 4.1). IKEv2 creates tunnel mode unless both sides agree to USE_TRANSPORT_MODE (RFC 7296, section 1.3.1). For site-to-site, tunnel mode is what you'll see.

Step 5: lifetimes, PFS and rekeying

When a tunnel comes up fine and then drops after a fixed amount of time, look at rekeying.

New Child SAs and rekeys both use the CREATE_CHILD_SA exchange, which either side can start. RFC 7296 notes that "some of its function was referred to as a Phase 2 exchange in IKEv1" (section 1.3). A rekey creates the new SA first and deletes the old one afterwards (section 2.8).

Two facts explain many rekey surprises:

• Lifetimes aren't negotiated in IKEv2. Each end enforces its own lifetime policy, and the end with the shorter lifetime ends up doing the rekeying (RFC 7296, section 2.8). Mismatched lifetimes aren't an error, but they make it harder to predict which side will start a rekey. NIST's table of recommended options lists 24 hours for the IKE SA and 8 hours for the IPsec SA (SP 800-77r1, Table 1).
• A PFS mismatch hides until the first rekey. A CREATE_CHILD_SA request may carry its own KE payload, a fresh Diffie-Hellman exchange that gives the new Child SA stronger forward secrecy (section 1.3). The first Child SA, though, is built inside IKE_AUTH with no separate DH exchange. So, as NIST puts it, any DH group mismatch "will only become apparent during a rekey message exchange hours later" (SP 800-77r1, section 6.4). If a tunnel fails like clockwork after its Child SA lifetime, compare the PFS setting on both ends first.

Liveness checks matter here too. IKEv2 checks whether a peer is alive with an empty INFORMATIONAL request, which must be answered (RFC 7296, sections 1.4 and 2.4). NIST advises against setting dead-peer-detection probes below one minute: on a congested link, dropped probes can tear down a healthy tunnel and add re-establishment traffic to a link that's already struggling (SP 800-77r1, section 6.4).

Choosing the algorithms

There are two good references. NIST's Table 1 recommends IKEv2 (IKEv1 is listed as legacy), AES in GCM, CTR, CBC or CCM modes, HMAC-SHA-256/384/512, and DH groups 14 to 21. It also recommends that PFS use the same or a stronger group than the initial IKE exchange (SP 800-77r1, Table 1). For anyone moving off IKEv1, NIST says 3DES, MD5, SHA-1 and DH groups 2 and 5 "should not be used". It points to AES-CBC with HMAC-SHA-2, or AES-GCM, with DH group 14 or an ECDH group (19, 20 or 21) instead (section 3.9).

The IETF's implementation requirements say much the same thing. For IKEv2, RFC 8247 marks DH group 14 as MUST, group 19 as SHOULD, and groups 2 and 5 as SHOULD NOT. For ESP, RFC 8221 marks AES-GCM with a 16-byte ICV as MUST and 3DES as SHOULD NOT. These tables describe what implementations must support, which isn't quite the same as what you should pick, but they point the same way.

And IKEv1 itself? RFC 9395 (April 2023) moved it to Historic status, and NIST says IKEv1 "should not be used for new deployments" (SP 800-77r1, section 3.11). One small trap NIST mentions: vendors mean different things by "SHA-2". Some mean only SHA2-256, others mean the whole family, so compare the exact hash on both ends (section 6.4).

The troubleshooting order

Here is the whole walk-through folded into a checklist. The order matters: each step assumes the one before it worked.

Step 6 deserves a note. NIST describes two classic traps. First, if internal traffic is NATed to the gateway's public address before the IPsec policy is checked, the source no longer matches the policy and the packet may go out unencrypted. Second, with route-based VPNs, if the routes into the tunnel don't match on both ends, traffic can be encrypted in one direction but not the other (SP 800-77r1, sections 6.4 and 6.5). NIST's own position is to prefer policy-based VPNs scoped to specific subnets over a 0.0.0.0/0 selector plus routing, despite the extra management. If you do run a route-based tunnel, keep the routes and the security policy on both ends deliberately in step.

A habit that saves time: when the tunnel is down, ask "which step am I on?" before changing anything. One look at the IKE log usually answers it.

Hardening remote access VPNs

Everything above applies to remote access VPNs too, but those gateways face the whole internet, which raises the stakes. In September 2021 the NSA and CISA published a joint information sheet, Selecting and Hardening Remote Access VPN Solutions. The NSA's announcement says VPN servers are attractive targets because they are entry points into protected networks, and that multiple nation-state APT actors have weaponised public CVEs against VPN devices. Exploitation can let an attacker steal credentials, run code remotely, weaken the encryption of traffic, hijack sessions and read sensitive data from the device (NSA press release; CISA alert).

Their top recommendations are:

• choose standards-based VPNs and tested, validated products (the NIAP Product Compliant List is the example given);
• use strong authentication, such as multi-factor authentication;
• apply patches and updates promptly;
• reduce the attack surface by disabling features the VPN doesn't need.

NIST adds two points for IKE-based remote access in particular: group PSKs are strongly discouraged, and remote access should use EAP-TLS or machine-certificate authentication (SP 800-77r1, sections 3.3.4 and 7.2.2).

Wrapping up

IKEv2 looks complicated mostly because vendors spread its settings across several screens. Underneath, it's four messages to get a working tunnel, one exchange type for new and rekeyed SAs after that, and an ESP format whose current specification dates from December 2005 (RFC 4303). Once you can name the step a tunnel is stuck on, IPsec troubleshooting becomes a checklist rather than guesswork.

Sources

• RFC 4301: Security Architecture for the Internet Protocol
• RFC 4303: IP Encapsulating Security Payload (ESP)
• RFC 7296: Internet Key Exchange Protocol Version 2 (IKEv2)
• RFC 7383: IKEv2 Message Fragmentation
• RFC 3948: UDP Encapsulation of IPsec ESP Packets
• RFC 8221: Cryptographic Algorithm Implementation Requirements and Usage Guidance for ESP and AH
• RFC 8247: Algorithm Implementation Requirements and Usage Guidance for IKEv2
• RFC 9395: Deprecation of the IKEv1 Protocol and Obsoleted Algorithms
• NIST SP 800-77 Rev. 1: Guide to IPsec VPNs (publication page) and PDF
• Palo Alto Networks: Proxy ID for IPSec VPN
• NSA: NSA, CISA Release Guidance on Selecting and Hardening Remote Access VPNs
• CISA: CISA and NSA Release Guidance on Selecting and Hardening VPNs