Run racoon in non-daemon mode with debugging enabled.

Put a third machine between the client and the server. You should be then use that machine to sniff the communication between the client and the server with a network monitoring program such as tcpdump or Ethereal. The packets should be encrypted. If you see unencrypted packets (e.g. plain text L2TP), there is something wrong in your setup.

The L2TP/IPsec packet structure looks like the example below. The PPP Payload contains the original IP datagram, and the italicized text represents what is encrypted with IPsec

Use nmap (or any other good portscanner) on the client and scan for open UDP ports on the server (nmap -sU 10.0.2.1). You should not see the L2TP daemon (UDP port 1701). The only IPsec related open ports should be UDP 500 (IKE) and optionally UDP 4500 (NAT-T).

Enable debugging of IKE negotiations on the Windows/XP side as described in "Enabling detailed tracing for IKE negotiations" ^[45]. This will generate a file called OAKLEY.LOG.

Enable debugging of PPP negotiations on the Windows/XP side as described in "HOW TO: Enable PPP Logging in Windows (Q234014)" ^[46]. This will generate a file called PPP.LOG. Note that set tracing * enable will log even more)

Install the Network Monitor Driver and then use Netcap ^[47] to write the communication to a file. This file can then be imported in Ethereal.