Configuring racoon/ipsec-tools to use x509 Certificates

This paper reviews the process to configure racoon/ipsec-tools to authenticate IPSec communications via x509 certificates.

Our example does not cover howto configure a Certificate Authority (CA). This process will be examined in another paper. Our example also assumes the clients are both NetBSD, however the procedures are nearly the same for Linux, FreeBSD and MacOSX as well.

Our example assumes a host to host IPSec connection, however this has no bearing on the x509 authentication and could just as easily be a gateway to gateway connection.

In our example, the hosts have the IP addresses and

You will need to configure your IPSec rules, these rules are placed in /etc/ipsec.conf. On /etc/ipsec.conf should contain:

  spdadd any -P out ipsec esp/tunnel/;
  spdadd any -P in ipsec esp/tunnel/;

Likewise on, /etc/ipsec.conf should contain:

  spdadd any -P out ipsec esp/tunnel/;
  spdadd any -P in ipsec esp/tunnel/;

Now on to the certificates, first you will want to place a copy of your Certificate Authority cert into /etc/racoon/certs. This location is configurable via “path certificate” in racoon.conf, more on this below.

You will also need to create a symbolic link for the hash value of the certificate since that is how openssl queries the appropriate certificate.

  # cp /path/to/cacert.pem /etc/racoon/certs
  # cd /etc/racoon/certs
  # ln -s cacert.pem `openssl x509 -hash -noout -in cacert.pem`.0

If a certificate should become compromised, it will be necessary to ensure that the compromised certificate can no longer be used to authenticate. This is done via Certificate Revocation Lists, these will be explained in greater detail in a future paper. However, if you have a crl, it can be implemented as shown below.

Note: You are linking the crl to the cacert hash and not the crl hash, the file extension is also .r0 rather than .0

  # cp /path/to/crl.pem /etc/racoon/certs
  # cd /etc/racoon/certs
  # ln -s crl.pem `openssl x509 -hash -noout -in cacert.pem`.r0

You will also want to place your host’s certificate and private key into /etc/racoon/certs as well, in our scenario these files are and respectively. It is necessary that the private key is not encrypted, directions to unencrypt your private key can be found here.

Now, you will need to modify /etc/racoon/racoon.conf to accomodate x509 certificate authentication, see the following from the host:

  path certificate "/etc/racoon/certs" ;

  remote anonymous

        #exchange_mode main,aggressive,base;
        exchange_mode main,base;

        verify_cert on;
        my_identifier asn1dn;
        certificate_type x509 "" "" ;

        lifetime time 24 hour ; # sec,min,hour

        #initial_contact off ;
        #passive on ;

        # phase 1 proposal (for ISAKMP SA)
        proposal {
                encryption_algorithm 3des ;
                hash_algorithm sha1;
                authentication_method rsasig ;
                dh_group 2 ;

        # the configuration makes racoon (as a responder) to obey the
        # initiator's lifetime and PFS group proposal.
        # this makes testing so much easier.
        proposal_check obey;
  sainfo anonymous
        pfs_group 2;
        lifetime time 12 hour ;
        encryption_algorithm 3des, cast128, blowfish 448, des, rijndael ;
        authentication_algorithm hmac_sha1, hmac_md5 ;
        compression_algorithm deflate ;

Repeat these steps for other clients using their respective host certificate (and key) and you should be up and running in no time.

One thought on “Configuring racoon/ipsec-tools to use x509 Certificates”

Leave a Reply

Your email address will not be published. Required fields are marked *