Interoperability between the KAME and FreeS/WAN IPsec implementations.


Objective:  Enable IP layer encryption and authentication between a Linux system and a BSD system using x509 certificates for IKE authentication.

Software: NetBSD 1.5.2 with racoon (snapshot 20011215a).  Linux 2.4.17 with FreeS/WAN 1.95.  OpenSSL (0.9.6c on Linux, 0.9.5a on NetBSD).

Initial configuration

I initially configured the systems to authenticate using pre-shared keys.  I was able to accomplish this with the help of a post to a NetBSD mailing list. That post included working configuration files for both racoon and FreeS/WAN, which was helpful, and it also documented some of the differences between the IPsec implementations.  These differences are:
You may wish to refer to the config files included in that post, and even use them as a basis for your own configuration.  I found them to be very complete and functional, and was able to get IPsec working with pre-shared keys in just a few minutes using them.

CA Creation

In order to use x.509 certificates for IPsec (or anything else, really), you need to have a trusted authority sign the certificates. In the commercial world, an entity such as Verisign or Thawte plays this part.  However, they charge commercial prices, which are far out of the realm of most mere mortals. Fortunately, OpenSSL includes all the necessary functionality to create a certificate authority (CA) ourselves.  In my case, I used the NetBSD system as the CA, since it will be able to fill that role in my network.  Any system can be a CA, however it's good to keep such a system as absolutely secure as possible since access to the CA keys allows one to sign certificates.

The OpenSSL "req" command is used to create the new CA certificate and private key:
openssl req -x509 -days 1095 -newkey rsa:2048 -keyout caKey.pem -out caCert.pem
This creates a new 2048 bit RSA keypair and stores the public part in an x.509 certificate in PEM format called caCert.pem.  This certificate is valid for 1095 days, or 3 years. In order for this command to properly generate a certificate, OpenSSL's openssl.cnf file must exist and have valid contents.  A sample openssl.cnf file is included with the OpenSSL distribution and provides a framework for you to modify.  Most modifications must be made in the CA_default section.  Be sure that the files referenced there actually exist and that they have valid contents.  See the OpenSSL ca(1) manual page for more info.

Once the CA certificate has been created, it must be installed on the client machines in such a way that they know to find it when they need to verify signatures.  On the Linux system, the CA certificate goes in /etc/ipsec.d/cacerts.  On the NetBSD system, it goes in the directory specified by the "path certificate" line in racoon.conf.  Once in place, it must either be renamed or be pointed to by an appropriately named symbolic link.  I installed this link with the following command:
ln -s caCert.pem `openssl x509 -noout -hash -in caCert.pem`.0
Certificate creation and signing

Each host must now generate a certificate request.  The request must then be signed by the CA in order to be a valid authentication key.  The request generation command is the same for both systems:
openssl req -newkey rsa:1024 -keyout hostKey.pem -out hostReq.pem
hostReq.pem must then be signed by the CA:
openssl ca -in hostReq.pem -days 365 -out hostCert.pem -notext
This creates a PEM formatted x.509 certificate in hostCert.pem that is valid for 1 year.  This certificate should then be copied back to the host that made the request and installed such that the IPsec implementation knows where to find it.

On Linux, the certificate must be converted to DER format and copied to /etc/x509cert.der.  The conversion is done with the following OpenSSL command:
openssl x509 -in hostCert.pem -outform DER -out x509cert.der
On the BSD system, the certificate can be anywhere, and you'll need to properly configure racoon.conf to tell racoon where to find it.  That will be described in more depth later.

Configuring the systems for x.509 authentication

At this point, the certificates are generated and in the necessary locations for use by their respective systems.  What remains is to configure the systems to use these certs to identify each other.  On the Linux side, this involves the addition of a couple of new fields to /etc/ipsec.conf.  An example connection definition is
conn freeswan-kame
        auto=start
        authby=rsasig
        type=transport
        left=192.168.2.99
        leftnexthop=192.168.2.1
        leftrsasigkey=%cert
        leftid="C=US/ST=Massachusetts/O=morgul.net/CN=freeswan/Email=root@morgul.net"
        right=10.0.0.1
        rightrsasigkey=%cert
        rightid="C=US/ST=Massachusetts/O=morgul.net/CN=kame/Email=root@morgul.net"
In this example, the "left" host is the Linux FreeS/WAN system. The authentication type is set to "rsasig", just as it would be if you were using standard unsigned RSA keys for authentication.  The key, however, is set to the special value '%cert'. The 'leftid' field is the subject field from the FreeS/WAN system's certificate. This value can be obtained via the following openssl command:
openssl -in /etc/x509cert.der -inform DER -noout -subject
Be sure to remove the leading '/' from the string that is output by that command.

On the BSD system, racoon.conf will contain essentially the same information, but it's formatted differntly.
remote 192.168.2.99
{
        exchange_mode main,aggressive;
        situation identity_only;
        my_identifier asn1dn "C=US/ST=Massachusetts/O=morgul.net/CN=kame/Email=root@morgul.net";
        peers_identifier asn1dn "C=US/ST=Massachusetts/O=morgul.net/CN=freeswan/Email=root@morgul.net";

        certificate_type x509 "myHostCert.pem" "myHostKey.pem";

        passive off;

        nonce_size 16;
        lifetime time 30 min;   # sec,min,hour
        initial_contact on;
        support_mip6 on;
        proposal_check obey;    # obey, strict or claim

        proposal {
                encryption_algorithm 3des;
                hash_algorithm sha1;
                authentication_method rsasig;
                dh_group 2 ;
        }
}
One thing that you may notice here is that the 'certificate_type' line contains information not found anywhere in the Linux configuration.  That's because this is where racoon is told about the location for its certificate and key files.  In Linux, the certificate used is always /etc/x509cert.der, and they private key path is given in /etc/ipsec.secrets.  Otherwise, the information here should be fairly straightforward.

Adjusting configuration parameters

When configuring my IPsec environment, I had several Linux systems that all needed to talk to a single NetBSD system, so it made more sense to configure the BSD system to more closely resemble the FreeS/WAN defaults.  This would make it much easier to add more FreeS/WAN systems to the network in the future.  The FreeS/WAN developers chose to violate certain RFC-specified requirements in an attempt to force a higher level of security.  For example, FreeS/WAN does not support DES encryption, since at this point that algorithm is considered to have far too small a keyspace to be safe from brute-force attacks.  By default, FreeS/WAN only supports 3DES encryption, although patches exist to allow it to support AES, among others.  It also only supports key sizes of 1024 bits or more for the Diffie-Hellman key exchange, which is why you see 'dh_group 2' in the proposal subsection above.  This change must be made in the 'sainfo' section of racoon.conf as well.  Another important adjustment in racoon.conf was to set the SA lifetime to 3600 seconds in the 'sainfo' section of the file.