Chapter 6. Certificates and PKI

Table of Contents

Why it is needed
How do certificates work
Certificates in organizations
CA service providers
Certificate management protocols
Flows and feeds
OpenSSL as CA
Setting up the CA
Daily handling
Scripted approach
Mail service

Why it is needed

Now that some of the basic network services have been installed, it is time to look at certificate management.

Certificates are used to either authenticate people or clients to a service, or to support encryption of data sent across the network. For small networks, administrators usually create self-signed certificates, but the problem with self-signed certificates is that every client that wants to communicate with a service using a self-signed certificate needs to include this certificate in its trust store, and that the owner of the certificate usually does not keep track of when the certificate is about to expire.

How do certificates work

Assume there are two parties (Alice and Bob are the two names often used in encryption fields) willing to communicate with each other over a secured, encrypted channel. They generally need to use an encryption key which only those two parties know. Once they have such an encryption key, they can use it to communicate with each other securely:

  • because they are the only two with the key, others cannot decrypt the information being sent between the two

  • because no-one else has the key, each of those parties is certain that information received from the other one is indeed from the other party (as it would otherwise not be properly encrypted)

  • if the data stream also contains time stamp information (or challenge information) the parties can also ascertain that the information is "fresh" and not resent from a previous communication between the two

The major difficulty comes from getting a "shared" encryption key without having to send it over the (clear) network so that a malicious person (Carol or Charles, let's use the gender-neutral "Chris" ;-) cannot obtain it.

Symmetric keys

Most secure communication uses a symmetric key, often called a session key.

A symmetric key is an encryption key which is used for encryption and decryption. In the previous example of a secure communication, the symmetric key is the secret encryption key that the two parties (and only those two parties) should have a hold off. Such a symmetric key is usually used with an algorithm that is fairly simple (and thus has good performance) in regard of encryption and decryption.

This good performance requirement is needed because it will be used to encrypt and decrypt all traffic sent between the two services, which can be a fair amount of data. If the encryption and decryption algorithms used would be very CPU intensive, the communication would be far too slow (or use too many resources on the systems).

A way to get both parties to know a common, secret key is to each use a secret set of algorithms (or known algorithms with secret data sets) which have the property that, whatever order is used to apply the algorithms, the result is always the same. Extremely simple examples are addition or multiplication, but far more complex ones exist as well. A non-mathematical example would be to add in colored paint to a pre-agreed color: if you first add in blue and then red, or first red and then blue, the result is always the same.

In the (very simple) case of addition, both Alice and Bob have a secret and will exchange the results of their algorithms:

   Alice                 Bob
  "1000"                "1000"
  +7428                 +5879
 = 8428                = 6879
   6879                  8428
  +7428                 +5879
= 14307               = 14307  <-- the symmetric key

The number to begin with may be made public (although it is safer when its not) because, even with Chris knowing the numbers "1000, 8428, 6879", if that person does not know the algorithm (or data) used, it will be difficult for him to find out what the common secret key will be. Of course, in this example, addition is quite simple, but at least it gives the idea on what such key exchange can be like.

Public and private keys

The idea behind public-key cryptography is that, unlike a symmetric key, there are now two keys in play:

  • the private key is only known by the target service (say Alice)

  • the public key is known by everyone who wants to communicate with the target

The two keys play such a role that encryption of data happens with the public key, but it is not possible to decrypt that data with the public key. To decrypt it, the private key is needed. Another aspect is that it is possible to encrypt with the private key and decrypt with the public, allowing people or systems to verify that data was indeed encrypted by that person (in reality this is used to "sign" data, which is to make a checksum of the file and then encrypt this checksum).

In the previous communication problem from before, using public/private keys would make it a whole lot easier: Bob just takes Alice' public key, encrypts the data he wants to send to Alice using the public key, and he's done. Alice can do the same with Bob's public key so that two-way communication is possible. However, there are a few drawbacks here:

  • Public-key cryptographic functions are quite resource intensive.

    For this reason, they are only used to exchange a symmetric key or part of a symmetric key (like the initial number "1000" used in the previous example).

  • A malicious person could create his own public/private key pair and tell the world that his public key is the public key of Alice. If Bob would want to communicate with Alice but uses Chris' public key, he would encrypt the data so that Chris can only read it. Chris then decrypts the data, encrypts it with Alice' real public key and sends it to Alice. A communication channel between Alice and Bob is made, but Chris is "in the middle" reading (and perhaps even manipulating) the data. This is called a Man-In-The-Middle or MITM attack.

The second part is the major "downside" of a public key - how to ensure that this public key is actually of the target person or service that the communication is targeted towards? This is where certificates come into play...


A certificate is this same public key, together with data that identifies who the public/private key belongs to. This certificate also has signatures attached that are generated by the private keys of other certificates. The idea is that, if both Alice and Bob know a public key (certificate) of a party they both trust (and know that this public key is indeed of the trusted party), then Alice can send her own certificate, signed by this trusted party, to Bob.

Figure 6.1. Certificates and CAs in a nutshell

Certificates and CAs in a nutshell

Bob then validates the signature on this certificate (using the public key he has of the trusted party). If the signature indeed pans out, then Bob verifies if the certificate isn't in the list of revoked certificates (which is managed by the Certificate Authority). If that isn't the case, then Bob knows that the certificate he got is indeed from Alice (because the trusted party says so) and of which the private key is not known to be lost or stolen (otherwise it would have been mentioned in the revocation list).

The list of certificates that are trusted (the certificates of the Certificate Authority) are stored in a trust store.

A malicious person now has a more difficult task. If a wrong certificate is generated, then the trusted party will probably not sign it. As a result, Chris cannot "fake" a certificate because both Alice and Bob will check the signature of the certificate and the certificate revocation list before they agree that it is a valid certificate.

Certificates in organizations

In a larger organization or network, certificates play an important role. Many servers use SSL (or TLS) encryption. In this case, the client connects to the server and receives that servers' certificate information. The client validates this certificate with the keys he has of trusted "authorative" actors. If the certificate is valid, then the client knows he is communicating with the correct service. Some magic occurs then to generate a session key only the client and the server know (isearch on the internet for "ssl handshake pre-master" for the details on this) so they can then use symmetric encryption algorithms for the rest of the communication.

If self-signed certificates would be used, then all the clients should have the public keys of all these systems in their own list of trusted keys (the trust store), which is a nightmare to manage. And if this would be managed, why not just keep the symmetric keys then and do not use public key infrastructure... It is also possible to have the clients accept self-signed certificates, but then these systems are vulnerable for MITM attacks.

This is where good certificate management comes into play. Serious organizations need a way to "sign" certificates used in the architecture with a single key (or very limited set of keys) and distribute this (trusted) public key to all the clients. The trust store of these clients is then much smaller.

The service that signs certificates is called the Certificate Authority or CA. Often, a chain of keys is used: a top key called the Root CA which is extremely heavily protected and is used to sign a small set of subkeys or signing keys. These signing keys are then used to sign the certificates. Often, these signing keys have a different purpose (keys for signing certificates that will be used for code signing, keys for signing certificates that will be used for authenticating people or clients, keys for signing certificates that will be used for internet-facing services, etc.)

Problems with Certificate Authorities

On the Internet, many popular CA services exist, like, Verizon, Geotrust, and more. All these companies try to become this "trusted party" that others seek and offer signing services: they will sign certificates you create (some of them even generate public/private key pairs if you want). These services try to protect their own keys and only sign certificates from customers they can validate are proper customers (with correct identities).

Although this seems like a valid model, it does have its flaws:

  • Such popular services are more likely target for hackers and crackers. Once their keys are compromised, their public keys should be removed from all trust stores (or, if proper validation of keys against a revocation list is implemented, have these keys available in the revocation list). However, that also means that all certificates used for services that are signed by these companies will effectively stop working as the clients will not trust the certificates anymore.

  • These services have a financial incentive to sign certificates. If they do not do proper validation of the certificate requests they get (because "volume" might be more important to them than integrity), they might sign a certificate from a malicious person (who is pretending to be a valid customer).

  • The more of these keys that are in the clients' trust store, the more chance that a malicious certificate is seen by the client as valid.

The latter is important to remember. Assume that the clients' trust store has 100 certificates that it "trusts", even though the organization only uses a single one. If a malicious user creates a certificate that identifies itself as one of your services and gets it signed by one of those 100 CAs, then the clients will trust the connection if it uses this (malicious) certificate (because it is signed with one of the trusted keys).

Keeping a lid on CAs

To keep the risk sufficiently low that a malicious certificate is being used, find a CA that is trusted fully, and keep the trust store limited to that CA.

And who else to trust better than yourself ;-)

By using management software for certificates internally, many important activities can be handled without having to seek for and pay a large CA vendor:

  • Create a CA store

  • Sign requests of certificate requests that users have sent in

  • Revoke certificates (or at least, signatures made on certificates) for end user certificates that should not be trusted anymore (for instance because they have been compromised)

  • Have users create a certificate (in case they don't have the proper tools on their systems, although it is not recommend to generate private and public keys on a remote server)

  • Have users submit a certificate signing request to the system for processing, and then download the signed certificate from the site

CA service providers

If managing a private CA is seen as a bit too difficult, one can also opt to use an online CA service provider. There are many paid-for service providers as well as free ones. Regardless of the provider chosen, make sure that the internal certificates are only signed by a root certificate authority that is specific to this environment, and that the provider does not sign certificate requests by others with the root CA trusted internally.

For this reason, be careful which service provider to choose, and make a clear distinction between certificates that will be used internally (not exposed to others) and those that are facing external environments.

Certificate management protocols

The Online Certificate Status Protocol (OCSP) is a simple protocol that clients can use to check if a certificate is still valid, without needing to (re)download the Certificate Revocation List (CRL) over and over again and parsing the list. An advantage is that you don't need to expose "all the bad certificates" that you know of (which might be a sort of information leakage) and that clients don't need to parse the CRL themselves (as it is now handled on a server level).


To provide a private CA, openssl will be used as the main interface. The system itself should of course be very secure, with an absolute minimum of users having access to the system (and even those users should not have direct access to the private key). A mail-driven system is used to allow users to request signing of their certificates; the certificate signing requests are stored in a folder where one of the certificate administrators can then sign the requests and send them back to the client.

Also, an OCSP daemon will be running to provide the necessary revocation validation.

Flows and feeds

In the next picture, the two main flows are shown: the OCSP (to check the validity of a certificate) and mail (with certificate signing request).

Figure 6.2. Flows and feeds for the CA server

Flows and feeds for the CA server


Administration-wise, all actions are handled through logons on the system. The usual configuration management services apply (such as using Puppet for general system administration).


An important aspect in the setup is auditing. All CA activities should be properly audited, as well as operating system activities.


Figure 6.3. Operations on a CA server

Operations on a CA server


Three roles are identified for CA operations.

Figure 6.4. User definitions for CA operations

User definitions for CA operations

The first one is the regular system administrator. His job is to keep the system available and has privileged access to the system. However, the mandatory access control system in place prevents the admin from (easily) reaching the private key(s) used by the CA software.

The CA admin has no system privileges, but is allowed to call and interact with the CA tools. The CA admin can generate new private keys & certificates, sign certificates, etc. However, he too cannot reach the private keys.

The security admin has a few system privileges, but most importantly has the ability to update the MAC policy and read the private keys. The role is not involved in the daily operations though.


The mandatory access control in place (SELinux) prevents direct access to the private key. Of course, next to the MAC policy, other security requirements should be in place as well, such as limited role assignment (only a few people in the organization should be security admin), proper physical security of the system, etc.

OpenSSL as CA

As CA management software, the presented architecture uses the command line interface of openssl. Although more user friendly interfaces seem to exist, they are either not properly maintained or very difficult to manage in the long term.

The OpenSSL stack is a very common, well maintained library for handling encryption and encryption-related functions, including certificate handling. Most, if not all Linux/Unix systems have it installed.

Setting up the CA

Setting the defaults

The default settings for using OpenSSL are stored in the /etc/ssl/openssl.cnf file. Below shows a few changes suggested when dealing with CA certificates.

dir = ./genficCA
default_days = 7305 # 20 years

default_bits = 2048

Setting up the root CA

The root CA is the top-level key, which will be "ultimately trusted". If an HSM device would be used, then this key would be stored in the HSM device itself and never leave. But in case this isn't possible, create a root CA as follows:

# cd /etc/ssl/private
# openssl genrsa -des3 -out root-genfic.key 2048

This generates root-genfic.key, which is the private key and will be used as the root key.

Next, create a certificate from this key. In the example, a 20-year lifespan is used. The shorter the lifespan, the faster there is a need to refresh the key stores within the (entire) organization, which can be a costly activity. However, the longer the period, the more time malicious persons have to get to the key before a new one is generated.

# openssl req -new -x509 -days 7205 -key root-genfic.key -out root-genfic.crt
Country Name (2 letter code) [AU]:BE
State or Province Name (full name) [Some-State]:Antwerp
Locality Name (eg, city) []:Mechelen
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Gentoo Fictional, Inc.
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:GenFic Root CA
Email Address []:

This provides the root-genfic.crt file, which is the certificate (public key of the key pair created before, together with identity information and key information). To view the certificate in its glory, use openssl x509:

# openssl x509 -noout -text -in root-genfic.crt

Finally, create the certificate revocation list (which is empty for now):

# mkdir genficCA
# touch genficCA/index.txt
# echo 01 > genficCA/crlnumber
# openssl ca -gencrl -crldays 365 -keyfile root-genfic.key -cert root-genfic.crt \
    -out root-genfic.crl

Now put all files in the location(s) as defined in openssl.cnf.

Multilevel (or hierarchical) certificate authorities

In many environments, the root key itself isn't used to sign "end" user (or system) certificates: a hierarchy is established to support a more flexible approach on certificates. Often, this hierarchy is to reflect the use of the certificates (end user certificates, system certificates) and, if the organization has multiple companies, these companies often have an intermediate certificate authority. In this example, a simple hierarchy is used: below the root certificate, two signing certificates are used: one for end users and one for systems.

root CA
+- user CA
`- system CA

The root certificate (created above) will be stored offline (not reachable through the network, and preferably in a shut down state or on a HSM device) together with the certificates created for the user and system CA's. This is needed in case these certificates need to be revoked, since the process of revoking a certificate (see later) requires the certificate.

To support the two additional CA's, edit openssl.cnf accordingly. Add sections for each CA to support - copy the "[ CA_default ]" settings and edit the directory and other settings where necessary. For instance, call them CA_root, CA_user and CA_system to identify the different certificate authorities.

Then, create the keys:

# openssl genrsa -des3 -out user-genfic.key 2048
# openssl genrsa -des3 -out system-genfic.key 2048

Next, create certificate requests. Unlike the root CA, these will not be signed with the same key, but rather with the root CA key.

# openssl req -new -days 1095 -key user-genfic.key -out user-genfic.csr
# openssl req -new -days 1109 -key system-genfic.key -out system-genfic.csr

If the command asks for a challenge password, leave that empty. The purpose of the challenge password is that, if the certificate is to be revoked, the challenge password needs to be given again as well. This gives some assurance that rogue administrators can't revoke certificates that they don't own, but as we will store the root key offline (rather than keep it available as a service) revocation of the certificates requires (physical) access to the keys anyhow.

The .csr files (Certificate Signing Request) contain the public key of the key pair generated, as well as identity information. Based on the CSR, the root CA will sign and create a certificate:

# openssl ca -name CA_root -days 1095 -extensions v3_ca -out system-genfic.crt \
    -infiles system-genfic.csr
# openssl ca -name CA_root -days 1109 -extensions v3_ca -out user-genfic.crt \
    -infiles user-genfic.csr

Notice that there are different validation periods for the certificates given. This is to support the (plausible) management activities that result when certificates are suddenly expired. If the organization "forgets" that the certificates expire, their user certificates will expire first, and two weeks later the system certificates. Not only will this allow the continued servicing of the various systems while the user CA certificate is being updated, but will also allow the organization to prepare for the system CA update in time (since they now have 14 days as they "noticed" that the user CA certificate was expired and now have time until the system CA certificate expires).

When the certificate signing requests are handled, the genficCA/newcerts directory contains the two certificates. This allows for the CA to revoke the certificates when needed.

Finally, copy the signed certificates and prepare the directory structure for the two certificate authorities as well.

# mkdir genficSystemCA
# cd genficSystemCA
# touch index.txt
# echo 01 > serial
# echo 01 > crlnumber
# mkdir newcerts crl private
# mv ../system-genfic.key private/

Protecting the root CA

As mentioned before, it is good practice to protect the root CA. This can be done by handling the system as a separate, offline system (no network connection) although, with the other CAs in place, there is little reason for the root CA to be permanently "available". In other words, it is fine to move it to some offline medium (and even print it out) and store this in a (very) safe location. For instance put it on a flash disk (or tape) and put it in a safe (or even better, two flash disks in two safes).

Starting the OCSP Server

OpenSSL has an internal OCSP daemon that can be used to provide OCSP services internally.

One way to start it is in a screen session:

$ screen -S ocsp-server
$ cd /etc/ssl/private/genficUserCA
$ openssl ocsp -index index.txt -CA user-genfic.crt \
  -rsigner user-genfic.crt -rkey private/user-genfic.key \
  -port 80
Waiting for OCSP client connections..

This way, the OCSP daemon will listen on port 80 and within the screen session called ocsp-server.

If the administrator ever needs to get to this screen session, he can run "screen -x ocsp-server" and he's attached to the session again.

Of course, an init script for this can be created as well (instead of using screen).

Daily handling

With the certificate authority files in place, let's look at the daily operational tasks involved with certificates.

User certificates

A user certificate is used by a person to identify himself towards services. Such certificates can be used to authenticate a user for operating system access, but a more common use of user certificates is access towards websites: the user has a key (and certificate) loaded in the browser (or to some store that the browser has access to, in case the key itself is on a smartcard or other HSM) and sites that want the user to identify the user can ask for the user certificate.

A user can create his own key using the following openssl command, and its accompanying certificate signing request:

# openssl req -newkey rsa:2048 -keyout amber.key -out amber.req

The signing request (.req) is sent to the certificate authority, which then validates and signs the request, generating a signed certificate:

# openssl ca -name CA_user -days 396 -out amber.crt -infiles amber.req

This generates a certificate that is valid for 13 months.

System certificates

System certificates are created similarly to the user certificates, but depending on the system that uses it, the key might not be encrypted (-nodes). Encrypting a private key is a useful measure to protect the keys more (especially user keys are best used with encryption or on a HSM device with password protection since they are in many cases "mobile"), but not all hosts that need a system certificate can deal with encrypted keys.

# openssl req -newkey rsa:2048 -nodes -keyout w03443ad.key -out w03443ad.req

It is a good practice to use, as e-mail address, the e-mail address of the requestee (or a "shared" e-mail address that is known to remain fixed throughout the lifetime of the system). This information is used later to send e-mails regarding expiration dates or other important news.

# openssl ca -name CA_system -days 3650 -out w03443ad.crt -infiles w03443ad.req

Listing certificate expirations

To list the expiration date(s) of the certificates signed by a CA, iterate over the files in the newcerts location, showing the subject and its expiration date:

# openssl x509 -noout -subject -enddate -in 01.pem

Revoking a certificate

Suppose a user reports that his or her private key is lost (or even worse, compromised).

First, find this users' (signed) certificate. Look at the index.txt file within the CA directory to find out what number the certificate has, and then use this number for the revocation:

# grep amber genficUserCA/index.txt
V ... 23    unknown /C=BE/.../CN=Amber McStone/emailAddress=...
# openssl ca -name CA_user -revoke genficUserCA/newcerts/23.pem

As a result, the database is now updated, showing that the certificate itself is revoked. Next, publish the new certificate revocation list:

# openssl ca -name CA_user -gencrl -out genficUserCA/crl/user-genfic.crl

The resulting certificate revocation list then needs to be published where all users can find it.

Scripted approach

Directly using openssl is prone to errors or to the use of different parameters that might lead to confusion later. It is advised to use a script or tool that simplifies this. <command> [<options>] <value>

Command can be one of:
  -r, --create-root             Create a root CA key named <value>
  -c, --create-child            Create a child CA key named <value>
  -s, --sign-request            Sign a certificate request (<value> is input)
  -R, --create-request          Create a key and signing request named <value>.req
  -x, --revoke                  Revoke the key matching <value>

Options can be one of:
  -p, --parent <parent>         Use <parent> as the parent key value
  -o, --output <file>           Save resulting file as <file>
  -v, --valid <days>            Number of days that the certificate is valid

For instance, first a root key pair is created, valid for 20 years:

$ sudo -r genfic -v 7300

Next, child key pairs are created - one for system certificates and one for user certificates. Both are valid for 10 years:

$ sudo -p genfic -c genfic-system -v 3650
$ sudo -p genfic -c genfic-user -v 3650

New certificates (and key pairs) for systems can be generated from the script itself (by creating a key and signing request, and then signing):

$ sudo -R /var/db/ca/self/myhostname
$ sudo -p genfic-system -s /var/db/ca/self/myhostname.req -o /var/db/ca/self/myhostname.crt

Revoking certificates can be done easily as well:

$ sudo -p genfic-system -x

SELinux security

With a scripted approach, the script itself can be protected by SELinux mandatory access control policies, where the policy is designed such that:

  • no-one except the security administrator can access the private keys generated

  • no-one except the security administrator can update SELinux policies

  • no-one except the security administrator can relabel scripts to the ca_cli_t domain (used for or relabel files from the private key type (ca_private_key_t)

  • system administrators have no direct access to memory or disk devices

Mail service

To pull in the certificate signing requests through e-mail, configure fetchmail to pull in the mails, and procmail to handle the attachments (extract them from the mail and store them in a specific directory).


The fetchmail daemon is provided by the net-mail/fetchmail package. Once installed, create a Linux system user (say "csrpull") with a configuration file similar to the following as ~/.fetchmailrc:

set daemon 60

poll protocol IMAP with option interval 1:
  user "" is csrpull here
  password 'thisIsOneHellOfAPassword'
  mda "/usr/bin/procmail -d %T"

Once created, make sure it is only readable by the csrpull user itself:

$ chmod 0600 /home/csrpull/.fetchmailrc

Finally, start the fetchmail daemon. Don't forget to add it to the default runlevel too.

# run_init rc-service fetchmail start
# rc-update add fetchmail default


From fetchmail, mails that are pulled in will be sent towards procmail, so configure procmail to automatically extract the certificate signing request files from the mails and drop them in the proper directory (like /home/csrpull/requests). Create a .procmailrc file in the csrpull home directory with the following content:

* ^To:
| munpack -q -C "$CSR_DROP_DIR"

These rules can be made as powerful as necessary. For instance, if the requests don't need to be signed manually, a procmail recipe can be created that calls the proper script which, after extracting the request, signs it automatically and sends it back to the sender (or to the address set in the certificate itself).

With this file in place, fetchmail will automatically forward any mails received towards the procmail recipe, which will then extract the attachment(s) into the requests folder.