↑↑ Home ↑ Net & Web  

Decrypting S/MIME mails with a PKCS11 smart card from the command line

How it works in principle

As other operations based on public-key systems, S/MIME does not directly encrypt the message with the public key of the recipient. Rather, the message is encrypted with a symmetric encryption method, and the randomly generated message encryption key is encrypted with the public key of the recipient and included in the mail. This has two main advantages: Public key operations are computationally expensive, so having to encrypt and decrypt only the message key with the public key is more efficient. And the message may have multiple recipients and should also be readable to the sender from the Fcc folder, so encrypting only the message key with the required public keys saves space compared to multiple encrypted versions of the whole message which would otherwise be necessary.

When the private key needed for decryption is located on a smart card, it gets more complicated still. The smart card may contain multiple public-private key pairs that share a key ID. But the mail message contains only the serial number of the public key. So a two-stage search is required: First the keys on the card are searched for a match of one of the public keys from the mail message (remember: multiple keys for sender + recipient(s)). Then the ID of the matching public key is read, and the card is searched for the private key with the same ID. To my knowledge there is no command-line program that performs this double search, so decrypting such a message on the command line requires several manual steps and/or some custom coding.

Packages and tools

The following software packages will be required or at least helpful:

Dissecting the S/MIME mail

Save the S/MIME encrypted mail to a file, msg.mail. It should have the content type (possibly with an additional name= attribute):

Content-Type: application/pkcs7-mime; smime-type=enveloped-data

Otherwise it may not be an encrypted mail, or differ in other ways from the one I have investigated, and the following may not apply to it. (The procedure for analysing it may still be of use.)

First we list the structure of the "enveloped" message. This requires the preprocessing steps:

openssl smime -in msg.mail -pk7out -out msg.pem
openssl pkcs7 -inform pem -outform der -in msg.pem -out msg.der

The abstract structure of an S/MIME mail is defined by ASN.1 (the X.580 standard). The first of the openssl commands above generates a file in PEM format, which is a base64-encoded version of the binary DER format, which we obtain with the second openssl command. Simply base64-decoding the mail body also results in a DER-format file, but in my experience it can have some peculiarities that our subsequent commands do not like. The openssl smime command tolerantly decodes the S/MIME contents and combines sections of the encrypted message if they are stored in several blocks, which will be useful later.

Now we can list the internal structure of the DER file. The more widely available way of doing that is with openssl:

openssl asn1parse -inform der -in msg.der

The output looks somewhat like this:

    0:d=0  hl=4 l=3626 cons: SEQUENCE          
    4:d=1  hl=2 l=   9 prim: OBJECT            :pkcs7-envelopedData
   15:d=1  hl=4 l=3611 cons: cont [ 0 ]        
   19:d=2  hl=4 l=3607 cons: SEQUENCE
   23:d=3  hl=2 l=   1 prim: INTEGER           :00
   26:d=3  hl=4 l=1326 cons: SET
   30:d=4  hl=4 l= 438 cons: SEQUENCE
   34:d=5  hl=2 l=   1 prim: INTEGER           :00
   37:d=5  hl=3 l= 157 cons: SEQUENCE          
   40:d=6  hl=3 l= 140 cons: SEQUENCE          
   43:d=7  hl=2 l=  11 cons: SET               
   45:d=8  hl=2 l=   9 cons: SEQUENCE          
   47:d=9  hl=2 l=   3 prim: OBJECT            :countryName
   52:d=9  hl=2 l=   2 prim: PRINTABLESTRING   :DE
   56:d=7  hl=2 l=  15 cons: SET               
   58:d=8  hl=2 l=  13 cons: SEQUENCE          
   60:d=9  hl=2 l=   3 prim: OBJECT            :stateOrProvinceName
[... localityName, organizationName, organizationalUnitName, commonName ...]
  183:d=6  hl=2 l=  12 prim: INTEGER           :B7A699164CABAFE0826D9B31
  197:d=5  hl=2 l=  13 cons: SEQUENCE
  199:d=6  hl=2 l=   9 prim: OBJECT            :rsaEncryption
  210:d=6  hl=2 l=   0 prim: NULL
  212:d=5  hl=4 l= 256 prim: OCTET STRING      [HEX DUMP]:...
  472:d=4  hl=4 l= 438 cons: SEQUENCE
[...]
 1356:d=3  hl=4 l=2270 cons: SEQUENCE
 1360:d=4  hl=2 l=   9 prim: OBJECT            :pkcs7-data
 1371:d=4  hl=2 l=  29 cons: SEQUENCE
 1373:d=5  hl=2 l=   9 prim: OBJECT            :aes-256-cbc
 1384:d=5  hl=2 l=  16 prim: OCTET STRING      [HEX DUMP]:CEEB7B64723F5366A1629C91402BC488
 1402:d=4  hl=4 l=2224 prim: cont [ 0 ]

The first number is the byte offset in the DER file. The three variables are the structure nesting level d, the header length hl and the content length l. Adding the offset, the header length and the content length together gets you the offset of the following item, skipping all deeper nesting levels. On the right after the colon is the type or value of an item (if any).

A nicer output with indentation is provided by the dumpasn1 program, though it omits the header lengths and prints some cryptic ASN.1 number tuples. It should be run from the same directory as its configuration file. I learned about it from this answer, which also deals with S/MIME, but not with smart cards.

Going back to the listing, the SEQUENCE at offset 30 contains the symmetric message key encrypted with a recipient's public key and associated information. The sub-SEQUENCE at offset 37 contains the issuer and serial number of the certificate key used for encrypting the message key. In this context public keys come with a certificate because an authority certifies that the public key belongs to a certain person (unlike in PGP where there is no such authority). The certificate issuer is given by the strings labeled with "countryName", "stateOrProvinceName", "localityName", "organizationName", "organizationalUnitName" and "commonName" and wrapped into the SEQUENCE at offset 40. The certificate serial number is the INTEGER at offset 183, B7A69916....

After the certificate issuer and serial number, at offset 197 the encrypted message key follows. The label "rsaEncryption" indicates the encryption method, and the data at offset 212 are the ciphertext (i.e. the public-key encrypted message key, not the symmetrically encrypted message). Actually, the ciphertext to decrypt is the content of the OCTET STRING item not including the header, which starts at offset 216 and is 256 bytes long.

After the end of the SEQUENCE from offset 30 that contains all the above, there is another SEQUENCE of the same length. I have not included it in the example listing, but it has exactly the same structure. It contains the certificate issuer and serial number and the encrypted message key for a different recipient of the mail message (or for the sender, so they can read their own Fcc). A recipient has to search these blocks for a key to which they have the corresponding private key. If a smart card wasn't involved, the recipient could use openssl pkeyutl (formerly rsautl) to decrypt the message key with their private key. How to decrypt it with a smart card is described below.

After the encrypted message key blocks, at offset 1356 the message ciphertext and associated information follows. The label aes-256-cbc gives the encryption method used, and the octet string following it is the initial value that this encryption method needs. When you have the symmetric message key, you can decrypt the message with:

openssl enc -d -aes-256-cbc -K <symmetric key> -iv <initial value> -in <message ciphertext> -out <output file>

<symmetric key> and <initial value> are given as strings of hexadecimal digits; <message ciphertext> and <output file> are file names.

To process data from the DER file further, you will have to extract parts of it. This can be done with one of the following equivalent commands:

tail -c +<offset+1> < msg.der | head -c <length> > <filename>
dd if=msg.der bs=1 skip=<offset> count=<length> of=<filename>

It is worth noting that openssl asn1parse also works on PEM files (which are base64-encoded DER files). But the offsets in its output always refer to the DER file, and to extract the data you need the DER format in any case.

Check access to your smart card is working

The details in this section, and smart card stuff in general, may depend on your type of card and card reader. I will describe what worked for me.

First, the pcscd daemon has to be running to access a cryptographic smart card. Install pcsclite and start the daemon. It is not necessary for a pcscd process to run permanently. It is usually started on demand when a program tries to access a cryptographic token via the library libpcsclite.so.

Libraries distributed with commercial smart cards usually link to some version of libpcsclite.so. If you have such a library, you have to give its path on the following command lines. I will call it vendor.so.

One tool for accessing smart cards is pkcs11-tool from the opensc package. The most basic command to try is listing the available card slots:

pkcs11-tool -L

If this fails, your card reader is probably not supported. If a slot is found, but your smart card requires a vendor library, it will output "token not recognized" after the card slot. With the vendor library, information about the smart card is printed, and you can use -T instead of -L to omit output about unoccupied slots:

pkcs11-tool --module /path/to/vendor.so -T

Next we can list the keys on the smart card with one of the following commands:

pkcs11-tool --module /path/to/vendor.so -O
pkcs11-tool --module /path/to/vendor.so -O -l

The -l option logs in to the smart card after prompting for your password. How much information is displayed with or without login will depend on your card. My card allows listing only the public keys and certificates without a login, but with -l the private keys are also displayed.

If these commands work as expected, you should be good regarding smart card access.

As an alternative to pkcs11-tool, p11tool from gnutls or p11-kit from the eponymous package can be used to test card access. Both depend on p11-kit and require configuration of your vendor library in a file in ~/.config/pkcs11/modules/ or /etc/pkcs11/pkcs11.conf. The format is described in the manual page pkcs11.conf (5), but this single line should suffice:

module: /path/to/vendor.so

Then your vendor library and smart card should show up in the list of modules (and tokens):

p11-kit list-modules

With p11tool you can list the tokens with:

p11tool --list-tokens

Using the URL from the list of tokens, you can list the public and private keys with:

p11tool --list-all <URL>
p11tool --login --list-all-privkeys <URL>

Decrypting the message key with a smart card

As explained above, we have to find the key ID matching the certificate serial number. As there seems to be no standard tool that helps with that, I have written this small program. It doesn't do much, but it lists the serial numbers and IDs of all objects stored on the card. Its only mandatory argument is the path to the vendor library:

sclist /path/to/vendor.so
sclist -l /path/to/vendor.so

The -l option lets you log in to the card and may display a few more objects. Search for a certificate object with one the serial numbers with which the message key was encrypted. (grep -C 1 is a great help.) Note the ID of the matching certificate, as this is also the ID of the private key required to decrypt the message key.

To decrypt the symmetric message key, you first have to extract it to a file, say msgkey.cipher, as described above. Obviously you have to choose the encrypted message key from the same ASN.1 structure that contained your certificate serial number. Then you can decrypt it with your card using pkcs11-tool:

pkcs11-tool --module /path/to/vendor.so -l --decrypt --id <keyID> -i msgkey.cipher -o msgkey.clear

The <keyID> has to be given as a hexadecimal string without separators (which should be documented but isn't). The parsing function for such arguments is apparently meant to support colon separators but doesn't due to a bug.

With a hexdump (again without separators) of the binary message key from msgkey.clear you can then decrypt the message itself as described above.

Debugging smart card operations

Finding out more about your smart card vendor library

When using some tools (and also when coding your own) it can be useful to know which version of the PKCS11 standard your vendor library supports. As the different versions have a slightly different list of API functions, this can be determined by listing the symbols defining them, or counting them:

nm -D /path/to/vendor.so | grep " T C_"
nm -D /path/to/vendor.so | grep " T C_" | wc -l

nm lists symbols, i.e. labels, in binary executables and libraries. The -D options lists the symbols in a shared library. The letter T in the output indicates symbols in the text section of the object file, which executable code is located. And including "C_" in the grep pattern catches all PKCS11 functions, which all start with that prefix.

For comparison, download the header files pkcs11f.h from the include file directories of different versions of the PKCS11 standard (links below). Then you can get a short list of the API functions with:

grep CK_PKCS11_FUNCTION_INFO pkcs11f.h

PKCS11 V3.0 has 92 API functions, older versions have around 70 or so (I have encountered 68 and 72 functions, possibly accounted for by different 2.x versions). If you want to compare the lists one on one, piping into | sort | nl may help.

If you have several versions of your vendor library, it may be that they report the properties of the smart card slightly differently. I have encountered one library giving the model of the same card as a number, and another as a name. This may affect you if you use p11tool or another program using p11-kit and you have to specify the card.

Tracing smart card actions

The opensc package contains the library pkcs11-spy.so that acts as a man in the middle and detailed function call logger for another PKCS11 library. It is very useful if you have a program that can access your smart card with your vendor library but you cannot get it to work with other software. The application in question must allow choosing an arbitrary PKCS11 library for accessing the card.

You generate a log as follows: First it is advisable to run the script program to save the copious output to a log file. Then set the PKCS11SPY environment variable to the vendor library path. Then start the application that can access your card from the same terminal and tell it to use /usr/lib/pkcs11-spy.so (or the correct path on your system) as the card access library. As you use the application, plenty of output should be generated about the PKCS11 functions called and their arguments and return values. After closing the application and exiting from the shell, the log file will be closed by script.

Caution: If you log in to the card, the dump will contain your password among the parameters to the C_Login function! Sanitise the dump if you want to pass it on to ask for advice!

A less important caveat is that pkcs11-spy.so seems to work only with libraries that support the same PKCS11 standard version as your opensc package. When I tried to use it with a mismatching vendor library, smart card operations simply did not work, probably because pkcs11-spy.so could not pass through function calls that it (or the vendor lib) was not aware of. A simple way to find out which PKCS11 version your pkcs11-spy.so supports is to count the PKCS11 API function symbols as described above for the vendor library.

Links to standards documents

Except for the PKCS11 standard documents, I did not find them overly helpful and have hardly read them to debug my problem. But if you need them, here are direct links to the relevant standards I could find. Not all of them are on the open web any more, so some point to the Internet Archive Wayback Machine.


TOS / Impressum