Certificate Authority (CA) Database

Fedora's database of trusted CA certificates adopts the Mozilla CA Certificate Store as its core and additionally allows custom adjustments on the local host. It aims to furnish a consolidated repository of CA certificates available to any PKI-savvy application wishing to verify an unfamiliar certificate. Broadly, there are two paths to this database. Hip applications speaking Cryptoki consult the CA database through the PKCS#11 Kit Trust Module, which directly loads the CA database into its memory. GnuTLS and NSS, for example, load the CA database this way. Old-school applications instead consult this CA database by reading its certificate bundles from files. OpenSSL takes this latter route (unless you provide, configure, and load a PKCS#11 engine).

Package ca-certificates installs the Mozilla CA Store. Package p11-kit-trust together with p11-kit provides a Cryptoki interface to the CA database. The PKCS#11 Kit Trust Module combines the CA database and Cryptoki access to the database. It is often abbreviated as p11-kit-trust. Here, "module" is understood in terms of the Cryptoki/PKCS#11 framework.


The CA database comprises two directory trees, /etc/pki/ca-trust/source and /usr/share/pki/ca-trust-source. In the latter directory, file ca-bundle.trust.p11-kit holds the complete Mozilla CA store in a format that module p11-kit-trust accepts. This format accommodates trust settings, and the database thereby includes both anchor and blacklisted certificates; mostly anchor. Out-of-the box, ca-certificates does not augment the Mozilla CA Store, and hence ca-bundle.trust.p11-kit is the main attraction. A system administrator can incorporate additional certificates simply by placing their PEM files under one of these two directories and running companion command update-ca-trust to set things straight. Should the same certificate appear in both source directories but with different trust or policy settings, the dictates under /etc/pki/ca-trust/source prevail.

The man page update-ca-trust has the details on the organization of these directories and the addition of other certificates.

You can get the gist of the format and contents of ca-bundle.trust.p11-kit by browsing this ASCII file in your favorite text viewer.

Module p11-kit-trust loads all of the certificates it finds in source repositories /etc/pki/ca-trust/source and /usr/share/pki/ca-trust-source. You can use its companion utility trust to look inside. For example, to see the whole enchilada:

-> trust list
    type: certificate
    label: ACCVRAIZ1
    trust: anchor
    category: authority

The current CA consists of 156 certificates, of which 154 are trusted and 2 are blacklisted:

-> trust list | grep 'pkcs11:id' | wc --lines
-> trust list | grep 'type:' | sort | uniq
    type: certificate
-> trust list | grep 'category:' | sort | uniq
    category: authority
-> trust list | grep 'trust:' | sort | uniq
    trust: anchor
    trust: blacklisted
-> trust list | grep 'trust: anchor' | wc --lines
-> trust list | grep 'trust: blacklisted' | wc --lines

You can see all of the data for a given certificate:

-> trust dump --filter="pkcs11:id=%d2%87⋯%3f%bd;type=cert" | less

(The quotation marks enclosing the ID keep the shell happy in the presence of that semicolon.)

The repository paths appear to be hardwired into the p11-kit-trust module; for example:

-> strings /usr/lib64/pkcs11/p11-kit-trust.so | grep /pki/

Nor does the configuration file indicate options (see /usr/share/p11-kit/modules/p11-kit-trust.module).

Legacy applications know nothing of Crytpoki, and other applications may reject Cryptoki for reasons of their own. To accommodate these applications, package ca-certificates renders the Mozilla CA store as certificate bundles under directory tree /etc/pki/ca-trust/extracted:

 -> ls /etc/pki/ca-trust/extracted/
java/  openssl/  pem/  README

These three sub-directories bundle certificates with different formats.

Directory pem bundles TLS certificates into file tls-ca-bundle.pem consisting of 138 "CERTIFICATE" blocks:

-> grep BEGIN pem/tls-ca-bundle.pem | uniq
-> grep BEGIN pem/tls-ca-bundle.pem | wc --lines

It also provides bundles email-ca-bundle.pem (113) and objsign-ca-bundle.pem (0) for S/MIME certificates and software certificates, respectively.

Directory openssl bundles trusted certificates into file ca-bundle.trust.crt

-> grep BEGIN openssl/ca-bundle.trust.crt | uniq
-> grep BEGIN openssl/ca-bundle.trust.crt | wc --lines

These format embeds the permitted uses, and so there are no additional bundles corresponding to those under directory pem. See subsequent note for details.

Directory java provides a Java keystore in file cacerts, which spans 138 certificates:

-> echo "" | keytool -list -keystore java/cacerts 2>/dev/null | head -4 | tail -1
Your keystore contains 138 entries

(The keytool command prompts for a password, and echo feeds it an empty string. Although keytool accepts this null password, it warns about the integrity of the information it subsequently reports. Redirection of stderr quells that warning. The head and tail pipes merely pluck the number of entries from the top of the lengthy output.)

Shell script update-ca-trust creates the certificate bundles under /etc/pki/ca-trust/extracted by retrieving all certificates from module pk11-kit-trust. For example, this command demonstrates how tls-ca-bundle.pem is created:

-> trust extract --filter=ca-anchors --purpose server-auth --format=pem-bundle --overwrite /tmp/a.pem
-> grep BEGIN /tmp/a.pem  | wc --lines

See the short, straightforward script for the complete command sequence; see the trust man page for details. (Actually, update-ca-trust calls p11-kit, but the latter's man page notes that extraction is now handled by trust.)

OpenSSL has its own notion of a "trusted" certificate, which PEM-block label "TRUSTED CERTIFICATE" reflects. From the x509 man page:

A trusted certificate is an ordinary certificate which has several additional pieces of information attached to it such as the permitted and prohibited uses of the certificate and an "alias".

A quick comparison of the first certificate in each of the bundles pem/tls-ca-bundle.pem and openssl/ca-bundle.trust.crt elucidates this description. First, note that each bundle starts with the same certificate:

-> openssl x509 -in pem/tls-ca-bundle.pem -noout -issuer -serial
-> openssl x509 -in openssl/ca-bundle.trust.crt -noout -issuer -serial

Next, extract the details as text reports for easy comparison:

-> openssl x509 -in pem/tls-ca-bundle.pem       -noout -text > /tmp/acert.txt
-> openssl x509 -in openssl/ca-bundle.trust.crt -noout -text > /tmp/acert-trusted.txt

Now the differences are easy to show:

-> wc --lines /tmp/acert.txt /tmp/acert-trusted.txt
  107 /tmp/acert.txt
  111 /tmp/acert-trusted.txt
  218 total
-> diff /tmp/acert.txt /tmp/acert-trusted.txt
> Trusted Uses:
>   E-mail Protection, TLS Web Server Authentication
> No Rejected Uses.
> Alias: ACCVRAIZ1


-> openssl x509 -in pem/tls-ca-bundle.pem -noout -alias
<No Alias>
-> openssl x509 -in openssl/ca-bundle.trust.crt -noout -alias

The x509 command has options to add and clear this extra information about a certificate; see section Trust Settings of its man page.

Package ca-certificates recognizes the reality that legacy applications using OpenSSL or Java may expect to find CA bundles in specific files, which it refers to as "classical" files. It addresses this reality by linking classical filenames to corresponding bundles under /etc/pki/ca-trust/extracted. Thereby, old-school and new-school players are on the same page.

For example, this OpenSSL build expects to find CA certificates under /etc/pki/tls:

-> openssl version -d
OPENSSLDIR: "/etc/pki/tls"

Package ca-certificates populates this directory with the required links:

=> rpm --query --list ca-certificates | grep -P "/etc/pki/tls"
-> ls -l /etc/pki/tls/cert.pem 
⋯ /etc/pki/tls/cert.pem -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
-> ls -l /etc/pki/tls/certs/*.crt
⋯ /etc/pki/tls/certs/ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
⋯ /etc/pki/tls/certs/ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt

(See: What certificate authorities does OpenSSL recognize?)

And similarly for Java's keystore:

=> rpm --query --list ca-certificates | grep -P "/etc/pki/java"
-> ls -l /etc/pki/java/cacerts
⋯ /etc/pki/java/cacerts -> /etc/pki/ca-trust/extracted/java/cacerts

The curl project provides a bundle of CA certificates (cacert.pem) extracted from Mozilla's CA store. You can download this file and use it with certtool and openssl for tinkering.

-> curl --silent https://curl.haxx.se/ca/cacert.pem > cacert-curl.pem
-> grep BEGIN cacert-curl.pem | uniq
-> grep BEGIN cacert-curl.pem | wc --lines
-> grep 'as of' cacert-curl.pem 
## Certificate data from Mozilla as of: Wed Jan 17 04:12:05 2018 GMT

To address legacy issues with GnuTLS and OpenSSL, Fedora's trust database retains a handful of CA certificates that have been removed from the Mozilla CA Store. These are listed on the project's home page. You can use command ca-legacy (as root) to enable or disable trust in these certificates.

-> ca-legacy check
Legacy CAs are set to ENABLED in file /etc/pki/ca-trust/ca-legacy.conf (affects install/upgrade)
Status of symbolic link /etc/pki/ca-trust/source/ca-bundle.legacy.crt:
-> grep 'BEGIN TRUSTED CERTIFICATE' /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.enable.crt | wc --lines
-> ca-legacy disable
-> ca-legacy check
Legacy CAs are set to DISABLED in file /etc/pki/ca-trust/ca-legacy.conf (affects install/upgrade)
Status of symbolic link /etc/pki/ca-trust/source/ca-bundle.legacy.crt:
-> grep 'BEGIN TRUSTED CERTIFICATE' /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt | wc --lines 

PKCS#11 Kit Trust Module

The PKCS#11 Kit Trust Module, aka p11-kit-trust, represents the CA database of ca-certificates as a database consulted through the Cryptoki/PKCS#11 API. On initialization, this module loads the certificates it finds in directories /etc/pki/ca-trust/source and /usr/share/pki/ca-trust-source. In terms of Cryptoki, p11-kit-trust models the certificates of these directories as two tokens, the System Trust token for /etc/pki/ca-trust/source (in slot 0 or 0x18) and the Default Trust token for /usr/share/pki/ca-trust-source (in slot 1 or 0x19). The Trust Module is implemented as library p11-kit-trust.so, which PKI-savvy applications link. To query the CA database, such applications simply use the Cryptoki interface to the library.

If you're curious for a look-see, you can query the Trust Module with commands pkcs11-dump, p11tool, trust, and others; illustrated below.

Library p11-kit-trust.so provides the PKCS#11 Kit Trust Module:

-> pkcs11-dump info /usr/lib64/p11-kit-trust.so 
⋮ copyright info

Provider Information:
               cryptokiVersion: 2.40
                manufacturerID: PKCS#11 Kit                     
                         flags: 00000000
                libraryVersion: 0.23
            libraryDescription: PKCS#11 Kit Trust Module  

(Package pkcs11-dump provides command pkcs11-dump.)

The PKCS#11 Kit Trust Module has two tokens, labeled "System Trust" and "Default Trust":

-> p11tool --list-tokens
Token 0:
	URL: pkcs11:model=p11-kit-trust;manufacturer=PKCS%2311%20Kit;serial=1;token=System%20Trust
	Label: System Trust
	Type: Trust module
	Manufacturer: PKCS#11 Kit
	Model: p11-kit-trust
	Serial: 1
	Module: p11-kit-trust.so
Token 1:
	URL: pkcs11:model=p11-kit-trust;manufacturer=PKCS%2311%20Kit;serial=1;token=Default%20Trust
	Label: Default Trust
	Type: Trust module
	Manufacturer: PKCS#11 Kit
	Model: p11-kit-trust
	Serial: 1
	Module: p11-kit-trust.so

(Package gnutls-utils provides command p11tool.)

Absent local adjustments to the ca-certificates database, the Default Trust has all the action:

-> p11tool --list-all-certs "pkcs11:token=Default%20Trust" | grep Object | tail -1
Object 163:
-> p11tool --list-all-certs "pkcs11:token=System%20Trust"
No matching objects found

Of the 164 certificates in the Default Store, 154 are trusted:

-> p11tool --list-all-trusted "pkcs11:token=Default%20Trust" | grep Object | tail -1
Object 153:
(Object numbering begins at zero.)

The PKCS#11 Kit Trust Module has two slots, numbered 18 and 19. The two source directories of the ca-certificates database attach to these slots:

-> pkcs11-dump slotlist /usr/lib64/p11-kit-trust.so 
⋮ copyright info

18	/etc/pki/ca-trust/source                                        
19	/usr/share/pki/ca-trust-source

Slot 18 holds the token for the System Trust, and slot 19 holds the token for the Default Trust:

-> pkcs11-dump dump /usr/lib64/p11-kit-trust.so 18 - 2>&1 | grep label
Please enter PIN: any key
                         label: System Trust
-> pkcs11-dump dump /usr/lib64/p11-kit-trust.so 19 - 2>&1 | grep label
Please enter PIN: any key
                         label: Default Trust

That is, the System Trust token represents /etc/pki/ca-trust/source, and the Default Trust token represents /usr/share/pki/ca-trust-source.

Package p11-kit-trust provides library p11-kit-trust.so (in /usr/lib64/pkcs11). It also provides companion utility trust, which extracts certificates from the database and exports them in several formats. The package's /usr/lib64/libnssckbi.so is a circuitous link back to p11-kit-trust.so. It facilitates substituting p11-kit-trust.so as a drop-in replacement for NSS's library of the same name; see below.

Network Security Services (NSS)

Although Network Security Services (NSS) includes a built-in trust module for the Mozilla CA Store, called CKBI, Fedora replaces this module with the PKCS#11 Kit Trust Module. This switch transparently insures that NSS-based applications see any local adjustments to the core Mozilla CA Store that are present in the system's CA database. This switch does not affect a user's own NSS database of certificates.

Package nss includes Cryptoki trust module CKBI, in libnssckbi.so:

-> pkcs11-tool --module /usr/lib64/nss/libnssckbi.so --show-info --list-slots
Cryptoki version 2.20
Manufacturer     Mozilla Foundation
Library          NSS Builtin Object Cryptoki Modu (ver 2.18)
Available slots:
Slot 0 (0x1): NSS Builtin Objects
  token state:   uninitialized
Using slot 0 with a present token (0x1)

It has 173 certificates:

-> pkcs11-tool --module /usr/lib64/nss/libnssckbi.so --list-objects --type cert
Using slot 0 with a present token (0x1)
Certificate Object; type = X.509 cert
  label:      GlobalSign Root CA
  ID:         3000
Certificate Object; type = X.509 cert
  label:      SSL.com EV Root Certification Authority ECC
  ID:         3000

-> pkcs11-tool --module /usr/lib64/nss/libnssckbi.so --list-objects --type cert | grep 'label:' | wc --lines
Using slot 0 with a present token (0x1)

Fedora replaces library libnssckbi from NSS with p11-kit-trust.so, above, via alternatives:

-> file /usr/lib64/libnssckbi.so 
/usr/lib64/libnssckbi.so: symbolic link to /etc/alternatives/libnssckbi.so.x86_64
-> alternatives --display libnssckbi.so.x86_64
libnssckbi.so.x86_64 - status is auto.
 link currently points to /usr/lib64/pkcs11/p11-kit-trust.so
/usr/lib64/pkcs11/p11-kit-trust.so - priority 30
/usr/lib64/nss/libnssckbi.so - priority 10
Current `best' version is /usr/lib64/pkcs11/p11-kit-trust.so.
-> file /usr/lib64/pkcs11/p11-kit-trust.so
/usr/lib64/pkcs11/p11-kit-trust.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV),
dynamically linked, BuildID[sha1]=7249986c6cd198fb6f597b1c34dbbe81e9a0cba9, stripped

This legerdemain allows Firefox, Thunderbird, and other NSS supplicants to transparently access the system-wide CA database in addition to any parochial NSS-based stores in a user's custom profiles.

An NSS-based application can let users augment the system CA database with their own set of certificates. These are stored in a file named either cert8.db or cert9.db somewhere under the user's home directory. With Firefox, for example:

-> cd ~/.mozilla/firefox/*.default*
-> file cert?.db
cert8.db: Berkeley DB 1.85 (Hash, version 2, native byte-order)
cert9.db: SQLite 3.x database, last written using SQLite version 3020001

The files differ by the database engine, either Berkeley DB or SQLite. Current versions of NSS favor SQLite because this engine allows a user to concurrently share a single file among multiple applications, perhaps using common ~/.pki/nssdb/cert9.db for both Firefox and Thunderbird. Both files coexist peacefully but independently; an application picks one. You can communicate your choice of engine by setting environment variable NSS_DEFAULT_DB_TYPE to "sql" or "dbm". In the absence of this guidance, NSS uses the legacy DB file.

You can use certutil (package nss-tools) to have a look at your application's CAs:

-> cd ~/.mozilla/firefox/*.default*
-> certutil -d . -L | tail --lines +5 | sort
AlphaSSL CA - SHA256 - G2                                    ,,   
Verizon Akamai SureServer CA G14-SHA2                        ,,   
-> certutil -d . -L | tail --lines +5 | wc --lines

Option -d tells certutil where to look for cert8.db or cert9.db, and option -L tells certutil to list the certificates in the file. (The tail pipe trims the initial five header lines, which are not informative here.) You can then get the skinny on a specific certificate, given its name above; for example:

-> certutil -d . -L -n "Let's Encrypt Authority X3"
        Version: 3 (0x2)
        Serial Number:
⋮ lot's more omitted

When managing certificates, certutil checks environment variable NSS_DEFAULT_DB_TYPE for the database type, given as "sql" or "dbm". In the absence of the latter guidance, it goes with DB. It balks when it tries the wrong format:

-> NSS_DEFAULT_DB_TYPE=dbm certutil -L -d $HOME/.pki/nssdb
certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.

You can easily lend a hand by prepending "sql:" or "dbm:" to the directory name:

-> certutil -d sql:$HOME/.pki/nssdb -L

You can also use certutil to add, modify, or delete certificates; the man page has details. Applications like Firefox and Thunderbird provide GUI management of your custom NSS certificates.