Fedora's database of trusted CA certificates adopts the Mozilla CA Certificate Store as its core and additionally allows custom adjustments on the local host. It aims to furnish a consolidated repository of CA certificates available to any PKI-savvy application wishing to verify an unfamiliar certificate. Broadly, there are two paths to this database. Hip applications speaking Cryptoki consult the CA database through the PKCS#11 Kit Trust Module, which directly loads the CA database into its memory. GnuTLS and NSS, for example, load the CA database this way. Old-school applications instead consult this CA database by reading its certificate bundles from files. OpenSSL takes this latter route (unless you provide, configure, and load a PKCS#11 engine).
Package ca-certificates installs the Mozilla CA Store. Package p11-kit-trust together with p11-kit provides a Cryptoki interface to the CA database. The PKCS#11 Kit Trust Module combines the CA database and Cryptoki access to the database. It is often abbreviated as p11-kit-trust. Here, "module" is understood in terms of the Cryptoki/PKCS#11 framework.
The CA database comprises two directory trees, /etc/pki/ca-trust/source and /usr/share/pki/ca-trust-source. In the latter directory, file ca-bundle.trust.p11-kit holds the complete Mozilla CA store in a format that module p11-kit-trust accepts. This format accommodates trust settings, and the database thereby includes both anchor and blacklisted certificates; mostly anchor. Out-of-the box, ca-certificates does not augment the Mozilla CA Store, and hence ca-bundle.trust.p11-kit is the main attraction. A system administrator can incorporate additional certificates simply by placing their PEM files under one of these two directories and running companion command update-ca-trust to set things straight. Should the same certificate appear in both source directories but with different trust or policy settings, the dictates under /etc/pki/ca-trust/source prevail.
The man page update-ca-trust has the details on the organization of these directories and the addition of other certificates.
You can get the gist of the format and contents of ca-bundle.trust.p11-kit by browsing this ASCII file in your favorite text viewer.
Module p11-kit-trust loads all of the certificates it finds in source repositories /etc/pki/ca-trust/source and /usr/share/pki/ca-trust-source. You can use its companion utility trust to look inside. For example, to see the whole enchilada:
-> trust list pkcs11:id=%d2%87%b4%e3%df%37%27%93%55%f6%56%ea%81%e5%36%cc%8c%1e%3f%bd;type=cert type: certificate label: ACCVRAIZ1 trust: anchor category: authority ⋮
The current CA consists of 156 certificates, of which 154 are trusted and 2 are blacklisted:
-> trust list | grep 'pkcs11:id' | wc --lines 156 -> trust list | grep 'type:' | sort | uniq type: certificate -> trust list | grep 'category:' | sort | uniq category: authority -> trust list | grep 'trust:' | sort | uniq trust: anchor trust: blacklisted -> trust list | grep 'trust: anchor' | wc --lines 154 -> trust list | grep 'trust: blacklisted' | wc --lines 2
You can see all of the data for a given certificate:
-> trust dump --filter="pkcs11:id=%d2%87⋯%3f%bd;type=cert" | less
(The quotation marks enclosing the ID keep the shell happy in the presence of that semicolon.)
The repository paths appear to be hardwired into the p11-kit-trust module; for example:
-> strings /usr/lib64/pkcs11/p11-kit-trust.so | grep /pki/ /etc/pki/ca-trust/source:/usr/share/pki/ca-trust-source
Nor does the configuration file indicate options (see /usr/share/p11-kit/modules/p11-kit-trust.module).
Legacy applications know nothing of Crytpoki, and other applications may reject Cryptoki for reasons of their own. To accommodate these applications, package ca-certificates renders the Mozilla CA store as certificate bundles under directory tree /etc/pki/ca-trust/extracted:
-> ls /etc/pki/ca-trust/extracted/ java/ openssl/ pem/ README
These three sub-directories bundle certificates with different formats.
Directory pem bundles TLS certificates into file tls-ca-bundle.pem consisting of 138 "CERTIFICATE" blocks:
-> grep BEGIN pem/tls-ca-bundle.pem | uniq -----BEGIN CERTIFICATE----- -> grep BEGIN pem/tls-ca-bundle.pem | wc --lines 138
It also provides bundles email-ca-bundle.pem (113) and objsign-ca-bundle.pem (0) for S/MIME certificates and software certificates, respectively.
Directory openssl bundles trusted certificates into file ca-bundle.trust.crt
-> grep BEGIN openssl/ca-bundle.trust.crt | uniq -----BEGIN TRUSTED CERTIFICATE----- -> grep BEGIN openssl/ca-bundle.trust.crt | wc --lines 157
These format embeds the permitted uses, and so there are no additional bundles corresponding to those under directory pem. See subsequent note for details.
Directory java provides a Java keystore in file cacerts, which spans 138 certificates:
-> echo "" | keytool -list -keystore java/cacerts 2>/dev/null | head -4 | tail -1 Your keystore contains 138 entries
(The keytool command prompts for a password, and echo feeds it an empty string. Although keytool accepts this null password, it warns about the integrity of the information it subsequently reports. Redirection of stderr quells that warning. The head and tail pipes merely pluck the number of entries from the top of the lengthy output.)
Shell script update-ca-trust creates the certificate bundles under /etc/pki/ca-trust/extracted by retrieving all certificates from module pk11-kit-trust. For example, this command demonstrates how tls-ca-bundle.pem is created:
-> trust extract --filter=ca-anchors --purpose server-auth --format=pem-bundle --overwrite /tmp/a.pem -> grep BEGIN /tmp/a.pem | wc --lines 138
See the short, straightforward script for the complete command sequence; see the trust man page for details. (Actually, update-ca-trust calls p11-kit, but the latter's man page notes that extraction is now handled by trust.)
OpenSSL has its own notion of a "trusted" certificate, which PEM-block label "TRUSTED CERTIFICATE" reflects. From the x509 man page:
A trusted certificate is an ordinary certificate which has several additional pieces of information attached to it such as the permitted and prohibited uses of the certificate and an "alias".
A quick comparison of the first certificate in each of the bundles pem/tls-ca-bundle.pem and openssl/ca-bundle.trust.crt elucidates this description. First, note that each bundle starts with the same certificate:
-> openssl x509 -in pem/tls-ca-bundle.pem -noout -issuer -serial issuer=CN = ACCVRAIZ1, OU = PKIACCV, O = ACCV, C = ES serial=5EC3B7A6437FA4E0 -> openssl x509 -in openssl/ca-bundle.trust.crt -noout -issuer -serial issuer=CN = ACCVRAIZ1, OU = PKIACCV, O = ACCV, C = ES serial=5EC3B7A6437FA4E0
Next, extract the details as text reports for easy comparison:
-> openssl x509 -in pem/tls-ca-bundle.pem -noout -text > /tmp/acert.txt -> openssl x509 -in openssl/ca-bundle.trust.crt -noout -text > /tmp/acert-trusted.txt
Now the differences are easy to show:
-> wc --lines /tmp/acert.txt /tmp/acert-trusted.txt 107 /tmp/acert.txt 111 /tmp/acert-trusted.txt 218 total -> diff /tmp/acert.txt /tmp/acert-trusted.txt 107a108,111 > Trusted Uses: > E-mail Protection, TLS Web Server Authentication > No Rejected Uses. > Alias: ACCVRAIZ1
-> openssl x509 -in pem/tls-ca-bundle.pem -noout -alias <No Alias> -> openssl x509 -in openssl/ca-bundle.trust.crt -noout -alias ACCVRAIZ1
The x509 command has options to add and clear this extra information about a certificate; see section Trust Settings of its man page.
Package ca-certificates recognizes the reality that legacy applications using OpenSSL or Java may expect to find CA bundles in specific files, which it refers to as "classical" files. It addresses this reality by linking classical filenames to corresponding bundles under /etc/pki/ca-trust/extracted. Thereby, old-school and new-school players are on the same page.
For example, this OpenSSL build expects to find CA certificates under /etc/pki/tls:
-> openssl version -d OPENSSLDIR: "/etc/pki/tls"
Package ca-certificates populates this directory with the required links:
=> rpm --query --list ca-certificates | grep -P "/etc/pki/tls" /etc/pki/tls /etc/pki/tls/cert.pem /etc/pki/tls/certs /etc/pki/tls/certs/ca-bundle.crt /etc/pki/tls/certs/ca-bundle.trust.crt -> ls -l /etc/pki/tls/cert.pem ⋯ /etc/pki/tls/cert.pem -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem -> ls -l /etc/pki/tls/certs/*.crt ⋯ /etc/pki/tls/certs/ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem ⋯ /etc/pki/tls/certs/ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
And similarly for Java's keystore:
=> rpm --query --list ca-certificates | grep -P "/etc/pki/java" /etc/pki/java /etc/pki/java/cacerts -> ls -l /etc/pki/java/cacerts ⋯ /etc/pki/java/cacerts -> /etc/pki/ca-trust/extracted/java/cacerts
-> curl --silent https://curl.haxx.se/ca/cacert.pem > cacert-curl.pem -> grep BEGIN cacert-curl.pem | uniq -----BEGIN CERTIFICATE----- -> grep BEGIN cacert-curl.pem | wc --lines 138 -> grep 'as of' cacert-curl.pem ## Certificate data from Mozilla as of: Wed Jan 17 04:12:05 2018 GMT
To address legacy issues with GnuTLS and OpenSSL, Fedora's trust database retains a handful of CA certificates that have been removed from the Mozilla CA Store. These are listed on the project's home page. You can use command ca-legacy (as root) to enable or disable trust in these certificates.
-> ca-legacy check Legacy CAs are set to ENABLED in file /etc/pki/ca-trust/ca-legacy.conf (affects install/upgrade) Status of symbolic link /etc/pki/ca-trust/source/ca-bundle.legacy.crt: /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.enable.crt -> grep 'BEGIN TRUSTED CERTIFICATE' /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.enable.crt | wc --lines 14 -> ca-legacy disable -> ca-legacy check Legacy CAs are set to DISABLED in file /etc/pki/ca-trust/ca-legacy.conf (affects install/upgrade) Status of symbolic link /etc/pki/ca-trust/source/ca-bundle.legacy.crt: /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt -> grep 'BEGIN TRUSTED CERTIFICATE' /usr/share/pki/ca-trust-legacy/ca-bundle.legacy.disable.crt | wc --lines 5
The PKCS#11 Kit Trust Module, aka p11-kit-trust, represents the CA database of ca-certificates as a database consulted through the Cryptoki/PKCS#11 API. On initialization, this module loads the certificates it finds in directories /etc/pki/ca-trust/source and /usr/share/pki/ca-trust-source. In terms of Cryptoki, p11-kit-trust models the certificates of these directories as two tokens, the System Trust token for /etc/pki/ca-trust/source (in slot 0 or 0x18) and the Default Trust token for /usr/share/pki/ca-trust-source (in slot 1 or 0x19). The Trust Module is implemented as library p11-kit-trust.so, which PKI-savvy applications link. To query the CA database, such applications simply use the Cryptoki interface to the library.
If you're curious for a look-see, you can query the Trust Module with commands pkcs11-dump, p11tool, trust, and others; illustrated below.
Library p11-kit-trust.so provides the PKCS#11 Kit Trust Module:
-> pkcs11-dump info /usr/lib64/p11-kit-trust.so ⋮ copyright info Provider Information: cryptokiVersion: 2.40 manufacturerID: PKCS#11 Kit flags: 00000000 libraryVersion: 0.23 libraryDescription: PKCS#11 Kit Trust Module
(Package pkcs11-dump provides command pkcs11-dump.)
The PKCS#11 Kit Trust Module has two tokens, labeled "System Trust" and "Default Trust":
-> p11tool --list-tokens Token 0: URL: pkcs11:model=p11-kit-trust;manufacturer=PKCS%2311%20Kit;serial=1;token=System%20Trust Label: System Trust Type: Trust module Manufacturer: PKCS#11 Kit Model: p11-kit-trust Serial: 1 Module: p11-kit-trust.so Token 1: URL: pkcs11:model=p11-kit-trust;manufacturer=PKCS%2311%20Kit;serial=1;token=Default%20Trust Label: Default Trust Type: Trust module Manufacturer: PKCS#11 Kit Model: p11-kit-trust Serial: 1 Module: p11-kit-trust.so
(Package gnutls-utils provides command p11tool.)
Absent local adjustments to the ca-certificates database, the Default Trust has all the action:
-> p11tool --list-all-certs "pkcs11:token=Default%20Trust" | grep Object | tail -1 Object 163: -> p11tool --list-all-certs "pkcs11:token=System%20Trust" No matching objects found
Of the 164 certificates in the Default Store, 154 are trusted:
-> p11tool --list-all-trusted "pkcs11:token=Default%20Trust" | grep Object | tail -1 Object 153:(Object numbering begins at zero.)
The PKCS#11 Kit Trust Module has two slots, numbered 18 and 19. The two source directories of the ca-certificates database attach to these slots:
-> pkcs11-dump slotlist /usr/lib64/p11-kit-trust.so ⋮ copyright info 18 /etc/pki/ca-trust/source 19 /usr/share/pki/ca-trust-source
Slot 18 holds the token for the System Trust, and slot 19 holds the token for the Default Trust:
-> pkcs11-dump dump /usr/lib64/p11-kit-trust.so 18 - 2>&1 | grep label Please enter PIN: any key label: System Trust -> pkcs11-dump dump /usr/lib64/p11-kit-trust.so 19 - 2>&1 | grep label Please enter PIN: any key label: Default Trust
That is, the System Trust token represents /etc/pki/ca-trust/source, and the Default Trust token represents /usr/share/pki/ca-trust-source.
Package p11-kit-trust provides library p11-kit-trust.so (in /usr/lib64/pkcs11). It also provides companion utility trust, which extracts certificates from the database and exports them in several formats. The package's /usr/lib64/libnssckbi.so is a circuitous link back to p11-kit-trust.so. It facilitates substituting p11-kit-trust.so as a drop-in replacement for NSS's library of the same name; see below.
Although Network Security Services (NSS) includes a built-in trust module for the Mozilla CA Store, called CKBI, Fedora replaces this module with the PKCS#11 Kit Trust Module. This switch transparently insures that NSS-based applications see any local adjustments to the core Mozilla CA Store that are present in the system's CA database. This switch does not affect a user's own NSS database of certificates.
Package nss includes Cryptoki trust module CKBI, in libnssckbi.so:
-> pkcs11-tool --module /usr/lib64/nss/libnssckbi.so --show-info --list-slots Cryptoki version 2.20 Manufacturer Mozilla Foundation Library NSS Builtin Object Cryptoki Modu (ver 2.18) Available slots: Slot 0 (0x1): NSS Builtin Objects token state: uninitialized Using slot 0 with a present token (0x1)
It has 173 certificates:
-> pkcs11-tool --module /usr/lib64/nss/libnssckbi.so --list-objects --type cert Using slot 0 with a present token (0x1) Certificate Object; type = X.509 cert label: GlobalSign Root CA ID: 3000 ⋮ Certificate Object; type = X.509 cert label: SSL.com EV Root Certification Authority ECC ID: 3000 -> pkcs11-tool --module /usr/lib64/nss/libnssckbi.so --list-objects --type cert | grep 'label:' | wc --lines Using slot 0 with a present token (0x1) 173
Fedora replaces library libnssckbi from NSS with p11-kit-trust.so, above, via alternatives:
-> file /usr/lib64/libnssckbi.so /usr/lib64/libnssckbi.so: symbolic link to /etc/alternatives/libnssckbi.so.x86_64 -> alternatives --display libnssckbi.so.x86_64 libnssckbi.so.x86_64 - status is auto. link currently points to /usr/lib64/pkcs11/p11-kit-trust.so /usr/lib64/pkcs11/p11-kit-trust.so - priority 30 /usr/lib64/nss/libnssckbi.so - priority 10 Current `best' version is /usr/lib64/pkcs11/p11-kit-trust.so. -> file /usr/lib64/pkcs11/p11-kit-trust.so /usr/lib64/pkcs11/p11-kit-trust.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=7249986c6cd198fb6f597b1c34dbbe81e9a0cba9, stripped
This legerdemain allows Firefox, Thunderbird, and other NSS supplicants to transparently access the system-wide CA database in addition to any parochial NSS-based stores in a user's custom profiles.
An NSS-based application can let users augment the system CA database with their own set of certificates. These are stored in a file named either cert8.db or cert9.db somewhere under the user's home directory. With Firefox, for example:
-> cd ~/.mozilla/firefox/*.default* -> file cert?.db cert8.db: Berkeley DB 1.85 (Hash, version 2, native byte-order) cert9.db: SQLite 3.x database, last written using SQLite version 3020001
The files differ by the database engine, either Berkeley DB or SQLite. Current versions of NSS favor SQLite because this engine allows a user to concurrently share a single file among multiple applications, perhaps using common ~/.pki/nssdb/cert9.db for both Firefox and Thunderbird. Both files coexist peacefully but independently; an application picks one. You can communicate your choice of engine by setting environment variable NSS_DEFAULT_DB_TYPE to "sql" or "dbm". In the absence of this guidance, NSS uses the legacy DB file.
You can use certutil (package nss-tools) to have a look at your application's CAs:
-> cd ~/.mozilla/firefox/*.default* -> certutil -d . -L | tail --lines +5 | sort AlphaSSL CA - SHA256 - G2 ,, ⋮ Verizon Akamai SureServer CA G14-SHA2 ,, -> certutil -d . -L | tail --lines +5 | wc --lines 66
Option -d tells certutil where to look for cert8.db or cert9.db, and option -L tells certutil to list the certificates in the file. (The tail pipe trims the initial five header lines, which are not informative here.) You can then get the skinny on a specific certificate, given its name above; for example:
-> certutil -d . -L -n "Let's Encrypt Authority X3" Certificate: Data: Version: 3 (0x2) Serial Number: 0a:01:41:42:00:00:01:53:85:73:6a:0b:85:ec:a7:08 ⋮ lot's more omitted
When managing certificates, certutil checks environment variable NSS_DEFAULT_DB_TYPE for the database type, given as "sql" or "dbm". In the absence of the latter guidance, it goes with DB. It balks when it tries the wrong format:
-> NSS_DEFAULT_DB_TYPE=dbm certutil -L -d $HOME/.pki/nssdb certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.
You can easily lend a hand by prepending "sql:" or "dbm:" to the directory name:
-> certutil -d sql:$HOME/.pki/nssdb -L ⋮
You can also use certutil to add, modify, or delete certificates; the man page has details. Applications like Firefox and Thunderbird provide GUI management of your custom NSS certificates.