bug 1570222 - avoid passing unrelated certificates to mozilla::pkix from NSSCertDBTrustDomain r=kjacobs

During path building, mozilla::pkix filters out candidate certificates provided
by trust domains where the subject distinguished name does not match the issuer
distinguished name of the certificate it's trying to find an issuer for.
However, if there's a problem decoding the candidate issuer certificate,
mozilla::pkix will make a note of this error, regardless of if that certificate
was potentially a suitable issuer. If no trusted path is found, the error from
that unrelated certificate may ultimately be returned by mozilla::pkix,
resulting in confusion.

Before this patch, NSSCertDBTrustDomain could cause this behavior by blithely
passing every known 3rd party certificate to mozilla::pkix (other sources of
certificates already filter on subject distinguished name). This patch adds
filtering to 3rd party certificates as well.

Differential Revision: https://phabricator.services.mozilla.com/D48120

--HG--
extra : moz-landing-system : lando

security/certverifier/NSSCertDBTrustDomain.cpp

@@ -272,6 +272,17 @@ Result NSSCertDBTrustDomain::FindIssuer(Input encodedIssuerName,
   }
 
   for (const auto& thirdPartyRootInput : mThirdPartyRootInputs) {
+    BackCert root(thirdPartyRootInput, EndEntityOrCA::MustBeCA, nullptr);
+    Result rv = root.Init();
+    if (rv != Success) {
+      continue;
+    }
+    // Filter out 3rd party roots that can't be issuers we're looking for
+    // because the subject distinguished name doesn't match. This prevents
+    // mozilla::pkix from accumulating spurious errors during path building.
+    if (!InputsAreEqual(encodedIssuerName, root.GetSubject())) {
+      continue;
+    }
     if (!geckoRootCandidates.append(thirdPartyRootInput)) {
       return Result::FATAL_ERROR_NO_MEMORY;
     }
@@ -279,6 +290,18 @@ Result NSSCertDBTrustDomain::FindIssuer(Input encodedIssuerName,
 
   for (const auto& thirdPartyIntermediateInput :
        mThirdPartyIntermediateInputs) {
+    BackCert intermediate(thirdPartyIntermediateInput, EndEntityOrCA::MustBeCA,
+                          nullptr);
+    Result rv = intermediate.Init();
+    if (rv != Success) {
+      continue;
+    }
+    // Filter out 3rd party intermediates that can't be issuers we're looking
+    // for because the subject distinguished name doesn't match. This prevents
+    // mozilla::pkix from accumulating spurious errors during path building.
+    if (!InputsAreEqual(encodedIssuerName, intermediate.GetSubject())) {
+      continue;
+    }
     if (!geckoIntermediateCandidates.append(thirdPartyIntermediateInput)) {
       return Result::FATAL_ERROR_NO_MEMORY;
     }