Bug 1612116 - turn NSS not setting an error code into SEC_ERROR_LIBRARY_FAILURE r=bbeurdouche

Sometimes SSL_ForceHandshake will return SECFailure without setting an error
code. When this happens, calling GetXPCOMFromNSSError on that not-an-error-code
will fail. This patch first checks for this situation and substitutes
SEC_ERROR_LIBRARY_FAILURE if applicable.

Differential Revision: https://phabricator.services.mozilla.com/D114908

security/manager/ssl/nsNSSIOLayer.cpp

@@ -321,7 +321,13 @@ nsNSSSocketInfo::DriveHandshake() {
 
   if (rv != SECSuccess) {
     PRErrorCode errorCode = PR_GetError();
-    MOZ_DIAGNOSTIC_ASSERT(errorCode, "handshake failed without error code");
+    MOZ_ASSERT(errorCode, "handshake failed without error code");
+    // There is a bug in NSS. Sometimes SSL_ForceHandshake will return
+    // SECFailure without setting an error code. In these cases, cancel
+    // the connection with SEC_ERROR_LIBRARY_FAILURE.
+    if (!errorCode) {
+      errorCode = SEC_ERROR_LIBRARY_FAILURE;
+    }
     if (errorCode == PR_WOULD_BLOCK_ERROR) {
       return NS_BASE_STREAM_WOULD_BLOCK;
     }