I think they're remnants from the past that we don't really need anymore.
And they're making things more complicated for some pending work of mine.
Differential Revision: https://phabricator.services.mozilla.com/D89687
The files are copied verbatim from upstream autoconf 2.13 (but only the
files we need) and old.configure is adapted to use the vendored version.
Differential Revision: https://phabricator.services.mozilla.com/D89554
When configuring without system NSPR the configuration variable
PKGCONF_REQUIRES_PRIVATE isn't ever set, leading to a .pc file that still
contains the @PKGCONF_REQUIRES_PRIVATE@ stub.
Ensure that we always define PKGCONF_REQUIRES_PRIVATE, by setting it to an
empty string in case no system-nsrp is enabled.
In this way, the pkg-config file stub will be always replaced.
Differential Revision: https://phabricator.services.mozilla.com/D88179
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
We don't anticipate end users will actually care to do this, but it's useful especially for unit tests. For example, after bug 1659539, Python `configure` tests will run in a new, non-`init_py3` `virtualenv`, and we'll want to target that `virtualenv` for `configure` rather than having it create a new `virtualenv` for no reason.
Differential Revision: https://phabricator.services.mozilla.com/D88661
2020-08-21 Kevin Jacobs <kjacobs@mozilla.com>
* automation/abi-check/previous-nss-release, lib/nss/nss.h,
lib/softoken/softkver.h, lib/util/nssutil.h:
Set version numbers to 3.57 Beta
[783f49ae6126]
2020-08-24 Kevin Jacobs <kjacobs@mozilla.com>
* gtests/ssl_gtest/ssl_auth_unittest.cc, lib/ssl/dtls13con.c,
lib/ssl/dtlscon.c, lib/ssl/ssl3con.c, lib/ssl/sslimpl.h,
lib/ssl/sslnonce.c:
Bug 1653641 - Cleanup inaccurate DTLS comments, code review fixes.
r=mt
[0e1b5c711cb9]
2020-08-24 Robert Relyea <rrelyea@redhat.com>
* lib/freebl/fipsfreebl.c, lib/softoken/fipstest.c,
lib/softoken/kbkdf.c, lib/softoken/lowpbe.c, lib/softoken/lowpbe.h,
lib/softoken/pkcs11c.c, lib/softoken/pkcs11i.h,
lib/softoken/sftkhmac.c, lib/softoken/sftkike.c:
Bug 1660304 New FIPS IG requires self-tests for approved kdfs.
r=ueno comments=kjacobs
FIPS guidance now requires self-tests for our kdfs. It also requires
self-tests for cmac which we didn't have in the cmac patch.
Currently only one test per kdf is necessary. Specifially for
SP-800-108, only one of the three flavors are needed (counter,
feedback, or pipeline). This patch includes more complete testing
but it has been turned off the currently extraneous tests under the
assumption that NIST guidance may require them in the future. HKDF
is currently not included in FIPS, but is on track to be included,
so hkdf have been included in this patch.
Because the test vectors are const strings, the patch pushes some
const definitions that were missing in existing private interfaces.
There are three flavors of self-tests: Function implemented in
freebl are added to the freebl/fipsfreebl.c Functions implemented in
pkcs11c.c have selftests completely implemented in
softoken/fipstest.c Functions implemented in their own .c file have
their selftest function implemented in that .c file and called by
fipstests.c These are consistant with the previous choices for
selftests.
Some private interfaces that took in keys from pkcs #11 structures
or outputted keys to pkcs #11 structures were modified to optionally
take keys in by bytes and output keys as bytes so the self-tests can
work in just bytes.
[5dca54fe61c2]
2020-08-25 Daiki Ueno <dueno@redhat.com>
* lib/softoken/manifest.mn:
Bug 1659252, disable building libnssdbm3.so if NSS_DISABLE_DBM=1,
r=rrelyea
Reviewers: rrelyea
Reviewed By: rrelyea
Bug #: 1659252
[4d55d36ca6ef]
2020-08-24 Kevin Jacobs <kjacobs@mozilla.com>
* lib/pk11wrap/pk11cxt.c, lib/softoken/pkcs11c.c, lib/softoken/sdb.c,
lib/softoken/sftkpwd.c:
Bug 1651834 - Fix various static analyzer warnings. r=rrelyea
[ab04fd73fd6d]
2020-08-28 Mike Hommey <mh@glandium.org>
* lib/freebl/blapii.h:
Bug 1661810 - Define pre_align/post_align based on the compiler.
r=jcj
Things worked fine before we upgraded to clang 11 presumably because
the stack was always 16-bytes aligned in the first place, or
something akin to that, and the lack of pre_align/post_align doing
anything didn't matter. The runtime misalignment of the stack may
well be a clang > 9 bug, but keeping pre_align/post_align tied to
the x86/x64 is a footgun anyways.
[c100e11991f6] [tip]
Differential Revision: https://phabricator.services.mozilla.com/D88876
config.guess infers information about the compiler using environment
variables, such as CC. However, we use such environment variables to
configure the tooling for the target.
Differential Revision: https://phabricator.services.mozilla.com/D88085
Previously, we would optimistically attempt to use a Rust toolchain that
matches the current C toolchain, and would throw an error if an
attempted compile with that Rust toolchain failed.
Instead, if we fail to detect a usable Rust toolchain, we now helpfully
inform users of their two options: change C toolchain, or install
matching Rust toolchain.
Differential Revision: https://phabricator.services.mozilla.com/D88084
config.guess infers information about the compiler using environment
variables, such as CC. However, we use such environment variables to
configure the tooling for the target.
Differential Revision: https://phabricator.services.mozilla.com/D88085
Previously, we would optimistically attempt to use a Rust toolchain that
matches the current C toolchain, and would throw an error if an
attempted compile with that Rust toolchain failed.
Instead, if we fail to detect a usable Rust toolchain, we now helpfully
inform users of their two options: change C toolchain, or install
matching Rust toolchain.
Differential Revision: https://phabricator.services.mozilla.com/D88084
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
We have a minimum rust version required for compilations. For both stable and beta rust compilers, we can trust that they will have all the stabilized features we're expecting.
However, for nightlies, they may "match" our minimum version, but may have been released in the version window before a certain feature we need has been stabilized.
So, when validating rustc version in configure, ensure that the nightly is at least one version newer than our expected version.
Differential Revision: https://phabricator.services.mozilla.com/D86889
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:
1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.
2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.
As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.
2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.
3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).
4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).
5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.
Differential Revision: https://phabricator.services.mozilla.com/D85916
This makes us use one less tool from MSVC, and removes one more use of wine
in cross builds.
We replace the call to either rc/llvm-rc or windres with a wrapper script.
While the script is not strictly needed for the latter, we use a wrapper
in that case anyway because it's one step towards fixing bug 1498414.
For llvm-rc, however, we need a wrapper because llvm-rc doesn't preprocess
on its own, so the wrapper does that too.
The wrapper script also allows to deal with the default flags passed to
llvm-rc or windres, rather than inherit them from old-configure.
We also need to explicitly pass the codepage to llvm-rc, which was not
necessary with rc (presumably, llvm-rc has a different default).
While here, remove the unused WINDRES subst from js/src/old-configure.in.
Also, while here, we remove --use-temp-file, because as described in the
linked bug and in the windres manual page, it was used to work around bugs
on Windows 98 and earlier.
Differential Revision: https://phabricator.services.mozilla.com/D86312
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
We are currently relying on two different ways to insert side-by-side
manifests in binaries on Windows: through resource files, or through
the use of MT. The latter is not supported on mingw builds, which is
not great.
Link.exe has options to add a manifest at link time without relying on
either method above, but that's not supported on mingw either.
So the best we can do is to move everything to using resource files.
This also avoids using MT, which, on cross builds, requires using wine.
Ideally, the manifests would be declared in moz.build, but that
complicates things for cases like TestDllInterceptor, where there are
multiple binaries in the same directory, but only one of them needs the
manifest. This keeps the status quo of getting the manifest
automatically from the source directory.
Differential Revision: https://phabricator.services.mozilla.com/D85382
We are currently relying on two different ways to insert side-by-side
manifests in binaries on Windows: through resource files, or through
the use of MT. The latter is not supported on mingw builds, which is
not great.
Link.exe has options to add a manifest at link time without relying on
either method above, but that's not supported on mingw either.
So the best we can do is to move everything to using resource files.
This also avoids using MT, which, on cross builds, requires using wine.
Ideally, the manifests would be declared in moz.build, but that
complicates things for cases like TestDllInterceptor, where there are
multiple binaries in the same directory, but only one of them needs the
manifest. This keeps the status quo of getting the manifest
automatically from the source directory.
Differential Revision: https://phabricator.services.mozilla.com/D85382
clang-11 has a new warning that fires when you pass a 256-bit vector type as a parameter or return value, and your compilation doesn't enable AVX. The aim is to warn you that the ABI is different depending on whether you enable AVX, which could become a problem if AVX and non-AVX files communicate through such parameters.
While the intent is good, it's not really worth 1800 lines of log spam for us. We have only a tiny number of AVX compilations in media codecs, and the only things they interchange with other code are buffer pointers.
Differential Revision: https://phabricator.services.mozilla.com/D86107
2020-07-27 Jan-Marek Glogowski <glogow@fbihome.de>
* lib/freebl/Makefile:
Bug 1652032 Disable all freebl assembler code for MSVC arm64
r=rrelyea,bbeurdouche
There are two places, where NSS tries to compile either x86_64 MSVC
assembler or GCC aarch64 code, which will fail the build. And also
drop the non-MSVC arch build flags for them.
AFAI could identify, there isn't any armasm64 compatible asm code in
the whole NSS library, so I don't even adapt AS for the build. The
cross-build finishes this way.
[d98bbb6168f4]
2020-07-24 Benjamin Beurdouche <bbeurdouche@mozilla.com>
* cmd/bltest/blapitest.c, coreconf/config.gypi, coreconf/config.mk,
lib/freebl/alg2268.c, lib/freebl/deprecated/alg2268.c,
lib/freebl/freebl_base.gypi, lib/freebl/ldvector.c,
lib/freebl/loader.c, lib/freebl/loader.h, lib/freebl/manifest.mn,
lib/softoken/lowpbe.c, lib/softoken/pkcs11c.c:
Bug 1652729 - Add build flag to disable RC2 and relocate to
lib/freebl/deprecated. r=kjacobs
[e6c6f1d2d544]
2020-07-27 Robert Relyea <rrelyea@redhat.com>
* gtests/softoken_gtest/manifest.mn,
gtests/softoken_gtest/softoken_dh_vectors.h,
gtests/softoken_gtest/softoken_gtest.cc,
gtests/softoken_gtest/softoken_gtest.gyp, lib/freebl/blapi.h,
lib/freebl/dh.c, lib/freebl/ldvector.c, lib/freebl/loader.c,
lib/freebl/loader.h, lib/softoken/manifest.mn,
lib/softoken/pkcs11.c, lib/softoken/pkcs11c.c,
lib/softoken/pkcs11i.h, lib/softoken/pkcs11u.c,
lib/softoken/sftkdhverify.c, lib/softoken/softoken.gyp:
Bug 1648822 Add stricter validation of DH keys when in FIPS mode.
Update: FIPS now also requires us to do y^q mod p testing on key
generation (always). We now do that in FIPS mode only, but in all
modes we do full DH verification for DH and ECDH. Because of this,
the path has now separated out the prime checks, which are now only
done for the DH operation if we aren't using a known prime and the
subprime value has been provided. I've also learned we can accept
keys that we do full validation on in FIPS mode, so I've added that
to this patch, though we still can't generate those kinds of keys
without adding the subprime at keygen time.
The new FIPS standard is dh operations must use approved primes.
Approved primes are those selected in the tls and ike RFCs.
Currently tls and ike have modes with checks whether the primes are
approved, but the check may not always happen. The safest thing to
do in FIPS mode is only allow those primes. In addition, FIPS
requires 1< y < p-1 (or technically 2<=y<=p-2, since y is an integer
those two tests are identical).
While making changes I realized we would want a mode where we can do
more strict checks on the prime while not requiring that the prime
be an approved prime. We already allow for strict checking if q is
supplied with the private key, but there were a couple of issues
with that check:
1. there was no way of actually setting q in the current NSS
pk11wrap interfaces. 2. If the prime was a safe prime, but g was an
actual generator, then we would fail the y^q mod p = 1 tests for 50%
of the keys, even though those keys are safe. 3. We weren't checking
primality of p and q.
So the old code:
if (q) { check y^q mod p = 1 if not fail }
check 1 <y < p-1 (done in DH_Derive).
New code:
if (! p is approved prime) { if (FIPS) fail; if (q) { y_test = y if
(p,q-> p is a safe prime) { y_test = 1 } check prime is prime Fail
if not check subprime is subprime fail if not y_test^q mod p = 1 } }
check 1 < y < p-1 (done in DH_Derive)
This means:
Existing code non-fips without setting the subprime continues to run
as before. Non-fips code which sets the subprime now runs slower,
but p and q are checked if p or q where not prime, the derive fails
(which it should). In FIPS mode only approved primes will succeed
now. Non-fips code can now set the subprime to q=(p-1)/2 if it
doesn't have an explicit q value (like in tls). If the derive
succeeds, we know that p is a safe prime. If p is approved, the
checks are skipped because we already know that p is a safe prime.
Code can optionally do a test derive on a new p and remember it's
safe so that we know longer need to check ever call (though if q is
not (p-1)/2, you will need to continue to do the checks each call
because y could still be a small subgroup).
This patch:
gtests/softoken_gtest
1. Added New dh tests to softoken_gtests. The tests were added to
softoken_gtests because we need to test both non-FIPS and FIPS mode.
Test vectors include a category, so the same test vectors can be
used in FIPS and non-FIPS even though each class may have different
results. Most of the test vectors where created either by dhparams
command in openssl, dsaparams in openssl, and the nss makepqg
command. Each vector includes a label, prime, base, optional
subprime, optional public key, test type, and key class (basically
size). 2. If public key is not supplied, we use a generated public
key. 3. If subPrime is supplied to wet it on the private key after
generation.
lib/freebl/dh.c
add primality tests to KEA_VerifyKey().
lib/softokn/
1. Allow CKA_SUBPRIME to be set after key generation or import.
This affects how we test for it's existance, since it is now always
there on the key, we check it's length to make sure it's non-zero.
2. We implement the psuedocode above as real code. 3. We create two
new functions: sftl_VerifyDH_Prime which return SECSuccess if Prime
is an approved prime. sftk_IsSafePrime which returns SECSuess of
both prime and subprime look reasonable, and sets a Bool to PR_TRUE
is subprime -> prime is safe (subprime = (prime-1)/2. These
functions are implemented in sftkdhverify.c 4.Cleanup incorrect
nominclature on primes (safe primes are not strong primes).
[0be91fa2217a]
* gtests/softoken_gtest/softoken_dh_vectors.h,
gtests/softoken_gtest/softoken_gtest.cc:
Fix more of the timeout issues on tests. (Drop expensive 4098 dh
tests ).
[4014c075a31b]
2020-07-29 Makoto Kato <m_kato@ga2.so-net.ne.jp>
* coreconf/config.gypi, lib/freebl/Makefile, lib/freebl/blinit.c,
lib/freebl/freebl.gyp, lib/freebl/sha1-armv8.c,
lib/freebl/sha_fast.c, lib/freebl/sha_fast.h:
Bug 1650702 - Use ARM's crypt extension for SHA1. r=kjacobs
ARM Crypto extension has SHA1 acceleration. Using this, SHA1 is 3
times faster on ARMv8 CPU. The following data is AWS's a1 instance
(Cortex-A72).
Before ====== ``` # mode in opreps cxreps context op time(sec)
thrgput sha1_e 954Mb 31M 0 0.000 10000.000 10.000 95Mb ```
After ===== ``` # mode in opreps cxreps context op time(sec) thrgput
sha1_e 2Gb 94M 0 0.000 10000.000 10.000 288Mb ```
[68b6eb737689]
2020-07-29 Jan-Marek Glogowski <glogow@fbihome.de>
* manifest.mn:
Bug 1653975 - Set "all" as the default Makefile target r=jcj,rrelyea
Just reorder the rules in manifest.mn, so all is again the first
rule. This restores pre-3.53 Makefile defaults.
[eb52747b7000]
2020-07-31 Makoto Kato <m_kato@ga2.so-net.ne.jp>
* lib/freebl/blapii.h, lib/freebl/blinit.c, nss-tool/hw-support.c:
Bug 1654142 - Add CPU feature detection for Intel SHA extension.
r=kjacobs
[e6b77a9c417a]
2020-08-03 Nathan Froyd <froydnj@mozilla.com>
* coreconf/detect_host_arch.py:
Bug 1656986 - special-case arm64 in detect_host_arch.py; r=jcj
This case comes up when attempting to build NSS on ARM64 Mac. If we
don't do this, we wind up detecting arm64 as "arm", with predictably
bad consequences.
[afa38fb2f0b5] [tip]
Differential Revision: https://phabricator.services.mozilla.com/D85888
The order in which toolchain binaries are resolved change based on environmental factors,
such as whether Firefox is being built in release mode or not.
An informative log was added in either case.
Differential Revision: https://phabricator.services.mozilla.com/D84439
It turns out setting CARGO_PROFILE_RELEASE_LTO has unwanted side
effects.
First it's not actually strictly equivalent to using `cargo rustc --
-Clto`. For instance, it apparently also enables cross-language LTO in
newer versions of cargo.
Second, it changes the rust computed hash for all the dependencies of
the crate being built with the variable set, which makes them diverge
from when the same dependencies are built through another crate in the
tree that is not LTOed. This effectively makes us build a _lot_ of
crates twice, many of which are not cacheable.
Since the original problem is that cargo >= 1.45 passes extra flags (`-C
embed-bitcode=no`) to rustc that are incompatible with `-Clto`, and while
it knows to adjust based on the `lto` setting in the build profile
(which CARGO_PROFILE_RELEASE_LTO overrides the default of), cargo
ignores flags passed via `cargo rustc -- ...` when making those
adjustments.
So, we need to override with `-C embed-bitcode=yes` on our own at the
same time we pass `-Clto`. But doing that through `cargo rustc -- ...`
is not enough because all the dependencies of the crate built with
`-Clto` need to be built with `-C embed-bitcode=yes`. So we need to
override with `RUSTFLAGS`, which will affect all the dependencies.
But we also need to do this consistently across all crates, not only the
dependencies of crates built with `-Clto`, otherwise we'd still end up
building crates twice (once with and once without the override).
Unfortunately, the `-C embed-bitcode=*` flag is also not supported in
versions older than 1.45, so we have to avoid adding it on older
versions.
We unfortunately support a large range of versions of rustc (albeit only
for tools/crashreporter), but we actually need to upgrade the smaller
supported version because rustc < 1.38 doesn't support our top-level
Cargo.lock. This makes the version check slightly less awful.
Differential Revision: https://phabricator.services.mozilla.com/D84652
Razvan Maries
Mike Hommey
Marco Trevisan
David Major
Ricky Stewart
Kevin Jacobs
Narcis Beleuzu
Mitchell Hentges
Hector Zhao
Bogdan Tara
Mihai Alexandru Michis
Tom Ritter
Dorel Luca
longsonr