This is bug 1639318 all over again except from Rust rather than C++. It's the same symptom, nsXPTCStub vtables aren't marked as valid targets.
In the earlier bug we disabled CFG for C++ on ARM64. Let's do the same for Rust. According to that bug, "It's not clear why this doesn't happen on x86 builds. Given priorities, I can't really justify investigating this, although I suspect that fixing the underlying issue would be pretty much bug 1483885."
Differential Revision: https://phabricator.services.mozilla.com/D99278
Bug 1680152 updated automation to use the 10.12 SDK, and shortly after,
bug 1678174 introduced a change that broke the build with the 10.11 SDK.
Considering we haven't actually supported running on macos 10.11 and
earlier since Firefox 79, and that still supporting building with the
10.11 SDK would mean adding and maintaining code that, in practice,
would never be used by users, I think it is fair at this point that we
just drop support for the 10.11 SDK entirely.
Differential Revision: https://phabricator.services.mozilla.com/D99181
On a very parallel debug build, I see a long time just waiting for
bindgen / style compilation / geckoservo.
Turns out that a bunch of this is just proc macros / build scripts.
Optimizing it saves between 10 and 17 seconds of my debug build. We
might want to consider running bindgen much like cbindgen rather than
rebuilding it all the time, which should help a lot more, but my guess
is that this should still help with the pretty hot custom derives that
the style crate runs.
This needs rust 1.41, so the requirement for tools/crashreporter needs
to be bumped as a consequence. To make things simpler, it was bumped
to 1.47 while we're at it.
Differential Revision: https://phabricator.services.mozilla.com/D98366
Windows being Windows, with its own spellings for everything, let's stay within the WINNT block regardless of whether we meet the compiler requirement.
Differential Revision: https://phabricator.services.mozilla.com/D98355
Code review followups for 1675600: Restrict the use of new pass manager during LTO to builds where we're using the new pass manager in general, and (on Windows) where lld-link is new enough to understand the flag.
Differential Revision: https://phabricator.services.mozilla.com/D97372
Upstream clang supports --target=aarch64-apple-darwin, and that matches
what config.sub canonicalizes to, and thus what the default toolchain
prefix ends up being, so it's better to use aarch64-apple-darwin when
not compiling with Xcode clang.
Differential Revision: https://phabricator.services.mozilla.com/D97698
There is no earlier SDK that supports it. It seems Xcode clang doesn't
care (maybe it defaults to 11.0 is MACOSX_DEPLOYMENT_TARGET is too low?),
but upstream clang does.
Differential Revision: https://phabricator.services.mozilla.com/D97565
And don't set it via mozconfig. The default to /System/Library/PrivateFrameworks
may also not have matched the used SDK previously, so the new default is
better.
Differential Revision: https://phabricator.services.mozilla.com/D97564
2020-11-18 Kevin Jacobs <kjacobs@mozilla.com>
* lib/ssl/ssl3con.c, lib/ssl/tls13con.c, lib/ssl/tls13ech.c:
Bug 1654332 - Fixup a10493dcfcc9: copy ECHConfig.config_id with
socket r=jcj
A late review change for ECH was for the server to compute each
ECHConfig `config_id` when set to the socket, rather than on each
connection. This works, but now we also need to copy that config_id
when copying a socket, else the server won't find a matching
ECHConfig to use for decryption.
[3eacb92e9adf] [tip]
2020-11-17 Kevin Jacobs <kjacobs@mozilla.com>
* automation/abi-check/expected-report-libssl3.so.txt,
cmd/tstclnt/tstclnt.c, cpputil/tls_parser.h,
gtests/ssl_gtest/libssl_internals.c,
gtests/ssl_gtest/libssl_internals.h, gtests/ssl_gtest/manifest.mn,
gtests/ssl_gtest/ssl_auth_unittest.cc,
gtests/ssl_gtest/ssl_custext_unittest.cc,
gtests/ssl_gtest/ssl_extension_unittest.cc,
gtests/ssl_gtest/ssl_gtest.gyp,
gtests/ssl_gtest/ssl_tls13compat_unittest.cc,
gtests/ssl_gtest/tls_agent.cc, gtests/ssl_gtest/tls_agent.h,
gtests/ssl_gtest/tls_connect.cc, gtests/ssl_gtest/tls_connect.h,
gtests/ssl_gtest/tls_ech_unittest.cc,
gtests/ssl_gtest/tls_esni_unittest.cc,
gtests/ssl_gtest/tls_filter.cc, gtests/ssl_gtest/tls_filter.h,
lib/ssl/SSLerrs.h, lib/ssl/manifest.mn, lib/ssl/ssl.gyp,
lib/ssl/ssl3con.c, lib/ssl/ssl3ext.c, lib/ssl/ssl3ext.h,
lib/ssl/ssl3exthandle.c, lib/ssl/ssl3exthandle.h,
lib/ssl/ssl3prot.h, lib/ssl/sslencode.c, lib/ssl/sslencode.h,
lib/ssl/sslerr.h, lib/ssl/sslexp.h, lib/ssl/sslimpl.h,
lib/ssl/sslinfo.c, lib/ssl/sslsecur.c, lib/ssl/sslsock.c,
lib/ssl/sslt.h, lib/ssl/tls13con.c, lib/ssl/tls13con.h,
lib/ssl/tls13ech.c, lib/ssl/tls13ech.h, lib/ssl/tls13esni.c,
lib/ssl/tls13esni.h, lib/ssl/tls13exthandle.c,
lib/ssl/tls13exthandle.h, lib/ssl/tls13hashstate.c,
lib/ssl/tls13hashstate.h:
Bug 1654332 - Update ESNI to draft-08 (ECH). r=mt
This patch adds support for Encrypted Client Hello (draft-ietf-tls-
esni-08), replacing the existing ESNI (draft -02) support.
There are five new experimental functions to enable this:
- SSL_EncodeEchConfig: Generates an encoded (not BASE64) ECHConfig
given a set of parameters.
- SSL_SetClientEchConfigs: Configures the provided ECHConfig to the
given socket. When configured, an ephemeral HPKE keypair will be
generated for the CH encryption.
- SSL_SetServerEchConfigs: Configures the provided ECHConfig and
keypair to the socket. The keypair specified will be used for HPKE
operations in order to decrypt encrypted Client Hellos as they are
received.
- SSL_GetEchRetryConfigs: If ECH is rejected by the server and
compatible retry_configs are provided, this API allows the
application to extract those retry_configs for use in a new
connection.
- SSL_EnableTls13GreaseEch: When enabled, non-ECH Client Hellos will
have a "GREASE ECH" (i.e. fake) extension appended. GREASE ECH is
disabled by default, as there are known compatibility issues that
will be addressed in a subsequent draft.
The following ESNI experimental functions are deprecated by this
update:
- SSL_EncodeESNIKeys
- SSL_EnableESNI
- SSL_SetESNIKeyPair
In order to be used, NSS must be compiled with
`NSS_ENABLE_DRAFT_HPKE` defined.
[a10493dcfcc9]
* lib/ssl/ssl3con.c, lib/ssl/sslencode.c, lib/ssl/sslencode.h,
lib/ssl/tls13con.c, lib/ssl/tls13con.h:
Bug 1654332 - Buffered ClientHello construction. r=mt
This patch refactors construction of Client Hello messages. Instead
of each component of the message being written separately into
`ss->sec.ci.sendBuf`, we now construct the message in its own
sslBuffer. Once complete, the entire message is added to the sendBuf
via `ssl3_AppendHandshake`.
`ssl3_SendServerHello` already uses this approach and it becomes
necessary for ECH, where we use the constructed ClientHello to
create an inner ClientHello.
[d40121ba59ba]
2020-11-13 J.C. Jones <jjones@mozilla.com>
* automation/abi-check/expected-report-libnss3.so.txt, automation/abi-
check/expected-report-libnssutil3.so.txt, automation/abi-check
/previous-nss-release, lib/nss/nss.h, lib/softoken/softkver.h,
lib/util/nssutil.h:
Set version numbers to 3.60 Beta
[5e7b37609f22]
Differential Revision: https://phabricator.services.mozilla.com/D97492
The SEH unwind info problem no longer happens because nowadays bug 1631929 prevents the code pattern that led to it. I've confirmed that bug 1626951's bustage doesn't come back, both in regular and beta-simulation builds.
This cleanup will allow me to reverse the dependency in bug 1677742 and have `lto` depend on `new_pass_manager_flags`.
Differential Revision: https://phabricator.services.mozilla.com/D97362
Currently, we report "10.0.0.or.more" for the latest release of Xcode,
but we know it's 10.0.0 for sure, which is when the clang version
reported by Xcode is exactly 12.0.0. We know that in the past a bump of
LLVM has always been accompanied with a bump of the minor version, so we
expect no LLVM version bump until at least 12.0.1.
Differential Revision: https://phabricator.services.mozilla.com/D97241
Currently, in LTO builds, we use the new pass manager during the initial translation to bitcode but not for the final optimization during linking.
On Linux, we can enable the new pass manager during LTO with a plugin option. I've landed a patch upstream to allow it on Windows as well, which is included here.
Switching the pass manager brings speed improvements on its own, but it also reduces code size by ~6%, which we can use a portion of as budget to increase the import limit (via the hot multiplier) for even more speed improvements.
Differential Revision: https://phabricator.services.mozilla.com/D96108
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
biplist broke with python 3.9, but we don't need this anymore because
Python3's plistlib allows reading binary plists since python 3.4.
I've tested this with all sdks that I have and we support (10.11-10.15).
Differential Revision: https://phabricator.services.mozilla.com/D94577
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
The support code for Android, which doesn't support shm_open and can't
use the memfd backend because of issues with its SELinux policy (see bug
1670277), has been reorganized to reflect that we'll always use its own
API, ashmem, in that case.
Differential Revision: https://phabricator.services.mozilla.com/D90605
The warning is broken in older versions of clang. It also turns out we
were disabling it locally for some directories because of this very
problem.
A few local disable rules stay under accessible/ because they do hide
actual warnings from code generated by MIDL.
Differential Revision: https://phabricator.services.mozilla.com/D94262
We used to have a complicated scheme to figure out the minimum supported
version of clang on OSX, based on some compiler feature, which wouldn't
allow to do other version checks further down the line.
The main blocker for better tests was to be able to distinguish between
Xcode clang and plain clang, which turns out to be possible with the
__apple_build_version__ define.
We still need to map versions manually, but it's better than the current
status quo.
Differential Revision: https://phabricator.services.mozilla.com/D94261
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
The support code for Android, which doesn't support shm_open and can't
use the memfd backend because of issues with its SELinux policy (see bug
1670277), has been reorganized to reflect that we'll always use its own
API, ashmem, in that case.
Differential Revision: https://phabricator.services.mozilla.com/D90605
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
This avoids a sort of duplication of work between both, because the
linker will eventually LTO-compile everything, so we technically don't
really need the extra step of the rust compiler doing an intermediate
LTO on the static libraries it produces.
Differential Revision: https://phabricator.services.mozilla.com/D94224
2020-10-13 Mike Hommey <mh@glandium.org>
* lib/freebl/freebl.gyp:
Bug 1670839 - Use ARM crypto extension for AES, SHA1 and SHA2 on
mac. r=kjacobs
AFAICT, the Makefile equivalent already does.
[58dc3216d518] [tip]
* lib/freebl/sha1-armv8.c:
Bug 1670839 - Only build sha1-armv8.c code when USE_HW_SHA1 is
defined. r=kjacobs
This matches what is done in sha256-armv8.c, and avoids
inconsistency with sha1-fast.c, which will define the same functions
in the case USE_HW_SHA1 is not defined.
[54be084e3ba8]
2020-10-16 J.C. Jones <jjones@mozilla.com>
* automation/abi-check/expected-report-libnss3.so.txt, automation/abi-
check/previous-nss-release, lib/nss/nss.h, lib/softoken/softkver.h,
lib/util/nssutil.h:
Set version numbers to 3.59 Beta
[d4b21706e432]
Differential Revision: https://phabricator.services.mozilla.com/D94070
Rustc >= 1.44 changed the file names of the static libraries it
produces with -windows-gnu targets, to match that of mingw clang/gcc.
Considering we still build on 1.43, the best fix would be to derive the
prefix/suffix based on the version of rust, but that actually turns into
a hard-to-solve problem because of configure tests for bindgen also
depending on the prefix/suffix value to be known.
On the other hand, we're soon due to an update to 1.47, so the simpler
solution is to just push mingw builds to require 1.44 (settling for the
smallest upgrade possible for now) and to remove the split between C and
rust library prefix/suffixes.
Differential Revision: https://phabricator.services.mozilla.com/D93726
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
Differential Revision: https://phabricator.services.mozilla.com/D90605
Now that we don't recurse into the js python configure, we don't need to
have a special treatment for the options that need to be passed down to
that subconfigure, which is what js_option was for.
Differential Revision: https://phabricator.services.mozilla.com/D92727
It was only meant to be used internally, when the top-level python
configure invoked the js python subconfigure. Now that this doesn't
happen, we can remove the option, and consolidate js_standalone and
building_js, which are now roughly synonyms.
Differential Revision: https://phabricator.services.mozilla.com/D92726
Instead, we now run js/src/old-configure from the top-level configure
after having run old-configure and extracted a few variables to inherit
from it.
Because we're now running from the top-level, $_objdir is always the
top-level objdir, which simplifies some things. The topobjdir in
js/src/config.status, however, needs to stay in js/src because of the
build frontend expecting it there.
When running js/src/old-configure, we used to need some special
treatment for a large number of variables for historic reasons, where
we'd take values from the assigned values before running old-configure
for some, or from AC_SUBSTs after running old-configure.
Now that both old-configure and js/src/old-configure get the same
assignments from old-configure.vars, we don't need anything special for
the former. And only a few remaining variables still need manual work
for the latter.
One notable difference, though, is that the new code doesn't try to
avoid running js subconfigure, which added complexity, and was actually
error-prone.
Differential Revision: https://phabricator.services.mozilla.com/D92725
Preparing to run both old-configure and js/src/old-configure from the
same python configure run, we refactor things such that shared parts are
separate.
Differential Revision: https://phabricator.services.mozilla.com/D92722
It is only really used in js/src/devtools/rootAnalysis/Makefile.in,
and even there, the way it is used seems wrong, so fix that at the
same time (binaries have been linked into $DIST/bin directly for a
while).
Differential Revision: https://phabricator.services.mozilla.com/D92721
In order to be able to run both old-configure and js/src/old-configure
from the same python configure run, we need to stop setting the items
set by old-configure into the global sandbox config, and instead store
them to be later handled by configure.py.
Differential Revision: https://phabricator.services.mozilla.com/D92718
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
Differential Revision: https://phabricator.services.mozilla.com/D90605
2020-09-18 Kevin Jacobs <kjacobs@mozilla.com>
* automation/abi-check/previous-nss-release, lib/nss/nss.h,
lib/softoken/softkver.h, lib/util/nssutil.h:
Set version numbers to 3.58 Beta
[c28e20f61e5d] [tip]
* .hgtags:
Added tag NSS_3_57_RTM for changeset cf7e3e8abd77
[a963849538ca] <NSS_3_57_BRANCH>
* lib/nss/nss.h, lib/softoken/softkver.h, lib/util/nssutil.h:
Set version numbers to 3.57 final
[cf7e3e8abd77] [NSS_3_57_RTM] <NSS_3_57_BRANCH>
Differential Revision: https://phabricator.services.mozilla.com/D91070
It was added in bug 1320656 because back then we were building as C++14
with warnings about future incompatibilities with C++17. Since then,
we've switched to C++17, which means we had to fix those
incompatibilities, and thus they don't exist anymore. A local build with
-Werror=noexcept-type finishes just fine.
This removes the only difference between top-level and js warning flags.
Differential Revision: https://phabricator.services.mozilla.com/D90521
I think they're remnants from the past that we don't really need anymore.
And they're making things more complicated for some pending work of mine.
Differential Revision: https://phabricator.services.mozilla.com/D89687
I think they're remnants from the past that we don't really need anymore.
And they're making things more complicated for some pending work of mine.
Differential Revision: https://phabricator.services.mozilla.com/D89687
The files are copied verbatim from upstream autoconf 2.13 (but only the
files we need) and old.configure is adapted to use the vendored version.
Differential Revision: https://phabricator.services.mozilla.com/D89554
When configuring without system NSPR the configuration variable
PKGCONF_REQUIRES_PRIVATE isn't ever set, leading to a .pc file that still
contains the @PKGCONF_REQUIRES_PRIVATE@ stub.
Ensure that we always define PKGCONF_REQUIRES_PRIVATE, by setting it to an
empty string in case no system-nsrp is enabled.
In this way, the pkg-config file stub will be always replaced.
Differential Revision: https://phabricator.services.mozilla.com/D88179
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
We don't anticipate end users will actually care to do this, but it's useful especially for unit tests. For example, after bug 1659539, Python `configure` tests will run in a new, non-`init_py3` `virtualenv`, and we'll want to target that `virtualenv` for `configure` rather than having it create a new `virtualenv` for no reason.
Differential Revision: https://phabricator.services.mozilla.com/D88661
2020-08-21 Kevin Jacobs <kjacobs@mozilla.com>
* automation/abi-check/previous-nss-release, lib/nss/nss.h,
lib/softoken/softkver.h, lib/util/nssutil.h:
Set version numbers to 3.57 Beta
[783f49ae6126]
2020-08-24 Kevin Jacobs <kjacobs@mozilla.com>
* gtests/ssl_gtest/ssl_auth_unittest.cc, lib/ssl/dtls13con.c,
lib/ssl/dtlscon.c, lib/ssl/ssl3con.c, lib/ssl/sslimpl.h,
lib/ssl/sslnonce.c:
Bug 1653641 - Cleanup inaccurate DTLS comments, code review fixes.
r=mt
[0e1b5c711cb9]
2020-08-24 Robert Relyea <rrelyea@redhat.com>
* lib/freebl/fipsfreebl.c, lib/softoken/fipstest.c,
lib/softoken/kbkdf.c, lib/softoken/lowpbe.c, lib/softoken/lowpbe.h,
lib/softoken/pkcs11c.c, lib/softoken/pkcs11i.h,
lib/softoken/sftkhmac.c, lib/softoken/sftkike.c:
Bug 1660304 New FIPS IG requires self-tests for approved kdfs.
r=ueno comments=kjacobs
FIPS guidance now requires self-tests for our kdfs. It also requires
self-tests for cmac which we didn't have in the cmac patch.
Currently only one test per kdf is necessary. Specifially for
SP-800-108, only one of the three flavors are needed (counter,
feedback, or pipeline). This patch includes more complete testing
but it has been turned off the currently extraneous tests under the
assumption that NIST guidance may require them in the future. HKDF
is currently not included in FIPS, but is on track to be included,
so hkdf have been included in this patch.
Because the test vectors are const strings, the patch pushes some
const definitions that were missing in existing private interfaces.
There are three flavors of self-tests: Function implemented in
freebl are added to the freebl/fipsfreebl.c Functions implemented in
pkcs11c.c have selftests completely implemented in
softoken/fipstest.c Functions implemented in their own .c file have
their selftest function implemented in that .c file and called by
fipstests.c These are consistant with the previous choices for
selftests.
Some private interfaces that took in keys from pkcs #11 structures
or outputted keys to pkcs #11 structures were modified to optionally
take keys in by bytes and output keys as bytes so the self-tests can
work in just bytes.
[5dca54fe61c2]
2020-08-25 Daiki Ueno <dueno@redhat.com>
* lib/softoken/manifest.mn:
Bug 1659252, disable building libnssdbm3.so if NSS_DISABLE_DBM=1,
r=rrelyea
Reviewers: rrelyea
Reviewed By: rrelyea
Bug #: 1659252
[4d55d36ca6ef]
2020-08-24 Kevin Jacobs <kjacobs@mozilla.com>
* lib/pk11wrap/pk11cxt.c, lib/softoken/pkcs11c.c, lib/softoken/sdb.c,
lib/softoken/sftkpwd.c:
Bug 1651834 - Fix various static analyzer warnings. r=rrelyea
[ab04fd73fd6d]
2020-08-28 Mike Hommey <mh@glandium.org>
* lib/freebl/blapii.h:
Bug 1661810 - Define pre_align/post_align based on the compiler.
r=jcj
Things worked fine before we upgraded to clang 11 presumably because
the stack was always 16-bytes aligned in the first place, or
something akin to that, and the lack of pre_align/post_align doing
anything didn't matter. The runtime misalignment of the stack may
well be a clang > 9 bug, but keeping pre_align/post_align tied to
the x86/x64 is a footgun anyways.
[c100e11991f6] [tip]
Differential Revision: https://phabricator.services.mozilla.com/D88876
config.guess infers information about the compiler using environment
variables, such as CC. However, we use such environment variables to
configure the tooling for the target.
Differential Revision: https://phabricator.services.mozilla.com/D88085
Previously, we would optimistically attempt to use a Rust toolchain that
matches the current C toolchain, and would throw an error if an
attempted compile with that Rust toolchain failed.
Instead, if we fail to detect a usable Rust toolchain, we now helpfully
inform users of their two options: change C toolchain, or install
matching Rust toolchain.
Differential Revision: https://phabricator.services.mozilla.com/D88084
config.guess infers information about the compiler using environment
variables, such as CC. However, we use such environment variables to
configure the tooling for the target.
Differential Revision: https://phabricator.services.mozilla.com/D88085
Previously, we would optimistically attempt to use a Rust toolchain that
matches the current C toolchain, and would throw an error if an
attempted compile with that Rust toolchain failed.
Instead, if we fail to detect a usable Rust toolchain, we now helpfully
inform users of their two options: change C toolchain, or install
matching Rust toolchain.
Differential Revision: https://phabricator.services.mozilla.com/D88084
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
We have a minimum rust version required for compilations. For both stable and beta rust compilers, we can trust that they will have all the stabilized features we're expecting.
However, for nightlies, they may "match" our minimum version, but may have been released in the version window before a certain feature we need has been stabilized.
So, when validating rustc version in configure, ensure that the nightly is at least one version newer than our expected version.
Differential Revision: https://phabricator.services.mozilla.com/D86889
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:
1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.
2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.
As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.
2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.
3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).
4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).
5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.
Differential Revision: https://phabricator.services.mozilla.com/D85916
This makes us use one less tool from MSVC, and removes one more use of wine
in cross builds.
We replace the call to either rc/llvm-rc or windres with a wrapper script.
While the script is not strictly needed for the latter, we use a wrapper
in that case anyway because it's one step towards fixing bug 1498414.
For llvm-rc, however, we need a wrapper because llvm-rc doesn't preprocess
on its own, so the wrapper does that too.
The wrapper script also allows to deal with the default flags passed to
llvm-rc or windres, rather than inherit them from old-configure.
We also need to explicitly pass the codepage to llvm-rc, which was not
necessary with rc (presumably, llvm-rc has a different default).
While here, remove the unused WINDRES subst from js/src/old-configure.in.
Also, while here, we remove --use-temp-file, because as described in the
linked bug and in the windres manual page, it was used to work around bugs
on Windows 98 and earlier.
Differential Revision: https://phabricator.services.mozilla.com/D86312
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
I noticed that the `objdir:build` entry in `build/virtualenv_packages.txt` entry was apparently unused. This originates from bug 841713, seven years ago, when the `objdir` handling was introduced. Today, this doesn't appear to be serving a purpose. There is no Python library in my `$objdir/build` directory; nor can I see anything in `build/moz.build` or any related files suggesting one could ever appear. I can only assume this feature has outlived its usefulness, so delete it and the relevant in-tree support.
This necessitates slightly changing the signature and implementation of the `activate_pipenv()` method; also update all callers.
Differential Revision: https://phabricator.services.mozilla.com/D85635
We are currently relying on two different ways to insert side-by-side
manifests in binaries on Windows: through resource files, or through
the use of MT. The latter is not supported on mingw builds, which is
not great.
Link.exe has options to add a manifest at link time without relying on
either method above, but that's not supported on mingw either.
So the best we can do is to move everything to using resource files.
This also avoids using MT, which, on cross builds, requires using wine.
Ideally, the manifests would be declared in moz.build, but that
complicates things for cases like TestDllInterceptor, where there are
multiple binaries in the same directory, but only one of them needs the
manifest. This keeps the status quo of getting the manifest
automatically from the source directory.
Differential Revision: https://phabricator.services.mozilla.com/D85382
David Major
Mike Hommey
Emilio Cobos Álvarez
Bogdan Tara
Cosmin Sabou
Sylvestre Ledru
Kevin Jacobs
Butkovits Atila
Mitchell Hentges
Steve Fink
Ricky Stewart
Jed Davis
Dorel Luca
J.C. Jones
Razvan Maries
Kartikaya Gupta
Marco Trevisan
Narcis Beleuzu
Hector Zhao
Mihai Alexandru Michis