On a very parallel debug build, I see a long time just waiting for
bindgen / style compilation / geckoservo.
Turns out that a bunch of this is just proc macros / build scripts.
Optimizing it saves between 10 and 17 seconds of my debug build. We
might want to consider running bindgen much like cbindgen rather than
rebuilding it all the time, which should help a lot more, but my guess
is that this should still help with the pretty hot custom derives that
the style crate runs.
This needs rust 1.41, so the requirement for tools/crashreporter needs
to be bumped as a consequence. To make things simpler, it was bumped
to 1.47 while we're at it.
Differential Revision: https://phabricator.services.mozilla.com/D98366
Windows being Windows, with its own spellings for everything, let's stay within the WINNT block regardless of whether we meet the compiler requirement.
Differential Revision: https://phabricator.services.mozilla.com/D98355
Code review followups for 1675600: Restrict the use of new pass manager during LTO to builds where we're using the new pass manager in general, and (on Windows) where lld-link is new enough to understand the flag.
Differential Revision: https://phabricator.services.mozilla.com/D97372
Upstream clang supports --target=aarch64-apple-darwin, and that matches
what config.sub canonicalizes to, and thus what the default toolchain
prefix ends up being, so it's better to use aarch64-apple-darwin when
not compiling with Xcode clang.
Differential Revision: https://phabricator.services.mozilla.com/D97698
There is no earlier SDK that supports it. It seems Xcode clang doesn't
care (maybe it defaults to 11.0 is MACOSX_DEPLOYMENT_TARGET is too low?),
but upstream clang does.
Differential Revision: https://phabricator.services.mozilla.com/D97565
And don't set it via mozconfig. The default to /System/Library/PrivateFrameworks
may also not have matched the used SDK previously, so the new default is
better.
Differential Revision: https://phabricator.services.mozilla.com/D97564
2020-11-18 Kevin Jacobs <kjacobs@mozilla.com>
* lib/ssl/ssl3con.c, lib/ssl/tls13con.c, lib/ssl/tls13ech.c:
Bug 1654332 - Fixup a10493dcfcc9: copy ECHConfig.config_id with
socket r=jcj
A late review change for ECH was for the server to compute each
ECHConfig `config_id` when set to the socket, rather than on each
connection. This works, but now we also need to copy that config_id
when copying a socket, else the server won't find a matching
ECHConfig to use for decryption.
[3eacb92e9adf] [tip]
2020-11-17 Kevin Jacobs <kjacobs@mozilla.com>
* automation/abi-check/expected-report-libssl3.so.txt,
cmd/tstclnt/tstclnt.c, cpputil/tls_parser.h,
gtests/ssl_gtest/libssl_internals.c,
gtests/ssl_gtest/libssl_internals.h, gtests/ssl_gtest/manifest.mn,
gtests/ssl_gtest/ssl_auth_unittest.cc,
gtests/ssl_gtest/ssl_custext_unittest.cc,
gtests/ssl_gtest/ssl_extension_unittest.cc,
gtests/ssl_gtest/ssl_gtest.gyp,
gtests/ssl_gtest/ssl_tls13compat_unittest.cc,
gtests/ssl_gtest/tls_agent.cc, gtests/ssl_gtest/tls_agent.h,
gtests/ssl_gtest/tls_connect.cc, gtests/ssl_gtest/tls_connect.h,
gtests/ssl_gtest/tls_ech_unittest.cc,
gtests/ssl_gtest/tls_esni_unittest.cc,
gtests/ssl_gtest/tls_filter.cc, gtests/ssl_gtest/tls_filter.h,
lib/ssl/SSLerrs.h, lib/ssl/manifest.mn, lib/ssl/ssl.gyp,
lib/ssl/ssl3con.c, lib/ssl/ssl3ext.c, lib/ssl/ssl3ext.h,
lib/ssl/ssl3exthandle.c, lib/ssl/ssl3exthandle.h,
lib/ssl/ssl3prot.h, lib/ssl/sslencode.c, lib/ssl/sslencode.h,
lib/ssl/sslerr.h, lib/ssl/sslexp.h, lib/ssl/sslimpl.h,
lib/ssl/sslinfo.c, lib/ssl/sslsecur.c, lib/ssl/sslsock.c,
lib/ssl/sslt.h, lib/ssl/tls13con.c, lib/ssl/tls13con.h,
lib/ssl/tls13ech.c, lib/ssl/tls13ech.h, lib/ssl/tls13esni.c,
lib/ssl/tls13esni.h, lib/ssl/tls13exthandle.c,
lib/ssl/tls13exthandle.h, lib/ssl/tls13hashstate.c,
lib/ssl/tls13hashstate.h:
Bug 1654332 - Update ESNI to draft-08 (ECH). r=mt
This patch adds support for Encrypted Client Hello (draft-ietf-tls-
esni-08), replacing the existing ESNI (draft -02) support.
There are five new experimental functions to enable this:
- SSL_EncodeEchConfig: Generates an encoded (not BASE64) ECHConfig
given a set of parameters.
- SSL_SetClientEchConfigs: Configures the provided ECHConfig to the
given socket. When configured, an ephemeral HPKE keypair will be
generated for the CH encryption.
- SSL_SetServerEchConfigs: Configures the provided ECHConfig and
keypair to the socket. The keypair specified will be used for HPKE
operations in order to decrypt encrypted Client Hellos as they are
received.
- SSL_GetEchRetryConfigs: If ECH is rejected by the server and
compatible retry_configs are provided, this API allows the
application to extract those retry_configs for use in a new
connection.
- SSL_EnableTls13GreaseEch: When enabled, non-ECH Client Hellos will
have a "GREASE ECH" (i.e. fake) extension appended. GREASE ECH is
disabled by default, as there are known compatibility issues that
will be addressed in a subsequent draft.
The following ESNI experimental functions are deprecated by this
update:
- SSL_EncodeESNIKeys
- SSL_EnableESNI
- SSL_SetESNIKeyPair
In order to be used, NSS must be compiled with
`NSS_ENABLE_DRAFT_HPKE` defined.
[a10493dcfcc9]
* lib/ssl/ssl3con.c, lib/ssl/sslencode.c, lib/ssl/sslencode.h,
lib/ssl/tls13con.c, lib/ssl/tls13con.h:
Bug 1654332 - Buffered ClientHello construction. r=mt
This patch refactors construction of Client Hello messages. Instead
of each component of the message being written separately into
`ss->sec.ci.sendBuf`, we now construct the message in its own
sslBuffer. Once complete, the entire message is added to the sendBuf
via `ssl3_AppendHandshake`.
`ssl3_SendServerHello` already uses this approach and it becomes
necessary for ECH, where we use the constructed ClientHello to
create an inner ClientHello.
[d40121ba59ba]
2020-11-13 J.C. Jones <jjones@mozilla.com>
* automation/abi-check/expected-report-libnss3.so.txt, automation/abi-
check/expected-report-libnssutil3.so.txt, automation/abi-check
/previous-nss-release, lib/nss/nss.h, lib/softoken/softkver.h,
lib/util/nssutil.h:
Set version numbers to 3.60 Beta
[5e7b37609f22]
Differential Revision: https://phabricator.services.mozilla.com/D97492
The SEH unwind info problem no longer happens because nowadays bug 1631929 prevents the code pattern that led to it. I've confirmed that bug 1626951's bustage doesn't come back, both in regular and beta-simulation builds.
This cleanup will allow me to reverse the dependency in bug 1677742 and have `lto` depend on `new_pass_manager_flags`.
Differential Revision: https://phabricator.services.mozilla.com/D97362
Currently, we report "10.0.0.or.more" for the latest release of Xcode,
but we know it's 10.0.0 for sure, which is when the clang version
reported by Xcode is exactly 12.0.0. We know that in the past a bump of
LLVM has always been accompanied with a bump of the minor version, so we
expect no LLVM version bump until at least 12.0.1.
Differential Revision: https://phabricator.services.mozilla.com/D97241
Currently, in LTO builds, we use the new pass manager during the initial translation to bitcode but not for the final optimization during linking.
On Linux, we can enable the new pass manager during LTO with a plugin option. I've landed a patch upstream to allow it on Windows as well, which is included here.
Switching the pass manager brings speed improvements on its own, but it also reduces code size by ~6%, which we can use a portion of as budget to increase the import limit (via the hot multiplier) for even more speed improvements.
Differential Revision: https://phabricator.services.mozilla.com/D96108
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
biplist broke with python 3.9, but we don't need this anymore because
Python3's plistlib allows reading binary plists since python 3.4.
I've tested this with all sdks that I have and we support (10.11-10.15).
Differential Revision: https://phabricator.services.mozilla.com/D94577
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
The support code for Android, which doesn't support shm_open and can't
use the memfd backend because of issues with its SELinux policy (see bug
1670277), has been reorganized to reflect that we'll always use its own
API, ashmem, in that case.
Differential Revision: https://phabricator.services.mozilla.com/D90605
The warning is broken in older versions of clang. It also turns out we
were disabling it locally for some directories because of this very
problem.
A few local disable rules stay under accessible/ because they do hide
actual warnings from code generated by MIDL.
Differential Revision: https://phabricator.services.mozilla.com/D94262
We used to have a complicated scheme to figure out the minimum supported
version of clang on OSX, based on some compiler feature, which wouldn't
allow to do other version checks further down the line.
The main blocker for better tests was to be able to distinguish between
Xcode clang and plain clang, which turns out to be possible with the
__apple_build_version__ define.
We still need to map versions manually, but it's better than the current
status quo.
Differential Revision: https://phabricator.services.mozilla.com/D94261
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
The support code for Android, which doesn't support shm_open and can't
use the memfd backend because of issues with its SELinux policy (see bug
1670277), has been reorganized to reflect that we'll always use its own
API, ashmem, in that case.
Differential Revision: https://phabricator.services.mozilla.com/D90605
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
This avoids a sort of duplication of work between both, because the
linker will eventually LTO-compile everything, so we technically don't
really need the extra step of the rust compiler doing an intermediate
LTO on the static libraries it produces.
Differential Revision: https://phabricator.services.mozilla.com/D94224
2020-10-13 Mike Hommey <mh@glandium.org>
* lib/freebl/freebl.gyp:
Bug 1670839 - Use ARM crypto extension for AES, SHA1 and SHA2 on
mac. r=kjacobs
AFAICT, the Makefile equivalent already does.
[58dc3216d518] [tip]
* lib/freebl/sha1-armv8.c:
Bug 1670839 - Only build sha1-armv8.c code when USE_HW_SHA1 is
defined. r=kjacobs
This matches what is done in sha256-armv8.c, and avoids
inconsistency with sha1-fast.c, which will define the same functions
in the case USE_HW_SHA1 is not defined.
[54be084e3ba8]
2020-10-16 J.C. Jones <jjones@mozilla.com>
* automation/abi-check/expected-report-libnss3.so.txt, automation/abi-
check/previous-nss-release, lib/nss/nss.h, lib/softoken/softkver.h,
lib/util/nssutil.h:
Set version numbers to 3.59 Beta
[d4b21706e432]
Differential Revision: https://phabricator.services.mozilla.com/D94070
Rustc >= 1.44 changed the file names of the static libraries it
produces with -windows-gnu targets, to match that of mingw clang/gcc.
Considering we still build on 1.43, the best fix would be to derive the
prefix/suffix based on the version of rust, but that actually turns into
a hard-to-solve problem because of configure tests for bindgen also
depending on the prefix/suffix value to be known.
On the other hand, we're soon due to an update to 1.47, so the simpler
solution is to just push mingw builds to require 1.44 (settling for the
smallest upgrade possible for now) and to remove the split between C and
rust library prefix/suffixes.
Differential Revision: https://phabricator.services.mozilla.com/D93726
This commit also allows `memfd_create` in the seccomp-bpf policy for all
process types.
`memfd_create` is an API added in Linux 3.17 (and adopted by FreeBSD
for the upcoming version 13) for creating anonymous shared memory
not connected to any filesystem. Supporting it means that sandboxed
child processes on Linux can create shared memory directly instead of
messaging a broker, which is unavoidably slower, and it should avoid
the problems we'd been seeing with overly small `/dev/shm` in container
environments (which were causing serious problems for using Firefox for
automated testing of frontend projects).
`memfd_create` also introduces the related operation of file seals:
irrevocably preventing types of modifications to a file. Unfortunately,
the most useful one, `F_SEAL_WRITE`, can't be relied on; see the large
comment in `SharedMemory:ReadOnlyCopy` for details. So we still use
the applicable seals as defense in depth, but read-only copies are
implemented on Linux by using procfs (and see the comments on the
`ReadOnlyCopy` function in `shared_memory_posix.cc` for the subtleties
there).
There's also a FreeBSD implementation, using `cap_rights_limit` for
read-only copies, if the build host is new enough to have the
`memfd_create` function.
Differential Revision: https://phabricator.services.mozilla.com/D90605
Now that we don't recurse into the js python configure, we don't need to
have a special treatment for the options that need to be passed down to
that subconfigure, which is what js_option was for.
Differential Revision: https://phabricator.services.mozilla.com/D92727
It was only meant to be used internally, when the top-level python
configure invoked the js python subconfigure. Now that this doesn't
happen, we can remove the option, and consolidate js_standalone and
building_js, which are now roughly synonyms.
Differential Revision: https://phabricator.services.mozilla.com/D92726
Emilio Cobos Álvarez
Bogdan Tara
Cosmin Sabou
Mike Hommey
David Major
Sylvestre Ledru
Kevin Jacobs
Butkovits Atila
Mitchell Hentges
Steve Fink
Ricky Stewart
Jed Davis
Dorel Luca
J.C. Jones
Razvan Maries
Kartikaya Gupta