Because of conflicts between gcov_flush from gcc and the one from llvm, we renamed llvm one into ___custom_llvm_gcov_flush.
Since we switched to clang for linux ccov builds, this workaround is now useless.
Differential Revision: https://phabricator.services.mozilla.com/D104990
In the case of toolchain tasks, the tooltool download script already
extracted the SDK in $MOZ_FETCHES_DIR, so no adjustment was required.
Only a Firefox mozconfig needs adaptation.
Differential Revision: https://phabricator.services.mozilla.com/D104646
Bug 1634204 bumped the maximum version of symbols allowed in our
dependency upon libstdc++, which effectively makes some of the
stdc++compat code dead. We can now remove it.
Differential Revision: https://phabricator.services.mozilla.com/D104617
When using the --sysroot argument to clang, clang changes where it
searches for libraries in its own directory, and excludes the lib and
lib32 subdirectories. So we need to move the gcc files to a place where
it does look (and that it also looks without --sysroot).
We however still keep a copy of libstdc++ in the lib directory for
runtime purposes.
Differential Revision: https://phabricator.services.mozilla.com/D104123
We used to only push to opt-level=2 on --enable-release builds, to make
local builds faster with opt-level=1. Years later, it seems opt-level=2 makes no noticeable
difference in build times vs. opt-level=1, neither on my Threadripper
workstation at -j64 or my M1 Macbook Air at -j4.
That's one less difference to carry.
Differential Revision: https://phabricator.services.mozilla.com/D103266
2021-01-22 Kevin Jacobs <kjacobs@mozilla.com>
* automation/abi-check/previous-nss-release, lib/nss/nss.h,
lib/softoken/softkver.h, lib/util/nssutil.h:
Set version numbers to 3.62 Beta
[680ec01577b9]
2021-01-23 Kevin Jacobs <kjacobs@mozilla.com>
* tests/chains/scenarios/nameconstraints.cfg,
tests/libpkix/certs/NameConstraints.ipaca.cert,
tests/libpkix/certs/NameConstraints.ocsp1.cert:
Bug 1686134 - Renew two chains libpkix test certificates. r=rrelyea
[3ddcd845704c]
2021-01-25 Kevin Jacobs <kjacobs@mozilla.com>
* gtests/common/testvectors/hpke-vectors.h,
gtests/pk11_gtest/pk11_hpke_unittest.cc, lib/pk11wrap/pk11hpke.c,
lib/pk11wrap/pk11hpke.h, lib/pk11wrap/pk11pub.h:
Bug 1678398 - Update HPKE to draft-07. r=mt
This patch updates HPKE to draft-07. A few other minor changes are
included:
- Refactor HPKE gtests for increased parameterized testing.
- Replace memcpy calls with PORT_Memcpy
- Serialization tweaks to make way for context Export/Import (D99277).
This should not be landed without an ECH update, as fixed ECH test
vectors will otherwise fail to decrypt.
[e0bf8cadadc7]
* automation/abi-check/expected-report-libnss3.so.txt,
gtests/pk11_gtest/pk11_hpke_unittest.cc, lib/nss/nss.def,
lib/pk11wrap/pk11hpke.c, lib/pk11wrap/pk11pub.h:
Bug 1678398 - Add Export/Import functions for HPKE context. r=mt
This patch adds and exports two new HPKE functions:
`PK11_HPKE_ExportContext` and `PK11_HPKE_ImportContext`, which are
used to export a serialized HPKE context, then later reimport that
context and resume Open and Export operations. Only receiver
contexts are currently supported for export (see the rationale in
pk11pub.h).
One other change introduced here is that `PK11_HPKE_GetEncapPubKey`
now works as expected on the receiver side.
If the `wrapKey` argument is provided to the Export/Import
functions, then the symmetric keys are wrapped with AES Key Wrap
with Padding (SP800-38F, 6.3) prior to serialization.
[8bcd12ab3b34]
* automation/abi-check/expected-report-libssl3.so.txt,
gtests/ssl_gtest/libssl_internals.c,
gtests/ssl_gtest/libssl_internals.h,
gtests/ssl_gtest/ssl_extension_unittest.cc,
gtests/ssl_gtest/tls_ech_unittest.cc, lib/ssl/ssl3con.c,
lib/ssl/ssl3ext.c, lib/ssl/ssl3ext.h, lib/ssl/sslexp.h,
lib/ssl/sslimpl.h, lib/ssl/sslsecur.c, lib/ssl/sslsock.c,
lib/ssl/sslt.h, lib/ssl/tls13con.c, lib/ssl/tls13con.h,
lib/ssl/tls13ech.c, lib/ssl/tls13ech.h, lib/ssl/tls13exthandle.c,
lib/ssl/tls13exthandle.h, lib/ssl/tls13hashstate.c,
lib/ssl/tls13hashstate.h:
Bug 1681585 - Update ECH to Draft-09. r=mt
This patch updates ECH implementation to draft-09. Changes of note
are:
- Acceptance signal derivation is now based on the handshake secret.
- `config_id` hint changes from 32B to 8B, trial decryption added on
the server.
- Duplicate code in HRR cookie handling has been consolidated into
`tls13_HandleHrrCookie`.
- `ech_is_inner` extension is added, which causes a server to indicate
ECH acceptance.
- Per the above, support signaling ECH acceptance when acting as a
backend server in split-mode (i.e. when there is no other local
Encrypted Client Hello state).
[ed07a2e2a124]
2021-01-24 Kevin Jacobs <kjacobs@mozilla.com>
* cmd/selfserv/selfserv.c:
Bug 1681585 - Add ECH support to selfserv. r=mt
Usage example: mkdir dbdir && cd dbdir certutil -N -d . certutil -S
-s "CN=ech-public.com" -n ech-public.com -x -t "C,C,C" -m 1234 -d .
certutil -S -s "CN=ech-private-backend.com" -n ech-private-
backend.com -x -t "C,C,C" -m 2345 -d . ../dist/Debug/bin/selfserv -a
ech-public.com -a ech-private-backend.com -n ech-public.com -n ech-
private-backend.com -p 8443 -d dbdir/ -X publicname:ech-public.com
(Copy echconfig from selfserv output and paste into the below
command) ../dist/Debug/bin/tstclnt -D -p 8443 -v -A
tests/ssl/sslreq.dat -h ech-private-backend.com -o -N <echconfig> -v
[92dcda94c1d4]
Differential Revision: https://phabricator.services.mozilla.com/D102982
For simplicity, this implements just on in `NO_TASKS` (the default) or
on in `ALL_TASKS` (opt-in). This disables all category registrations
when in background task mode; we'll selectively re-enable things as
appropriate.
The flag constants were chosen to smoothly extend to a (16-)bit set in
the future, should we want to add a `JUST_TASKS("task", "other-task")`
option in the future.
This also adds ython tests for gen_static_components.py exercising
categories, simply 'cuz it's easiest to see what this adds in such
tests. Functional tests will follow in patches that actually
implement the new background tasks functionality.
Differential Revision: https://phabricator.services.mozilla.com/D96654
remoteautomation.py is an old collection of code used by android mochitest and android reftest;
it survived the removal of automation.py. This patch removes remoteautomation.py, moving the
majority of the functionality to a new class in mozdevice. Some features are simplified or
removed, and the remainder moved into the remote mochitest/reftest harnesses.
Differential Revision: https://phabricator.services.mozilla.com/D102239
This adds a --enable-bootstrap build flag that will automatically update
cbindgen, node, clang, sccache, nasm, wine, lucetc, dump_syms, pdbstr,
and winchecksec if they are already installed in ~/.mozbuild.
Eventually, we'll want to allow to install toolchains that weren't
already install, but one step at a time.
This explicitly doesn't cover rustc, which is its own can of worms, or
android-{ndk,sdk}, which are not installed via toolchain artifacts
currently.
Differential Revision: https://phabricator.services.mozilla.com/D101723
Instead of adding all possible tool paths from ~/.mozbuild, we only
add the relevant paths for each of the tools we search for.
Differential Revision: https://phabricator.services.mozilla.com/D101718
The only thing that varies between toolchain_search_path and
host_toolchain_search_path is the path to the MSVC C/C++ compiler and
tools, because MSVC has a different compiler for each platform, and host
and target platforms may differ (when e.g. compiling for arm64 on
x86_64).
However, we don't use the MSVC compiler anymore, and the only thing we
use its path for is the assembler, which we don't use for host things
(and we don't have a HOST_AS), and to derive the path to some system
headers/SDK.
Differential Revision: https://phabricator.services.mozilla.com/D101714
There is only one place where it's used:
config/check_vanilla_allocations.py, which is only executed from
js/src/build/Makefile.in on the condition that the build is targeting
Linux and not LTO. But the LTO test is actually outdated, because we
don't build with `-flto`, but `-flto=thin`, so the exclusion doesn't
work anymore.
There is however no AC_CHECK_PROG, and we currently rely on NM to be
given, or fall back to "nm", which works in most cases, except LTO with
clang. It works on CI because in LTO builds we explicitly set NM to
llvm-nm (which can output symbols from LLVM bitcode objects), but we
could also do that automatically.
So we add a full detection of nm/llvm-nm to python configure, and limit
it to Linux, since we only ever use it there.
Differential Revision: https://phabricator.services.mozilla.com/D101681
js/src/aclocal.m4 contains includes starting with `../../`.
As explained in https://bugzilla.mozilla.org/show_bug.cgi?id=1680862#c8,
m4 will first try to resolve this path relative to the working directory
and only if that path doesn't exists, fall back to the location set by
`localdir` (from `-I`).
The working directory is usually MOZ_OBJDIR, an immediate subdirectory
of topsrcdir, so `../../` resolves to a location outside of topsrcdir.
Usually, that path does not exist, and m4 falls back to `localdir` that
was passed via `-I`.
But if that path existed and is incompatible with the current Gecko
checkout, then the build will fail (see bug report). To prevent this
from happening, this patch fixes the working directory to `localdir`,
so that m4 will immediately find the expected file.
Differential Revision: https://phabricator.services.mozilla.com/D101500
I missed these in bug 1682405.
Additionally, after this removal, llvmorg-10-init-5191-ga84b200e604-windows-pgo.patch also becomes unused, so it is deleted too.
Differential Revision: https://phabricator.services.mozilla.com/D101633
glean_parser depends on `iso8601`, but only on Python <= 3.6.
Add the `iso8601` hash to our frozen dependency hash list accordingly.
Differential Revision: https://phabricator.services.mozilla.com/D101494
In addition to warning on regular methods overloading virtual functions,
GCC also watches for static functions doing such overloads.
:andi confirmed that this is not valuable, so the warning is being
disabled for GCC.
Differential Revision: https://phabricator.services.mozilla.com/D101367
Only sanitizer builds require a native llvm-symbolizer executable.
Ideally, we'd build llvm-symbolizer from scratch, which would be faster,
but for now, let's go the easy route and just extract it from the
corresponding native clang builds.
We don't actually do anything with the llvm-symbolizer executable on
android builds, so we don't install it in $FINAL_TARGET, avoilding
the dependency on android builds (plus, we actually don't have an
android-native llvm-symbolizer, so even if it were already shipped, it
would be the wrong file).
Differential Revision: https://phabricator.services.mozilla.com/D101076
Passing `-flto=thin` worked previously but the value passed was just ignored
and full lto was performed. On newer versions of gcc passing an unknown value
causes failure. So this commit checks if `-flto=thin` is passed and fails with
an error message if so, else full lto is enabled if any other value is passed.
Differential Revision: https://phabricator.services.mozilla.com/D100953
Only sanitizer builds require a native llvm-symbolizer executable.
Ideally, we'd build llvm-symbolizer from scratch, which would be faster,
but for now, let's go the easy route and just extract it from the
corresponding native clang builds.
We don't actually do anything with the llvm-symbolizer executable on
android builds, so we don't install it in $FINAL_TARGET, avoilding
the dependency on android builds (plus, we actually don't have an
android-native llvm-symbolizer, so even if it were already shipped, it
would be the wrong file).
Differential Revision: https://phabricator.services.mozilla.com/D101076
In addition to the usual dot-release type of fixes, this also lets us drop a good amount of code that we had patched into our clang 11.
Differential Revision: https://phabricator.services.mozilla.com/D100959
Some distros include flags when they specify the location of a binary,
such as: `XARGS=xargs -r`.
This was confusing in `configure`, since:
* We require that environment variables contain only path
overrides (without flags).
* The error message thrown when configure chokes was unclear: "why
would `$ xargs -r` fail?"
This patch should make our "path-only" requirement more clear.
Differential Revision: https://phabricator.services.mozilla.com/D100044
Old clang shakes its fist when `auto&& item : range` is used with a
range
that returns values instead of references.
Modern `clang` doesn't warn for this scenario, so we disable the
warning.
Also removes pragmas that manually disable this warning.
Differential Revision: https://phabricator.services.mozilla.com/D100155
I was waiting for a better reason to do this, because the cbindgen
changes from 0.15.0 to 0.16.0 don't break trunk builds. But since
downstream has updated (see bug 1684180) and there's no reason not to,
let's do this to avoid future churn.
Differential Revision: https://phabricator.services.mozilla.com/D100499
2020-12-11 Kevin Jacobs <kjacobs@mozilla.com>
* automation/abi-check/expected-report-libssl3.so.txt, automation/abi-
check/previous-nss-release, lib/nss/nss.h, lib/softoken/softkver.h,
lib/util/nssutil.h:
Set version numbers to 3.61 Beta
[f277d2674c80]
* gtests/<...>
Bug 1677207 - Update Google Test to release-1.10.0 r=bbeurdouche
./gtests/google_test/update.sh release-1.10.0 && hg remove -A && hg
add gtests/google_test/*
[89141382df45]
* gtests/<...>
Bug 1677207 - Replace references to TestCase, which is deprecated,
with TestSuite r=bbeurdouche
grep -rl --exclude-dir=google_test INSTANTIATE_TEST_CASE_P gtests |
xargs sed -i '' s/INSTANTIATE_TEST_CASE_P/INSTANTIATE_TEST_SUITE_P/g
grep -rl --exclude-dir=google_test SetUpTestCase gtests | xargs sed
-i '' s/SetUpTestCase/SetUpTestSuite/g
[e15b78be87fa]
* gtests/ssl_gtest/ssl_ciphersuite_unittest.cc,
gtests/ssl_gtest/ssl_debug_env_unittest.cc,
gtests/ssl_gtest/ssl_extension_unittest.cc,
gtests/ssl_gtest/ssl_loopback_unittest.cc,
gtests/ssl_gtest/ssl_renegotiation_unittest.cc,
gtests/ssl_gtest/ssl_resumption_unittest.cc,
gtests/ssl_gtest/ssl_version_unittest.cc,
gtests/ssl_gtest/tls_ech_unittest.cc:
Bug 1677207 - Use GTEST_SKIP in ssl_gtests. r=bbeurdouche
[0772f1bf5fd6]
2020-12-17 Robert Relyea <rrelyea@redhat.com>
* gtests/common/testvectors/ike-aesxcbc-vectors.h,
gtests/common/testvectors/ike-sha1-vectors.h,
gtests/common/testvectors/ike-sha256-vectors.h,
gtests/common/testvectors/ike-sha384-vectors.h,
gtests/common/testvectors/ike-sha512-vectors.h,
gtests/common/testvectors_base/test-structs.h,
gtests/pk11_gtest/manifest.mn, gtests/pk11_gtest/pk11_gtest.gyp,
gtests/pk11_gtest/pk11_ike_unittest.cc, lib/softoken/sftkike.c:
Bug 1682071 IKE Quick mode IPSEC give you incorrect keys if you are
asking for keys smaller than the hash size.
IKE Appendix B fixes.
This patch fixes 2 problems.
If you run either ike v1 App B or quick mode asking for a key with
length
mod macsize = 0, you will generate an extra block that's not used
and overwrites the end of the buffer.
If you use quick mode, the function incorrectly subsets the
existing key
rather than generating a new key. This is correct behavior for
Appendix B, where appendix B is trying to take a generated key and
create a new longer key (with no diversification, just transform the
key into something that's longer), so if you ask for a key less than
or equal to, then you want to just subset the original key. In quick
mode you are taking a base key and creating a set of new keys based
on additional data, so you want to subset the generated data. This
patch only subsets the original key if you aren't doing quickmode.
Full test vectors have now been added for all ike modes in this
patch as well (previously we depended on the FIPS CAVS tests to test
ike, which covers basic IKEv1, IKEv1_psk, and IKEv2 but not IKEv1
App B and IKE v1 Quick mode).
[f4995c9fa185]
2020-12-18 Robert Relyea <rrelyea@redhat.com>
* gtests/common/testvectors/rsa_pkcs1_2048_test-vectors.h,
gtests/common/testvectors/rsa_pkcs1_3072_test-vectors.h,
gtests/common/testvectors/rsa_pkcs1_4096_test-vectors.h,
gtests/freebl_gtest/Makefile, gtests/freebl_gtest/manifest.mn,
gtests/freebl_gtest/rsa_unittest.cc, gtests/manifest.mn,
gtests/pk11_gtest/pk11_rsaencrypt_unittest.cc,
gtests/pk11_gtest/pk11_rsaoaep_unittest.cc, lib/freebl/alghmac.c,
lib/freebl/alghmac.h, lib/freebl/rsapkcs.c:
Bug 1651411 New tlsfuzzer code can still detect timing issues in RSA
operations.
This patch defeats Bleichenbacher by not trying to hide the size of
the decrypted text, but to hide if the text succeeded for failed.
This is done by generating a fake returned text that's based on the
key and the cipher text, so the fake data is always the same for the
same key and cipher text. Both the length and the plain text are
generated with a prf.
Here's the proposed spec the patch codes to:
1. Use SHA-256 to hash the private exponent encoded as a big-
endian integer to a string the same length as the public modulus.
Keep this value secret. (this is just an optimisation so that the
implementation doesn't have to serialise the key over and over
again) 2. Check the length of input according to step one of
https://tools.ietf.org/html/rfc8017#section-7.2.2 3. When provided
with a ciphertext, use SHA-256 HMAC(key=hash_from_step1,
text=ciphertext) to generate the key derivation key 4. Use SHA-256
HMAC with key derivation key as the key and a two-byte big- endian
iterator concatenated with byte string "length" with the big- endian
representation of 2048 (0x0800) as the bit length of the generated
string.
- Iterate this PRF 8 times to generate a 256 byte string 5. initialise
the length of synthetic message to 0 6. split the PRF output into 2
byte strings, convert into big-endian integers, zero- out high-order
bits so that they have the same bit length as the octet length of
the maximum acceptable message size (k-11), select the last integer
that is no larger than (k-11) or remain at 0 if no integer is
smaller than (k-11); this selection needs to be performed using a
side-channel free operators 7. Use SHA-256 HMAC with key derivation
key as the key and a two-byte big-endian iterator concatenated with
byte string "message" with the big-endian representation of k*8
- use this PRF to generate k bytes of output (right-truncate last HMAC
call if the number of generated bytes is not a multiple of SHA-256
output size) 8. perform the RSA decryption as described in step 2 of
section 7.2.2 of rfc8017 9. Verify the EM message padding as
described in step 3 of section 7.2.2 of rfc8017, but instead of
outputting "decryption error", return the last l bytes of the
"message" PRF, when l is the selected synthetic message length using
the "length" PRF, make this decision and copy using side-channel
free operation
[fc05574c7399]
2020-12-22 Robert Relyea <rrelyea@redhat.com>
* gtests/freebl_gtest/rsa_unittest.cc,
gtests/pk11_gtest/pk11_rsaoaep_unittest.cc, lib/freebl/alghmac.c,
lib/freebl/rsapkcs.c:
Restore lost portion of the bleichenbacher timing batch that
addressed review comments. All the review comments pertained to
actual code comments, so this patch only affects the comments.
[fcebe146314e]
2020-12-22 Kevin Jacobs <kjacobs@mozilla.com>
* lib/dev/devslot.c:
Bug 1682863 - Revert nssSlot_IsTokenPresent to 3.58 after ongoing Fx
hangs with slow PKCS11 devices. r=bbeurdouche
This patch reverts the `nssSlot_IsTokenPresent` changes made in bug
1663661 and bug 1679290, restoring the version used in NSS 3.58 and
earlier. It's not an actual `hg backout` because the comment in
lib/dev/devt.h is worth keeping. While removing the nested locking
did resolve the hang for some (most?) third-party modules, problems
remain with some slower tokens after an even further relaxation of
the locking, which defeats the purpose of addressing the races in
the first place.
The crash addressed by these patches was caused by the Intermediate
Preloading Healer in Firefox, which has been disabled. We clearly
have insufficient test coverage for third-party modules, and now
that osclientcerts is enabled in Fx Nightly, any problems caused by
these and similar changes is unlikely to be reported until Fx Beta,
well after NSS RTM. I think the best option at this point is to
simply revert NSS.
[97ef009f7a78] [tip]
Differential Revision: https://phabricator.services.mozilla.com/D100401
Since bug 1632542 we have a new major clang version, several new versions worth of rustc, new pass manager during LTO, and of course eight months worth of code change. Time for a refresh. Same process as the earlier bug.
Differential Revision: https://phabricator.services.mozilla.com/D99243
At the moment installing Python packages with native code is done by calling
`pip install <package>` and does not enforce any SHA hash for installed
dependencies, nor does it enforce a specific version to be installed.
This commit adds `requirements.in` and `requirements.txt` files for native
packages and changes these packages to be installed by running `pip install`
and passing the requirements file for the package. This allows us to pin the
SHA of the various dependencies. The `.txt` files are generated using
`pip-compile`.
We also add the new requirements files to the sparse profile for `mach`.
Differential Revision: https://phabricator.services.mozilla.com/D99912
In clang 11, the contents of `ast_type_traits` were moved to the `clang` namespace, but forwarders were kept to give a grace period for downstreams.
In this week's clang 12, the forwarders have been removed, and our builds will break.
Thanks to upstream kindly allowed this transition period, we can simply do the rename now to prepare for clang 12, and existing clang 11 builds still work.
Differential Revision: https://phabricator.services.mozilla.com/D99817
At this point it's pretty clear that we won't be reverting to clang-9.
This doesn't remove everything with clang-9 in the name. I did a mark-and-sweep GC by hand so this only removes unused entries. Some are still active, e.g. linux64-clang-9 is used to build a number of misc helper tools.
Differential Revision: https://phabricator.services.mozilla.com/D99721
Covers adding the new JS global `GleanPings` for JS, the new structs for C++ at
mozilla::glean_pings, ping-id and string-table-index codegen, the usual
boilerplate for JS and C++ stuff, and tests.
Unresolved:
* What happens if we call this on a non-parent process?
(This isn't a supported mode of operation)
Differential Revision: https://phabricator.services.mozilla.com/D98671
The gyp flag logic in nICEr is supposed to ensure that the code is instrumented
for libFuzzer because we have a related fuzzing target. However, libFuzzer
instrumentation must be completely disabled for TSan due to incompatibility.
The current logic fails in doing so and incorrectly falls back to legacy
trace-pc instrumentation causing the TSan fuzzing build to fail on startup.
Differential Revision: https://phabricator.services.mozilla.com/D99351
This is bug 1639318 all over again except from Rust rather than C++. It's the same symptom, nsXPTCStub vtables aren't marked as valid targets.
In the earlier bug we disabled CFG for C++ on ARM64. Let's do the same for Rust. According to that bug, "It's not clear why this doesn't happen on x86 builds. Given priorities, I can't really justify investigating this, although I suspect that fixing the underlying issue would be pretty much bug 1483885."
Differential Revision: https://phabricator.services.mozilla.com/D99278
Bug 1680152 updated automation to use the 10.12 SDK, and shortly after,
bug 1678174 introduced a change that broke the build with the 10.11 SDK.
Considering we haven't actually supported running on macos 10.11 and
earlier since Firefox 79, and that still supporting building with the
10.11 SDK would mean adding and maintaining code that, in practice,
would never be used by users, I think it is fair at this point that we
just drop support for the 10.11 SDK entirely.
Differential Revision: https://phabricator.services.mozilla.com/D99181
This is step 1 to make the docs here better, as discussed. Another followup
will come in a day or two to update the text/links of the docs to refer to
this and the latest version of repack-rust.
Differential Revision: https://phabricator.services.mozilla.com/D99128
In Bug 1671424 we changed the way we configure firefox and it broke the codeql database
generation job. This job wraps the entire build process in a way similar to
codeql --command="./mach build"
Specifically, the previous way we executed the configure shell script made codeql
disable itself because it was named configure (codeql disables itself during
configuration.)
codeql injects via LD_PRELOAD, and it opens a configuration file and a logging file
(getting fd 3 and 4 respectively.)
autoconf grabs file descriptor 4 and uses it a temporary redirection point either
to a file or stdout. When it does so, it closes the original file descriptor 4 and
points it at the new location, which also affects the codeql code, resulting in
undesired logging output going into the configure script.
Because this file descriptor trick is only used to avoid duplicating a few lines of
code, I removed the trick and duplicated the code.
Differential Revision: https://phabricator.services.mozilla.com/D98642
This matches the other paths further down in the same document.
I'm also adding some quotes because I was getting an "unknown target" error without them, but I'm not sure if the missing quotes were the reason or whether it just happened to succeed on the next try.
Depends on D98673
Differential Revision: https://phabricator.services.mozilla.com/D98674
On a very parallel debug build, I see a long time just waiting for
bindgen / style compilation / geckoservo.
Turns out that a bunch of this is just proc macros / build scripts.
Optimizing it saves between 10 and 17 seconds of my debug build. We
might want to consider running bindgen much like cbindgen rather than
rebuilding it all the time, which should help a lot more, but my guess
is that this should still help with the pretty hot custom derives that
the style crate runs.
This needs rust 1.41, so the requirement for tools/crashreporter needs
to be bumped as a consequence. To make things simpler, it was bumped
to 1.47 while we're at it.
Differential Revision: https://phabricator.services.mozilla.com/D98366
Windows being Windows, with its own spellings for everything, let's stay within the WINNT block regardless of whether we meet the compiler requirement.
Differential Revision: https://phabricator.services.mozilla.com/D98355
Don't fail to run all mach commands when old psutil directory
cannot be removed.
Glandium mentioned that there shouldn't be any negative effects from the
old directory lingering.
Differential Revision: https://phabricator.services.mozilla.com/D98352
If we encounter a single error in glxtest, we typically bail immediately
with one liner message. This patch makes it put more effort into
returning what information it is able to, as well as the current error
messages. If certain errors are correlated to specific devices, it would
be useful if we had the information to make the connection.
Differential Revision: https://phabricator.services.mozilla.com/D97861
Most of the deletions here come from bug 1481612, the `--with-windows-wheel` option to `mach vendor python`, which according to that commit message "is very single-purpose: it's intended to let us vendor an unpacked
wheel for psutil on Windows". Since vendoring `psutil` is something we're no longer doing, we can safely just delete that added code.
Differential Revision: https://phabricator.services.mozilla.com/D90919
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
Code review followups for 1675600: Restrict the use of new pass manager during LTO to builds where we're using the new pass manager in general, and (on Windows) where lld-link is new enough to understand the flag.
Differential Revision: https://phabricator.services.mozilla.com/D97372
Upstream clang supports --target=aarch64-apple-darwin, and that matches
what config.sub canonicalizes to, and thus what the default toolchain
prefix ends up being, so it's better to use aarch64-apple-darwin when
not compiling with Xcode clang.
Differential Revision: https://phabricator.services.mozilla.com/D97698
There is no earlier SDK that supports it. It seems Xcode clang doesn't
care (maybe it defaults to 11.0 is MACOSX_DEPLOYMENT_TARGET is too low?),
but upstream clang does.
Differential Revision: https://phabricator.services.mozilla.com/D97565
And don't set it via mozconfig. The default to /System/Library/PrivateFrameworks
may also not have matched the used SDK previously, so the new default is
better.
Differential Revision: https://phabricator.services.mozilla.com/D97564
2020-11-18 Kevin Jacobs <kjacobs@mozilla.com>
* lib/ssl/ssl3con.c, lib/ssl/tls13con.c, lib/ssl/tls13ech.c:
Bug 1654332 - Fixup a10493dcfcc9: copy ECHConfig.config_id with
socket r=jcj
A late review change for ECH was for the server to compute each
ECHConfig `config_id` when set to the socket, rather than on each
connection. This works, but now we also need to copy that config_id
when copying a socket, else the server won't find a matching
ECHConfig to use for decryption.
[3eacb92e9adf] [tip]
2020-11-17 Kevin Jacobs <kjacobs@mozilla.com>
* automation/abi-check/expected-report-libssl3.so.txt,
cmd/tstclnt/tstclnt.c, cpputil/tls_parser.h,
gtests/ssl_gtest/libssl_internals.c,
gtests/ssl_gtest/libssl_internals.h, gtests/ssl_gtest/manifest.mn,
gtests/ssl_gtest/ssl_auth_unittest.cc,
gtests/ssl_gtest/ssl_custext_unittest.cc,
gtests/ssl_gtest/ssl_extension_unittest.cc,
gtests/ssl_gtest/ssl_gtest.gyp,
gtests/ssl_gtest/ssl_tls13compat_unittest.cc,
gtests/ssl_gtest/tls_agent.cc, gtests/ssl_gtest/tls_agent.h,
gtests/ssl_gtest/tls_connect.cc, gtests/ssl_gtest/tls_connect.h,
gtests/ssl_gtest/tls_ech_unittest.cc,
gtests/ssl_gtest/tls_esni_unittest.cc,
gtests/ssl_gtest/tls_filter.cc, gtests/ssl_gtest/tls_filter.h,
lib/ssl/SSLerrs.h, lib/ssl/manifest.mn, lib/ssl/ssl.gyp,
lib/ssl/ssl3con.c, lib/ssl/ssl3ext.c, lib/ssl/ssl3ext.h,
lib/ssl/ssl3exthandle.c, lib/ssl/ssl3exthandle.h,
lib/ssl/ssl3prot.h, lib/ssl/sslencode.c, lib/ssl/sslencode.h,
lib/ssl/sslerr.h, lib/ssl/sslexp.h, lib/ssl/sslimpl.h,
lib/ssl/sslinfo.c, lib/ssl/sslsecur.c, lib/ssl/sslsock.c,
lib/ssl/sslt.h, lib/ssl/tls13con.c, lib/ssl/tls13con.h,
lib/ssl/tls13ech.c, lib/ssl/tls13ech.h, lib/ssl/tls13esni.c,
lib/ssl/tls13esni.h, lib/ssl/tls13exthandle.c,
lib/ssl/tls13exthandle.h, lib/ssl/tls13hashstate.c,
lib/ssl/tls13hashstate.h:
Bug 1654332 - Update ESNI to draft-08 (ECH). r=mt
This patch adds support for Encrypted Client Hello (draft-ietf-tls-
esni-08), replacing the existing ESNI (draft -02) support.
There are five new experimental functions to enable this:
- SSL_EncodeEchConfig: Generates an encoded (not BASE64) ECHConfig
given a set of parameters.
- SSL_SetClientEchConfigs: Configures the provided ECHConfig to the
given socket. When configured, an ephemeral HPKE keypair will be
generated for the CH encryption.
- SSL_SetServerEchConfigs: Configures the provided ECHConfig and
keypair to the socket. The keypair specified will be used for HPKE
operations in order to decrypt encrypted Client Hellos as they are
received.
- SSL_GetEchRetryConfigs: If ECH is rejected by the server and
compatible retry_configs are provided, this API allows the
application to extract those retry_configs for use in a new
connection.
- SSL_EnableTls13GreaseEch: When enabled, non-ECH Client Hellos will
have a "GREASE ECH" (i.e. fake) extension appended. GREASE ECH is
disabled by default, as there are known compatibility issues that
will be addressed in a subsequent draft.
The following ESNI experimental functions are deprecated by this
update:
- SSL_EncodeESNIKeys
- SSL_EnableESNI
- SSL_SetESNIKeyPair
In order to be used, NSS must be compiled with
`NSS_ENABLE_DRAFT_HPKE` defined.
[a10493dcfcc9]
* lib/ssl/ssl3con.c, lib/ssl/sslencode.c, lib/ssl/sslencode.h,
lib/ssl/tls13con.c, lib/ssl/tls13con.h:
Bug 1654332 - Buffered ClientHello construction. r=mt
This patch refactors construction of Client Hello messages. Instead
of each component of the message being written separately into
`ss->sec.ci.sendBuf`, we now construct the message in its own
sslBuffer. Once complete, the entire message is added to the sendBuf
via `ssl3_AppendHandshake`.
`ssl3_SendServerHello` already uses this approach and it becomes
necessary for ECH, where we use the constructed ClientHello to
create an inner ClientHello.
[d40121ba59ba]
2020-11-13 J.C. Jones <jjones@mozilla.com>
* automation/abi-check/expected-report-libnss3.so.txt, automation/abi-
check/expected-report-libnssutil3.so.txt, automation/abi-check
/previous-nss-release, lib/nss/nss.h, lib/softoken/softkver.h,
lib/util/nssutil.h:
Set version numbers to 3.60 Beta
[5e7b37609f22]
Differential Revision: https://phabricator.services.mozilla.com/D97492
Sometimes thread names bit rot in the list because there is no checker in place to ensure they are still active. The threads I removed have either:
-Been converted to use the background thread pool
-Been removed entirely
-Been renamed
And they no longer require entries in the list.
Differential Revision: https://phabricator.services.mozilla.com/D96846
The SEH unwind info problem no longer happens because nowadays bug 1631929 prevents the code pattern that led to it. I've confirmed that bug 1626951's bustage doesn't come back, both in regular and beta-simulation builds.
This cleanup will allow me to reverse the dependency in bug 1677742 and have `lto` depend on `new_pass_manager_flags`.
Differential Revision: https://phabricator.services.mozilla.com/D97362
Currently, we report "10.0.0.or.more" for the latest release of Xcode,
but we know it's 10.0.0 for sure, which is when the clang version
reported by Xcode is exactly 12.0.0. We know that in the past a bump of
LLVM has always been accompanied with a bump of the minor version, so we
expect no LLVM version bump until at least 12.0.1.
Differential Revision: https://phabricator.services.mozilla.com/D97241
Currently, in LTO builds, we use the new pass manager during the initial translation to bitcode but not for the final optimization during linking.
On Linux, we can enable the new pass manager during LTO with a plugin option. I've landed a patch upstream to allow it on Windows as well, which is included here.
Switching the pass manager brings speed improvements on its own, but it also reduces code size by ~6%, which we can use a portion of as budget to increase the import limit (via the hot multiplier) for even more speed improvements.
Differential Revision: https://phabricator.services.mozilla.com/D96108
Most of the deletions here come from bug 1481612, the `--with-windows-wheel` option to `mach vendor python`, which according to that commit message "is very single-purpose: it's intended to let us vendor an unpacked
wheel for psutil on Windows". Since vendoring `psutil` is something we're no longer doing, we can safely just delete that added code.
Differential Revision: https://phabricator.services.mozilla.com/D90919
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
CLOSED TREE
Backed out changeset baa1f7a615e4 (bug 1666345)
Backed out changeset b6646baa866d (bug 1661624)
Backed out changeset e4d550db6037 (bug 1667152)
Most of the deletions here come from bug 1481612, the `--with-windows-wheel` option to `mach vendor python`, which according to that commit message "is very single-purpose: it's intended to let us vendor an unpacked
wheel for psutil on Windows". Since vendoring `psutil` is something we're no longer doing, we can safely just delete that added code.
Differential Revision: https://phabricator.services.mozilla.com/D90919
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
Most of the deletions here come from bug 1481612, the `--with-windows-wheel` option to `mach vendor python`, which according to that commit message "is very single-purpose: it's intended to let us vendor an unpacked
wheel for psutil on Windows". Since vendoring `psutil` is something we're no longer doing, we can safely just delete that added code.
Differential Revision: https://phabricator.services.mozilla.com/D90919
Install `psutil` when setting up the `mach` `virtualenv`s and stop importing the in-tree version in the build.
Nothing in-tree currently assumes or mandates the installation of `psutil` (all uses of `psutil` are guarded with imports of the form `try : import psutil; except ImportError: psutil = None`), so there's no back-incompatibility concerns here. There will be an awkward period where telemetry will be lacking CPU/disk data for everyone until they re-run `mach bootstrap` or `mach create-mach-environment`, but that will come back as people gradually update their `virtualenv`s.
An alternative to circumvent that issue is REQUIRING that `psutil` be installed by adding an assertion in `mach` that `psutil` can be found (allowing us to remove all the conditional logic in-tree around whether `psutil` is installed), but I wouldn't claim that we're ready to do that and deal with whatever fallout might occur.
Differential Revision: https://phabricator.services.mozilla.com/D90914
Some of the surrounding lines recently changed so the patch refuses to apply. We can live with less context so I'd rather trim it than make a new clang-12 version of this patch.
Differential Revision: https://phabricator.services.mozilla.com/D96725
With PYTHONPATH containing other directories, such as the python ones,
bad things can happen when the python script that is being run then
goes on to subprocess.Popen a python process for a different virtualenv,
or a different version, or whatever.
Differential Revision: https://phabricator.services.mozilla.com/D96155
This supports one manifestparser expression per line in the 'skip-if',
'fail-if' and 'run-if' keys. As a side effect the:
skip-if = foo ||
bar
syntax is no longer supported. Instead it can be:
skip-if =
foo # bug 123
bar # bug 456
Differential Revision: https://phabricator.services.mozilla.com/D95927
This introduces a new mangled symbol of the form FILE_<hash>. It's defined
on line 1 of each visited source file, and it's referenced in places that
the preprocessor tells us there was an #include. Both angle-bracket includes
and quote includes are supported here.
Differential Revision: https://phabricator.services.mozilla.com/D95529
In the next patch I'll want to be able to provide the entire source rather than
having visitIdentifier just use the token at `Loc`. The existing call sites
create a SourceRange from the SourceLocation and indicate (by not setting the
LocRangeEndValid flag) that the visitIdentifier function should retain it's
old behaviour of using the token to figure out the range.
Depends on D95527
Differential Revision: https://phabricator.services.mozilla.com/D95528
This extracts a relativizePath function that normalizes a file path to be
relative to the source/objdir, and returns the type of the file.
Differential Revision: https://phabricator.services.mozilla.com/D95527
I'll need to update this file for clang-12, and in order to prevent the need for adding more of these ifdefs, let's just assume they are true and remove the checks. Our codebase has been on C++17 for a while now.
Differential Revision: https://phabricator.services.mozilla.com/D95592
Csoregi Natalia
Calixte Denizet
Mike Hommey
David Major
Cosmin Sabou
Tom Ritter
Kevin Jacobs
Nick Alexander
Daniel Le
Geoff Brown
Rob Wu
Aki Sasaki
Mitchell Hentges
Ryan VanderMeulen
Connor Sheehan
Razvan Maries
Sylvestre Ledru
Kartik Gautam
Markus Stange
Andi-Bogdan Postelnicu
Bogdan Tara
Emilio Cobos Álvarez
James Graham
Tarek Ziadé
Sebastian Hengst
Dave Townsend
Chris H-C
Dragana Damjanovic
Dimi Lee
Dorel Luca
Christian Holler
Butkovits Atila
Edmund Wong
Alexis Beingessner
Andrew Osmond
Ricky Stewart
Kris Wright
Noemi Erli
Mihai Alexandru Michis
smolnar
Brindusan Cristian
Andrew Halberstadt
Steve Fink
Kartikaya Gupta