I think they're remnants from the past that we don't really need anymore.
And they're making things more complicated for some pending work of mine.
Differential Revision: https://phabricator.services.mozilla.com/D89687
resolver_64.cc should have been on the list all along, because the InternalThunk
constructor runs before ASan init. It was probably just accident (maybe inlining?)
that we got away with not including it in the past.
Differential Revision: https://phabricator.services.mozilla.com/D89669
In addition to the existing build telemetry, also gather the stats and
report with Glean. This new telemetry is reported in tandem with the existing
telemetry to allow testing and confidence before a full roll-out.
Additionally, Glean isn't compatible with Python 2, so the new telemetry only runs
on Python 3 mach commands.
Differential Revision: https://phabricator.services.mozilla.com/D83572
The files are copied verbatim from upstream autoconf 2.13 (but only the
files we need) and old.configure is adapted to use the vendored version.
Differential Revision: https://phabricator.services.mozilla.com/D89554
This was originally meant to allow `virtualenv`s to use packages from a parent Python environment without having to re-install them. This turned out to not pan out as we would have liked, so we're going another way to solve the same problem. Bug 1660351 walked back a bunch of this logic; this patch deletes the rest of it.
Differential Revision: https://phabricator.services.mozilla.com/D89492
This commit does the following.
- Renames `slashslash` as `dumbComments`. As a result, it now comes before
`emptyLines` in alphabetical ordering, which means that if you apply both
`dumbComments` and `emptyLines`, lines that contain only comments will be
fully removed.
(I contemplated changing the filter ordering to match the order specified,
rather than using alphabetical ordering, but that was more invasive and not
obviously better.)
- Changes `dumbComments` so it only applies if the comment is at the start of
the line (with optional leading whitespace). This is so it can be used with
prefs files, which contain lines like `pref("foo", "https://mozilla.org");`
where the `//` must not be treated as a comment.
Note that `slashslash` wasn't being used anywhere.
Depends on D88240
Differential Revision: https://phabricator.services.mozilla.com/D88242
It's not used, probably because it's pretty strange and hard to imagine using
safely. (Stripping leading and trailing space could be useful, but collapsing
sequences of spaces? Hmm.)
Differential Revision: https://phabricator.services.mozilla.com/D88240
When configuring without system NSPR the configuration variable
PKGCONF_REQUIRES_PRIVATE isn't ever set, leading to a .pc file that still
contains the @PKGCONF_REQUIRES_PRIVATE@ stub.
Ensure that we always define PKGCONF_REQUIRES_PRIVATE, by setting it to an
empty string in case no system-nsrp is enabled.
In this way, the pkg-config file stub will be always replaced.
Differential Revision: https://phabricator.services.mozilla.com/D88179
widl output for dlldata has #defines immediately followed by #includes,
so looking for empty lines when we observer a #define doesn't work. We
instead look for #defines.
Differential Revision: https://phabricator.services.mozilla.com/D88938
The work in bug 1620133 ended up moving the execution of some of the
midl commands to the top-level, which was alleviated by adding the -out
midl command line option to make midl place its output in the expected
directory. Unfortunately, that option is not handled by widl, which is
the alternative command used in mingw builds.
So instead of using -out, we set the cwd for the midl command, and
readjust the command line arguments to be relative to that.
Differential Revision: https://phabricator.services.mozilla.com/D88937
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
We don't anticipate end users will actually care to do this, but it's useful especially for unit tests. For example, after bug 1659539, Python `configure` tests will run in a new, non-`init_py3` `virtualenv`, and we'll want to target that `virtualenv` for `configure` rather than having it create a new `virtualenv` for no reason.
Differential Revision: https://phabricator.services.mozilla.com/D88661
2020-08-21 Kevin Jacobs <kjacobs@mozilla.com>
* automation/abi-check/previous-nss-release, lib/nss/nss.h,
lib/softoken/softkver.h, lib/util/nssutil.h:
Set version numbers to 3.57 Beta
[783f49ae6126]
2020-08-24 Kevin Jacobs <kjacobs@mozilla.com>
* gtests/ssl_gtest/ssl_auth_unittest.cc, lib/ssl/dtls13con.c,
lib/ssl/dtlscon.c, lib/ssl/ssl3con.c, lib/ssl/sslimpl.h,
lib/ssl/sslnonce.c:
Bug 1653641 - Cleanup inaccurate DTLS comments, code review fixes.
r=mt
[0e1b5c711cb9]
2020-08-24 Robert Relyea <rrelyea@redhat.com>
* lib/freebl/fipsfreebl.c, lib/softoken/fipstest.c,
lib/softoken/kbkdf.c, lib/softoken/lowpbe.c, lib/softoken/lowpbe.h,
lib/softoken/pkcs11c.c, lib/softoken/pkcs11i.h,
lib/softoken/sftkhmac.c, lib/softoken/sftkike.c:
Bug 1660304 New FIPS IG requires self-tests for approved kdfs.
r=ueno comments=kjacobs
FIPS guidance now requires self-tests for our kdfs. It also requires
self-tests for cmac which we didn't have in the cmac patch.
Currently only one test per kdf is necessary. Specifially for
SP-800-108, only one of the three flavors are needed (counter,
feedback, or pipeline). This patch includes more complete testing
but it has been turned off the currently extraneous tests under the
assumption that NIST guidance may require them in the future. HKDF
is currently not included in FIPS, but is on track to be included,
so hkdf have been included in this patch.
Because the test vectors are const strings, the patch pushes some
const definitions that were missing in existing private interfaces.
There are three flavors of self-tests: Function implemented in
freebl are added to the freebl/fipsfreebl.c Functions implemented in
pkcs11c.c have selftests completely implemented in
softoken/fipstest.c Functions implemented in their own .c file have
their selftest function implemented in that .c file and called by
fipstests.c These are consistant with the previous choices for
selftests.
Some private interfaces that took in keys from pkcs #11 structures
or outputted keys to pkcs #11 structures were modified to optionally
take keys in by bytes and output keys as bytes so the self-tests can
work in just bytes.
[5dca54fe61c2]
2020-08-25 Daiki Ueno <dueno@redhat.com>
* lib/softoken/manifest.mn:
Bug 1659252, disable building libnssdbm3.so if NSS_DISABLE_DBM=1,
r=rrelyea
Reviewers: rrelyea
Reviewed By: rrelyea
Bug #: 1659252
[4d55d36ca6ef]
2020-08-24 Kevin Jacobs <kjacobs@mozilla.com>
* lib/pk11wrap/pk11cxt.c, lib/softoken/pkcs11c.c, lib/softoken/sdb.c,
lib/softoken/sftkpwd.c:
Bug 1651834 - Fix various static analyzer warnings. r=rrelyea
[ab04fd73fd6d]
2020-08-28 Mike Hommey <mh@glandium.org>
* lib/freebl/blapii.h:
Bug 1661810 - Define pre_align/post_align based on the compiler.
r=jcj
Things worked fine before we upgraded to clang 11 presumably because
the stack was always 16-bytes aligned in the first place, or
something akin to that, and the lack of pre_align/post_align doing
anything didn't matter. The runtime misalignment of the stack may
well be a clang > 9 bug, but keeping pre_align/post_align tied to
the x86/x64 is a footgun anyways.
[c100e11991f6] [tip]
Differential Revision: https://phabricator.services.mozilla.com/D88876
config.guess infers information about the compiler using environment
variables, such as CC. However, we use such environment variables to
configure the tooling for the target.
Differential Revision: https://phabricator.services.mozilla.com/D88085
Previously, we would optimistically attempt to use a Rust toolchain that
matches the current C toolchain, and would throw an error if an
attempted compile with that Rust toolchain failed.
Instead, if we fail to detect a usable Rust toolchain, we now helpfully
inform users of their two options: change C toolchain, or install
matching Rust toolchain.
Differential Revision: https://phabricator.services.mozilla.com/D88084
config.guess infers information about the compiler using environment
variables, such as CC. However, we use such environment variables to
configure the tooling for the target.
Differential Revision: https://phabricator.services.mozilla.com/D88085
Previously, we would optimistically attempt to use a Rust toolchain that
matches the current C toolchain, and would throw an error if an
attempted compile with that Rust toolchain failed.
Instead, if we fail to detect a usable Rust toolchain, we now helpfully
inform users of their two options: change C toolchain, or install
matching Rust toolchain.
Differential Revision: https://phabricator.services.mozilla.com/D88084
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
For not-well-understood reasons, ld's `--gc-sections` discards a large number of the PGO bookkeeping structures that enable us to keep track of function counters, and the effect gets worse in object files generated by clang-10.
As much as I'd like to understand this better, the investigations take way too much time. As a path of least resistance, we can disable `--gc-sections` for the instrumentation phase of PGO builds. It won't harm anything since users never see those builds, and it will improve the performance of the optimized phase greatly.
Differential Revision: https://phabricator.services.mozilla.com/D78112
This adds toolchain definitions for clang 11.0.0 rc2, so that developers can get a sneak peek, but nothing in automation uses these tasks yet. We'll make the switch in a later patch.
NB: most of `clang.yml` is rote copy-paste, except for `macosx64-clang-11` which makes a deliberate departure, described in a comment.
Differential Revision: https://phabricator.services.mozilla.com/D88189
This avoids a set of intermittent issues related to `zstd` decompression failures, which in the absence of these changes break the entire build.
This also requires [updating an environment variable](https://github.com/mozilla/sccache/pull/822), which we do in `client.mk` as well as documentation.
Differential Revision: https://phabricator.services.mozilla.com/D88184
Also define a scheme for storing the index of Glean definitions files in a file
separate from the build system for consumption by
* mach build
* mach doc
* (future) mozilla/probe-scraper
Differential Revision: https://phabricator.services.mozilla.com/D87600
We have a minimum rust version required for compilations. For both stable and beta rust compilers, we can trust that they will have all the stabilized features we're expecting.
However, for nightlies, they may "match" our minimum version, but may have been released in the version window before a certain feature we need has been stabilized.
So, when validating rustc version in configure, ensure that the nightly is at least one version newer than our expected version.
Differential Revision: https://phabricator.services.mozilla.com/D86889
`mach create-mach-environment` is what installs `glean_sdk` to the `mach` `virtualenv`. `create-mach-environment` runs on the system Python and we can't assume the system Python has `glean_sdk` installed.
Differential Revision: https://phabricator.services.mozilla.com/D87507
There are zero uses of this `mach` command over the past 90 days according to our telemetry. There are no external references to `mach python-safety` in-tree, and indeed if you track the history of the originating bug 1468394, it appears that once the `mach` command was created, none of the follow-up work that was discussed (i.e. running this in CI and triaging failures to appropriate owners) was done over the following 2 years.
If this ever does appear to be useful in the future, we can just resurrect this code from source control.
Differential Revision: https://phabricator.services.mozilla.com/D87351
In two different places we've been encountering issues regarding 1) how we configure the system Python environment and 2) how the system Python environment relates to the `virtualenv`s that we use for building, testing, and other dev tasks. Specifically:
1. With the push to use `glean` for telemetry in `mach`, we are requiring (or rather, strongly encouraging) the `glean_sdk` Python package to be installed with bug 1651424. `mach bootstrap` upgrades the library using your system Python 3 in bug 1654607. We can't vendor it due to the package containing native code. Since we generally vendor all code required for `mach` to function, requiring that the system Python be configured with a certain version of `glean` is an unfortunate change.
2. The build uses the vendored `glean_parser` for a number of build tasks. Since the vendored `glean_parser` conflicts with the globally-installed `glean_sdk` package, we had to add special ad-hoc handling to allow us to circumvent this conflict in bug 1655781.
3. We begin to rely more and more on the `zstandard` package during build tasks, this package again being one that we can't vendor due to containing native code. Bug 1654994 contained more ad-hoc code which subprocesses out from the build system's `virtualenv` to the SYSTEM `python3` binary, assuming that the system `python3` has `zstandard` installed.
As we rely more on `glean_sdk`, `zstandard`, and other packages that are not vendorable, we need to settle on a standard model for how `mach`, the build process, and other `mach` commands that may make their own `virtualenv`s work in the presence of unvendorable packages.
With that in mind, this patch does all the following:
1. Separate out the `mach` `virtualenv_packages` from the in-build `virtualenv_packages`. Refactor the common stuff into `common_virtualenv_packages.txt`. Add functionality to the `virtualenv_packages` manifest parsing to allow the build `virtualenv` to "inherit" from the parent by pointing to the parent's `site-packages`. The `in-virtualenv` feature from bug 1655781 is no longer necessary, so delete it.
2. Add code to `bootstrap`, as well as a new `mach` command `create-mach-environment` to create `virtualenv`s in `~/.mozbuild`.
3. Add code to `mach` to dispatch either to the in-`~/.mozbuild` `virtualenv`s (or to the system Python 3 for commands which cannot run in the `virtualenv`s, namely `bootstrap` and `create-mach-environment`).
4. Remove the "add global argument" feature from `mach`. It isn't used and conflicts with (3).
5. Remove the `--print-command` feature from `mach` which is obsoleted by these changes.
This has the effect of allowing us to install packages that cannot be vendored into a "common" place (namely the global `~/.mozbuild` `virtualenv`s) and use those from the build without requiring us to hit the network. Miscellaneous implementation notes:
1. We allow users to force running `mach` with the system Python if they like. For now it doesn't make any sense to require 100% of people to create these `virtualenv`s when they're allowed to continue on with the old behavior if they like. We also skip this in CI.
2. We needed to duplicate the global-argument logic into the `mach` script to allow for the dispatch behavior. This is something we avoided with the Python 2 -> Python 3 migration with the `--print-command` feature, justifying its use by saying it was only temporarily required until all `mach` commands were running with Python 3. With this change, we'll need to be able to determine the `mach` command from the shell script for the forseeable future, and committing to this forever with the cost that `--print-command` incurs (namely `mach` startup time, an additional .4s on my machine) didn't seem worth it to me. It's not a ton of duplicated code.
Differential Revision: https://phabricator.services.mozilla.com/D85916
Andi-Bogdan Postelnicu
Razvan Maries
Mike Hommey
David Major
Dorel Luca
Doug Thayer
Mitchell Hentges
Mihai Alexandru Michis
Csoregi Natalia
Ricky Stewart
Nicholas Nethercote
Bogdan Tara
Gijs Kruitbosch
Kristen Wright
Tom Ritter
Marco Trevisan
Kevin Jacobs
Narcis Beleuzu
Chris H-C
Hector Zhao
Bob Clary