This patch is not ideal: if would be better to do the defaulting in
`toolkit/moz.configure`, but doing it there runs into problems with base
toolchain configurations, as the clang there is not new enough. So we
have this, doing everything with environment variables, which is easily
turned on or off, depending on the needs of the specific configuration.
The `mozconfig.no-compile` change is not really needed, as the wasm
sandboxing detection bits are not conditional on
`--enable-compile-environment`. Those bits should be, and I will tackle
doing that after the holidays.
Differential Revision: https://phabricator.services.mozilla.com/D58102
--HG--
extra : moz-landing-system : lando
This change introduces a "github-sync" component into tools,
which aims to support synchronizing both wgpu and WebRender with github.
~~It also features a "cargo test" job for standalone wgpu (bug 1596127)~~
The code is ported from "gfx/wr/scripts/wrupdater" folder. Changes are:
1. remove explicit WR parts and make them configurable by command line params
2. detect "mozilla-xxx" tags and use them in addition to the commits
As a follow up, wrupdater will be removed in favor of github-sync.
Status:
- [x] get the CI test job working
- [x] get @kats to fork "wgpu" github for "moz-gfx" bot
- [x] remove the wgpu testing CI job (into separate PR)
- [x] create new secret and reference it
Differential Revision: https://phabricator.services.mozilla.com/D57057
--HG--
extra : moz-landing-system : lando
This patch is not ideal: if would be better to do the defaulting in
`toolkit/moz.configure`, but doing it there runs into problems with base
toolchain configurations, as the clang there is not new enough. So we
have this, doing everything with environment variables, which is easily
turned on or off, depending on the needs of the specific configuration.
The `mozconfig.no-compile` change is not really needed, as the wasm
sandboxing detection bits are not conditional on
`--enable-compile-environment`. Those bits should be, and I will tackle
doing that after the holidays.
Differential Revision: https://phabricator.services.mozilla.com/D58102
--HG--
extra : moz-landing-system : lando
The doubling of `when` is needed: the `try_run` when ensures that we
only execute the compiler on OS X, and the `@depends` when ensures that
we don't look at the `None` result (which would normally indicate
failure) on non-OS X hosts.
Differential Revision: https://phabricator.services.mozilla.com/D57636
--HG--
extra : moz-landing-system : lando
This is needed to support min() / max() / clamp(), as LengthPercentage will have
a potential heap allocation and the current copy-assignment operators just do
bitwise copies.
Differential Revision: https://phabricator.services.mozilla.com/D57300
--HG--
extra : moz-landing-system : lando
-Wempty-init-stmt is a new clang diagnostic that diagnoses empty C++17 init-statements of if, switch, and range-based for statements:
void test() {
if(; // <- warning: init-statement of 'if' is a null statement
true)
;
switch (; // <- warning: init-statement of 'switch' is a null statement
x) {
...
}
for (; // <- warning: init-statement of 'range-based for' is a null statement
int y : S())
;
}
Differential Revision: https://phabricator.services.mozilla.com/D54866
--HG--
extra : moz-landing-system : lando
We can remove our backported objcopy patches, we'll need to update
llvm-mingw for toolchain fixes, and we need to add binutils to compile
this newer version of clang because the system binutils is too old.
Differential Revision: https://phabricator.services.mozilla.com/D50152
--HG--
rename : build/build-clang/clang-8-mingw.json => build/build-clang/clang-9-mingw.json
rename : taskcluster/scripts/misc/build-clang-8-mingw.sh => taskcluster/scripts/misc/build-clang-mingw.sh
extra : moz-landing-system : lando
We can remove our backported objcopy patches, we'll need to update
llvm-mingw for toolchain fixes, and we need to add binutils to compile
this newer version of clang because the system binutils is too old.
Differential Revision: https://phabricator.services.mozilla.com/D50152
--HG--
rename : build/build-clang/clang-8-mingw.json => build/build-clang/clang-9-mingw.json
rename : taskcluster/scripts/misc/build-clang-8-mingw.sh => taskcluster/scripts/misc/build-clang-mingw.sh
extra : moz-landing-system : lando
We've long handled chunks by defining the total number of chunks in our CI
configuration, and then passing that value down into the test harnesses at task
runtime (via the '--this-chunk' and '--total-chunks' parameters). The test
harness then runs an algorithm to determine which tests should be run in "this"
chunk.
There are several problems with this approach, but by far the biggest is that
we can't use test information in our scheduling algorithms. The information
simply isn't available yet. This patch switches things around such that we
determine which tests go in which tasks during the taskgraph generation. This
means we have perfect information around which tasks are running which tests,
and if e.g a ccov or machine learning algorithm deems a particular test
important, we can make sure to *only* schedule the tasks that contain that
test.
I'm planning to enable this a couple suites at a time so we don't accidentally
stop running tests. This specifically only enables this mode for
'mochitest-media', 'mochitest-browser-chrome' and 'mochitest-devtools-chrome'.
I chose these suites because they are the ones that are already using the
'chunk_by_runtime' algorithm.
Differential Revision: https://phabricator.services.mozilla.com/D52729
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
Now that we have Rust PGO, training on this set can actually make a difference (and in fact may be needed in order to see a difference from Rust PGO at all).
Differential Revision: https://phabricator.services.mozilla.com/D56751
--HG--
extra : moz-landing-system : lando
Nothing looks at the value of this option. We seem to have settled on the `--enable-profile-{generate,use}=cross` approach instead.
Differential Revision: https://phabricator.services.mozilla.com/D56588
--HG--
extra : moz-landing-system : lando
In order to avoid crashes when we're dumping gcda files, we backport a llvm/compiler-rt patch which fix this issue in adding a critical section around flush.
Differential Revision: https://phabricator.services.mozilla.com/D56559
--HG--
extra : moz-landing-system : lando
MOZ_PGO_PROFILE_USE is used to enable certain features in the automation
mozconfigs for profile-use builds, including the MOZ_PROFILE_USE
configure option. They aren't easily combined into one flag due to the
extra settings that are enabled only in automation mozconfigs.
Renaming the former variable to TASKCLUSTER_PGO_PROFILE_USE makes it
slightly more obvious where it comes from and avoids confusion.
Differential Revision: https://phabricator.services.mozilla.com/D56117
--HG--
extra : moz-landing-system : lando
pgomerge.py was needed for Windows MSVC PGO builds. Now that we use
clang for these builds, it's no longer used.
Differential Revision: https://phabricator.services.mozilla.com/D56113
--HG--
extra : moz-landing-system : lando
Because of conflicts between gcov_flush from gcc and the one from llvm, we renamed llvm one into ___custom_llvm_gcov_flush.
Since we switched to clang for ccov builds, this workaround is now useless.
Differential Revision: https://phabricator.services.mozilla.com/D56394
--HG--
extra : moz-landing-system : lando
This includes more bitflag operators, which means we can remove some slightly
ugly code.
Differential Revision: https://phabricator.services.mozilla.com/D56308
--HG--
extra : moz-landing-system : lando
Local PGO builds now use 3-tier machinery under the hood. Instead of a
single object directory that gets cleaned in between the instrumented
and profile-use builds, now the instrumented build uses a separate
'${objdir}/instrumented' directory. This makes it easier to handle
within mach since we can drive the two builds with environment variables
and keep all build artifacts separate, without needing to do manual
cleanup in between.
Differential Revision: https://phabricator.services.mozilla.com/D50098
--HG--
extra : moz-landing-system : lando
In automation this is normally handled by mozconfigs, but we should
disable it automatically during the profile-generate stage for users if
they have LTO enabled in their mozconfig while doing a 3-tier PGO build.
Differential Revision: https://phabricator.services.mozilla.com/D50097
--HG--
extra : moz-landing-system : lando
We're vendoring it already so no reason not to. Only a couple dependencies left
after these to get rid of 0.6.
Differential Revision: https://phabricator.services.mozilla.com/D56044
--HG--
extra : moz-landing-system : lando
This fixes quite a bit of historical baggage, and also goes
into a bit more details of what the l10n repacks actually do.
Differential Revision: https://phabricator.services.mozilla.com/D55807
--HG--
extra : moz-landing-system : lando
Bug 1601233 made cranelift bump its syn dependency to 1.0, breaking the
workspace hack. Some of the features were also stale from presumably
other updates.
Differential Revision: https://phabricator.services.mozilla.com/D55897
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
This change does not fully enable C++17, as we still need standard
library support from some platforms.
Differential Revision: https://phabricator.services.mozilla.com/D54202
--HG--
extra : moz-landing-system : lando
Guard against flaky results from the remote is_file and reduce adb traffic
by waiting once for the log file to be created rather than checking for it
throughout the test run.
(Minor ride-along change to reftest.jsm resolves javascript error introduced
by bug 1575266.)
Differential Revision: https://phabricator.services.mozilla.com/D55026
--HG--
extra : moz-landing-system : lando
Whereas:
- desktop tests don't make this check;
- the check for directory existence has been troublesome and almost never useful;
- bug classification of this condition has been troublesome;
- if a startup crash actually did occur before crashreporter init, there would still be an indication in logcat and possibly a tombstone, and the "No test summary found" check would definitely be triggered;
Let's stop checking for minidumps directory creation.
Differential Revision: https://phabricator.services.mozilla.com/D54755
--HG--
extra : moz-landing-system : lando
Make the $PYTHON3 build var point to a full virtualenv bootstrapped with
the same libraries as the $PYTHON Python 2 build var. This allows us to
upgrade build tasks from $PYTHON to $PYTHON3.
This patch adds some debug logging and documentation to the Python
2 virtualenv so that it is easier to diagnose issues that may arise
from running two different Python interpreters in re-entrant
multiprocess routines.
Differential Revision: https://phabricator.services.mozilla.com/D50819
--HG--
extra : moz-landing-system : lando
Whether ENABLE_INTL_API and ENABLE_TYPED_OBJECTS are defined, affects
the behaviour of JS_FOR_PROTOTYPES for the prototypes of Intl and
TypedObject. Therefore, these macros have to be available to embedders.
Rename them to JS_HAS_INTL_API and JS_HAS_TYPED_OBJECTS (in line with
the existing JS_HAS_CTYPES) everywhere they are used, and add them to
js-config.h.in.
Differential Revision: https://phabricator.services.mozilla.com/D52461
--HG--
extra : moz-landing-system : lando
Whether ENABLE_INTL_API and ENABLE_TYPED_OBJECTS are defined, affects
the behaviour of JS_FOR_PROTOTYPES for the prototypes of Intl and
TypedObject. Therefore, these macros have to be available to embedders.
Rename them to JS_HAS_INTL_API and JS_HAS_TYPED_OBJECTS (in line with
the existing JS_HAS_CTYPES) everywhere they are used, and add them to
js-config.h.in.
Differential Revision: https://phabricator.services.mozilla.com/D52461
--HG--
extra : moz-landing-system : lando
The "Localization" docs in tools/compare-locales are really mostly
build, so merging that content into the builds doc.
Removing Android parts, add Fluent there, noting DTD deprecation.
Moving the glossary to the actual l10n docs.
With proper top-level structure, show 2 levels of l10n and intl
on the front-matter page.
Differential Revision: https://phabricator.services.mozilla.com/D55126
--HG--
rename : tools/compare-locales/docs/glossary.rst => intl/l10n/docs/glossary.rst
extra : moz-landing-system : lando
Otherwise, Valgrind is liable to see false positives from mismatched
`new` where the `delete` has been inlined to `free` or vice versa.
Differential Revision: https://phabricator.services.mozilla.com/D55553
--HG--
extra : moz-landing-system : lando
The minidumps are stored in the temporary profile directory used by
profileserver.py. Before the temporary directory goes away, we use
mozcrash & minidump_stackwalk to process any crashreports and copy them
to the artifacts directory.
Differential Revision: https://phabricator.services.mozilla.com/D52394
--HG--
extra : moz-landing-system : lando
With clang 10 and up `getTemporary` has been replaced with the more versatile `getSubExpr`.
The commit that added this behaviour is: b0561b3346
Differential Revision: https://phabricator.services.mozilla.com/D54101
--HG--
extra : moz-landing-system : lando
Detect if Unix utilities on win32 are being picked up from a foreign
installation of MinGW, such as the tools packaged with Git for Windows.
If autoconf dies during `./mach configure` and foreign tools are found
in $PATH then warn the user that their $PATH may need to change to fix
the problem.
Differential Revision: https://phabricator.services.mozilla.com/D52960
--HG--
extra : moz-landing-system : lando
Detect if Unix utilities on win32 are being picked up from a foreign
installation of MinGW, such as the tools packaged with Git for Windows.
If autoconf dies during `./mach configure` and foreign tools are found
in $PATH then warn the user that their $PATH may need to change to fix
the problem.
Differential Revision: https://phabricator.services.mozilla.com/D52960
--HG--
extra : moz-landing-system : lando
This ought to overcome a regression from 1592981 as well as make this test suite less noisy about toolchain updates in the future.
While here, tidy up the PGO list: sort alphabetically, stop using a hardcoded test index for the Speedometer time extension, and stop using js-input/ for the virtual mappings since js-input/ is an actual directory in our repo.
Differential Revision: https://phabricator.services.mozilla.com/D52888
--HG--
extra : moz-landing-system : lando
We need this for "full" C++17 support (everything is supported, but some
C++17 features still have bugs) and this change also brings Linux into
parity with our Mac requirements.
MANUAL PUSH: build toolchains on inbound to avoid clogging autoland
Differential Revision: https://phabricator.services.mozilla.com/D51450
When running Cargo on Windows, `HOST_LINKER` is required by
`cargo-host-linker.bat`. So we should use windows.configure to set HOST_LINKER
when host OS is Windows and target OS isn't Windows.
Differential Revision: https://phabricator.services.mozilla.com/D37241
--HG--
extra : moz-landing-system : lando
Restore the env= argument to the subprocess.Popen() call that runs
autoconf in old.configure. The argument appears to have been removed by
accident in changeset f47e966507eb.
Differential Revision: https://phabricator.services.mozilla.com/D52293
--HG--
extra : moz-landing-system : lando
With Bug 1579189 we are going to raise the minimum clang version to 5. But in clang 5
and clang 6 an issue has been introduced where the `Decl` nodes from the `AST` don't
contain all of the annotation attributes. The missing attributes can cause static
analysis failures. We are therefore going to disable the static analysis for the
base-toolchain clang builds as a workaround.
Differential Revision: https://phabricator.services.mozilla.com/D52025
--HG--
extra : moz-landing-system : lando
When we first enabled ThinLTO on our builds, we got great performance gains, but also large size increases due to aggressive inlining. There is an LLVM option called -import-instr-limit that limits the size of functions that may be imported (the threshold is subject to modification by PGO). Chromium found a good balance between speed and performance by using a value of 10. In initial testing, on Windows and Linux that value can save us many megabytes from libxul without noticeable speed regressions. For Mac, which doesn't yet have PGO, we have to use a higher limit to avoid over-restricting the optimizer which caused slowdowns on my try pushes.
Differential Revision: https://phabricator.services.mozilla.com/D51463
--HG--
extra : moz-landing-system : lando
In the future we should re-write this checker but for now this solution is acceptable.
Differential Revision: https://phabricator.services.mozilla.com/D51242
--HG--
extra : moz-landing-system : lando
This is just drive-by, but it confused me. We require at least clang 4 to build
nowadays.
Differential Revision: https://phabricator.services.mozilla.com/D50646
--HG--
extra : moz-landing-system : lando
sccache-dist / icecream preprocess the file and then send it to a builder.
That means that the file contains #line directives or equivalents in order for
diagnostics and such to work correctly.
Unfortunately our clang-plugin build fails catastrophically if you include a
third-party header, as it doesn't account for them.
Use SourceManager::getPresumedLoc to get filenames, as it accounts for them,
unlike just getFilename.
Differential Revision: https://phabricator.services.mozilla.com/D50633
--HG--
extra : moz-landing-system : lando
"here" is a bad click target, giving it a proper name helps everyone.
Depends on D50035
Differential Revision: https://phabricator.services.mozilla.com/D50036
--HG--
extra : moz-landing-system : lando
Ciure Andrei
Ricky Stewart
Sylvestre Ledru
shindli
Nathan Froyd
Dzmitry Malyshau
Mike Hommey
Nicholas Nethercote
Andreea Pavel
Chris Manchester
Narcis Beleuzu
Jonathan Watt
shravanrn@gmail.com
Brindusan Cristian
Ryan VanderMeulen
Oana Pop Rus
Emilio Cobos Álvarez
Calixte Denizet
Tyson Smith
Chris Peterson
Noemi Erli
Tom Prince
Cosmin Sabou
Tom Ritter
Andrew Halberstadt
Māris Fogels
David Major
Dorel Luca
Mike Shal
Tarek Ziadé
Axel Hecht
Razvan Maries
Jesse Schwartzentruber
Gurzau Raul
Sebastian Hengst
Mihai Alexandru Michis
Andi-Bogdan Postelnicu
Geoff Brown
Philip Chimento
Christian Holler
Makoto Kato
Nihanth Subramanya
Edwin Takahashi
Marco Castelluccio
Jan-Erik Rediger
Wes Kocher
Daniel Varga