Make 52 -> 60 updates signed with the old mar scheme (instead of mar_384)
Differential Revision: https://phabricator.services.mozilla.com/D4841
--HG--
extra : moz-landing-system : lando
All the Linux builds using GCC uses the binutils bundled with GCC. This
gives us some leeway to update the binutils used for clang builds (using
the binutils toolchain as of bug 1486998) separately.
Since we only ship builds using GCC, we're more free to upgrade
binutils for clang builds, without worrying about the next merge.
This upgrades to the last released version of binutils, and applies the
patch from https://sourceware.org/bugzilla/show_bug.cgi?id=23591 on top,
so that asan fuzzing builds don't fail.
The GPG key used to sign the upstream tarball is unfortunately not
connected to the web of trust. I verified the contents matched what's in
the Debian archive (which has a different tarball, because some files
are removed/modified in Debian for license reasons ; there were no
differences besides those).
Differential Revision: https://phabricator.services.mozilla.com/D4748
--HG--
extra : moz-landing-system : lando
Last time it was updated is bug 1436208, and the crashes we patched it
for back then has been fixed upstream a few months later.
For some reason, they renamed the executable from llvm-dsymutil to
dsymutil.
Differential Revision: https://phabricator.services.mozilla.com/D4741
--HG--
extra : moz-landing-system : lando
Automatic changes by ESLint, except for manual corrections for .xml files.
Differential Revision: https://phabricator.services.mozilla.com/D4439
--HG--
extra : moz-landing-system : lando
Upgrading to 4.7.1 will apply a fix to a performance regression
that was introduced in 4.7.
Differential Revision: https://phabricator.services.mozilla.com/D4961
--HG--
extra : rebase_source : 0139856b2f2a22c5a0e8876d5e34cc8f7e8e0307
extra : source : 1b49fa4d48428ad7bac0cca69fb5f5f3767c43b9
Bumping the Mercurial version to 4.7.1 to apply a fix for a
performance regression introduced in 4.7.
Differential Revision: https://phabricator.services.mozilla.com/D5193
--HG--
extra : rebase_source : b06eae5268fe7f38f85f8644da7f08516840a827
run xperf on windows10 using new run-as-administrator feature
Differential Revision: https://phabricator.services.mozilla.com/D5658
--HG--
extra : rebase_source : d462f2aa55c284a336ccce79794675dd84236782
Prior to this patch, getHSTSPreloadList.js would queue an XHR for every preload
list candidate site. This meant that there would be ~50,000 requests in flight
simultaneously. Simply processing these requests caused them to all time out,
and no useful work was done. This patch resolves them in batches of 250 to
avoid this issue.
Differential Revision: https://phabricator.services.mozilla.com/D3622
--HG--
extra : source : e0f0d5824dd8aaaaf1395e569cec1806b028b12e
extra : amend_source : 62f1cd6fc49cbae39c7691c5712906f775862887
Prior to this patch, getHSTSPreloadList.js would queue an XHR for every preload
list candidate site. This meant that there would be ~50,000 requests in flight
simultaneously. Simply processing these requests caused them to all time out,
and no useful work was done. This patch resolves them in batches of 250 to
avoid this issue.
Differential Revision: https://phabricator.services.mozilla.com/D3622
--HG--
extra : moz-landing-system : lando
Rather than trying to parse strings, just pass a json blob. This will allow us
to easily do things like mark artifacts to be left unextracted.
Differential Revision: https://phabricator.services.mozilla.com/D3553
--HG--
extra : rebase_source : 4e762c65d1c9f13361d5bae2e4608ba09bb39a91
This also moves the call to 'fetch_artifacts' in run-task down inside the
try/finally block. This way if something goes wrong, we'll still cleanup
MOZ_FETCHES_DIR.
Differential Revision: https://phabricator.services.mozilla.com/D4152
--HG--
extra : moz-landing-system : lando
When different translation units contain the same symbol name, all
static, but one of them non-static, ld64 wrongfully link the references
to the static data with the non-static data, or vice versa. With libaom
and libvpx sharing data structures with the same name but different
contents, that leads to interesting failures/crashes at runtime.
This was apparently fixed in Apple ld64 from Xcode 9, but the last open
sourced version is the one from Xcode 8, so I ended up digging in the
ld64 source code and fixed the issue.
This work was merged to cctools-port upstream in
https://github.com/tpoechtrager/cctools-port/pull/59.
For the same reason as invoked in bug 1478917, though, updating to
cctools-port master is more involved than just changing a commit sha1
(as it requires building apple-libtapi, which in turn builds parts of
LLVM, which should probably be avoided), so just cherry-pick the fix.
When different translation units contain the same symbol name, all
static, but one of them non-static, ld64 wrongfully link the references
to the static data with the non-static data, or vice versa. With libaom
and libvpx sharing data structures with the same name but different
contents, that leads to interesting failures/crashes at runtime.
This was apparently fixed in Apple ld64 from Xcode 9, but the last open
sourced version is the one from Xcode 8, so I ended up digging in the
ld64 source code and fixed the issue.
This work was merged to cctools-port upstream in
https://github.com/tpoechtrager/cctools-port/pull/59.
For the same reason as invoked in bug 1478917, though, updating to
cctools-port master is more involved than just changing a commit sha1
(as it requires building apple-libtapi, which in turn builds parts of
LLVM, which should probably be avoided), so just cherry-pick the fix.
This includes a fix for the style build script hang where pthreads fork
subprocesses, as well as a fix for ignoring the icecream file lock.
MozReview-Commit-ID: 29eNcbNtwB1
Differential Revision: https://phabricator.services.mozilla.com/D4139
--HG--
extra : moz-landing-system : lando
This patch completely disables *ccov, and *jsdcov builds and tests when scheduling them through try option syntax as these build variations use a lot of resources and are rarely needed to be scheduled. The only way to schedule code coverage from now on will be with the |mach try fuzzy| tool.
Differential Revision: https://phabricator.services.mozilla.com/D3921
--HG--
extra : moz-landing-system : lando
run xperf in os groups=administrators and support os_groups in taskcluster
Differential Revision: https://phabricator.services.mozilla.com/D4001
--HG--
extra : moz-landing-system : lando
In order to support operator==() for tagged enum, we have to bump the version to
0.6.2.
Differential Revision: https://phabricator.services.mozilla.com/D3932
--HG--
extra : moz-landing-system : lando
move wasm-misc and unity3d benchmarks to tier2; run wasm-misc in different preferences
Differential Revision: https://phabricator.services.mozilla.com/D3903
--HG--
extra : moz-landing-system : lando
This unbreaks some tier 3 raptor tasks. There are a few fixes rolled together here:
1) Stop overwriting the 'env' in mozharness_test.py's 'native-engine' implementation
2) Set the workdir to /home/cltbld (which makes sure the fetches are downloaded to there)
3) Download the fetches via mozharness in the 'raptor' script (since they don't use run-task anymore)
Depends on D3651
Differential Revision: https://phabricator.services.mozilla.com/D3652
--HG--
extra : moz-landing-system : lando
This makes chunks and timeouts on the MacOSX64 coverage build closer to what we have defined for the Windows build.
Differential Revision: https://phabricator.services.mozilla.com/D2149
--HG--
extra : moz-landing-system : lando
This includes a patch to enable ccache support.
MozReview-Commit-ID: 9FzMQ2XX4ca
Differential Revision: https://phabricator.services.mozilla.com/D3363
--HG--
extra : moz-landing-system : lando
This change switches most CI builds to clang, with a few exceptions:
- valgrind builds, until bug 1481670 is figured out.
- PGO and nightly builds, until that's fully tested.
- coverage builds, per bug 1471339 comment 17.
- base toolchain builds, to keep some builds on GCC even when we're
fully switched to clang.
- any build that doesn't use build/unix/mozconfig.linux (e.g. probably
all those driven by autospider.py, maybe others).
This runs the jsshell benchmarks against Google's V8 engine in addition to spidermonkey.
Both shells will run in the same task to keep things simple and decluttered. Though we
could split them out to separate tasks at a later date if needed.
Differential Revision: https://phabricator.services.mozilla.com/D3356
--HG--
extra : moz-landing-system : lando
Followup to bug 1478995 to ensure node is available for mac repackage tasks for partner and eme-free repacks.
Differential Revision: https://phabricator.services.mozilla.com/D3383
--HG--
extra : moz-landing-system : lando
When the HSTS preload script was reworked to use async/await in bug 1436369,
`fetchstatus` would create an asynchronous xml http request and then attempt to
access a response header from it. However, there was nothing to ensure that the
request had completed before this code ran. This patch ensures that the request
has completed before the response header is used.
This patch also replaces a lingering instance of `Ci.nsISSLStatusProvider` that
should have been changed to `Ci.nsITransportSecurityInfo` in bug 1475647.
Finally, this patch removes the old, redundant getHSTSPreloadList.js in
security/manager/tools as well as the unused nsSTSPreloadList.errors file in
security/manager/ssl.
Differential Revision: https://phabricator.services.mozilla.com/D2807
--HG--
extra : moz-landing-system : lando
And require it for taskcluster build already, because it doesn't harm and lets
me put all the yml changes in the same commit.
I gave up cross-compiling for OSX after a few tries and after realizing it
wasn't enough with cctools and such, but that I also needed the Mac SDK, for
which I don't have permission...
Differential Revision: https://phabricator.services.mozilla.com/D2664
--HG--
extra : moz-landing-system : lando
This beetmover transform is only relevant for Fennec and *-Source platforms at this time.
Differential Revision: https://phabricator.services.mozilla.com/D2754
--HG--
extra : moz-landing-system : lando
This change switches most CI builds to clang, with a few exceptions:
- valgrind builds, until bug 1481670 is figured out.
- PGO and nightly builds, until that's fully tested.
- coverage builds, per bug 1471339 comment 17.
- base toolchain builds, to keep some builds on GCC even when we're
fully switched to clang.
- any build that doesn't use build/unix/mozconfig.linux (e.g. probably
all those driven by autospider.py, maybe others).
Enables the benchmark in CI, uses a fetch task to download the benchmark.
Depends on D2307.
Differential Revision: https://phabricator.services.mozilla.com/D2469
--HG--
extra : moz-landing-system : lando
raptor linux tests are currently run on AWS VM images, these should be run on raw hardware
Differential Revision: https://phabricator.services.mozilla.com/D2994
--HG--
extra : moz-landing-system : lando
When iterating on taskgraph changes, the exact number of chunks that
test-verify runs usually isn't important, so skip it when going fast.
Differential Revision: https://phabricator.services.mozilla.com/D2730
--HG--
extra : rebase_source : 4d46eee982e9868050f1201aba74b020045d9ec1
extra : histedit_source : 744948fa80ae8e3b18212e840843906577fd38ec
The taskgraph code for test-verify currently looks at locally changed files to
determine how many chunks should be run. This code exists so that
`mach try fuzzy` show the same chunks that would be run on a try push.
This changes it, so that local commts are only considered on try and when
called from try-select. This makes generating the taskgraph locally faster,
when not using `mach try`. It also makes test-verfiy not consider too many
files, if the try push happens to contain commits that have landed but havent
been pushed to try yet (i.e. the first push to try after a merge, or beta try
pushes).
Differential Revision: https://phabricator.services.mozilla.com/D2698
--HG--
extra : rebase_source : 68b1ea583730ff3086949aa6c7b6a1046b406d23
extra : histedit_source : 68bbc7ca2062c7f425353e6caf6b8959786dc42d
Currently, `mach try fuzzy` generates a taskgraph that is configured exactly
like the most recent push to mozilla-central. This isn't always desirabe, so
pass some configuration down, to allow the taskgraph to behave differently.
Differential Revision: https://phabricator.services.mozilla.com/D2729
--HG--
extra : rebase_source : 99d6958b33211697227e65df17edc1eb337f63a4
extra : histedit_source : 69b5ff6805bc8409340eb71323a1f6fc637259d7
Enable geckoview-junit, mochitest-cl, mochitest-gpu, crashtests, and jsreftests on
the new packet.net platform. Run only on mozilla-central for now, since we have
limited packet.net capacity in place at this time. Run as tier 3 since this is
preliminary, geckoview is under development, and we are not running on integration
branches.
This will apply to cron tasks, action tasks, and decision tasks. It is a
distinct retrigger implementation because (a) we do not want to follow
dependencies, and (b) it takes a lot of scopes to create a decision task, so we
need to limit access to this action.
MozReview-Commit-ID: 21DVSiagcrO
--HG--
extra : rebase_source : 6f027e349e245e4aa4dbed81145a0a5d75218cb1
extra : histedit_source : eff99aee5a0e7496b0734748b29739480eb0e3fb
This additionally reconsiders the order of all of the actions, spacing them 50
"units" apart and putting the more common actions first.
MozReview-Commit-ID: 98IOYKVMcGU
--HG--
extra : rebase_source : 1273a8b86625bd8e4dc3bddab80c6912241f88c8
extra : histedit_source : 16314284a2b4e0368da843b036e22aaedf485307
TOOLTOOL_CHECKOUT is typically `.`, which doesn't work so great for
adding things to $PATH. We need to turn everything we're adding to
$PATH into absolute paths, so $PATH actually works properly.
Otherwise it can't be used as a context manager since it
doesn't have __enter__ or __exit__.
Differential Revision: https://phabricator.services.mozilla.com/D2672
--HG--
extra : moz-landing-system : lando
This commit points the `install_mercurial.sh` script at the
newest Mercurial distributions uploaded to tooltool.
Differential Revision: https://phabricator.services.mozilla.com/D2626
--HG--
extra : amend_source : a9cd500cc86106750aa5b5472e624a00128cac68
extra : histedit_source : 7a06d9c77b6f7a7b2110dc6bc15858c7aa3b7242
This is what a lot of programs do.
We do logging in a helper function so we can flush after every write.
Differential Revision: https://phabricator.services.mozilla.com/D2526
--HG--
extra : rebase_source : 98563aee129c16662a783122241623b8ed2fe457
Previously, we told `tar` or `unzip` to operate on an explicit file.
This worked when `tar` understood the compression format of the file.
And this worked in the majority of cases.
But `tar` does not support zstandard compression (at least not outside
extremely new versions, which aren't yet widely deployed). And not all
versions of `tar` support the `-a` argument.
This commit changes our invocation of `tar` so input data is piped
to it from Python. In the case of `tar`, we perform decompression in
Python, if possible. This allows us to support zstandard and `tar`
binaries that don't support `-a` to auto-detect the compression format.
I wanted to be consistent and always pipe the raw data via stdin.
But `unzip` doesn't appear to like this. Oh well.
We also refactor the logic around detecting archives. We have a
function to identify the archive type based on a filename. We then
pass the archive type to the extraction function and key off that
logic within. We also conditionally call extract_archive() and
fail hard in extract_archive() when things fail. This will make
future archive code easier to reason about.
Differential Revision: https://phabricator.services.mozilla.com/D1576
--HG--
extra : rebase_source : 1c66396cced1b2a94a959386eecc3f512b033308
It was kept on clang 5 explicitly in bug 1467658 because of bug
1467673, now fixed.
--HG--
extra : rebase_source : 8de52e6967bb1f249b7e59d83b90ecfb291a9c44
While fiddling with clang (upgrading it and applying some miscompilation
patches), my mac LTO builds started to fail because ld64 would crash
during configure.
It turns out, it was crashing trying to print a warning it shouldn't
even print out, about failure to create a cache path.
This, in turn, is due to a pointer not being initialized in the ld64
code. I sent this upstream, and this was promptly fixed:
https://github.com/tpoechtrager/cctools-port/pull/57
However, since our last update of cctools-port, upstream landed a change
that broke support for tbd files if you don't compile against the new
libtapi library. Doing so is more work than I'm ready to put here,
so we just cherry-pick the fix.
--HG--
extra : rebase_source : 131952a5233bc379943c8eb124d377525f54202f
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
This also enables the py2 linter which will help maintain compatibility
with both 2 and 3.
Differential Revision: https://phabricator.services.mozilla.com/D1884
--HG--
extra : moz-landing-system : lando
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
Johan Lorenzo
Geoff Brown
Peter Moore
Tom Prince
Ciure Andrei
Coroiu Cristina
Mike Hommey
ui.manish
Margareta Eliza Balazs
Jesse Schwartzentruber
Mark Banner
Dave Townsend
Daniel Varga
Connor Sheehan
Gurzau Raul
Joel Maher
James Graham
Cosmin Sabou
Tom Ritter
arthur.iakab
Sebastian Hengst
Justin Wood
Andreea Pavel
David Keeler
Chris Manchester
ffxbld
Dorel Luca
Raymond Forbes
Andrew Halberstadt
Kris Maglione
Mike Shal
Greg Mierzwinski
Boris Chiou
Csoregi Natalia
Emilio Cobos Álvarez
shindli
Calixte Denizet
Tiberius Oros
Nick Thomas
Rob Wood
Steve Fink
Ionut Goldan
Bogdan Tara
Benjamin Bouvier
Noemi Erli
Ms2ger
Brindusan Cristian
Tudor-Gabriel Vîjială
Chris AtLee
Kartikaya Gupta
Dustin J. Mitchell
Narcis Beleuzu
Nathan Froyd
Dan Mosedale
Gregory Szorc
Jacek Caban
Simon Fraser
Geoff Lankow
Ben Hearsum
Nick Alexander
Mihai Tabara
Aki Sasaki
Robert Bartlensky
rahul-shiv
Christian Holler
Andi-Bogdan Postelnicu
Roland Mutter Michael