This updates the rust-size build to reference the latest version [1] that
includes a work-around for calculating the .bss of a PE file that has a
writable section with a smaller on-disk size than in-memory size.
[1] ab659d93b1
--HG--
extra : rebase_source : 24b451d244cd0533ca550688ccc05976cae410c7
Last time it was updated is bug 1436208, and the crashes we patched it
for back then has been fixed upstream a few months later.
For some reason, they renamed the executable from llvm-dsymutil to
dsymutil.
Differential Revision: https://phabricator.services.mozilla.com/D4741
--HG--
extra : moz-landing-system : lando
Rather than trying to parse strings, just pass a json blob. This will allow us
to easily do things like mark artifacts to be left unextracted.
Differential Revision: https://phabricator.services.mozilla.com/D3553
--HG--
extra : rebase_source : 4e762c65d1c9f13361d5bae2e4608ba09bb39a91
This also moves the call to 'fetch_artifacts' in run-task down inside the
try/finally block. This way if something goes wrong, we'll still cleanup
MOZ_FETCHES_DIR.
Differential Revision: https://phabricator.services.mozilla.com/D4152
--HG--
extra : moz-landing-system : lando
When different translation units contain the same symbol name, all
static, but one of them non-static, ld64 wrongfully link the references
to the static data with the non-static data, or vice versa. With libaom
and libvpx sharing data structures with the same name but different
contents, that leads to interesting failures/crashes at runtime.
This was apparently fixed in Apple ld64 from Xcode 9, but the last open
sourced version is the one from Xcode 8, so I ended up digging in the
ld64 source code and fixed the issue.
This work was merged to cctools-port upstream in
https://github.com/tpoechtrager/cctools-port/pull/59.
For the same reason as invoked in bug 1478917, though, updating to
cctools-port master is more involved than just changing a commit sha1
(as it requires building apple-libtapi, which in turn builds parts of
LLVM, which should probably be avoided), so just cherry-pick the fix.
When different translation units contain the same symbol name, all
static, but one of them non-static, ld64 wrongfully link the references
to the static data with the non-static data, or vice versa. With libaom
and libvpx sharing data structures with the same name but different
contents, that leads to interesting failures/crashes at runtime.
This was apparently fixed in Apple ld64 from Xcode 9, but the last open
sourced version is the one from Xcode 8, so I ended up digging in the
ld64 source code and fixed the issue.
This work was merged to cctools-port upstream in
https://github.com/tpoechtrager/cctools-port/pull/59.
For the same reason as invoked in bug 1478917, though, updating to
cctools-port master is more involved than just changing a commit sha1
(as it requires building apple-libtapi, which in turn builds parts of
LLVM, which should probably be avoided), so just cherry-pick the fix.
This includes a fix for the style build script hang where pthreads fork
subprocesses, as well as a fix for ignoring the icecream file lock.
MozReview-Commit-ID: 29eNcbNtwB1
Differential Revision: https://phabricator.services.mozilla.com/D4139
--HG--
extra : moz-landing-system : lando
In order to support operator==() for tagged enum, we have to bump the version to
0.6.2.
Differential Revision: https://phabricator.services.mozilla.com/D3932
--HG--
extra : moz-landing-system : lando
This includes a patch to enable ccache support.
MozReview-Commit-ID: 9FzMQ2XX4ca
Differential Revision: https://phabricator.services.mozilla.com/D3363
--HG--
extra : moz-landing-system : lando
And require it for taskcluster build already, because it doesn't harm and lets
me put all the yml changes in the same commit.
I gave up cross-compiling for OSX after a few tries and after realizing it
wasn't enough with cctools and such, but that I also needed the Mac SDK, for
which I don't have permission...
Differential Revision: https://phabricator.services.mozilla.com/D2664
--HG--
extra : moz-landing-system : lando
TOOLTOOL_CHECKOUT is typically `.`, which doesn't work so great for
adding things to $PATH. We need to turn everything we're adding to
$PATH into absolute paths, so $PATH actually works properly.
Otherwise it can't be used as a context manager since it
doesn't have __enter__ or __exit__.
Differential Revision: https://phabricator.services.mozilla.com/D2672
--HG--
extra : moz-landing-system : lando
This is what a lot of programs do.
We do logging in a helper function so we can flush after every write.
Differential Revision: https://phabricator.services.mozilla.com/D2526
--HG--
extra : rebase_source : 98563aee129c16662a783122241623b8ed2fe457
Previously, we told `tar` or `unzip` to operate on an explicit file.
This worked when `tar` understood the compression format of the file.
And this worked in the majority of cases.
But `tar` does not support zstandard compression (at least not outside
extremely new versions, which aren't yet widely deployed). And not all
versions of `tar` support the `-a` argument.
This commit changes our invocation of `tar` so input data is piped
to it from Python. In the case of `tar`, we perform decompression in
Python, if possible. This allows us to support zstandard and `tar`
binaries that don't support `-a` to auto-detect the compression format.
I wanted to be consistent and always pipe the raw data via stdin.
But `unzip` doesn't appear to like this. Oh well.
We also refactor the logic around detecting archives. We have a
function to identify the archive type based on a filename. We then
pass the archive type to the extraction function and key off that
logic within. We also conditionally call extract_archive() and
fail hard in extract_archive() when things fail. This will make
future archive code easier to reason about.
Differential Revision: https://phabricator.services.mozilla.com/D1576
--HG--
extra : rebase_source : 1c66396cced1b2a94a959386eecc3f512b033308
While fiddling with clang (upgrading it and applying some miscompilation
patches), my mac LTO builds started to fail because ld64 would crash
during configure.
It turns out, it was crashing trying to print a warning it shouldn't
even print out, about failure to create a cache path.
This, in turn, is due to a pointer not being initialized in the ld64
code. I sent this upstream, and this was promptly fixed:
https://github.com/tpoechtrager/cctools-port/pull/57
However, since our last update of cctools-port, upstream landed a change
that broke support for tbd files if you don't compile against the new
libtapi library. Doing so is more work than I'm ready to put here,
so we just cherry-pick the fix.
--HG--
extra : rebase_source : 131952a5233bc379943c8eb124d377525f54202f
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
This removes the 'use-artifacts' mechanism in favour of fetches. There are a
few pieces here that need to land atomically:
1. Remove use-artifact related code
2. Call 'fetch-content' from the run-task script
3. Convert existing tasks on top of fetches (jsshell, python unittest)
4. Stop calling 'fetch-content' from toolchain setup tasks (as this now gets handled in run-task)
Depends on D2166.
Differential Revision: https://phabricator.services.mozilla.com/D2167
--HG--
extra : moz-landing-system : lando
Currently 'fetch' artifacts are all extracted in the same directory, this could
make the extdir messy, or in the worst case, cause file name collisions.
Some artifacts are ok to extract into the same directory as they're already
bundled within the archive. But other artifacts are not. This patch keeps the
default behaviour (extracting everything into the same directory), but allows
task authors to specify per-artifact directories to extract into.
The syntax is:
path[>dest]@<task>
The 'dest' value will be a subdirectory of the MOZ_FETCHES_DIR environment
variable.
Depends on D2102.
Differential Revision: https://phabricator.services.mozilla.com/D2166
--HG--
extra : moz-landing-system : lando
Instead of clang 4, which they were the last to use, so remove the
clang 4 toolchain.
--HG--
extra : rebase_source : d03a083e9217aeb6c1d2c91decb978426f0e8d1a
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : source : 0b941cbdca76fb2fbb98dc5bbc1a0237c69954d0
extra : histedit_source : a3e43bdd8a9a58550bef02fec3be832ca304ea93
After this change, we consistently import GPG keys from files in
the GCC build scripts.
MozReview-Commit-ID: BcyvCQoGbMS
--HG--
extra : source : 5fce34a460b51e45ac280a9f0cb8bad896fbcff1
extra : histedit_source : 01621ea8111315c251a9493a11efca72c2ba3c7d
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : rebase_source : 4918b8c3bac53d63665006802054038bfbca0314
After this change, we consistently import GPG keys from files in
the GCC build scripts.
MozReview-Commit-ID: BcyvCQoGbMS
--HG--
extra : rebase_source : 657ccce8e242cabdfaff396fd0d6439754a3f364
Right now artifacts from previous tasks are left lying around. We should clean these up
in case a task accidentally uses an artifact from the wrong dependency.
Ideally we'd cache these properly based on the taskId they came from, but that can be
follow-up fodder.
MozReview-Commit-ID: HUgvNlqyFav
--HG--
extra : rebase_source : fb9c6723598223619993c2695fb588ead3325edb
Version 2.25.1's libiberty can choke on some symbols. That was fixed in
2.27. As of writing, the last version is 2.30. Conservatively go with
2.28.1, which is the same major version (2.28) as what is currently in
Debian stable.
--HG--
extra : rebase_source : 9e5ba94421a1568f662cfd98af7168ea1c890934
There is a superficial check in the run-task script which requires root. Simply
removing this check allows a native-engine task (which isn't running as root)
to proceed.
MozReview-Commit-ID: 44XavXAwxxn
--HG--
extra : rebase_source : bd1f01ce1c2feb4029838e07314493d449a4f46e
This adds an optional 'workdir' key to all job schemas. It still defaults to
/builds/worker, but can be overriden by individual tasks or schema
implementations.
MozReview-Commit-ID: LY20xfBhbCP
--HG--
extra : rebase_source : 7ac76ebf55d33d30c2aad73484421c6b4002cd33
This adds a 'use-artifacts' key to the run_task schema. Tasks can specify artifacts to download like this:
run:
using: run-task
use_artifacts:
build:
- target.tar.bz2
- target.common.tests.zip
- target.mochitest.tests.zip
This will cause the run-task script to download those three artifacts from the task's 'build' dependency.
If the task doesn't have a 'build' dependency, taskgraph generation will error. The artifacts will be
downloaded into $USE_ARTIFACT_PATH. It is up to the task to do whatever extracting/setup may be required.
E.g this setup could go in the task's command.
At this time, only 'run-task' tasks using docker-worker are supported.
MozReview-Commit-ID: 3f02oCys62i
--HG--
extra : rebase_source : e8a85040e45042b537d4119334c4a8b7280b295c
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : source : 94a9641c5a018cfe729ebe748e75a7c4373e4322
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : source : 4902cab3ce5dab2d1756cf0cd5c95f40603c0a0e
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : source : 19fe5702cf6d018b743108b35e86d1750f205a76
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : source : ef477a048b575958be74287a2273830813b385f1
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : rebase_source : d1450979e636387a5fdbd80ba19a92ec4fd31088
extra : histedit_source : 62910e29cc2fff0a1db0e9521cc58b200406d393
Python 3 is the future.
MozReview-Commit-ID: APuu4Q3mimj
--HG--
extra : rebase_source : 79839dc7f3ec130e467f1d0aa268bf3912cc1c8f
extra : histedit_source : 4be41658b5debaa6165228f60f09a9d1bf53ddbb
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : rebase_source : aa50a6cd1337fe604015131b053234a2db642e57
extra : histedit_source : 8481b23bde4f408a59c623b8be43ee0611492afd
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : rebase_source : 1a45b158fb625c7fd86701736b16da8df122218d
extra : histedit_source : f66fb22e86caf9b6b3cb301bedeab0b709685ef3
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : rebase_source : 074cbf7daaed820bd330b6850d28acd766b3801a
extra : histedit_source : e10c0a14f55148dd3e57a194d262ded41c038ee5
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : rebase_source : bbf3a0bd6cc881002d38c58eef64636c5efa6712
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : rebase_source : 6bda98911cf32ce1bc3d651805b473aff18bf1f9
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : rebase_source : 38d5626574d924c25e3fb4eecf29d379c2678e0d
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : rebase_source : b13d859ffaa2d10795312ff5a0f3ebbd5ca06835
Our normal ubuntu 16.04 test image is suitable for hosting an Android x86
emulator with these minor updates: Install kvm and make sure /dev/kvm
rw permissions are open for everyone. Note that /dev/kvm is generally
only visible when running docker with --privileged; its permissions
cannot be modified in the Dockerfile, only at run-time: run-task is the
first opportunity.
Volumes are a docker-worker concept. They shouldn't be encountered on
Windows, which uses generic-worker.
MozReview-Commit-ID: KUdSxVHVJQ
--HG--
extra : rebase_source : ff54131d471ae2ae2465b11ca5ba6e787f5d11de
main() is quite long. And the control flow will become more complicated
as we support Windows.
Let's move the bulk of the cache configuration code into a standalone
function so main() is less cluttered.
MozReview-Commit-ID: xredCubr1E
--HG--
extra : rebase_source : 385fe6fe9e083cf585edc922b1fa26355d580c02
The code for cache and volume normalization makes assumptions that uid
and gid are defined. They are not defined if not running as root and
Python would crash in these code paths.
So, presumably this means that all tasks using run-task are running as
root. Let's codify that requirement.
This requirement is arbitrary. But let's not scope bloat run-task to
support scenarios until we need them.
MozReview-Commit-ID: 2uW4OSovzWi
--HG--
extra : rebase_source : 56ced8ecbd0eef3a55dc68413f4eab7601756cc0
I want to make run-task work on Windows. The script is currently very
POSIX oriented, as it assumes the existence of the grp and pwd modules,
that user IDs are numeric, and that system calls like setresgid() and
setresuid() are available, etc.
This commit starts to make some of the POSIX-centric code conditional
on running on POSIX.
Code for uid/gid extraction has been moved to its own function. Some
error messages were tweaked slightly as part of the move. Otherwise,
the changes should be pretty straightforward.
There are still other parts of this file that won't work on e.g.
Windows. But this gets us a big step closer.
MozReview-Commit-ID: HNyytKcBbBo
--HG--
extra : rebase_source : 01d653c801865e36e562a8de4c9e6d6962498f19
In preparation for making it usable on Windows, after which point
having it in a directory with "docker" in it doesn't make much sense.
MozReview-Commit-ID: Hgu0buFyJwF
--HG--
rename : taskcluster/docker/recipes/run-task => taskcluster/scripts/run-task
extra : rebase_source : 3c0b502d28b5aad54bd04069efbfda88e25bbb20
This option shouldn't be used for local builds (see bug 1294157). Set
the option from the crate's taskcluster script instead, so that it's
used only for automated builds.
--HG--
extra : rebase_source : 94b398a0f1fac60094269d755735c426ecc63e17
This will allow mozharness configs to be specified exclusively in the taskgraph.
Differential Revision: https://phabricator.services.mozilla.com/D1017
--HG--
extra : rebase_source : a3a9b6cc9d1004c4bd396fccc3e4354a7316651d
extra : source : 10acd193df92b7c495789dc24157b85f116ade5e
And adapt the build-gcc.sh script for the changes to
contrib/download_prerequisites.
--HG--
rename : taskcluster/scripts/misc/build-gcc-6-linux.sh => taskcluster/scripts/misc/build-gcc-7-linux.sh
extra : rebase_source : b1d785777b8c141c0eb0f52a73734abd2db21b94
The URL is now being redirected to
https://www.openssl.org/source/old/1.1.0/openssl-1.1.0g.tar.gz. Let's
add a -L so we follow redirects automatically.
MozReview-Commit-ID: AuZ98jGidzl
--HG--
extra : rebase_source : 07e61558024e789df45d8e2ab67ab5ad9d3d355b
Build the latest tup master branch with the LD_PRELOAD dependency
checker.
MozReview-Commit-ID: ALfnnmOZrky
--HG--
extra : rebase_source : 529d4392ef73e03f66fb76f089f8b88f45b44972
Note that static analysis was the only remaining user of the 32-bit toolchain, so I've removed win32-clang-cl (or rather, renamed it to win32-clang-cl-st-an).
--HG--
rename : build/build-clang/clang-win32.json => build/build-clang/clang-win32-st-an.json
rename : build/build-clang/clang-win64.json => build/build-clang/clang-win64-st-an.json
rename : taskcluster/scripts/misc/build-clang32-windows.sh => taskcluster/scripts/misc/build-clang32-st-an-windows.sh
rename : taskcluster/scripts/misc/build-clang64-windows.sh => taskcluster/scripts/misc/build-clang64-st-an-windows.sh
Ensure better determinism when creating rust toolchain packages
by rejecting generic channels like 'stable' or 'nightly'. Instead,
insist on a specific version or date.
The current valid dates for beta and nightly can be obtained with:
curl -s https://static.rust-lang.org/dist/channel-rust-beta.toml | grep ^date
curl -s https://static.rust-lang.org/dist/channel-rust-nightly.toml | grep ^date
MozReview-Commit-ID: I0DXw1KJGZz
--HG--
extra : rebase_source : 92e158193072582b8568d9c9f00ffdefa0af1a9c
The Proguard dependency is now managed by Gradle.
MozReview-Commit-ID: EOvKSE5z28P
--HG--
extra : rebase_source : 760b117f500cc639cc8c24e9c02933990f358dd7
We'd like to install the NDK through the Android SDK manager. But we
can't pin versions of the NDK with the SDK manager, and so Google
can silently upgrade the NDK on us. Since that is undesirable, this is
the next best thing.
With the toolchain task in hand, we can make all the relevant tasks
depend on the toolchain task and remove the download of the NDK from
tooltool as well.
I tested this on automation and the build went on, though I couldn't test
the bindgen output because the build right now is busted on one dependent crate
with rust beta, which is the first toolchain that has this package, and will go
to release shortly.
This should work though! If I need more changes I'll adjust them in bug 1432153.
You can test the repackage manually with repack_rust.py --toolchain beta, for
example.
MozReview-Commit-ID: GI2f6vGVqTe
Don't build ucl when building upx, Debian stretch has a recent enough
version. In fact, the last upstream version doesn't build with GCC in
Debian stretch (http://bugs.debian.org/811707)
--HG--
extra : rebase_source : aae67773b9dd3b99f6ddf9ab7f59a628037e6925
New Android-Gradle plugins pin the build-tools version, and we want to
be consistent between Gradle and moz.build.
MozReview-Commit-ID: ApWS4rHzPuH
--HG--
extra : rebase_source : 22008e9333b15c594ce26c2a52f67396d6e3ab84
extra : source : f918500d9cf5112b70bc8e0a120df435b02252b7
Turns out Google's maven repository doesn't publish checksums. I
can't imagine why not, but there it is. We have to think more about
whether to trust the artifacts downloaded from maven.google.com.
MozReview-Commit-ID: CdWijorq1IV
--HG--
extra : rebase_source : 6c66cf1444876624f10409ea6437863e2c2ea9b0
extra : source : 0850b319efd43ac8f4d61485451722975da55ca1
Bump mingw version to get the newest commit and do not include the
un-needed dw-extras.h on MinGW (thanks Jacek!)
MozReview-Commit-ID: OjO93XHCxs
--HG--
extra : rebase_source : 933bbb385004988a23d1069c9cd3241b3a3b336e
llvm-symbolizer is necessary to get symbols in llvm-dsymutil crash
dumps. While we could use the one from clang during the build, it's
better if the llvm-dsymutil toolchain is standalone for local testing.
--HG--
extra : rebase_source : 5cd234a3e14ab52a4ce759821e0e756e68167797
When I originally wrote the llvm-dsymutil build script in bug 1430315,
I wasn't setting CMAKE_BUILD_TYPE to Release, and was ending up with
a very large binary (> 300MB), so I stripped it.
When I later set CMAKE_BUILD_TYPE to Release, I left the manual
stripping on, but that removes symbols that are useful for stacktraces
when dsymutil crashes (the Release type still leaves out debug info).
--HG--
extra : rebase_source : 802daadc24c0090574b1a44ea8b4e6c25735f703
By default, wget prints dots every 1k bytes. This can render a
lot of output for large files. We switch to the "mega" style, which
makes each dot represent 64k, thus reducing output by up to 64x.
We also force the use of dot display. By default, it uses "bar"
which attempts to use terminal formatting if possible. Since most
of this code executes in CI and terminal control characters can
interfere with logged output, we force the use of "dot." (Although
wget appears to automatically switch to dot in TC today. But
consistency is good.)
MozReview-Commit-ID: IpTWJdcauTV
--HG--
extra : rebase_source : 5c9aa1bbdcd78eaa0b31347ad026a2c1beaedc03
We've had problems with crashes in llvm-dsymutil for a while, and while
they are, in essence, due to the fact that rustc produces bad debug
info, they are a hurdle to our builds. The tool comes along clang, and
updating clang is not necessarily easy (witness bug 1409265), so, so
far, we've relied on backporting fixes, which can be time confusing
(witness bug 1410148).
OTOH, llvm-dsymutil is a rather specific tool, that doesn't strictly
need to be tied to clang. It's only tied to it because it uses the llvm
code to do some of the things it does, and it's part of the llvm source
tree. But it could just as well be a separate tool, like it was(is?) on
OSX.
So, we add a toolchain job to build it from the llvm source,
independently from clang, so that we can update it separately, if we
hit new crashes that happen to already be fixed on llvm trunk. It will
also allow to more easily update after upstream fixes crashes after we
report them.
--HG--
extra : rebase_source : b814353b4b4632e46646a24b8f54c5300618ff49
New Android-Gradle plugins pin the build-tools version, and we want to
be consistent between Gradle and moz.build.
MozReview-Commit-ID: ApWS4rHzPuH
--HG--
extra : rebase_source : 38a9781c472d858f3300cbbcbdc6d2311c465713
Turns out Google's maven repository doesn't publish checksums. I
can't imagine why not, but there it is. We have to think more about
whether to trust the artifacts downloaded from maven.google.com.
MozReview-Commit-ID: CdWijorq1IV
--HG--
extra : rebase_source : a884971e51ce7b1ff993754b130f462c476646ab
I don't intend to install ccache in the Debian build images. Hazard
builds are the only builds running on the desktop-build image still
using ccache somehow, and that gains them nothing, since the ccache
directory is not set to a cached directory on taskcluster, meaning
the build always starts with an empty cache. If anything, this currently
makes the build slower.
Eventually, those builds should be able to use sccache, once the
necessary setup moves out of mozconfig.cache.
--HG--
extra : rebase_source : fba6dab78b25ea61892cbe6127ead36da395b0e0
It was failing to build with the GCC/binutils on the CentOS-based docker
image, but it doesn't with the Debian-based one, so we can remove the
dependency on the gcc toolchain task. This allows sccache to remain
untouched when we change the gcc build scripts, and more importantly,
this allows it to depend on no toolchain that requires building things.
This makes it now possible to use sccache as a dependency for all other
toolchains jobs that compile, if that's beneficial (which might not be
the case, given the current sccache retention time, but at least it's a
viable option, now)
New Android-Gradle plugins pin the build-tools version, and we want to
be consistent between Gradle and moz.build.
MozReview-Commit-ID: ApWS4rHzPuH
--HG--
extra : rebase_source : 5a5730b4b9ce84af40a7c73c4f1abba017103f02
Turns out Google's maven repository doesn't publish checksums. I
can't imagine why not, but there it is. We have to think more about
whether to trust the artifacts downloaded from maven.google.com.
MozReview-Commit-ID: CdWijorq1IV
--HG--
extra : rebase_source : e4373273e7aea7df79d70b5fbc233233a84d2360
There is no /lib64 on Debian. OTOH, one doesn't need to give the full
path to a system library in LDFLAGS, so just use -l syntax instead.
--HG--
extra : rebase_source : b795f97ab209499824afa5ef1aee9da52657ceb9
libcrypto, part of OpenSSL, and that dmg links against, has a varying
ABI, and something built against libcrypto on Centos won't run on Debian
and vice versa. It might not even work between versions of the same OS
(e.g. Debian 7 vs. Debian 9).
Because of that, it is desirable to statically link it.
This incorporates https://github.com/mozilla/libdmg-hfsplus/pull/1
and sets OPENSSL_USE_STATIC_LIBS when building libdmg-hfsplus.
--HG--
extra : source : 78f2064b3811db58b364c32ce9b58a3f2dcaf8f8
The spidermonkey mozjs and rust-bindings builds run sed on
$topsrcdir/.cargo/config.in to generate the cargo config they use, but
they previously only replaced the @top_srcdir@ substitution. This patch
makes them replace any other substitutions with an empty value to add
a bit of future-proofing.
MozReview-Commit-ID: 1DzP9vXxHMD
--HG--
extra : rebase_source : e8c0268a2a6e91ca2000b340beee2dcff0636591
We're about to remove some tooltool manifests, so we need those calls to
work properly when TOOLTOOL_MANIFEST is not set.
--HG--
extra : rebase_source : 89d41021a87915dc9133e61543352e3bda1dace4
We're about to remove some tooltool manifests, so we need those calls to
work properly when TOOLTOOL_MANIFEST is not set.
--HG--
extra : rebase_source : 38ca0e3b894097ed3667901b05af79062a6c82c2
On Debian, the autoconf binary is autoconf2.13 while it is autoconf-2.13
on Centos.
In make-source-package.sh, we need to run autoconf to generate the
old-configure to include in the package, so try both.
In hazard-analysis.sh, we actually don't need autoconf itself, so just
copy configure.in to configure.
--HG--
extra : rebase_source : d21075394c69cd7cd6738da645173eb29f4a1259
The "contract" for toolchains is that extracting foo.tar.xz creates a
directory named foo/. That is however not true for mingw32.tar.xz, which
extracts into gcc/, possibly overwriting files from the gcc.tar.xz
archive (which is also used for mingw builds, for the host part).
This is also not true for nsis.tar.xz, but it reportedly has problems
when it's not in the same directory as mingw32.
But mingw32 doesn't actually need to be mixed with gcc, so it's better
to separate them as they are supposed to be.
--HG--
extra : rebase_source : 30d90af64459bbb31bc076e48f3c661fa9cd4a79
With all of our builds in Taskcluster now, we should never be uploading
symbols from build tasks. Unfortunately Windows builds were still doing so.
This patch removes MOZ_AUTOMATION_UPLOAD_SYMBOLS from all the in-tree
mozconfigs and a few other places so that it should always default off
(per moz-automation.mk). The rest of the uploadsymbols bits will be
removed once Thunderbird fixes their automation.
This patch was mostly autogenerated by running:
rg --files-with-matches UPLOAD_SYMBOLS browser/config/mozconfigs/ mobile/android/config/mozconfigs/ | xargs sed -ri '/.*UPLOAD_SYMBOLS.*/d'
sed -ri '/.*UPLOAD_SYMBOLS.*/d' build/unix/mozconfig.linux build/mozconfig.win-common build/macosx/local-mozconfig.common build/mozconfig.automation
Then mobile/android/config/mozconfigs/common and
taskcluster/scripts/builder/build-linux.sh were hand-edited.
MozReview-Commit-ID: Cy8kSEodSg4
--HG--
extra : rebase_source : 01caf1651b4eb428313e1f371aa585f8f34c4151
These scripts are included by hazard-analysis.sh. That's their only
reference in repo.
We could probably inline these scripts. But let's start by moving them
out of mozharness since no active mozharness based task is using them.
MozReview-Commit-ID: 13oen42Txmh
--HG--
rename : testing/mozharness/scripts/spidermonkey/build.browser => taskcluster/scripts/builder/hazard-browser.sh
rename : testing/mozharness/scripts/spidermonkey/build.shell => taskcluster/scripts/builder/hazard-shell.sh
extra : rebase_source : 782f7b3f3537cfefb51b0e5f1b459c8ad0daca5b
The goal is to use a newer Android-Gradle build plugin version (2.3.3
is latest stable). That requires a modern Gradle (anything 3.3+, but
3.4.1 is the default from my Android Studio), and also a newer
build-tools (25.0.3 is latest stable).
The locations of lint output changed, and we want to use the standard
output location because it's difficult to accommodate variant details
in custom names. We change the location of findbugs output to follow
suit.
This requires either:
- fixing lint errors
- adding to the lint whitelist
- using the new lint baseline
It's best to use the new lint baseline, which will happen in the next commit.
MozReview-Commit-ID: D19FzIDCJrE
--HG--
extra : rebase_source : 12d132c0c3e0dbe2b8873b31360ea96d612de44c
This is simply diagnostic; it's nice to see what the versions of the
Android packages installed are, like:
Installed packages:
Path | Version | Description | Location
------- | ------- | ------- | -------
<snip>
platforms;android-23 | 3 | Android SDK Platform 23 | platforms/android-23/
tools | 26.0.1 | Android SDK Tools 26.0.1 | tools/
This is really useful because it's not possible to pin most packages
to specific versions; that is, we can only install latest "tools", not
"tools;26.0.1".
MozReview-Commit-ID: HgZLGCAObEs
--HG--
extra : rebase_source : 33e5c9f81d05551c9e167eac62009f1de023b410
I don't think (the output of) this script is used anywhere.
MozReview-Commit-ID: DwMFtpozjNL
--HG--
extra : rebase_source : 005086039f520ba116534ab47ee49616c6958e85
The only tricky piece here is that the resulting toolchain archive is
private, and uses a newly allocated Task Cluster scope
(queue:get-artifact:project/gecko/android-sdk/*) to restrict access to
the archive. All SCM levels (1, 2, 3) have been given the new scope:
see https://tools.taskcluster.net/auth/roles/moz-tree:level:1 and
friends.
MozReview-Commit-ID: CcDqDOHODpe
--HG--
extra : rebase_source : 81dbb065f2a3c4e7733e964be66adb1733db52c6
I don't think (the output of) this script is used anywhere.
MozReview-Commit-ID: DwMFtpozjNL
--HG--
extra : rebase_source : 36b3cbe1a6a9e5cd163782c1c13653be8558a03a
The only tricky piece here is that the resulting toolchain archive is
private, and uses a newly allocated Task Cluster scope
(queue:get-artifact:project/gecko/android-sdk/*) to restrict access to
the archive. All SCM levels (1, 2, 3) have been given the new scope:
see https://tools.taskcluster.net/auth/roles/moz-tree:level:1 and
friends.
MozReview-Commit-ID: CcDqDOHODpe
--HG--
extra : rebase_source : 062bca8c65556f0f46e9c9cc6cd81eb04cf2b522
When adding sccache toolchain jobs in bug 1381772, building with gcc
failed, and building with clang worked, so I just went with the path of
least resistance. That's however a suboptimal position in the dependency
graph, so it's still preferable to use gcc if possible.
Looking exactly how it fails, it turns out it's because without CC being
set, ring wants to build with "cc", which ends up being the system gcc
instead of ours (our gcc archive doesn't provide "cc", only "gcc"), and
it is too old to support the compiler flags ring uses.
So setting CC does the trick.
--HG--
extra : rebase_source : 4c657664957dff1f7aebe470e0440a52c9e280e5
Copy the repack_rust.py from the rust-build docker container
so it can be used more generally by other taskcluster jobs.
Add --host, --target, and --suffix switches, allowing control
of the packaged toolchain and standard library builds from
the command line.
This drops the previous default behaviour of packaging all
supported platforms and targets.
Add a hard-coded copy of the Rust release signing key to
the script and add it to the running user's gpg config
so we can validate downloaded artifacts from the project
in automation.
Remove the keybase artifact validation since it requires
out-of-project network services and doesn't provide much
advantage in automation.
Calculate the SHA-2 checksum during download and remove
the dependency on shasum/sha256sum command-line tools.
Use more python for filesystem an process interaction
in general.
Create a generic rustc.tar.* package to correctly match
the unversioned unpack dirctory name.
Add support for copying the package to an output directory
if the UPLOAD_DIR environment variable is set. This lets
us hook up the script to taskcluster toolchain jobs without
an external wrapper.
MozReview-Commit-ID: 68LmY3QVU8V
--HG--
extra : rebase_source : f6329056d518ad2cd25faa5c71b22130cbc65c8f
CentOS 6 is pinned to glibc 2.12, but newer Android build-tools (like
aapt) require glibc 2.14. It's not possible to safely upgrade CentOS
6 distributions to glibc 2.14.
CentOS 7 is pinned to glibc 2.17, which is new enough for newer
Android build-tools. However, I had great difficulty bringing forward
our existing centos:6 Docker image to centos:7. In particular,
installing recent enough Mercurial, git, Python, and pip versions was
difficult enough that I elected to not pursue this approach.
Instead, I've elected to follow glandium's suggestion from
https://bugzilla.mozilla.org/show_bug.cgi?id=1370119#c5: base on
Debian with snapshots.debian.org for reproducibility.
The most significant changes here:
- using Debian's snapshots repository
- using Python and related tools provided by Debian and baked into the
build image
- using the JDK and JRE provided by Debian and baked into the build
image, rather than versions from tooltool (or eventually a toolchain
build)
Moving the builds over to use this image will follow in the patches
ahead.
I also snuck in some last-minute assertions and minor fixes into this patch:
- don't stop reporting for a callee if we've seen it already (or rather, make the reachable set local to a root rather than global to all roots). This slows down runs with hundreds of hazards, but results in every problematic root being reported, for a more accurate count.
- annotate away some thread assertions
- special-case annotation for bug 1400435 since it's a whole family of hazards
--HG--
extra : rebase_source : ac7335d45e3e0772d34cb42cc6a3f628564fd3d1
CLOSED TREE
--HG--
extra : amend_source : 84120d6bacb5d72a9fbe41e4c3b405d63825da7c
extra : histedit_source : 8320c2193761b745f10850055ee74a3c9ac73615%2Cfbc2a28d8c5004a53305ef858ca5aea4245691e0
CentOS 6 is pinned to glibc 2.12, but newer Android build-tools (like
aapt) require glibc 2.14. It's not possible to safely upgrade CentOS
6 distributions to glibc 2.14.
CentOS 7 is pinned to glibc 2.17, which is new enough for newer
Android build-tools. However, I had great difficulty bringing forward
our existing centos:6 Docker image to centos:7. In particular,
installing recent enough Mercurial, git, Python, and pip versions was
difficult enough that I elected to not pursue this approach.
Instead, I've elected to follow glandium's suggestion from
https://bugzilla.mozilla.org/show_bug.cgi?id=1370119#c5: base on
Debian with snapshots.debian.org for reproducibility.
The most significant changes here:
- using Debian's snapshots repository
- using Python and related tools provided by Debian and baked into the
build image
- using the JDK and JRE provided by Debian and baked into the build
image, rather than versions from tooltool (or eventually a toolchain
build)
Moving the builds over to use this image will follow in the patches
ahead.
Our current sccache build links in openssl's libraries dynamically. The
sonames of the dynamic libraries linked in are specific to the
CentOS/Fedora-ish systems that we build on; attempting to run the
generated sccache binaries on different systems (e.g. Debian-ish) will
result in failure. All of our current automation images are
CentOS-based, but for various reasons, Debian-based images may be used
in the future, and it would be great to have an sccache binary to run on
such systems as well. (It might also be interesting to distribute the
sccache binary we use to local developers as well, but that's a bit
further off.)
Therefore, this patch alters the sccache build on Linux to use static
linking for openssl. We cannot use the system openssl we build on
because the system openssl links to libkrb5, and the distribution we use
for the system images does not provide static libraries of libkrb5.
Building openssl ourself enables us to eliminate the libkrb5 dependency.
An sccache binary from builds with this patch depends on the following
libraries:
froydnj@hawkeye:~$ ldd sccache2/sccache
linux-vdso.so.1 => (0x00007ffe02b39000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff0e7403000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff0e71fb000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff0e6fdd000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ff0e6dc6000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff0e69fc000)
/lib64/ld-linux-x86-64.so.2 (0x0000557c8540b000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff0e66f2000)
which are standard on any Linux distribution.
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
arthur.iakab
Benjamin Bouvier
Eric Rahm
Mike Hommey
Tom Ritter
Tom Prince
Andrew Halberstadt
Mike Shal
Boris Chiou
Emilio Cobos Álvarez
Daniel Varga
Steve Fink
Narcis Beleuzu
Nathan Froyd
Gregory Szorc
Nick Alexander
Jacek Caban
Robert Bartlensky
Tudor-Gabriel Vîjială
Sebastian Hengst
Marco Castelluccio
Gurzau Raul
Chris AtLee
Ciure Andrei
Geoff Brown
Robin Templeton
Masatoshi Kimura
Ted Mielczarek
David Major
Ralph Giles
Jesse Schwartzentruber
shindli
Dorel Luca
Ryan VanderMeulen
Coroiu Cristina
Csoregi Natalia
Ting-Yu Chou
Christian Holler
Brindusan Cristian
Stephen A Pohl
Nick Fitzgerald
Chris Manchester
Wander Lairson Costa