for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1406
--HG--
extra : rebase_source : 207d1c25e37ab2619a09fb209282ffe55025de26
The crash reporter symbol files are the easiest cross-platform way to
find static initializers. While some types of static initializers (e.g.
__attribute__(constructor) functions) don't appear there in a notable
way, the static initializers we do care the most about for tracking do
(static initializers from C++ globals). As a matter of fact, there is
only a difference of 2 compared to the currently reported count of 125
on a linux64 build, so this is a good enough approximation. And allows
us to easily track the count on Android, OSX and Windows builds, which
we currently don't do.
The tricky part is that the symbol files are in
dist/crashreporter-symbols/$lib/$fileid/$lib.sym, and $fileid is hard to
figure out. There is a `fileid` tool in testing/tools, but it is a
target binary, meaning it's not available on cross builds (OSX,
Android).
So the simplest is just to gather the data while creating the symbol
files, which unfortunately requires to go through some hoops to make it
happen for just the files we care about.
--HG--
extra : rebase_source : 458fed1ffd6f9294eefef61f10ff7a284af0d986
Thunderbird releases need to look at comm-beta/comm-esr* branches for old
locale/version information.
Differential Revision: https://phabricator.services.mozilla.com/D1413
--HG--
extra : rebase_source : 76625ea5859d25f270b9fbec577f9075988bf2b7
PyYAML is not fast, so the fewer times we parse the same file, the better
MozReview-Commit-ID: KuYKFY7hFXp
--HG--
extra : rebase_source : 55df7c515db8864ee6d01895d444f7f26229bc2f
When running jobs on try, we shouldn't have access to the partials, so
don't try to give them.
Differential Revision: https://phabricator.services.mozilla.com/D1352
--HG--
extra : source : 90d19b52f956ce509525ca160ef5a2af80eea1a9
extra : histedit_source : b303ac9a4117a4e8e047d6459a7ba6477b74b762
For kind=hook, the spec doesn't include this value as it's untrustworthy.
For kind=task, it's still untrustworthy, but there is no privilege escalation
so that's not important. Still, it dramatically expands the size of the task
definition.
MozReview-Commit-ID: 6scQ2ZwxP10
--HG--
extra : rebase_source : 4dc34390a510091ddc26023755992995fe358e47
In a post-actions-as-hooks world, users will not have scopes to create tasks,
so this mode of action definition will not be possible. This is not currently
used from Treeherder (it links to
https://tools.taskcluster.net/tasks/<taskid>/interactive instead)
This drops support for the JSON-e-only interactive action; that action is not
currently used from treeherder, so that should have no impact for users.
MozReview-Commit-ID: 9i3POpjahAc
--HG--
extra : rebase_source : e6de03389a0c5c67d5332d2b1c97e1d4bf6a22d3
support_vcs_checkout() always sets the environment variables that
were set by this deleted function. In addition, support_vcs_checkout()
also adds caches and scopes - at least for docker-worker and
docker-engine. For generic-worker - which was used in all call sites
of docker_worker_add_gecko_vcs_env_vars() (yes, the "docker_worker"
bit of the name was completely wrong - probably a legacy holdover) -
support_vcs_checkout() was *almost* exactly equivalent to
docker_worker_add_gecko_vcs_env_vars(). The only difference is that
support_vcs_checkout() adds the
secrets:get:project/taskcluster/gecko/hgfingerprint scope in addition
to setting the environment variables.
MozReview-Commit-ID: 8fl3u9be5fT
--HG--
extra : rebase_source : 0eec2f143f903a3fcc5502b60026f5d8061100ea
This functionality was implemented at least 3 times. Let's consolidate
it to a central function.
Returning multiple command strings is kind of funky. I preserved
existing behavior and mozharness jobs are the only ones printing the
forensic logging. We should probably move this logging into
robustcheckout so we don't need to involve taskgraph with this. But
that can be deferred to another day.
MozReview-Commit-ID: I2LglJvfI6
--HG--
extra : rebase_source : 7cb413694aee4e46a6522febe9daa4b73b5307ca
extra : source : 096d7d374af427ee950c7a550878781eebad4135
The geckoview-junit tests require the OSS audio backend for the Android
4.3 ARM emulator, but mochitests don't work well with the OSS audio
backend. Therefore, use a different config file for the geckoview-junit
tests.
MozReview-Commit-ID: 20tzjtVdTuB
The email address used for notification on try was templated,
but nothing actually evaluated the template. This applies the same
templating that applies to the message to any emails specified.
Differential Revision: https://phabricator.services.mozilla.com/D1297
--HG--
extra : source : dd32a78ddf2196436f2098b4bc8bd3dc5c77b526
extra : amend_source : 8eac858e2b658bb2d8c3dacabe4c7fa3c077d9bc
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : source : 94a9641c5a018cfe729ebe748e75a7c4373e4322
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : source : 4902cab3ce5dab2d1756cf0cd5c95f40603c0a0e
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : source : 19fe5702cf6d018b743108b35e86d1750f205a76
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : source : ef477a048b575958be74287a2273830813b385f1
Our normal ubuntu 16.04 test image is suitable for hosting an Android x86
emulator with these minor updates: Install kvm and make sure /dev/kvm
rw permissions are open for everyone. Note that /dev/kvm is generally
only visible when running docker with --privileged; its permissions
cannot be modified in the Dockerfile, only at run-time: run-task is the
first opportunity.
download-and-compress isn't very complicated and should work on Python 3
with minimal effort. So let's switch it to use Python 3.
MozReview-Commit-ID: 9G1WfcbbKEY
--HG--
extra : rebase_source : 3a6bab06c8500a90413e8b7642a7bf7bdff04a46
python-zstandard 0.9 has an API that exposes a file object interface
for compression and decompression. This means we can remove our
stream wrapper in order to consume a zstandard compressed tar file.
MozReview-Commit-ID: DeWWKnigJVa
--HG--
extra : rebase_source : b510b9c7cf4471df835c755299a7842d13188b67
The latest python-zstandard uses a newer zstandard that is faster.
It also has wheels available, which means installation doesn't require
Python development headers, etc.
MozReview-Commit-ID: 5gRq81KYmX4
--HG--
extra : rebase_source : 96ccc64e9707c6b4815c1bfa5c1a98b9a428b387
Version 0.9.0 bundles a newer version of the zstandard library, which
is a little faster and has a few minor bug fixes (none that we were
likely hitting, however).
MozReview-Commit-ID: 9YgSZ0G41eg
--HG--
extra : rebase_source : 8f5a68323b1e1fe7e9f1dd1a92e132434972d21d
We want Python 3 available everywhere because it is 2018.
MozReview-Commit-ID: L3wufNXKdnp
--HG--
extra : rebase_source : c260923e3c13f8b28e30eaaf6e1bd38f79500052
Address Sanitizer builds use --disable-crashreporter so they don't have
symbols zip files to upload. Don't run symbol upload tasks for these builds.
MozReview-Commit-ID: GeQgRZF3m8t
--HG--
extra : rebase_source : 6ac59c5c96b5fb5ddbbe8c60af3a203d02ea9883
Mainly so searching "toolchain" + "gcc" yields something useful in the
taskgraph.
MozReview-Commit-ID: HWiT3AwwYQ2
--HG--
extra : rebase_source : b1ba0dfb4f99b6f8abe42506e8b37db68ed03590
Instead of having a special test-set for linux64-qr we can just use the
regular test set, and explicitly disable the individual tests that are
failing.
MozReview-Commit-ID: 8MUj1YdtOsH
--HG--
extra : rebase_source : 5b4398ccedd208c97fe2c58024d98bfdb759c932
Instead of having a special test-set for linux64-qr we can just use the
regular test set, and explicitly disable the individual tests that are
failing.
MozReview-Commit-ID: 8MUj1YdtOsH
--HG--
extra : rebase_source : 4d38dd3ea7a6934c84e57d6a20c7dc457f06c2da
Note in particular that tasks that were previously set to run on just
['mozilla-central', 'try'] will now also run on inbound and autoland, in
addition to mozilla-beta and other release branches. In some cases (e.g.
for talos tests) this might result in a significant increase in load on
CI infrastructure. For the tasks that were already running on ['trunk',
'try'] the extra load from the release branches should be relatively
small and will only take effect once 62 moves off nightly into beta.
MozReview-Commit-ID: 6sn9q6rCxOK
--HG--
extra : rebase_source : c228adf059a950aec3e311ae11915caf345e854f
Mainly so searching "toolchain" + "gcc" yields something useful in the
taskgraph.
MozReview-Commit-ID: HWiT3AwwYQ2
--HG--
extra : rebase_source : b1ba0dfb4f99b6f8abe42506e8b37db68ed03590
Summary:
In essence, we're allowing a new field in the task definition,
which is trusted over the one that's passed in with the config. This
wouldn't make much sense if it had a real date in, but allows us to
set an empty string for the kind that needs it
Reviewers: bhearsum
Reviewed By: bhearsum
Bug #: 1458854
Differential Revision: https://phabricator.services.mozilla.com/D1214
--HG--
extra : rebase_source : 82145a94fa91957ffe57112a1c0d327d99e32b23
This Dockerfile downloads non-deterministic remote content (by cloning a
Git repo) and then executes code from it. Part of that code is
executing Python package installs.
Since this Docker image was generated, it appears the remote code
requires new build dependencies. This commit adds those package
dependencies.
Not having deterministic Docker image builds is a bug. I'll file a
follow-up so we pin the Git commit used for building so this type
of failure doesn't occur again.
--HG--
extra : amend_source : 533a95abeb7cf7ddc9a1329549f5d294baf983f5
Volumes are a docker-worker concept. They shouldn't be encountered on
Windows, which uses generic-worker.
MozReview-Commit-ID: KUdSxVHVJQ
--HG--
extra : rebase_source : ff54131d471ae2ae2465b11ca5ba6e787f5d11de
main() is quite long. And the control flow will become more complicated
as we support Windows.
Let's move the bulk of the cache configuration code into a standalone
function so main() is less cluttered.
MozReview-Commit-ID: xredCubr1E
--HG--
extra : rebase_source : 385fe6fe9e083cf585edc922b1fa26355d580c02
The code for cache and volume normalization makes assumptions that uid
and gid are defined. They are not defined if not running as root and
Python would crash in these code paths.
So, presumably this means that all tasks using run-task are running as
root. Let's codify that requirement.
This requirement is arbitrary. But let's not scope bloat run-task to
support scenarios until we need them.
MozReview-Commit-ID: 2uW4OSovzWi
--HG--
extra : rebase_source : 56ced8ecbd0eef3a55dc68413f4eab7601756cc0
I want to make run-task work on Windows. The script is currently very
POSIX oriented, as it assumes the existence of the grp and pwd modules,
that user IDs are numeric, and that system calls like setresgid() and
setresuid() are available, etc.
This commit starts to make some of the POSIX-centric code conditional
on running on POSIX.
Code for uid/gid extraction has been moved to its own function. Some
error messages were tweaked slightly as part of the move. Otherwise,
the changes should be pretty straightforward.
There are still other parts of this file that won't work on e.g.
Windows. But this gets us a big step closer.
MozReview-Commit-ID: HNyytKcBbBo
--HG--
extra : rebase_source : 01d653c801865e36e562a8de4c9e6d6962498f19
In preparation for making it usable on Windows, after which point
having it in a directory with "docker" in it doesn't make much sense.
MozReview-Commit-ID: Hgu0buFyJwF
--HG--
rename : taskcluster/docker/recipes/run-task => taskcluster/scripts/run-task
extra : rebase_source : 3c0b502d28b5aad54bd04069efbfda88e25bbb20
Summary: Now with correct phabricator usernames
Reviewers: aki
Reviewed By: aki
Bug #: 1458855
Differential Revision: https://phabricator.services.mozilla.com/D1133
--HG--
extra : rebase_source : fd8aa5de1c1aaedacbbad53fa692e6c3132a0a94
The python3-minimal package provides /usr/bin/python3 on Debian.
This commit installs this package so a `python3` executable is
provided.
This required backporting the package to wheezy. The final patch
is trivial. But I wasted a bit of time figuring out why `mk-build-deps`
wasn't working. It would no-op and exit 0 and then the build would
complain about missing dependencies!
glandium's theory is that the ":any" multiarch support on wheezy
isn't complete. Removing ":any" seems to make things "just work."
MozReview-Commit-ID: FBicpK4SmkQ
--HG--
extra : rebase_source : a28ce731024e8ed6a43fb30e2ed57da2abb50d0f
If we're not passed `release_partner_config` in the input, let's poll github for it in the action.
MozReview-Commit-ID: 2swx76vhTE5
--HG--
extra : rebase_source : d16c517348e506519bc6e6296ad7a0cb2f90527c
extra : source : 3ad2e53b051d48a54dfb1cf88b743ed4d9571ff6
We originally had this logic here, and called it from the `partner_repack` transform. This kept the config more hidden, but had the downsides of a) being difficult to test, and b) hitting the network during transforms, which we're trying to avoid.
We moved this code to release-runner3, and passed in the `release_partner_config` as input, and saved it as a parameter. Parameterizing the partner config means that we can refer to it easily throughout taskgraph generation and in local testing, and we don't have to hit the network during transforms. The downsides include potentially having to generate this config in multiple places (rr3, ship-it-v2, the partner hook), and risking hitting the 20k gpg cleartext character limit in the `ACTION_INPUT` environment variable.
Now I'm moving this code back into util.partners, but I'm calling it from the action, not from the transform. The action populates the `release_partner_config` parameter, so we can still access the config from anywhere in the taskgraph generation code, more easily test, and avoid hitting the network during transforms. It also means that release-runner3, ship-it-v2, and the partner hook can all use the partner config generation code from a single location, rather than having to duplicate it.
Hoping this is the last major change we need to make here for a while.
MozReview-Commit-ID: 8UmvmcAoZgD
--HG--
extra : rebase_source : 57464354ea294bec972d6ff44b8c9426b31fc644
extra : source : 21289762391e66c9330f5c6f93a417f8d608c6e9
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : rebase_source : d1450979e636387a5fdbd80ba19a92ec4fd31088
extra : histedit_source : 62910e29cc2fff0a1db0e9521cc58b200406d393
Python 3 is the future.
MozReview-Commit-ID: APuu4Q3mimj
--HG--
extra : rebase_source : 79839dc7f3ec130e467f1d0aa268bf3912cc1c8f
extra : histedit_source : 4be41658b5debaa6165228f60f09a9d1bf53ddbb
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : rebase_source : aa50a6cd1337fe604015131b053234a2db642e57
extra : histedit_source : 8481b23bde4f408a59c623b8be43ee0611492afd
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : rebase_source : 1a45b158fb625c7fd86701736b16da8df122218d
extra : histedit_source : f66fb22e86caf9b6b3cb301bedeab0b709685ef3
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : rebase_source : 074cbf7daaed820bd330b6850d28acd766b3801a
extra : histedit_source : e10c0a14f55148dd3e57a194d262ded41c038ee5
A try push converting run-task to Python 3 seemed to complete without
error.
Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.
We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.
MozReview-Commit-ID: 4vATLZ6Si2e
--HG--
extra : rebase_source : bbf3a0bd6cc881002d38c58eef64636c5efa6712
Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.
MozReview-Commit-ID: KpvZGegBkYn
--HG--
extra : rebase_source : 6bda98911cf32ce1bc3d651805b473aff18bf1f9
This required a lot of attention to bytes versus strings.
The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.
There are still some failures in run-task on Python 3. But we're a big
step closer.
MozReview-Commit-ID: 4FJlTn3q9Ai
--HG--
extra : rebase_source : 38d5626574d924c25e3fb4eecf29d379c2678e0d
The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.
MozReview-Commit-ID: nY9A13Pt3E
--HG--
extra : rebase_source : b13d859ffaa2d10795312ff5a0f3ebbd5ca06835
Right now artifacts from previous tasks are left lying around. We should clean these up
in case a task accidentally uses an artifact from the wrong dependency.
Ideally we'd cache these properly based on the taskId they came from, but that can be
follow-up fodder.
MozReview-Commit-ID: HUgvNlqyFav
--HG--
extra : rebase_source : fb9c6723598223619993c2695fb588ead3325edb
Previously we would only generate upload-symbols tasks for nightly builds,
since we only want to upload symbols for builds we ship to users.
On try, developers may want to use the symbol server but very rarely want
to do nightly builds, so allow uploading symbols from any build type there.
MozReview-Commit-ID: IYs9mZii3DN
--HG--
extra : rebase_source : 15acb889510bb07ed3d29ea802997f11796dad9f
Tasks like upload-symbols that have a dependency on a build job want to
copy the treeherder.platform from the build job but it gets lost in
the task transform currently. This simply copies it into
an extra.treeherder-platform key to make life easier.
MozReview-Commit-ID: H4PtC4mvIYA
--HG--
extra : rebase_source : 5588bdadfcff8eb8f35935dc5c05dcde5658da83
filter_upload_symbols is a relic of task configurations that were written
before we had a better handle on taskgraph generation. We should only be
uploading symbols for nightly builds anyway, so this is better served using
newer filtering methods.
upload-symbols tasks were specified to run on non-nightly build types in the
kind.yml, but those were filtered out in filter_upload_symbols. I believe
these were simply an artifact of the initial upload-symbols implementation
happening before nightly builds were running in Taskcluster.
MozReview-Commit-ID: Je1NytrVPT8
--HG--
extra : rebase_source : a961c17d329af848fa7bb64c5186135d37dd412f
This is a follow up to 9c941daebe9fb3e79066ee4def16ed5ce0eb10a9
Differential Revision: https://phabricator.services.mozilla.com/D1140
--HG--
extra : rebase_source : 4c56ea683a7bcb3a80cd032e5491d3c13e891368
extra : source : 72e960bcc54751b15d144e0a5550c65649a2ad7d
We currently set GECKO_HEAD_REV to the revision being built, but the build
system already looks for MOZ_SOURCE_CHANGESET so set that as well. Additionally,
set MH_BRANCH for mozharness tasks on generic-worker to match docker-worker.
MozReview-Commit-ID: 52B3SSQpSwU
--HG--
extra : rebase_source : d65ca493764e6a5fbc7cca1d018b21cd6c82b6a0
extra : histedit_source : d7bd585e69fcc9b781550f89571250729882c8e8
This moves testing/profiles/prefs_general.js to
testing/profiles/common/user.js. It also adds an 'extensions' folder to the
common profile. Dropping extension files here will get them installed in all
test harnesses (useful for testing on try).
The idea is that all test harnesses will eventually use this 'common' profile.
We'll also create some new per harness profiles, e.g testing/profiles/mochitest
and testing/profiles/reftest. This way there will be a single location
developers can go to set preferences, both for a specific harness, and across
all harnesses.
MozReview-Commit-ID: 8sqBqLiypgU
--HG--
rename : testing/profiles/prefs_general.js => testing/profiles/common/user.js
extra : rebase_source : 72a4a4b691e93c77479c7e70647b0ca373862c51
This moves testing/profiles/prefs_general.js to
testing/profiles/common/user.js. It also adds an 'extensions' folder to the
common profile. Dropping extension files here will get them installed in all
test harnesses (useful for testing on try).
The idea is that all test harnesses will eventually use this 'common' profile.
We'll also create some new per harness profiles, e.g testing/profiles/mochitest
and testing/profiles/reftest. This way there will be a single location
developers can go to set preferences, both for a specific harness, and across
all harnesses.
MozReview-Commit-ID: 8sqBqLiypgU
--HG--
rename : testing/profiles/prefs_general.js => testing/profiles/common/user.js
extra : rebase_source : 7599913e547533f2f57b597ad0f238c6cd391c82
Previously, we installed the latest version of pip and virtualenv.
This commit pins the pip and virtualenv version so we install known
working versions (pip 10 breaks the image build for some reason).
MozReview-Commit-ID: hOAMencdcr
--HG--
extra : rebase_source : 2cb44c2ef55e29c55cf3d1b354e90d6fb5414cce
Justin Wood
Tom Prince
Csoregi Natalia
Mathieu Leplatre
Mike Hommey
Nick Thomas
Ciure Andrei
Kartikaya Gupta
Joel Maher
Dorel Luca
Coroiu Cristina
Bogdan Tara
Brindusan Cristian
Aki Sasaki
Mike Shal
Dustin J. Mitchell
Andrew Halberstadt
Johan Lorenzo
Margareta Eliza Balazs
Andreea Pavel
Ahilya Sinha
Chris AtLee
Sylvestre Ledru
Gregory Szorc
Raymond FOrbes
Sebastian Hengst
Jim Chen
Steve Fink
Cosmin Sabou
Geoff Brown
Andi-Bogdan Postelnicu
Ted Mielczarek
Maja Frydrychowicz
Narcis Beleuzu
Simon Fraser
Tom Ritter
Marco Castelluccio
Rob Wood
Ben Hearsum
Mihai Tabara
Dave Hunt