75c48c9453
Bug 1518313 - Removes --geckoProfile from talos-test mach command and replaces it with --gecko-profile; r=AlexandruIonescu,perftest-reviewers
...
Also removes import argparse as per https://bugzilla.mozilla.org/show_bug.cgi?id=1518313#c15
Addresses defect found in code analysis as per https://code-review.moz.tools/#/diff/242580
Changes parameters value as per https://phabricator.services.mozilla.com/D66380#2132590
Differential Revision: https://phabricator.services.mozilla.com/D66380
2020-06-09 13:22:15 +00:00
61d97f3f00
Bug 1642351 Set tasks_for in merge automation action to prevent source-tests r=rail
...
Differential Revision: https://phabricator.services.mozilla.com/D78672
2020-06-08 16:46:25 +00:00
eeadf81538
Bug 1643284 - require --full to schedule talos/raptor profiling jobs on try. r=davehunt,ahal
...
require --full to schedule talos/raptor profiling jobs on try.
Differential Revision: https://phabricator.services.mozilla.com/D78250
2020-06-08 15:19:55 +00:00
72a33db2ac
Backed out changeset 2ed99d13012d (bug 1591466) on ahal's request
2020-06-06 06:49:13 +03:00
c40751d405
Bug 1641131 - [taskgraph] Filter out devedition builds and tests with |mach try auto|, r=marco
...
Differential Revision: https://phabricator.services.mozilla.com/D78569
2020-06-05 20:28:44 +00:00
7ea3ea9209
Bug 1591466 - split raptor tests into tier-1 (high value) and tier-2 (lower value). r=sparky
...
split raptor tests into tier-1 (high value) and tier-2 (lower value)
Differential Revision: https://phabricator.services.mozilla.com/D78247
2020-06-05 21:07:21 +00:00
7e299a95cf
Backed out 3 changesets (bug 1640278) for verify lint failure CLOSED TREE
...
Backed out changeset e63b60fe21fe (bug 1640278)
Backed out changeset dbf1e5788c22 (bug 1640278)
Backed out changeset 27ae032f17be (bug 1640278)
2020-06-05 22:59:20 +03:00
7fa31e9752
Bug 1640278 - Fix bustage because empty {} is a dict not a set CLOSED TREE
...
Differential Revision: https://phabricator.services.mozilla.com/D78564
2020-06-05 19:40:48 +00:00
fe8c054562
Bug 1640278 - Fixup run-on-projects verification failures. r=sylvestre
...
This is in-part a followup to 1638260 which fixed some cases of this problem.
Differential Revision: https://phabricator.services.mozilla.com/D76746
2020-06-05 14:52:03 +00:00
b1ec9e9331
Bug 1640278 - Do validation of run-on-projects. r=catlee
...
Differential Revision: https://phabricator.services.mozilla.com/D76540
2020-06-05 16:41:21 +00:00
191a0b1695
Bug 1641131 - [taskgraph] Fix the 'push-interval-*' schema to not require an argument, r=jmaher
...
The Backstop optimization doesn't take any arguments, yet the schema for 'push-optimization-*'
requires a 'schedules-component' that goes ignored. Fix this.
Differential Revision: https://phabricator.services.mozilla.com/D77058
2020-06-05 15:38:24 +00:00
9e95f1a2e5
Bug 1637544 - Add a 'project' to the fake parameters used during taskgraph tests. r=tomprince DONTBUILD
...
Differential Revision: https://phabricator.services.mozilla.com/D78399
2020-06-04 21:52:34 +00:00
525a47b947
Bug 1637544 - Run no-sccache verification only on release projects. r=tomprince
...
Differential Revision: https://phabricator.services.mozilla.com/D77248
2020-06-04 19:40:54 +00:00
b2e275bf31
Bug 1637544 - Support project-specific verifications in taskgraph. r=tomprince
...
Differential Revision: https://phabricator.services.mozilla.com/D77247
2020-06-04 19:40:46 +00:00
0f1bc1a064
Bug 1633866 - [taskgraph] Improve 'guess_mozinfo_from_task' for more balanced chunks, r=gbrown
...
Adds a few properties to the mozinfo guess that were previously missing.
Differential Revision: https://phabricator.services.mozilla.com/D78156
2020-06-04 18:35:07 +00:00
ddaf15c72f
Bug 1633866 - [taskgraph] Implement a multiplier to tweak the duration for certain platforms / suites, r=gbrown
...
This is needed because xpcshell runs in parallel. Therefore we need to know the number
of threads are used (which varies by platform) in order to accurately determine how
many chunks are needed.
This allows us to specify a platform / suite specific multiplier to compensate via trial
and error.
Differential Revision: https://phabricator.services.mozilla.com/D78153
2020-06-04 15:34:51 +00:00
d4001d553e
Bug 1633866 - [taskgraph] Truncate runtimes that exceed DYNAMIC_CHUNK_DURATION so they don't artificially increase chunks, r=gbrown
...
For example, assume we have three manifests:
foo -> 98 minutes
bar -> 1 minute
baz -> 1 minute
And let's say we want average chunk lengths of ~20 minutes. The calculation
would currently say we need 5 chunks ((98 + 1 + 1) / 20). Even if set the max
chunks to the # of manifests, it would still return 3. Clearly this only needs
at most 2 chunks to run.
By truncating the runtime, we'd have:
foo -> 20 minutes (truncated)
bar -> 1 minute
baz -> 1 minute
So we'd actually only assign one chunk (round((20 + 1 + 1) / 20)). Despite
having a single 100 minute chunk, this outcome is desirable as we now avoid the
overhead of a second chunk that only runs 2 minutes worth of tests.
Depends on D77871
Differential Revision: https://phabricator.services.mozilla.com/D77873
2020-06-03 21:00:43 +00:00
b3fbd0cf70
Bug 1633866 - [taskgraph] Always use chunk number as symbol if it would otherwise be empty, r=jmaher
...
Depends on D76523
Differential Revision: https://phabricator.services.mozilla.com/D77871
2020-06-02 18:56:54 +00:00
27d78d7059
Bug 1633866 - [taskgraph] Create a 'bugbug' based test manifest loader, r=marco
...
Loads manifests using bugbug's push/schedules endpoint. For now we use as low
confidence threshold to select manifests. This is to try and both:
1) Improve regression detection rate
2) Save resources
at the same time. This way we theoretically shouldn't regress either dimension.
Note that we still optimize with CT_MEDIUM. My thinking is that at least all
of the "extra" stuff is still guaranteed to be relevant. Though we may want to
consider making these two thresholds match.
Differential Revision: https://phabricator.services.mozilla.com/D76523
2020-06-04 18:45:09 +00:00
be4e9089ec
Bug 1633866 - [taskgraph] Simplify how 'suites' are handled in manifest resolving and chunking, r=egao
...
This change is a straight refactor and results in identical full taskgraphs.
Differential Revision: https://phabricator.services.mozilla.com/D77543
2020-06-02 20:00:46 +00:00
dd5a8a4ba7
Bug 1642713: allow diff-artifact tasks to be optimized by push interval r=tomprince
...
Differential Revision: https://phabricator.services.mozilla.com/D77859
2020-06-04 14:27:42 +00:00
9eac413f54
Bug 1642446: Add an attribute to annotate toolchain tasks used by `mach bootstrap`; r=firefox-build-system-reviewers,rstewart
...
Toolchains that are used for local development need to be built on a level-3
branch to installable via `mach bootstrap`. Add an attribute to track the fact
that a toolchain is used that way, and:
- ensure that everything installed via `mach boostrap` has that attribute set
- ensure that everything with that attribute set is built on trunk projects
We could additionally verify that attribute is only set on things used by
bootstrap, but bootstrap doesn't currently have an exhaustive list of things
that it might install, making that difficult.
Differential Revision: https://phabricator.services.mozilla.com/D77706
2020-06-03 15:55:26 +00:00
40348063b3
Bug 1643029 - remove build-.*-aarch64 from being hidden behind --full; r=ahal
...
It's not good to hide Android AArch64 builds, a tier 1 platform, behind `--full`.
Differential Revision: https://phabricator.services.mozilla.com/D78070
2020-06-03 16:15:43 +00:00
6b213d3c19
Bug 1637810 - Define a new strategy that optimizes builds away if they are not selected by bugbug. r=ahal DONTBUILD
...
Differential Revision: https://phabricator.services.mozilla.com/D75621
2020-06-03 09:53:17 +00:00
8073961f7e
Bug 1641971: [taskgraph] Use task-reference to refer to the decision task in task; r=Callek
...
Differential Revision: https://phabricator.services.mozilla.com/D77552
2020-06-03 02:35:00 +00:00
ca6c9020de
Bug 1641971: [taskgraph] Always pass an explicit decision task id to create_tasks; r=Callek
...
Differential Revision: https://phabricator.services.mozilla.com/D77551
2020-06-03 02:35:03 +00:00
9f3450be95
Bug 1641971: [taskgraph] Pass explicit decision task id to action.json generation; r=Callek
...
Differential Revision: https://phabricator.services.mozilla.com/D77550
2020-06-03 02:35:05 +00:00
86711ecee8
Bug 1641971: Use `task-reference` to pass the decision task id to shadow schedulers; r=ahal
...
Differential Revision: https://phabricator.services.mozilla.com/D77549
2020-06-03 02:35:07 +00:00
581eba974b
Bug 1641971: Pass `mach perftest` options explicitly, rather than inpsect parameters; r=tarek
...
Differential Revision: https://phabricator.services.mozilla.com/D77548
2020-06-03 02:35:09 +00:00
6617f03f05
Bug 1641971: Allow references to decision task via task-reference/artifact-reference; r=Callek
...
Differential Revision: https://phabricator.services.mozilla.com/D77547
2020-06-03 02:43:34 +00:00
01e244f62c
Bug 1641971: Allow making references to a task itself in the task definition; r=Callek
...
This is to support linking to the a specific task in taskcluster.
Differential Revision: https://phabricator.services.mozilla.com/D77529
2020-06-03 02:35:14 +00:00
479c45bbea
Backed out 7 changesets (bug 1641971) for breaking Gecko Decision Task opt . CLOSED TREE
...
Backed out changeset a039e60a26d2 (bug 1641971164197116419711641971164197116419711641971
2020-06-03 05:30:51 +03:00
6f0a7bbba6
Bug 1641971: [taskgraph] Use task-reference to refer to the decision task in task; r=Callek
...
Differential Revision: https://phabricator.services.mozilla.com/D77552
2020-06-02 18:23:23 +00:00
f32e4a4f4d
Bug 1641971: [taskgraph] Always pass an explicit decision task id to create_tasks; r=Callek
...
Differential Revision: https://phabricator.services.mozilla.com/D77551
2020-06-02 18:23:19 +00:00
7c6a413ca9
Bug 1641971: [taskgraph] Pass explicit decision task id to action.json generation; r=Callek
...
Differential Revision: https://phabricator.services.mozilla.com/D77550
2020-06-02 18:23:20 +00:00
73b2e6f745
Bug 1641971: Use `task-reference` to pass the decision task id to shadow schedulers; r=ahal
...
Differential Revision: https://phabricator.services.mozilla.com/D77549
2020-06-02 18:23:11 +00:00
11aedca9b2
Bug 1641971: Pass `mach perftest` options explicitly, rather than inpsect parameters; r=tarek
...
Differential Revision: https://phabricator.services.mozilla.com/D77548
2020-06-02 18:23:07 +00:00
602da9cdcd
Bug 1641971: Allow references to decision task via task-reference/artifact-reference; r=Callek
...
Differential Revision: https://phabricator.services.mozilla.com/D77547
2020-06-02 18:50:10 +00:00
de6126516d
Bug 1641971: Allow making references to a task itself in the task definition; r=Callek
...
This is to support linking to the a specific task in taskcluster.
Differential Revision: https://phabricator.services.mozilla.com/D77529
2020-06-02 18:22:55 +00:00
4925d4d08e
Bug 1638990: Use requests for fetching parameters r=tomprince
...
Differential Revision: https://phabricator.services.mozilla.com/D76133
2020-05-28 19:01:42 +00:00
b6edad89bf
Bug 1638990: Sort values in taskgraph in order to make comparing output easier r=rstewart
...
Differential Revision: https://phabricator.services.mozilla.com/D77320
2020-05-29 15:01:50 +00:00
98dd864200
No bug: Don't use proxy for accessing taskcluster index; r=Callek
...
The index is not protected by scopes, so don't use the proxy to access it
Differential Revision: https://phabricator.services.mozilla.com/D77537
2020-05-29 21:37:46 +00:00
f0e5f6ed7b
Bug 1633866 - [taskgraph] Create a system to choose how test manifests are loaded via a parameter, r=tomprince
...
Currently test manifests are loaded by instantiating a TestResolver and
traversing moz.build files to find manifests.
With 'manifest-scheduling', we'll want to grab the manifests directly from the
bugbug service instead. Initially we'll want to enable manifest-scheduling with
|mach try auto|, but not on autoland yet.
This patch will allow |mach try auto| to set the parameter that causes bugbug
to be used (see future commits in this bug).
Differential Revision: https://phabricator.services.mozilla.com/D76522
2020-05-29 18:51:14 +00:00
c72f5589b1
Backed out changeset 2c18e64584af (bug 1641131) for gecko decision task bustage CLOSED TREE
2020-05-29 18:33:23 +03:00
ec4749227c
Bug 1641131 - [taskgraph] Make 'repackage' and 'repackage-signing' tasks copy their dependency's optimization, r=Callek
...
Differential Revision: https://phabricator.services.mozilla.com/D77346
2020-05-29 14:43:29 +00:00
12b0e45ee9
Bug 1641131 - [taskgraph] Make 'upload-symbols' tasks copy their dependency's optimization r=Callek
...
Differential Revision: https://phabricator.services.mozilla.com/D77048
2020-05-29 14:42:51 +00:00
ad57d3a5df
Bug 1641131 - [taskgraph] Make 'upload-generated-sources' tasks copy their dependency's optimization r=Callek
...
This should ensure that they are never the cause of their dependency being scheduled
and only run when they do.
Differential Revision: https://phabricator.services.mozilla.com/D76988
2020-05-29 14:42:44 +00:00
bd5660af34
Bug 1641131 - [taskgraph] Fix the 'push-interval-*' schema to not require an argument, r=jmaher
...
The Backstop optimization doesn't take any arguments, yet the schema for 'push-optimization-*'
requires a 'schedules-component' that goes ignored. Fix this.
Differential Revision: https://phabricator.services.mozilla.com/D77058
2020-05-29 14:48:07 +00:00
1b2e865f8c
Bug 1641283 - [taskgraph.optimize] Increase debug logging of optimization phase, r=tomprince
...
For each task, this sheds a bit of light on why it was removed or kept.
Differential Revision: https://phabricator.services.mozilla.com/D77147
2020-05-29 14:48:07 +00:00
e5ab1dc839
Bug 1641283 - [taskgraph] Upload bugbug push schedules API data to an artifact rather than the log, r=marco
...
Differential Revision: https://phabricator.services.mozilla.com/D77140
2020-05-29 14:39:38 +00:00
Kunchanapalli Shriram
Simon Fraser
Joel Maher
Bogdan Tara
Andrew Halberstadt
Justin Wood
Marco Castelluccio
Ben Hearsum
Tom Prince
Nathan Froyd
Narcis Beleuzu
Chris AtLee