While some builds have a PERFHERDER_EXTRA_OPTIONS environment set on the
taskcluster side, many others have the equivalent set at the mozharness
level. But only the former are actually linted against, which,
unsurprisingly, translates to conflicting values between some of the
mozharness configs.
So we move those configurations to taskcluster, enable the lint on all
the kinds that look like builds (based on them using the build_attrs
transform), and adjust the values to stop conflicting. Notably, for
searchfox and static-analysis-autotest.
--HG--
extra : rebase_source : 097333608e61e1df66e5d8f914e15784f35e58f2
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1468
--HG--
rename : testing/mozharness/configs/single_locale/mozilla-central.py => testing/mozharness/configs/single_locale/autoland.py
rename : testing/mozharness/configs/single_locale/mozilla-central_android-api-16.py => testing/mozharness/configs/single_locale/autoland_android-api-16.py
rename : testing/mozharness/configs/single_locale/mozilla-central.py => testing/mozharness/configs/single_locale/mozilla-inbound.py
rename : testing/mozharness/configs/single_locale/mozilla-central_android-api-16.py => testing/mozharness/configs/single_locale/mozilla-inbound_android-api-16.py
extra : rebase_source : 93c818ec8b949d211545fd57f2f3cd6ae5ed5fa3
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1409
--HG--
extra : rebase_source : 2798c5bc3e3153f8c293846d5a3d786e18bbdc34
This patch fixes baseline test addition. It allows all baselines to run in all test suites. Previously, only the test suite which had too many tests executed in it, was searched to the end for any baselines. Now, we continue to all other test suites before we finish.
MozReview-Commit-ID: FWMAsHb22CO
--HG--
extra : rebase_source : 7dbcdb3977bd351367ff9fa7892e41c2b86a2dce
The crash reporter symbol files are the easiest cross-platform way to
find static initializers. While some types of static initializers (e.g.
__attribute__(constructor) functions) don't appear there in a notable
way, the static initializers we do care the most about for tracking do
(static initializers from C++ globals). As a matter of fact, there is
only a difference of 2 compared to the currently reported count of 125
on a linux64 build, so this is a good enough approximation. And allows
us to easily track the count on Android, OSX and Windows builds, which
we currently don't do.
The tricky part is that the symbol files are in
dist/crashreporter-symbols/$lib/$fileid/$lib.sym, and $fileid is hard to
figure out. There is a `fileid` tool in testing/tools, but it is a
target binary, meaning it's not available on cross builds (OSX,
Android).
So the simplest is just to gather the data while creating the symbol
files, which unfortunately requires to go through some hoops to make it
happen for just the files we care about.
--HG--
extra : rebase_source : 458fed1ffd6f9294eefef61f10ff7a284af0d986
Thunderbird releases need to look at comm-beta/comm-esr* branches for old
locale/version information.
Differential Revision: https://phabricator.services.mozilla.com/D1413
--HG--
extra : rebase_source : 76625ea5859d25f270b9fbec577f9075988bf2b7
When running talos locally with --geckoProfile set, the latest gecko-profile archive will automatically be loaded in perf-html.io using the view-gecko-profile tool. To disable this automatic perf-html.io launching, set TALOS_DISABLE_PROFILE_LAUNCH=1.
MozReview-Commit-ID: 8tpLnsPAXD9
--HG--
extra : rebase_source : 66d03b55103e9771c4c8c4c70ff67212f24c1124
This patch prevents baseline coverage tests from being skipped when too many tests are being run.
MozReview-Commit-ID: JVTOYZAXbwf
--HG--
extra : rebase_source : dedd6a323445f030b60180805c6c6adf5d10771b
This adds section size metrics in order to track finer grained improvements
and regressions in binary size. Currently it implements tracking of:
- XUL
- NSS
- NSPR
- mozavcodec
- mozavutil
The sections tracked are limited in order to avoid too much noise:
- .text
- .data
- .rodata
- .data.rel.ro
- .bss
Currently this is limited to measure Linux and Android builds, but can be
easily extended to support other platforms once we have a `size`-like tool
available.
--HG--
extra : rebase_source : 494922e60c1ea47392e3121425d7aacef6c3003a
Missed some references to run_command_m, get_output_from_command_m when
removing the mock mixin.
--HG--
extra : amend_source : 03626d4f65d06eaffab0ee754a9547d5624baa09
extra : histedit_source : 13bacfc1ccff78a1e17c996f66465439a7aa16dc
The geckoview-junit tests require the OSS audio backend for the Android
4.3 ARM emulator, but mochitests don't work well with the OSS audio
backend. Therefore, use a different config file for the geckoview-junit
tests.
MozReview-Commit-ID: 20tzjtVdTuB
This patch fixes the addition of baseline tests into self.suites when the suite doesn't exist, and we are iterating over self.suites. Now, we find all baseline tests to run, then add them into self.suites when we are not iterating over self.suites.
MozReview-Commit-ID: GPQeCw1J9P3
--HG--
extra : rebase_source : c6fdfddea15a9e261431f03266784a932318b12f
This patch enables differencing test coverage reports with baseline reports so that test reports contain what is unique to the test and excludes what is common to all tests (based on test file types: html, xul, js).
MozReview-Commit-ID: LHzlFZ72Ufi
--HG--
extra : rebase_source : ef4e0b5505cc85e95e1717d056d38c00d78872c5
The tests added in this patch will be used as a baseline for coverage that is common to all tests. They are added to each chunk being run in the test-coverage suite only if a test with an associated baseline type exists.
MozReview-Commit-ID: 1CrRZ1Ev2Mz
--HG--
extra : rebase_source : e2e4f9a51c82c637c8892dd8a5dbd84704b67b54
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1409
--HG--
extra : rebase_source : 1fd925beca600888ccf77f44f48a3c34b0d54c75
We need testvars for the base case as well.
--HG--
extra : rebase_source : 0fbe7b3e206262d47f78e1289dceb02878ad987b
extra : histedit_source : fd1eabcc887e91dff8f4ec2e00b03fcd8bb68c9b
This commit adds CI tasks to perform "plain" builds. These builds use
the same toolchains used by official builds. But that's about it. The
mozconfig changes are minimal and only set up paths to toolchain
artifacts. sccache is not used.
The main goal of these builds is to have a "reference" build that
matches an out-of-the-box build environment as much as possible. We want
this mainly so we have timing and behavior information that matches what
developers are doing.
The Windows/generic Taskcluster worker doesn't like it when you specify
an artifact directory that doesn't exist. So we needed to add a
mozharness step to ensure UPLOAD_PATH exists to prevent those tasks from
erroring.
I'm not super thrilled about using mozharness here. We definitely don't
really need mozharness. But the main thing it is providing that we care
about is the Perfherder metrics data. I don't feel like scope bloating
to move that out of mozharness at this time.
I only implemented Linux64 and Windows64 because I'm not convinced
coverage on 32-bit build variations is useful. Tasks only run on trunk
because they are informational only and we don't need to waste resources
running these on release repos and on Try. They are tier 2 because they
aren't critical to shipping Firefox.
MozReview-Commit-ID: Gl6hGYbFX9b
--HG--
extra : rebase_source : 05360d2f6e5dbfed5543726a1be4b0e5d00e0b3d
On ESR[60] stub installer isn't designed to work, and we expect most users of esr will have no use for a stub installer. However this poses some problems where the taskgraph assumes that any Nightly generates a stub installer and thus it should be available along the way.
The patch hardcodes the list of branches that do not need a stub installer, though we'll want to clean up this specification in some way, as Thunderbird will likely need it to be cleaner and we may want on-change builds to use this logic (e.g. for on-change l10n).
Differential Revision: https://phabricator.services.mozilla.com/D1082
--HG--
extra : source : 2f091908b8839650961c3968b6beee1dd8d1084b
Since it is run with `mach python`, it uses the environment defined by
`build/virtualenv_packages.txt`. Thus, we don't need to create a separate
virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D1015
--HG--
extra : rebase_source : 023095f82d7219a10dffb3579276c5285db37cfc
extra : source : a2999a5b9b7aa08a7c5c2ba6384724853d14b9c5
This patch stops test-coverage (TC/TCw) from trying to create a zip file of the per-test coverage reports when no tests are run.
MozReview-Commit-ID: 6m3TR4oUCLx
--HG--
extra : rebase_source : cc9ae3b90c940913be3526a79461af0cd5617da6
This will allow mozharness configs to be specified exclusively in the taskgraph.
Differential Revision: https://phabricator.services.mozilla.com/D1017
--HG--
extra : rebase_source : a3a9b6cc9d1004c4bd396fccc3e4354a7316651d
extra : source : 10acd193df92b7c495789dc24157b85f116ade5e
With the transition to Taskcluster, "uploads" are artifacts in a local
directory. So we don't need to support uploading to a remote server
using SCP.
This commit removes all the code to support uploading to a remote
server.
And since property files were only written out for the remote case, all
that code can be deleted as well.
Since UPLOAD_HOST no longer means anything, we no longer set it in
mozharness configs.
MozReview-Commit-ID: 66gkM8erkGk
--HG--
extra : rebase_source : ee85bb927cfb98e1ca383ab2591febfc1f0ce5cd
With the transition to Taskcluster, "uploads" are artifacts in a local
directory. So we don't need to support uploading to a remote server
using SCP.
This commit removes all the code to support uploading to a remote
server.
And since property files were only written out for the remote case, all
that code can be deleted as well.
Since UPLOAD_HOST no longer means anything, we no longer set it in
mozharness configs.
MozReview-Commit-ID: 66gkM8erkGk
--HG--
extra : rebase_source : 15143a18598370eeaaa98f8f0ac8458d758db51f
We should always be using sccache these days. We shouldn't need
ccache on PATH in any modern builds.
MozReview-Commit-ID: 1czGAcHhRGC
--HG--
extra : rebase_source : 6c498c429d7c9f79fee4d668e2656e83efb7d550
This is a buildbot holdover and should no longer be relevant in a
Taskcluster/Docker world.
MozReview-Commit-ID: NeIN0jnW5I
--HG--
extra : rebase_source : 9fe63c448323f91ca2067d4520e7ac65703cc2be
First, we shouldn't need Git for most jobs. Second, this path shouldn't
exist in a post-buildbot world using Taskcluster/Docker.
MozReview-Commit-ID: 7S8jsPiD2dT
--HG--
extra : rebase_source : a7cde452225c0e27b7153cc297b26eaa3ccbfade
This path should be dead in a post-buildbot Taskcluster/Docker world.
MozReview-Commit-ID: KRsRlfFADvD
--HG--
extra : rebase_source : 75156921aa99c89a181b6c6716b31bb560b64453
This path shouldn't exist in a post buildbot world.
MozReview-Commit-ID: 1O7A4xAonHz
--HG--
extra : rebase_source : 96d4c4e082b3454c66b9312b702ec141afc9ed52
This path was used on buildbot. It doesn't exist in a Taskcluster/Docker
world.
MozReview-Commit-ID: KHeRGcfIftO
--HG--
extra : rebase_source : 75232e7a5f65db4f2ea216349c20922bdf5c408b
To ease investigation of failures the gecko log should be streamed
to stdout so it will be part of the default log. It helps with
correlating tracing output with the appropriate test.
MozReview-Commit-ID: JnH64bhhtgk
--HG--
extra : rebase_source : b50707189c181a865ab66dac8b3cb4e258a8e427
This allows Thunderbird to keep branch specific options in comm-central.
Differential Revision: https://phabricator.services.mozilla.com/D936
--HG--
extra : rebase_source : 5d6c552c52939433868bcf6c91789ac512bd8597
extra : amend_source : 135130754a8c4a9d78000983ee407e0de719858a
Talos uses an in-tree module that it installs in code, ignoring the config value.
Differential Revision: https://phabricator.services.mozilla.com/D986
--HG--
extra : rebase_source : d58e4b047224b5109ad00319f6469f7acd4ea0d8
extra : amend_source : a0b3dc220db32d77e8076b12d55d9dce0fedc9f9
New versions of pip don't install entrypoint scripts with a `-script.py` on
windows. Instead, just call the bare script name everywhere. On unix-like
systems, this uses the shebang; on Windows, this executes the `.exe` wrapper.
Differential Revision: https://phabricator.services.mozilla.com/D967
--HG--
extra : rebase_source : 16e7bcd9efc2376f9a948f948bc81fd6596486d0
extra : source : 891a59eaf8e5e8267154d1d3872322705efd8e1a
We now vendor pip via the vendored copy of virtualenv. Stop trying to install a
specific version of pip elsewhere, particularly stop downgrading it.
Differential Revision: https://phabricator.services.mozilla.com/D962
--HG--
extra : rebase_source : abb5c076747f229aa4379fa3989c9abf36b45d0b
extra : source : 747d0b814dc1be3e5ac04e080361ab0b0fc034f9
This makes the changes necessary to use TestRunnerActivity when geckoview
is installed and requested, but we do not yet attempt to run any such
test tasks in automation.
If you happen to be unfortunate enough to be testing changes that crash
xpcshell during the xpcshell self-tests, you'll be presented with error
messages like:
PROCESS-CRASH | test_child_assertions.js | application crashed [unknown top frame]
Crash dump filename: c:\users\task_1522676338\appdata\local\temp\xpc-other-mtot6h\cfaea928-e995-4430-baf9-0066c6b91be9.dmp
MINIDUMP_STACKWALK binary not found: z:\build\build\tools\breakpad\win64\minidump_stackwalk.exe
and then be told that, essentially, "your test failed because it
failed", which is unhelpful for figuring out what went wrong.
Apparently we had MINIDUMP_STACKWALK set at one point, and then it got
moved. Let's fix this situation by a) downloading minidump_stackwalk
out of tooltool (our test harnesses do this already); and b) pointing
MINIDUMP_STACKWALK at the correct location. With these changes, we can
once again get crash stacks if xpcshell (and probably a few other
things) fail their self tests.
robustcheckout from revision f0c7808d6c00e2bbf3d7325ac4cb8eb0a2b0e939
of version-control-tools is vendored without modifications.
MozReview-Commit-ID: HYqmEoFmRYq
--HG--
extra : rebase_source : e147e27d6b01fa71b77a74345f713b1a9cb908bc
OSX jittests have been running as a no-op for a while, due to a mozharness mis-configuration.
This fixes the configuration so that the test suite actually runs.
Android jit tests are not ready to run yet: They may not run green, there are concerns about
capacity, etc. I am adding this support now to make it more convenient to debug on try.
--HG--
extra : rebase_source : 00bc5bf21fec3c130133b0de322b1f37250893c3
This small change is actually very significant. Previously, |mach
package| for mobile/android had two jobs:
1) produce a final APK
2) rebuild parts of the APK that might have been silently modified by
l10n mechanisms, both from multi-locale builds and single-locale
repacks
This second part has never been sensible but has been difficult to
alter until recently, since the l10n mechanisms have been out of
mozilla-central and difficult to modify and test. That's less true
now.
This patch:
a) removes the rebuild parts (the step labeled 2) above (which I
generally refer to as the "nodeps mechanism")
b) uses the APKs produced by Gradle directly, without the copying
indirection from m/a/base/Makefile.in
c) does the rebuild for multi-locale builds as an explicit step in the
appropriate mozharness script
d) does the rebuild for each single-locale repack as another step in
the existing `installers-%` target in m/a/locales/Makefile.in (it's
not easy to remove this from the Makefile, since the repackage is
invoked immediately after (it's the `repackage-zip-$*` target))
The new m/a/gradle.py file will grow additional tasks in tickets to
follow, hence the lock file and pre-factored form.
MozReview-Commit-ID: IKflLdmHR3P
--HG--
extra : rebase_source : fdabe340b6f0896a0ebb9da2951f10753deb5ff5
You can still run them on a --disable-stylo build, as long as that works
(presumably not for long).
I think I haven't missed anything, but please double-check.
MozReview-Commit-ID: 3BIAEjgTLo5
This backs out the source readme changes, and gets us to the original source tarball, but massaged for taskcluster's signing+beetmover model.
MozReview-Commit-ID: QIaeQ9LdLb
--HG--
extra : rebase_source : 4677497da550e98a4d07c16169c0949c3ec495b9
In 605111b6f51f, we removed a bunch of unused actions. However, now that we're recreating the source tarball, some of those are no longer unused. This patch brings them back.
MozReview-Commit-ID: 5WZMEeuatup
--HG--
extra : rebase_source : f725e6cacd692357bc8e4194afb052e2c29b99b1
So, it seems we've (since I landed windows l10n in taskcluster) been using http://relengapi/tooltool/ as the tooltool url, this has only been working because if we have the relevent file in local cache we don't query against tooltool.
With TC's windows spinup process it fetches a set of tooltool files to cache locally, and this has been saving us, meaning we're very lucky.
However maple (beta-based) and central have just diverged in manifests, because central now uses a newer VS version than what we have on maple.
This patch will fix that to use the new tooltool location that we currently use for Windows builds, and aligns the repackage jobs to use the newer url too.
--HG--
extra : rebase_source : 4a50d23fcccade23a52c09265f06d978d0090608
We shouldn't really have this problem, and shouting about it early will
make sure that we address issues like multiple instances of omni.ja,
rather than not recording information that we should have been.
Unlike Android installers, installers for desktop versions of Firefox
contain two copies of omni.ja, and the code to check for omni.ja was
ignoring both copies. Let's special-case omni.ja so we get better
numbers for our desktop platforms.
The zipfile and tarfiles paths for finding interesting files in the
installer have duplicated code. We can eliminate this duplicated code
by factoring out a function to just get the paths and sizes for files
contained in the installer. If we need to make changes to how paths are
processed, we now only have to make the change in a single place, and we
can add other kinds of installers easily in the future.
The configuration doesn't need to be dynamic, so removed the code that existed
to support it.
Differential Revision: https://phabricator.services.mozilla.com/D542
--HG--
extra : rebase_source : 34b317b846ed4d8c54344b2379bbd4100a8623e5
extra : source : 36171867cedd5160dc230470033a60d31a7fbee5
Note that this still looks at "buildbot properties" set by mozharness itself.
Differential Revision: https://phabricator.services.mozilla.com/D576
--HG--
extra : rebase_source : 7e4771d4f1ffdf0959068462ead1b9b2e1154c64
The upload is handled by taskcluster, which is handled by copying files around,
so remove support for specifying alternative commands.
Differential Revision: https://phabricator.services.mozilla.com/D540
--HG--
extra : rebase_source : 968249c0a308178b62af77d48e6aa307d4192e5a
These are overriden in the `tc_*.py` config files.
Differential Revision: https://phabricator.services.mozilla.com/D535
--HG--
extra : rebase_source : d57bfb74a61c9d8d279769ca2dbf39e8bdd9f6e1
On Windows, some harnesses may produce incorrect test paths if a command
line argument specifies a test path in native format, like some\\dir\\test.js.
With this patch, test-verify consistently specifies test paths in posix
compliant format, like some/dir/test.js.
Previously, we were downloading tooltool.py from random servers.
Considering tooltool.py is used to secure the download of future
components, downloading tooltool.py from potentially 3rd party
services was a major lapse in our end-to-end security, as a
compromised tooltool.py wouldn't honor integrity checks.
This commit copies the already vendored copy of tooltool.py into
the mozharness directory. A copy needs to be in the mozharness
directory because then a copy of mozharness without access to
a source checkout will have access to it.
We modify the code in mozharness that fetches tooltool to use
the copy from mozharness (unless `mach artifact toolchain` is
available).
Since a copy of tooltool.py is always reliably available, we
can remove all config entries related to tooltool.py.
MozReview-Commit-ID: C7ls1xWrPMq
--HG--
rename : python/mozbuild/mozbuild/action/tooltool.py => testing/mozharness/external_tools/tooltool.py
extra : rebase_source : d7b48d837805f9312c97b6e21c6527cc5f5018dc
This is a new issue that gets linted with flake8 3.5.0. Basically you should
never use a blank except: statement.
This will catch all exceptions, including KeyboardInterrupt and SystemExit
(which is likely not intended). If a catch all is needed, use
`except: Exception`. If you *really* mean to also catch KeyboardInterrupt et
al, use `except: BaseException`.
Of course, being specific is often better than a catch all.
MozReview-Commit-ID: FKx80MLO4RN
--HG--
extra : rebase_source : 7c74a7d0d81f2c984b47aff3a0ee3448b791177b
In general, there is no simple mapping between apk name (geckoview_example.apk) and
package name (org.mozilla.geckoview_example). With bug 1434411, it will be relatively
easy to add or modify tasks to use a geckoview apk in taskcluster tests. At the
mozharness level, scripts are expected to expand mozharness configurations containing
"%(app)" into package names. For Firefox, android_emulator_unittest extracts and
reads package_name.txt, but there is no such file in the geckoview apk. In future
we might add package_name.txt to the geckoview apk, or possibly use a tool like aapt,
but for our immediate needs this simple hack does the job: If "geckoview" is in
the apk name, assume we are installing org.mozilla.geckoview_example.
When the emulator is in good shape, emulator verification on shutdown
only takes a few seconds. However, if the emulator is hung or otherwise
unresponsive, emulator verification can take several minutes. In the
pathological worst case, verification slows down the task enough to
exceed the task timeout, resulting in a failure that would otherwise
have been handled as a task retry. Emulator verification on shutdown
provides little value, so it is simply eliminated.
The MACHTYPE bash variable is an odd thing that returns e.g.
x86_64-redhat-linux-gnu on a CentOS system, but x86_64-pc-linux-gnu
on a Debian system, and possibly something different on other distros.
mach valgrind-test is the only place actually relying on MACHTYPE.
Others rely on information from python modules. Uniformize that, and use
the more generic 'pc' rather than 'redhat'.
--HG--
rename : build/valgrind/i386-redhat-linux-gnu.sup => build/valgrind/i386-pc-linux-gnu.sup
rename : build/valgrind/x86_64-redhat-linux-gnu.sup => build/valgrind/x86_64-pc-linux-gnu.sup
extra : rebase_source : ad94ce69e8094d2b9ddae97a3d261945886c0a61
When we branched to beta, the MinGW build started failing with a strange error
like "could not find the robustcheckout Mercurial extension". We disabled it
for a while, but now we're re-enabling it (and fixing the problem.)
The root culprit of this was that we were using the incorrect mozconfig. MinGW
does all sorts of stuff in the mozconfig, but the beta branch overrides the
mozconfig using platform_overrides in
testing/mozharness/configs/builds/branch_specifics.py
We avoid this override by changing the MinGW platform so it doesn't match
and the mozconfig doesn't get overridden.
MozReview-Commit-ID: JkETWCRHacO
--HG--
extra : rebase_source : 5ef7d96804acd3a4eaa6dd6cc8b483f3de438b61
It is no longer used as part of the build, now that publishing is handled in
separate tasks.
Differential Revision: https://phabricator.services.mozilla.com/D373
--HG--
extra : rebase_source : a6e31fe5d554076d4da584f0993c21b6c1a8ebd0
ccache was disabled in Bug 1426788 (ba9e7b695619) so setting variable to
control it isn't needed anymore.
Differential Revision: https://phabricator.services.mozilla.com/D368
--HG--
extra : rebase_source : 1db59b9eaa5bb887dfdb7382bcd940b647af967f
Chris AtLee
Cosmin Sabou
Tom Prince
Noemi Erli
Marco Castelluccio
Mike Hommey
arthur.iakab
Justin Wood
Gregory Mierzwinski
Csoregi Natalia
Nick Thomas
shindli
Geoff Brown
Joel Maher
Ciure Andrei
Rob Wood
Dorel Luca
Eric Rahm
Andreea Pavel
Chris Manchester
Margareta Eliza Balazs
Raymond FOrbes
Jim Chen
Narcis Beleuzu
Andi-Bogdan Postelnicu
Maja Frydrychowicz
Brindusan Cristian
Aki Sasaki
Bogdan Tara
Ben Hearsum
Johan Lorenzo
Gregory Szorc
Venkatesh
vprabhu
Henrik Skupin
Mihai Tabara
David Major
Mike Shal
Nathan Froyd
Emilio Cobos Álvarez
Sylvestre Ledru
Nick Alexander
Rail Aliiev
Coroiu Cristina
William Lachance
Gurzau Raul
Jared Wein
Jordan Lund
Bob Clary
Tom Ritter
Jesse Schwartzentruber
Andrew Halberstadt
Ionut Goldan
vinicius