This refactors the mozharness configs such that a specific flavor of an overall suite can still use the old
DesktopUnittestOutputParser. This is necessary because mochitest-jetpack still does not use the structured
logger.
MozReview-Commit-ID: E8EpSLH4xt2
--HG--
extra : rebase_source : be18e3e4aa1f693b563a82b90fdda2f93201d9ee
3fd83c9c0548 introduced the use of --trusted-host for use with pip.
Unfortunately, it unconditionally added --trusted-host, even when
easy_install is being used. This broke modules using easy_install
(which is currently limited to pywin32). Removing easy_install
and installing pywin32 is tracked in bug 1306691.
MozReview-Commit-ID: ExiO22EUr0B
--HG--
extra : rebase_source : 25dcded06a47ec8c0e61637bdd425131fe20a543
For Mozharness' download and extract action we're blocking when trying to read the contents from a url.
This mostly happens on Mac machines.
urllib2.urlopen()'s documentation mentions about a global timeout, however, it does not seem
to kick in or perhaps this won't do what we expect it to.
MozReview-Commit-ID: L9IDnDSjgzO
--HG--
extra : rebase_source : 595c0e99342a99d6afbe0303922bd5f067e26884
In the case for 'ondemand' symbols, we're OK to proceed without getting a hold of the
symbols from the get go. However, in other cases we need to at least retry
before giving up (e.g. debug tests need symbols)
MozReview-Commit-ID: 7LHflKhpze3
--HG--
extra : rebase_source : 9061fd9827a85e4b0c761d34041743aad87386cc
We also include the example APK, since it will be helpful to be able
to regression test the example without downloading the AAR and
manually building the example with that AAR.
MozReview-Commit-ID: CMtA1FRS0Rf
--HG--
extra : rebase_source : 42bab43c69093bd008242ec96e74f53bde150583
We need to bump the Gradle Deps task, which fetches dependencies, to
include new test dependencies; and use freshly uploaded tooltool
archives (manually uploaded) containing the new test dependencies.
MozReview-Commit-ID: 8bNOVQPHlk6
--HG--
extra : rebase_source : 0c80117fb58e43f9c857027941f0a14f03b97f13
partial_env only works if the class inherits from some other base
class, which apparently not all callers of this method do. So just
pass a copy of the environ dict with PYTHONUNBUFFERED added.
MozReview-Commit-ID: Ag75x28NR4D
--HG--
extra : rebase_source : f19b2be2db0a4b321542cc353a4599481ba60146
We did the same thing for run-task in bug 1304964 because otherwise
output may get buffered, triggering "no output in N seconds"
failures in automation.
We're starting to execute mozharness scripts from source checkouts
in automation. This means we can stop downloading and extracting
files that are already available in the source checkout and just
reference files from the checkout.
Web platform tests (WPT) are a logical place to start because they
are pretty well isolated. This commit modifies the mozharness script
for WPT execution to use files from a source checkout (previous
commits have guaranteed that a pristine source checkout is available
to the test execution environment).
As part of this, we also need to define an explicit mozinfo.json
path because previously it was relying on parent directory traversal
to find it (WPT tests were under a directory containing the
mozinfo.json file).
MozReview-Commit-ID: C1dlKC4eSzr
--HG--
extra : rebase_source : af6f5ad0f88739efc5f5e0c74106a50e845564f6
4 WPT config files all contained the same config options boilerplate.
Inline it into the WPT mozharness script.
MozReview-Commit-ID: 5Lba8QeKMTA
--HG--
extra : rebase_source : a99f8e25d04a3e3344db1cf280fd79c47177ccb4
list.extend() is favored over +=. Also use single quotes, fixup
indentation, and factor out a common variable.
MozReview-Commit-ID: 3qVDGrkYhVe
--HG--
extra : rebase_source : 7573dca951eca0611c86cbb42e3cf53ccae078f4
We add a mozharness action to the TestingMixin base class that ensures
we're running from a VCS checkout and we add this action to the WPT
script.
This ensures that we always have a source checkout available, even
in buildbot. (Before, we only had a source checkout in TaskCluster.)
MozReview-Commit-ID: 26NxwDZywXr
--HG--
extra : rebase_source : 6aea0390b0c9ff43972ef6fe6592f104401cc3fc
Rather than clearing actions in volatile_config, add in actions from the artifact config's
default_actions. Incompatible actions are then skipped based on 'forced_artifact_build'
config value.
MozReview-Commit-ID: IZuDvxcQ7cN
--HG--
extra : rebase_source : 265f973959d031617beb11852bb51e7d5f90c8bc
Now that we've removed support for using easy_install, we no longer
need the "install_method" argument to specify how we want to install
packages since there is only one method: pip. So remove that code.
MozReview-Commit-ID: BmjerQtfHov
--HG--
extra : rebase_source : 44427108c5a043ed929747323ea539dcda10c1cb
Support for easy_install was added in bug 761809 as part of supporting
pywin32. We just removed support for pywin32. And there are no in-tree
consumers using the "easy_install" install method. Furthermore,
easy_install is effectively deprecated as a package install mechanism:
pip should always be used.
So, we remove support for easy_install from mozharness.
MozReview-Commit-ID: CN1meLukqY6
--HG--
extra : rebase_source : 883e427f0b5b634a519c3564dd31577e9b164414
pywin32 was removed as a requirement to run Talos in bug 726700,
~3 years ago. The references in mozharness were never updated,
apparently.
MozReview-Commit-ID: FMYxLCNa63H
--HG--
extra : rebase_source : 424b9b301a1c615acd3fd221df50e10a6c00d2cb
Proxxy is only configured in buildbot land. Don't enable it in
TaskCluster.
Ideally, we'd only enable proxxy if we detect we're in a buildbot
environment. But the change in this commit is more conservative
and aligns with existing behavior.
MozReview-Commit-ID: HBCdQ6MkYGa
--HG--
extra : rebase_source : 08c753d7af7a4ee95c557d9deb6401c4f2da4547
So we can detect when we're running on TaskCluster. This will
be used to adjust environment settings accordingly.
MozReview-Commit-ID: JEG1E3tWsc5
--HG--
extra : rebase_source : 2acb70bd9accbde44ccb8530002ba1e892b94ce2
Downloads geckodriver from tooltool when wdspec tests are being run,
and adds the --webdriver-binary argument
MozReview-Commit-ID: AJeP0YDk7Yl
--HG--
extra : rebase_source : 497f25c5af32b1851adf3a6f0b90a20640b6ccc6
Failing to find symbols in this case should be turned into a warning rather than dumping the traceback
since we're going to rely on mozcrash doing the right thing later on.
This will reduce unnecessary reporting of symbols not being available.
MozReview-Commit-ID: GXO01B7vzGV
--HG--
extra : rebase_source : 99ff82ffca6eed209ce6fd31ab747239d7100516
The virtualenv is placed in the "work dir" by default. If we
clobber the "work dir" at the beginning of the job, the parent
directory of the virtualenv may not exist and virtualenv creation
will fail because we set cwd to the work dir.
Fix that by ensuring the work dir / cwd always exists when
creating the virtualenv.
MozReview-Commit-ID: FAZPP1Sm22k
--HG--
extra : rebase_source : 126443cbcd5c83aeb47848bfc90ae28be9c9f596
From changeset 3282813aa892f0fc247181a33ce0bde2b751da50 from the
version-control-tools repo. File added without modifications.
Upstream change was peer reviewed.
Failing to find symbols in this case should be turned into a warning rather than dumping the traceback
since we're going to rely on mozcrash doing the right thing later on.
This will reduce unnecessary reporting of symbols not being available.
MozReview-Commit-ID: GXO01B7vzGV
--HG--
extra : rebase_source : 5fa15dcf89bedea2b4e6ff52f6d06461fe5e208d
download_unpack() is managing to download files correctly, however, sometimes we get an exception
that the zip file is corrupted.
This change adds more logging and saves the fetched file to disk in order to get uploaded as an artifact
for inspection.
MozReview-Commit-ID: 2KCK6qGNor4
--HG--
extra : rebase_source : 05f29616c90f36991582d285c6fa00d62fe06b40
In automation, we try to use pypi.pvt.build.mozilla.org nearly
everywhere. This hostname doesn't resolve in TaskCluster and
outside of buildbot automation.
A consequence of work in bug 1304176 and using a modern pip is
that we attempt to connect to all defined pip links. This was resulting
in pip attempting connections to pypi.pvt.build.mozilla.org. And
since pip was attempting retries, this resulted in a several
second delay for all pip operations if that host didn't resolve.
This commit adds a DNS lookup in mozharness before using a pip
--link. We spend a little bit of overhead in mozharness doing a
DNS lookup. In return, we guarantee we'll avoid a multiple second
pause if any links don't resolve. This is somewhat hacky.
But it seems like the easiest solution.
MozReview-Commit-ID: EecqBQSW75R
--HG--
extra : rebase_source : 9664a3694232e4ce2dec216b036ba29783c2842d
Check try message for --artifact even if fx_desktop_build.py is run with
--skip-buildbot-actions
We can't rely on buildbot config. Add checks to TryToolsMixin._extract_try_message so
that it works even if self.buildbot_config is None.
MozReview-Commit-ID: 1xErjuOArBe
--HG--
extra : rebase_source : 2f3204b37e67fd9a77dbff0fa93ab894b08181c1
From changeset 872711d144202b3f95e090a95f45cc1c45831caf from the
version-control-tools repo. File added without modifications.
Upstream change was peer reviewed.
In bug 1305752 we discovered that we download a zip file into memory without any issues, however,
when we tried to unzip we discovered that we have an invalid zip file.
The information in the logs is not sufficient to determine what could be the root issue.
MozReview-Commit-ID: DKwDUCmUhFF
--HG--
extra : rebase_source : 7b1ea32dcdc497831a387986d4449fffa37f54da
We're starting to execute mozharness scripts from source checkouts
in automation. This means we can stop downloading and extracting
files that are already available in the source checkout and just
reference files from the checkout.
Web platform tests (WPT) are a logical place to start because they
are pretty well isolated. This commit modifies the mozharness script
for WPT execution to use files from a source checkout (previous
commits have guaranteed that a pristine source checkout is available
to the test execution environment).
As part of this, we also need to define an explicit mozinfo.json
path because previously it was relying on parent directory traversal
to find it (WPT tests were under a directory containing the
mozinfo.json file).
MozReview-Commit-ID: C1dlKC4eSzr
--HG--
extra : rebase_source : fad7905abafaf126329aa25a96efafc40b01051f
4 WPT config files all contained the same config options boilerplate.
Inline it into the WPT mozharness script.
MozReview-Commit-ID: 5Lba8QeKMTA
--HG--
extra : rebase_source : c613db4107128ad2fd4ba568836ec27e0a9a92a8
list.extend() is favored over +=. Also use single quotes, fixup
indentation, and factor out a common variable.
MozReview-Commit-ID: 3qVDGrkYhVe
--HG--
extra : rebase_source : 0c5faa30f5be09165f61943300ecfb2e9c7080f4
We add a mozharness action to the TestingMixin base class that ensures
we're running from a VCS checkout and we add this action to the WPT
script.
This ensures that we always have a source checkout available, even
in buildbot. (Before, we only had a source checkout in TaskCluster.)
MozReview-Commit-ID: 26NxwDZywXr
--HG--
extra : rebase_source : d9c0fade450ab14c0b52be674c3c92bf670d2d3b
These are the only 2 definitions of the hg.mozilla.org certificate
fingerprint in mozilla-central. The certificate changed on
2016-09-26. So update the fingerprints.
This /might/ be a cargo cult because automation should be pinning
the hg.mozilla.org certificate everywhere. An alternative to this
commit would be to remove the fingerprint pinning from these
scripts. But if these scripts are run by humans, we may want to keep
the pinning in.
MozReview-Commit-ID: 4FUhAGc2oqx
--HG--
extra : rebase_source : fa8889ffbb70c14270acde67121192f7c1932258
Paul Yang
Gregory Szorc
James Graham
Armen Zambrano Gasparnian
Joel Maher
Simon Fraser
Phil Ringnalda
Sebastian Hengst
Geoff Brown
Nick Alexander
Wes Kocher
Maja Frydrychowicz