Python processes with a TTY have stdout line buffered by default.
Python processes without a TTY have buffered output.
Mercurial inherits whatever Python's output buffering behavior is.
This means if we invoke Mercurial without a TTY, stdout and stderr
will be fully buffered. This means output may not be sent until
there is enough output to flush the output buffer.
A consequence of this is that timings reported for `hg` commands
invoked by run-task are inaccurate. In addition, output order is
incorrect. This is because Mercurial's progress indicators print
to stderr and flush when written. This means stderr output is
getting seen by run-task but stdout remains buffered.
This commit forces Python/Mercurial to not buffer stderr and stdout
by setting the PYTHONUNBUFFERED=1 environment variable.
MozReview-Commit-ID: 7lMdrjRMBqz
--HG--
extra : rebase_source : 198ced0053fe6071a45c9df8b044b24983c225cc
We don't need a variable to hold the result. Just use return.
The "virtualenv_path" option has a default value, so it should always
be set. Add code confirming that. And refactor the code to use
less indentation. And remove a branch that can never occur since
the virtualenv path is guaranteed to be defined.
MozReview-Commit-ID: DZ6LnlxZJFj
--HG--
extra : rebase_source : 46682e9d33beb43e0b4fc181b9163afd373e7f70
Not sure why we support this. The code goes all the way back to the
import of mozharness 0.4 into the old mozharness Mercurial repo in
bug 651974.
Having fewer variations makes it easier to search for usage. So nuke
the variation.
MozReview-Commit-ID: IgwLMdvXGB0
--HG--
extra : rebase_source : 0f9e25b2d23a670154a024eee0264b545606cc80
The Python code is now intelligent enough to add this flag on the
command line if supported. Eliminate the copy pasta and help prevent
cargo culting.
MozReview-Commit-ID: H4rbjbbgtRd
--HG--
extra : rebase_source : 2f3acf666d73260ed97ad877f365b3091186f9e2
If mozharness is running from a source checkout, it has access to a
modern virtualenv+pip/setuptools vendored as part of the source
checkout.
This commit changes the virtualenv creation code to use the vendored
virtualenv when it is available.
A side effect of this change is that a modern version of pip will
now be used by mozharness when a source checkout is available. This
has a number of consequences.
First, modern versions of pip automatically create and cache wheels
when building packages. This should make automation faster since it can
now reuse cached wheels instead of having to download and rebuild
packages all the time.
Second, modern versions of pip support pinning package hashes. This
opens the door to use having more secure package downloads and more
determinism in our test environment.
Third, modern versions of pip require connections to package servers
be secure by default. Plaintext connections are disallowed by
default. A --trusted-host option or environment variable can be used
to override this behavior.
Since upgrading pip resulted in some jobs failing due to disallowed
connections to insecure servers, code to sniff the pip version and
add --trusted-host where it is needed/supported. This retains the
existing behavior. This is insecure. But fixing that is for another
bug.
As part of testing this, we were getting IOError inside virtualenv.py
when installing distutils:
IOError: [Errno 13] Permission denied: '/builds/slave/test/build/venv/lib/python2.7/distutils/__init__.py'
We worked around this by adding --always-copy to the virtualenv.py
invocation.
MozReview-Commit-ID: D29ao9ZASei
--HG--
extra : rebase_source : 031b2561a64ab1f89d25a3bfb8cf486a58b9f308
Now that we can detect when we're running from a source checkout,
we can start using things from source checkouts instead of relying
on host machine state or grabbing files from another server.
We start by using the vendored tooltool.py if available. This
avoids non-determinism. It avoids a possible 3rd party hosting
dependency on github.com. It avoids a possible MitM attack vector.
Wins all around.
MozReview-Commit-ID: L6hLveHZxBR
--HG--
extra : rebase_source : 67cc9d53fc0b3f92710ce41cc9f6556aa3ebbf99
We're going to start executing more mozharness scripts from a source
checkout. Rather than add config options to specify the location of
a source checkout - something that must be added to every mozharness
invocation - we teach BaseScript.__init__ to recognize when we're
running from a source checkout and set self.topsrcdir accordingly.
This will allow any script or class to check for self.topsrcdir
and change behavior accordingly.
MozReview-Commit-ID: 3uxOjol7ntR
--HG--
extra : rebase_source : 40795fe231a908b42a13581db3ee079c13138412
Without this, process output is buffered by default. This means
timestamps that mozharness prefixes to process output aren't
accurate unless the process is spewing enough output to flush the
output buffer.
Output buffering could lead to bad things. For example, a process
could emit output that would cause mozharness's output monitor to
abort the process. However, if that output is caught in limbo in
the output buffer, mozharness may take several seconds or even
minutes to react.
With this change, the mozharness process receives process output
as soon as that process writes to its standard file descriptors.
Once a newline is seen, mozharness will process it immediately.
Note that this only impacts the case where there is no output
timeout, as the existing code for output timeout uses mozprocess
and I'm pretty sure mozharness doesn't buffer output.
MozReview-Commit-ID: HBkYnfEw7Hb
--HG--
extra : rebase_source : e17b44d88f27c16b054a64c3cc2b3415297daf3b
Move packaging for Marionette from make to test_archiver by using root manifest files.
MozReview-Commit-ID: 1cxEBYQeJ2Z
**
--HG--
extra : rebase_source : 372a407d9207bfbccbfb88c689df60b8cc1abcaf
There were two assumptions preventing this output from being logged, both
related to the case a test passes and xpcshell returns 0. The first was
that we would not find crash dumps in this case, and would therefore not
need to log the full output of the test process (in the case xpcshell
returned non-zero or a test failed, we would log this output prior to checking
for crashes). The second was that if a test was eligible to retry, we wouldn't
need to store a test's output at all, because this output would only relate to
a failure that we would consider non-fatal.
The first assumption does not hold because it's possible to fatally assert
at shutdown in tests spawning child processes without causing a test failure
or non-zero exit code.
The second assumption followed from the first, and is violated when the first
is violated, because in this case we would consider a found crash fatal even
when a test was eligible to retry.
This patch reverses these assumptions and logs the full output of a test that
passes but produces crash dumps. It's not clear that the existing code intended
for a crash to ever be considered fatal when a test was eligible to retry, but
to change this criteria now would reduce our effective test coverage by
ignoring crashes that are now considered fatal, so after this patch we continue
to consider this scenario fatal. If it is determined these crashes are related
to these tests running in parallel with other tests, or they are not relevant
for some other reason, these tests should be run sequentially, or this criteria
should be changed.
MozReview-Commit-ID: 2PaFSGx2MVR
--HG--
extra : rebase_source : 34c0d1f13f4256928906729b1f3667bc395b2c56
We can assume that if middle button's click event on a link isn't consumed by any event handlers including system event group's, it will cause open new tab. With this assumption, we can avoid using setTimeout which causes random orange.
However, unfortunately, in e10s mode, the default is NOT consumed at window in bubbling phase but consumed at that time. So, when not working the link is expected, we cannot check Event.defaultPrevented. But fortunately, we can check if the page is loaded after that.
Note that for testing this, the test needs to check if an event handler which is either in default group or system group consumed a click event. However, this runs as mochitest-plain. Therefore, Event.defaultPrevented returns false if the event is consumed only in the system group's event listener. For avoiding this issue, this patch adds defaultPreventedInAnyGroups() into SpecialPowers. In SpecialPowers, Event.defaultPrevented is accessed from chrome context. Therefore, we can get the result what this test needs.
MozReview-Commit-ID: Cfn4lFR1dfI
--HG--
extra : rebase_source : 51feb768bd38f62cc19c2f4aecaaea0135190599
This also adds E402 (no imports at top of file) to the global ignore list. The
other error codes added were previously ignored by default, but now that we
have started a custom ignore list, need to be listed explicitly.
MozReview-Commit-ID: RtMuVEX6i5
--HG--
extra : rebase_source : 939bc9354f5891c680513d7e9068d0438e169132
Gregory Szorc
Grigory Kruglov
Carsten "Tomcat" Book
Andrew Osmond
Henrik Skupin
Chris Manchester
Masayuki Nakano
Wes Kocher
Phil Ringnalda
Tom Schuster
John Dai
Maja Frydrychowicz
CuriousLearner
Gabriele Svelto
Prabhjyot Singh Sodhi
Kim Moir
Rob Thijssen
Ted Mielczarek
Dão Gottwald
Domenic Denicola
Boris Zbarsky
Kilik Kuo
Steve Fink
Matthew Noorenberghe
Rail Aliiev
prakhar0409
bhuvi96