There's some trade-offs in play here: the major issue is that we can't
pin `psutil`'s because it's pre-installed on some CI workers with a
different version (5.4.2).
Additionally, we can't "just" bump that version, because CI workers jump
between different revisions to do work, so a specific pinned version
won't work when we try to update such a package.
One option is that we could avoid validating package versions in CI, but
then that will cause surprises (heck, I didn't know we were still using
`psutil==5.4.2` instead of `5.8.0` until now). By doing validation, we
make it more explicit and avoid accidentally depending on behaviour of
too new of such a package.
However, in most cases, we manage the installation environment and can
pin dependencies. So, I've made the top-level Mach virtualenv the _only_
one that is able to use requirement specifiers other than "==".
Differential Revision: https://phabricator.services.mozilla.com/D122889
Rather than re-implementing it as `search_path()`, use the existing
`MachEnvRequirements` tool to parse `mach_virtualenv_requirements.txt`
Differential Revision: https://phabricator.services.mozilla.com/D126280
There's some trade-offs in play here: the major issue is that we can't
pin `psutil`'s because it's pre-installed on some CI workers with a
different version (5.4.2).
Additionally, we can't "just" bump that version, because CI workers jump
between different revisions to do work, so a specific pinned version
won't work when we try to update such a package.
One option is that we could avoid validating package versions in CI, but
then that will cause surprises (heck, I didn't know we were still using
`psutil==5.4.2` instead of `5.8.0` until now). By doing validation, we
make it more explicit and avoid accidentally depending on behaviour of
too new of such a package.
However, in most cases, we manage the installation environment and can
pin dependencies. So, I've made the top-level Mach virtualenv the _only_
one that is able to use requirement specifiers other than "==".
Differential Revision: https://phabricator.services.mozilla.com/D122889
Rather than re-implementing it as `search_path()`, use the existing
`MachEnvRequirements` tool to parse `mach_virtualenv_requirements.txt`
Differential Revision: https://phabricator.services.mozilla.com/D126280
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.
5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.
To produce this patch I did all of the following:
1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.
2. Run ./mach lint --linter black --fix
3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.
4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).
# ignore-this-changeset
Differential Revision: https://phabricator.services.mozilla.com/D94045
`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.
The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).
Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.
So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.
And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.
Differential Revision: https://phabricator.services.mozilla.com/D93741
`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.
The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).
Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.
So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.
And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.
Differential Revision: https://phabricator.services.mozilla.com/D93741
Instead, we now run js/src/old-configure from the top-level configure
after having run old-configure and extracted a few variables to inherit
from it.
Because we're now running from the top-level, $_objdir is always the
top-level objdir, which simplifies some things. The topobjdir in
js/src/config.status, however, needs to stay in js/src because of the
build frontend expecting it there.
When running js/src/old-configure, we used to need some special
treatment for a large number of variables for historic reasons, where
we'd take values from the assigned values before running old-configure
for some, or from AC_SUBSTs after running old-configure.
Now that both old-configure and js/src/old-configure get the same
assignments from old-configure.vars, we don't need anything special for
the former. And only a few remaining variables still need manual work
for the latter.
One notable difference, though, is that the new code doesn't try to
avoid running js subconfigure, which added complexity, and was actually
error-prone.
Differential Revision: https://phabricator.services.mozilla.com/D92725
In order to be able to run both old-configure and js/src/old-configure
from the same python configure run, we need to stop setting the items
set by old-configure into the global sandbox config, and instead store
them to be later handled by configure.py.
Differential Revision: https://phabricator.services.mozilla.com/D92718
I think they're remnants from the past that we don't really need anymore.
And they're making things more complicated for some pending work of mine.
Differential Revision: https://phabricator.services.mozilla.com/D89687
I think they're remnants from the past that we don't really need anymore.
And they're making things more complicated for some pending work of mine.
Differential Revision: https://phabricator.services.mozilla.com/D89687
Rather than always printing instructions at the end of the bootstrap phase, we will now create a mozconfig
file if one doesn't exist and there's configuration to be written.
Differential Revision: https://phabricator.services.mozilla.com/D78417
This also does a few remaining python 2 incompatible changes to
.configure files.
Differential Revision: https://phabricator.services.mozilla.com/D69538
--HG--
extra : moz-landing-system : lando
Many values we get out from configure are of types that look like
lists/tuples, that write_indented_repr will serialize as lists... but
only in python 2, because the alternative implementation for python 3
is not doing that. So sanitize first.
Differential Revision: https://phabricator.services.mozilla.com/D69949
--HG--
extra : moz-landing-system : lando
If configure is invoked manually, clobber.py is bypassed and the CLOBBER
file doesn't get touched. The clobber check in Makefile.in gets
triggered causing the build to stop.
Moving the objdir/CLOBBER file creation into configure.py should cause
it to be created regardless of how configure is invoked.
Depends on D53290
Differential Revision: https://phabricator.services.mozilla.com/D53291
--HG--
extra : moz-landing-system : lando
Now that there is no bytes strings in it, we don't need to store
config.status in the system encoding to keep those valid. Moreover, the
system encoding is lossy, which utf-8 is not.
Differential Revision: https://phabricator.services.mozilla.com/D42731
--HG--
extra : moz-landing-system : lando
With bug 1575135, we ensure nothing gets out of the configure sandbox
as a bytes string. We can thus now avoid the encode() pass in
configure.py. We still need, however, to normalize the configuration
so that it doesn't contain unexpected types, and conformning to what
indented_repr does to the configuration in config.status.
While here, convert some obj.iteritems() to six.iteritems(obj).
And remove the now unused encode function.
Differential Revision: https://phabricator.services.mozilla.com/D42630
--HG--
extra : moz-landing-system : lando
Make it a hard error when the sandbox returns non-unicode strings.
This should help quickly catch any remaining non-unicode string that
are not caught by automation.
Differential Revision: https://phabricator.services.mozilla.com/D42607
--HG--
extra : moz-landing-system : lando
This is necessary for any attempt at @import('six') or doing any similar py2+py3 work
in the configure sandbox.
Differential Revision: https://phabricator.services.mozilla.com/D29604
--HG--
extra : moz-landing-system : lando
Since js configure is also python configure, we can actually create
a ConfigureSandbox directly, with the right environment and arguments.
Depends on D16668
Differential Revision: https://phabricator.services.mozilla.com/D16669
--HG--
extra : moz-landing-system : lando
Since MozbuildObject.from_environment() reads from mozinfo.json, tup
picks up that file as a dependency for anything that imports buildconfig
(eg: all generated files). Using FileAvoidWrite when creating
mozinfo.json will help avoid unnecessary work after re-running configure
in the tup backend.
MozReview-Commit-ID: EEOPQYJA1MV
--HG--
extra : rebase_source : 136a0579090776dc55ea5cee870574f13cc27c58
So the file can be more easily consumed without this variable defined.
MozReview-Commit-ID: DF3ASwx4SZP
--HG--
extra : rebase_source : 7d760a71c8f0fa389c2f31f1b383a1f298220545
The config.statusd directory is created alongside config.status, which
contains the same information but is split across many files instead of
all in a single file. This allows the build system to track dependencies
on individual configure values.
MozReview-Commit-ID: 2DbwKCJuNSX
--HG--
extra : rebase_source : 8b6257fd9c75cff3e4b6513d69048c0e3fdda5f4
This generalizes the monkey-patching of the main module added
in bug 914563 to allow multiprocessing to be used from config.status
(which triggers the same bug as when it's used with `mach` as main).
MozReview-Commit-ID: AdOdpKzmbsp
As of Python 3, decimal notations of octal values for permission modes
are no longer permitted and will result in a `SyntaxError` exception
(`invalid token`).
Using the proper octal notation which is also Python 2.7 compatible will
fix this issue.
--HG--
extra : rebase_source : 2e897c51f04ad0ee69071f84b98df224f3af72d3
We want local source directory paths to take precedence over system
paths so system installed packages don't get used.
MozReview-Commit-ID: FIZ0RxqZj78
--HG--
extra : rebase_source : e8d484ea0c8a0bdb1df10375317f6b2c349b9162
The mozconfig detection logic has bitten us on many occasions in the
past. The following changes are made to tentatively improve the
situation:
- The API is modified such that autodetection of the mozconfig has
to be a conscious decision made by the caller, and not triggered
any time there is no mozconfig given, which could be a conscious
decision of the opposite.
- mozinfo.json now stores the actual mozconfig (or lack thereof) used
during configure.
--HG--
extra : rebase_source : c7a632afd414f25daf7bbe7e1a66c3736c26e039
Historically, a mozinfo for js standalone build has not been necessary,
but with the move towards a) having things work with mach and b)
buildconfig using the MozbuildObject.from_environment in next patch,
mozinfo becomes necessary (it's required for
MozbuildObject.from_environment to find the directory is an objdir).
Interestingly, hazard builds do both a js standalone build and a desktop
Firefox build at the same time, both of which are done with MOZCONFIG
set in the environment... with the Firefox mozconfig. The result of now
writing a mozinfo for js standalone builds is that in that case, they
end up with a reference to the mozconfig, which the build system then
reads, and finds a MOZ_OBJDIR, which then doesn't match the js
standalone build objdir. So for those builds, reset MOZCONFIG.
Mitchell Hentges
criss
Ricky Stewart
Bogdan Tara
Dorel Luca
Mike Hommey
Narcis Beleuzu
Razvan Maries
Anthony Ramine
Mike Shal
Justin Wood
Justin Wood
Chris Manchester
Gregory Szorc
Sylvestre Ledru
Elias Probst