Ideally, this kind of manual work wouldn't be required, but the frontend
doesn't provide enough information and while backend could correlate the
information, they currently don't.
Differential Revision: https://phabricator.services.mozilla.com/D80606
While ideally we'd just move all of them at the top-level, there are
other things that depend on them that wouldn't then get the right
dependencies. One of them is install steps in dist/something, but there
are others, and that's a rather long pole of things requiring many
changes along with chances to break things.
So instead, we go with a simpler and more limited approach, where we
still recurse into directories to run their export tier (to run whatever
else than GeneratedFiles they have), but ensure the GeneratedFiles we
moved at the top-level are built before recursing by making
directory/export depend on them.
Differential Revision: https://phabricator.services.mozilla.com/D80608
Ideally, this kind of manual work wouldn't be required, but the frontend
doesn't provide enough information and while backend could correlate the
information, they currently don't.
Differential Revision: https://phabricator.services.mozilla.com/D80606
While ideally we'd just move all of them at the top-level, there are
other things that depend on them that wouldn't then get the right
dependencies. One of them is install steps in dist/something, but there
are others, and that's a rather long pole of things requiring many
changes along with chances to break things.
So instead, we go with a simpler and more limited approach, where we
still recurse into directories to run their export tier (to run whatever
else than GeneratedFiles they have), but ensure the GeneratedFiles we
moved at the top-level are built before recursing by making
directory/export depend on them.
Differential Revision: https://phabricator.services.mozilla.com/D80608
Ideally, this kind of manual work wouldn't be required, but the frontend
doesn't provide enough information and while backend could correlate the
information, they currently don't.
Differential Revision: https://phabricator.services.mozilla.com/D80606
This uses sys.exit() instead of throwing an exception since:
1. It isn't an error, and bug 1646406 isn't finished yet
2. The output is more straightforward for the user
This verification happens separately from the other python-verification logic, but that's
because I wanted the validation to happen as soon as possible - before the user invests too
much time, and before too much of the script ran with the wrong interpreter.
Differential Revision: https://phabricator.services.mozilla.com/D80726
1.38 changed the Cargo.lock format, and the current minimum (1.37) can't
read it, so the minimum version was already wrong.
1.41 has a problem vendoring nom for some reason, that appears to be
fixed in 1.42.
Differential Revision: https://phabricator.services.mozilla.com/D80593
This adds a way to exclude certain file patterns when extracting Python
packages in |mach vendor python|. For now I've excluded likely test and doc
dirs, though we may want to expand on the list in the future.
Differential Revision: https://phabricator.services.mozilla.com/D80209
When linking a weak symbol in an object against a library where the
symbol is provided with a version, the final binary get a weak versioned
symbol reference.
It turns out weak versioned symbols still make the dynamic linker need
the symbol version, even if all symbols needed with that version are
weak.
Practically speaking, that means with bug 1634204, we now end up with
a weak versioned symbol reference to __cxa_thread_atexit_impl with
version GLIBC_2.18, and glibcs without the symbol can't fulfil that
version, even though the weak symbol is the only thing we need from that
version.
This means the check_binary changes in bug 1634204 are too
relaxed, so we revert them (although we keep the easier to read
conditions in check_dep_versions).
We also introduce a hack in stdc++compat.cpp (although it's not
technically entirely about libstdc++ compat) so that we avoid the weak
symbol reference while keeping the intended baseline for libstdc++ and
glibc.
Differential Revision: https://phabricator.services.mozilla.com/D79773
This patch adds a `--test-date` option to mozperftest that lets us specify the date that we are testing on. This lets us trigger multi-commit tests on past dates, and also ensures that retriggers of the multi-commit tests use the same day that they were pushed on (giving us reproducibility).
Differential Revision: https://phabricator.services.mozilla.com/D80173
They were necessary some time ago, but:
- there is no js/xpconnect/src/export target anymore,
- accessibility/xpcom/export had a dependency on xpcom/xpidl/export for
the ply cache files, but they're not used as of bug 1633156.
- the recursive make backend was adding automatic dependencies on
xpcom/xpidl/export for the ply cache files.
Differential Revision: https://phabricator.services.mozilla.com/D80150
When linking a weak symbol in an object against a library where the
symbol is provided with a version, the final binary get a weak versioned
symbol reference.
It turns out weak versioned symbols still make the dynamic linker need
the symbol version, even if all symbols needed with that version are
weak.
Practically speaking, that means with bug 1634204, we now end up with
a weak versioned symbol reference to __cxa_thread_atexit_impl with
version GLIBC_2.18, and glibcs without the symbol can't fulfil that
version, even though the weak symbol is the only thing we need from that
version.
This means the check_binary changes in bug 1634204 are too
relaxed, so we revert them (although we keep the easier to read
conditions in check_dep_versions).
We also introduce a hack in stdc++compat.cpp (although it's not
technically entirely about libstdc++ compat) so that we avoid the weak
symbol reference while keeping the intended baseline for libstdc++ and
glibc.
Differential Revision: https://phabricator.services.mozilla.com/D79773
Sentry needs to be able to send data in a minimal environment, and intelligently determining
the "topobjdir" (without leaning on Mach) is tough.
Instead, since the "topobjdir" usually falls under the "topsrcdir", just leaning on <topsrcdir>
patching should be sufficient.
Differential Revision: https://phabricator.services.mozilla.com/D80122
This is a documented feature of `mach clobber python`, which I never noticed until bug 1645889 landed. I think it's fair to say that this is not how one would intuitively expect the command to work given that I'm the "owner" of this `mach` command and I never noticed this quirk until today. :)
`python/` shouldn't contain compiled C extensions or anything and the existing `*.py[cdo]` and `*/__pycache__` clauses should get rid of most, if not all, of the relevant garbage in the directory.
If the original behavior turns out to be desirable, we can add it back in (maybe we can use the `--full` command line option?)
Differential Revision: https://phabricator.services.mozilla.com/D80204
This patch fixes the tests in testing/performance/ that hadn't been updated to
the new API, as well as fixing a couple issues with the `--proxy` option:
* `policy.json` wasn't being logged correctly, since mozlog was treating the
data as a format string
* Proxy prefs weren't being set correctly for the profile
Differential Revision: https://phabricator.services.mozilla.com/D79937
When linking a weak symbol in an object against a library where the
symbol is provided with a version, the final binary gets a weak versioned
symbol reference.
It turns out weak versioned symbols still make the dynamic linker need
the symbol version, even if all symbols needed with that version are
weak.
Practically speaking, that means with bug 1634204, we now end up with
a weak versioned symbol reference to __cxa_thread_atexit_impl with
version GLIBC_2.18, and glibcs without the symbol can't fulfil that
version, even though the weak symbol is the only thing we need from that
version.
This means the check_binary changes in bug 1634204 are too
relaxed, so we revert them (although we keep the easier to read
conditions in check_dep_versions).
We also introduce a hack in stdc++compat.cpp (although it's not
technically entirely about libstdc++ compat) so that we avoid the weak
symbol reference while keeping the intended baseline for libstdc++ and
glibc.
Differential Revision: https://phabricator.services.mozilla.com/D79773
This patch fixes the tests in testing/performance/ that hadn't been updated to
the new API, as well as fixing a couple issues with the `--proxy` option:
* `policy.json` wasn't being logged correctly, since mozlog was treating the
data as a format string
* Proxy prefs weren't being set correctly for the profile
Differential Revision: https://phabricator.services.mozilla.com/D79937
Additionally, recommends the installation of AdoptOpenJDK instead of the Oracle JDK.
This required a different "JAVA_HOME" detection mechanism, which is implemented in this revision.
Differential Revision: https://phabricator.services.mozilla.com/D79547
The `key` function to the `sort` call here was janky, so this produced an error if you tried to invoke this command on a path that doesn't have an owner (`TypeError: '<' not supported between instances of 'TypedTuple' and 'NoneType'`).
Differential Revision: https://phabricator.services.mozilla.com/D80043
There is some remaining code in central originating from bug 1184405, which sought to associate source files with their "affected" test files. That ended up not panning out, and bug 1644228 removed a lot of that code, but left some remnants in the `Files` object which are still referenced in a couple different places. I'm deleting all of that code in `context.py` plus everything that references it for the following reasons:
1. Right now, `Files.{test_files,test_tags,test_flavors}` do get populated, but only ever with "default" values -- namely `moz.build` files that are above the files in question in the directory hierarchy. This is a heuristic that doesn't actually have anything to do with mapping source files to their corresponding test files, which is misleading.
2. Those attributes are accessed in two places. The first is in the `mach file-info dep-tests` command. This command isn't referenced anywhere else in tree and I don't have any evidence anyone ever uses it. Even if they do, I would claim that doing so is a mistake (because the results of the command aren't meaningful and are just populated by the "defaults" described above), and that person's workflow should be migrated to something else that *is* meaningful.
3. The second place where this metadata is accessed is in `testing/mozbase/moztest/moztest/resolve.py`; that method is invoked in `tools/tryselect/selectors/syntax.py`, but only if you pass `--detect-paths` to `mach try syntax`. This is [entirely broken](https://bugzilla.mozilla.org/show_bug.cgi?id=1614614), and even if we made an effort to fix it, it wouldn't do anything resembling what the documentation of `--detect-paths` suggests it would do (again, because the data isn't populated meaningfully). So I'm deleting the command line option entirely.
Differential Revision: https://phabricator.services.mozilla.com/D79711
When linking a weak symbol in an object against a library where the
symbol is provided with a version, the final binary gets a weak versioned
symbol reference.
It turns out weak versioned symbols still make the dynamic linker need
the symbol version, even if all symbols needed with that version are
weak.
Practically speaking, that means with bug 1634204, we now end up with
a weak versioned symbol reference to __cxa_thread_atexit_impl with
version GLIBC_2.18, and glibcs without the symbol can't fulfil that
version, even though the weak symbol is the only thing we need from that
version.
This means the check_binary changes in bug 1634204 are too
relaxed, so we revert them (although we keep the easier to read
conditions in check_dep_versions).
We also introduce a hack in stdc++compat.cpp (although it's not
technically entirely about libstdc++ compat) so that we avoid the weak
symbol reference while keeping the intended baseline for libstdc++ and
glibc.
Differential Revision: https://phabricator.services.mozilla.com/D79773
This command, which dates from 7 years ago, depends on `pymake` which requires Python 2, but it runs in Python 3 as of bug 1632354. It's also not generally useful and appears to just be a thing for debugging/build system development, but evidently no one regularly uses it for that purpose because it fails unconditionally now. If something like this turns out to be useful then it can be resurrected, either depending on a py3-compatible version of `pymake` (see bug 1646112) or some other library.
Differential Revision: https://phabricator.services.mozilla.com/D79879
With the ongoing artifact issues, I find myself asking people to run `mach clobber python` quite frequently. Very often, those people come back and tell me that they tried to `mach clobber` and it didn't work, which is of course no surprise to me because `mach clobber` and `mach clobber python` do different things; so I have to repeat myself and we waste time going through the cycle multiple times for no reason.
It's not crazy to assume that `mach clobber` will do as much as, or more, work than `mach clobber python`, so this change should bring the actual functionality of the code in sync with what people tend to assume. People tend to instinctively `mach clobber` when something has gone wrong locally and since `mach clobber python` does actually fix a set of local issues that `mach clobber` does not, this could also result in fewer bugs being reported in the long run. This is technically a back-incompatible change, but unlike a clobber of the `objdir` which can require you to do an expensive full rebuild, nobody's going to miss their `.pyc` files being deleted.
Differential Revision: https://phabricator.services.mozilla.com/D79733
IMPACTED_TESTS was added in bug 1184405, but is under-utilized in current moz.build
annotations; also bug 1516847 removed use of IMPACTED_TESTS in 'mach test'. This
patch removes remaining support for IMPACTED_TESTS.
Differential Revision: https://phabricator.services.mozilla.com/D79141
To avoid sending identifying information, common absolute paths are patched with placeholder values. For example, devs
may place their Firefox repository within their home dir, so absolute paths are doctored to be prefixed with
"<topsrcdir"> instead.
Additionally, any paths including the user's home directory are patched to instead be a relate path from "~".
Differential Revision: https://phabricator.services.mozilla.com/D78962
To avoid sending identifying information, common absolute paths are patched with placeholder values. For example, devs
may place their Firefox repository within their home dir, so absolute paths are doctored to be prefixed with
"<topsrcdir"> instead.
Additionally, any paths including the user's home directory are patched to instead be a relate path from "~".
Differential Revision: https://phabricator.services.mozilla.com/D78962
This patch adds a prefix field to the mozperftest perfherder data. This is a non-standard data field so the performance-artifact-schema.json file isn't modified to include it.
Differential Revision: https://phabricator.services.mozilla.com/D79296
It is possible to have both a default command (with positional arguments) and
sub-commands (with arguments) in mach. If the subcommand exists, it
is dispatched to; if it doesn't the default command is called the positional
argument filled in.
However, when you run ./mach command --help, it will detect the subcommands
and only print out their help section. If the default command has arguments,
they were not printed out. Now they are.
Small papercuts in this patch are that the Default Command Arguments are
printed after the subcommands and that subcommand help without default
arguments have an extra newline after them. Both of these seem small
enough that the refactoring necessary to abate them is undesirable.
Differential Revision: https://phabricator.services.mozilla.com/D76505
Here we unify the 'run_after' section with 'file-updates' just naming
them 'update-actions'. This will allow a simpler schema and a clearer
picture of the order of actions that are taken.
Differential Revision: https://phabricator.services.mozilla.com/D76428
Additionally, raise voluptuous.Invalid errors so the message provided in
the Exception is shown to the user on error.
Differential Revision: https://phabricator.services.mozilla.com/D75697
Depends on D75696
This section is for expressing actions that occur after extracting
the files, before vendoring is completed. While we support running
scripts (or at least, we will...) this section can be used for simple
actions that don't need scripts.
Also, add the dav1d excluded files.
Differential Revision: https://phabricator.services.mozilla.com/D75696
Depends on D75695
Add license-file to allow one to specify a special file the
project's license is in.
Add 'vendor-directory' to allow vendoring the files into a
separate directory (e.g. under third_party.) Remove a contradictory
comment.
Remove 'revision' from 'vendoring'. This doesn't make sense to me:
'vendoring' is about how to update the library. In a future iteration
we may wish to restrict updates to a branch; but specifying a revision
is specifically about _not_ updating. It appears the intent of revision
was to identify what revision was currently in-tree; but that information
should live in the 'origin' section.
Differential Revision: https://phabricator.services.mozilla.com/D75694
Depends on D75896
Previously, python import syntax was used to refer users to the definition of the --application-choice arguments.
However, it isn't straightforward to the uninitiated, especially since it doesn't work in searchfox.
Instead, by providing a file reference, it should be more accessible.
Differential Revision: https://phabricator.services.mozilla.com/D79003
Such files are, at least as of this writing, only libraries, and we can
get by with installing them later. This change will also avoid having
them added to the dependent libraries list.
Differential Revision: https://phabricator.services.mozilla.com/D78618
We're going to reuse this machinery in a subsequent patch. I think this
change, by virtue of moving all the output to a single place, also makes
things clearer.
Differential Revision: https://phabricator.services.mozilla.com/D78617
Mike Hommey
Dorel Luca
Sylvestre Ledru
Mitchell Hentges
Andi-Bogdan Postelnicu
James Graham
Narcis Beleuzu
Andrew Halberstadt
Ricky Stewart
John Paul Adrian Glaubitz
Geoff Brown
Gregory Mierzwinski
Yue Zhang
Kagami Sascha Rosylight
Butkovits Atila
Jim Porter
Gijs Kruitbosch
Mihai Alexandru Michis
Razvan Maries
Andy Grover
Tom Prince
Simon Fraser
Tarek Ziadé
Bogdan Tara
Tom Ritter
Nathan Froyd
Noemi Erli
Rail Aliiev