Since bug 1784179, we make `make` keep going after errors. When an error
happens, `make` stops rules at the command that fails, but if there are
multiple double-colon rules for the same target, it still runs the
remaining rules, which has undesirable side effects in client.mk: it
stops an already stopped sccache, which leaves an error in the log.
Differential Revision: https://phabricator.services.mozilla.com/D161083
It can be a frustrating experience to push something to e.g. try, get
build errors, fix them, and realize on a subsequent push that there were
other build errors, but since the build failed as soon as it reached the
first error, the logs didn't show them.
While stopping at the first error locally is more helpful locally
(although it still stop immediately and errors are often buried in the
logs), it is actually desirable to avoid stopping on automation.
Differential Revision: https://phabricator.services.mozilla.com/D154327
Historically, client.mk was not invoked with -jn because it would create
race conditions, but that was actually mostly solved by the addition of
`.NOTPARALLEL` in bug 422986, although the mechanism of adding -jn via
`MOZ_MAKE_FLAGS` or `MOZ_PARALLEL_BUILD` has continued well past that.
Nowadays, client.mk is only invoked by mach (it will even bail out if
that's not the case) and only has one target (`build`) and no
dependencies.
This means we don't need to rely on `MOZ_PARALLEL_BUILD` to pass `-jn` in
some cases, and can just always invoke `make -f client.mk` with `-jn`, even
when we just want no parallelism, in which case we can use `-j1`.
This, in turn, allows to remove the extra allow_parallel argument to
`_run_make`, and only rely on `num_jobs`, and to remove some of the
multiple ways the `n` in `-jn` could be set.
Differential Revision: https://phabricator.services.mozilla.com/D124729
It was moved in bug 1495798 for rusttests, because they didn't use
client.mk, but as of bug 1683797, they do. And it turns out that when
sccache is really started as originally intended, inheriting the make
jobserver, the build is dead-locked until sccache quits (because sccache
still has a jobserver token for some reason). But sccache never quits
when we make it stop outside client.mk.
Differential Revision: https://phabricator.services.mozilla.com/D124728
`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.
The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).
Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.
So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.
And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.
Differential Revision: https://phabricator.services.mozilla.com/D93741
`mach configure` currently runs the equivalent to `make -f client.mk`.
This is history, and essentially does the following:
- Create `configure` and `js/src/configure` from `configure.in` and
`js/src/configure.in` respectively.
- Create the objdir.
- Run `configure` from the objdir.
The `configure` script is, nowadays, only really used as a means to set
OLD_CONFIGURE (and also for people who want to run `configure`,
literally, as in the `configure; make` workflow). `mach configure`
actually doesn't need it. Neither does recursing into `js/src` require
`js/src/configure`, since bug 1520340 (and now as of bug 1669633, we
don't even recurse).
Because configure.py can actually derive OLD_CONFIGURE on its own
(except for `js/src/configure`, but `mach configure` doesn't run that),
we don't really need `configure` for `mach configure`.
So all in all, we're at a point in history where it's straightforward to
just initiate configure.py from mach configure, so we just do that.
And in the hypothetical case where the `mach configure` code is somehow
running in python2, we get the mach virtualenv python3 and use it to
execute `configure.py`.
Differential Revision: https://phabricator.services.mozilla.com/D93741
This avoids a set of intermittent issues related to `zstd` decompression failures, which in the absence of these changes break the entire build.
This also requires [updating an environment variable](https://github.com/mozilla/sccache/pull/822), which we do in `client.mk` as well as documentation.
Differential Revision: https://phabricator.services.mozilla.com/D88184
This addresses a related issue along the way: a build that results in running
configure would not update the value of self.config_environment (and therefore
self.substs) as seen from the build driver, so out of date values would have
been used.
The changes to Makefile.in and client.mk made exploit the assumption that by
he time anything in the Make build is running, config.status is up to date.
Users running "make" without the benefit of "mach" will need to manually run
configure when necessary in order to take this into account.
Differential Revision: https://phabricator.services.mozilla.com/D8473
--HG--
extra : moz-landing-system : lando
In builds that use sccache, the sccache server would be shut down at the end
of the build step in client.mk. Unfortunately in rusttest builds the check
step winds up compiling more Rust code which restarts the sccache server but
without the proper configuration. This patch moves sccache shutdown to a
post-script step in mozharness so that the server will stay running through
the check step.
Differential Revision: https://phabricator.services.mozilla.com/D7651
--HG--
extra : moz-landing-system : lando
This patch makes it so we write the sccache log directly to the artifact
directory, so that it will be uploaded even if the build fails. It also makes
the log slightly more verbose. Both of these should help with diagnosing
sccache failures in CI.
The sccache log will no longer be explicitly gzip compressed, but some
Taskcluster client implementations will store it as gzip compressed.
Differential Revision: https://phabricator.services.mozilla.com/D6187
--HG--
extra : moz-landing-system : lando
Unfortunately, I wasn't able to figure out a way to make firefox build & run in
the intermediate stages of these commits. Because of this, I am going to just
delete most of the code which I am deleting in the first patch, as I figure that
those are somewhat uninteresting changes, and then make the other changes in the
following patches.
In total, the following things are deleted:
1. All of xpcom/typelib, except for `xpt/tools` - this directory is being
subsumed entirely into xpcom/reflect/xptinfo.
2. Most of the code in xpcom/reflect/xptinfo, it is being rewritten to avoid
allocating and contain all of the necessary data structures.
3. idl-parser's typelib.py XPT generator, as it will be replaced.
4. Most includes of files which have been deleted.
NOTE: xpcom/typelib/xpt/tools/xpt.py was not removed, as it is used by bundling
code & bundling tests, which we don't want to remove yet.
In order to determine if we need to re-run configure, we write
a JSON file representing the evaluated mozconfig. If this JSON
file changes, configure (and config.status for that matter) is
out of data and it is re-executed.
This commit moves the generation of that JSON file to Python.
MozReview-Commit-ID: 636rpSY7gOm
--HG--
extra : rebase_source : ee1defd74decfd64ffb66a45b053dada58de04fb
This is a pretty straightforward port of the logic. But we
even go a step further: we delete the file in the objdir if there is
no source mozconfig!
MozReview-Commit-ID: AHFFzy6mXRY
--HG--
extra : rebase_source : 1b9387bd72f5a8e9bf8274f5764b0db0176fdba2
extra : source : 0cab9a382d817e6fbab9daa37db0f23e7f73e71f
This target isn't referenced elsewhere in client.mk. It appears to be
dead.
When this target became orphaned, I don't know. I suspect at some point
some included .mk file attempted to "include autoconf.mk." But I'm not
sure what file that would have been or when it would have changed.
FWIW, autoconf.mk is generated by config.status, courtesy of it being
listed in CONFIGURE_SUBST_FILES.
MozReview-Commit-ID: AcPrihAou11
--HG--
extra : rebase_source : 68b1b19428d0754884515c42250353bd75407784
The file is a filtered version of the make file that we previously
started generating for client.mk. Why there is special casing for
UPLOAD_EXTRA_FILES, I'm not sure. This smells fishy and is something
I'd like to take a look at once all code is ported out of client.mk.
The removal of the logic from client.mk meant that we could remove
a bunch of code from client.mk related to loading mozconfig files.
We can now simply include the auto-generated make file directly and
be done with it.
MozReview-Commit-ID: 4M5NElQA7iR
--HG--
extra : rebase_source : 87ed98fa62513007c6fdd2df00eb871a5a29a146
extra : source : 247617a64b7c438528f97d10c86e2f7b8cb72237
We write the file that client.mk is printing from Python. We can
also log it from there pretty easily. So do that.
MozReview-Commit-ID: 7eeugdOJR5b
--HG--
extra : rebase_source : 308826e948fa20684bbc40c806322f802689627e
Now that we require mach to run client.mk and mach only invokes
client.mk for full builds (the "build" target) and configure
(the "configure" target), we should no longer have anyone using
client.mk as a proxy to make targets in the build backend.
So remove that feature.
MozReview-Commit-ID: BbaMdZHnRXy
--HG--
extra : rebase_source : 570e81a62e35cbe4ea27e011883a306f23f3024c
This was added before mach AFAICT. If people need this functionality,
we can add a mach command.
MozReview-Commit-ID: HL52nJE2SMQ
--HG--
extra : rebase_source : cdf67a997731e7f1f94e16aa36c3f806c09779e8
The clobber logic is already written in Python. Now that we
always use mach in front of client.mk, we can check the clobber
state before we execute client.mk.
Since we always check the clobber state, we can remove the
CLOBBER files from various dependencies in client.mk. The
clobberer code should ensure everything is in a good state.
The refactor of the clobber Python code required some changes to
its testing. We drop some support for verifying output strings.
But testing this correctly would require a bit of effort. I don't
think it is worth it.
MozReview-Commit-ID: 69CoImCgtNm
--HG--
extra : rebase_source : c925bb49fd54fe6a5abaa4ac9dc0833e139c6a57
I'm pretty sure this was related to the now removed feature for
building multiple projects.
MozReview-Commit-ID: 4AG7lsk6BLS
--HG--
extra : rebase_source : b677a141161255ffeea57358d4f497fc05c8fb1c
Now that mach is being used to invoke client.mk, we can perform
objdir directory creation there.
Removing the use of mkdir_deps meant that we could remove some
included make files which AFAICT were only used to provide
$(mkdir_deps).
MozReview-Commit-ID: 4ZRToz8xqZy
--HG--
extra : rebase_source : 8d0d2430b33863e1dec8cee84e72178307d1c6e0
extra : source : d223afd90123eba035714288d5da9394b2dbb8d8
The auto-generated make file that we include (and the `mach environment`
output that we included before that) should not contain comment lines.
I think it is safe to remove the code that filters them out.
It is possible a multi-line value in mozconfigs could contain lines
looking like comments and this may cause problems. I'm inclined to
believe that this scenario doesn't exist. If someone complains and
we need to bring back support, we could certainly do that.
MozReview-Commit-ID: 8kKw91HH4ms
--HG--
extra : rebase_source : 3fab8af7f713c91632f3d5172664832af10b555f
This should have the same net result.
TBH, I'm not 100% convinced we need this export. It is only needed
to send variables to sub-makes. And the only make file reading
FOUND_MOZCONFIG is client.mk. Since the code that evals the
auto-generated make file is always executed in client.mk, we
shouldn't need this export.
All this code is going away soon anyway. So I'm inclined to cargo
cult this just in case.
MozReview-Commit-ID: DqF1BU702A
--HG--
extra : rebase_source : 31859c0d4bb6ceb06367bf0ca554d79d57d2d0c3
Currently, client.mk calls `mach environment` to obtain a make file
to be evaluated in the context of client.mk. The reason it is
implemented this way is because client.mk could be an entrypoint to
the build system.
With recent changes that require the use of mach to use client.mk,
we are now guaranteed to have Python code running before client.mk
is invoked. This means we don't need to invoke `mach` from client.mk.
This commit ports the code for generating a client.mk suitable make
file from `mach environment` to the build dispatcher. We now write out
a new .mozconfig-client-mk file in the objdir. client.mk is changed
to cat this file and to include it as a native make file.
The OBJDIR environment variable is also set so client.mk knows where
to read the auto-generated file from.
This commit should be backwards compatible.
Hopefully it is obvious, but this new make file is only temporary.
As soon as the remaining mozconfig logic is moved out of client.mk,
we should be able to simplify down to a single "include" in client.mk.
MozReview-Commit-ID: BEfWo76Z1qA
--HG--
extra : rebase_source : 752df93f816b95bda108a3c787d7f4941b136bbb
Instead of evaluating config.guess in client.mk, we evaluate it
in Python. The Python code also looks for CONFIG_GUESS in the
mozconfig. This still happens in client.mk courtesy of evaling
the mozconfig's relevant parts.
MozReview-Commit-ID: 87NmQiB2ccX
--HG--
extra : rebase_source : 368bc7bf1375a3943ce62fbb77458c40091a7092
MozbuildObject._run_make() also invokes baseconfig.mk as part of
validating the make binary that will be executed.
Since mach calls this Python code and mach is now required to
invoke client.mk, the validation in client.mk is redundant and can
be removed.
autoconf.mk includes baseconfig.mk. So even once we eliminate
client.mk, the make backend itself will still validate make.
We should probably move this validation to moz.configure (if it
isn't already there). But that can be done another time.
MozReview-Commit-ID: DPCBdz253S6
--HG--
extra : rebase_source : 8588738c517bd636039555299095981c71722600
We will soon begin the process of moving logic from client.mk
into mach. Once we start this process, it won't be possible to
invoke client.mk in isolation.
This commit introduces a check in client.mk that will error unless
client.mk is executed from mach. This should help more easily
identify users of client.mk during the transition period - before
client.mk is removed and callers start getting missing file errors.
Obviously the check can be bypassed easily. If someone does this,
upcoming changes will quickly break that workflow. So this isn't a
big deal.
MozReview-Commit-ID: JB7hXlvTKWe
--HG--
extra : rebase_source : 97774adcf9253ec8f6a3a2a4576dfb9c66a52d7c
AFAICT this is unused.
There are some mozharness consumers that evaluate an echo-variable-*
target. However, I believe they are hitting the target from debugmake.mk
and not going through client.mk.
MozReview-Commit-ID: 9XjykFMRpsT
--HG--
extra : rebase_source : edc9be971ddb0384408652fd04f550d7d8e51aa6
The last use of PERL was removed recently.
The last use of comma was removed by c8276e89a513 in 2008.
The last use of SH was... never AFAICT. The variable was
defined but unused in the initial import of client.mk from CVS
in 1998.
MozReview-Commit-ID: LFqcVRf36wf
--HG--
extra : rebase_source : 5a32c1969870c29065c80a94ba3f20ba3873569f
After removing sccache.mk, there are no more references to these
variables. Let's nuke them.
MozReview-Commit-ID: LH1oHm59SnU
--HG--
extra : rebase_source : 5138ddd703350caa5f54fe8941a2e0dcb9dc2c17
sccache.mk is the only thing that uses MOZ_PREFLIGHT_ALL and
MOZ_POSTFLIGHT_ALL. We're trying to kill client.mk and these
variables are next on the chopping block.
This commit essentially moves the logic of sccache.mk inline into
client.mk. Initially, I wanted to move the management of the
sccache daemon to Python as part of what `mach build` invokes.
However, the sccache daemon needs access to the make jobserver.
Since we don't have an active make process in `mach`, this is
obviously problematic. The sccache daemon is also used by
configure, which is launched from client.mk. So the best we
can do right now is move the sccache daemon logic into client.mk.
As part of the port, we pass the path to the sccache binary via
an environment variable. This feels slightly better than hardcoding
the path that automation uses.
MozReview-Commit-ID: zcOYR4I1OG
--HG--
extra : rebase_source : 305a237dc9f5bd96e4aa83d491ac2498fe3c3ba0
It was just an alias for "clean," which is proxied to the root
Makefile. `mach clobber` should be used instead.
MozReview-Commit-ID: A8tHaNb7H1z
--HG--
extra : rebase_source : 455206b5a609e5a598c9b6453869d115538c5d19
extra : source : e71fba9d0818cde1f83782b4a18f253f84b1a70e
This doesn't need to be in client.mk.
Also, we inline the info to help people correct the failure, as this
results in a better user experience.
MozReview-Commit-ID: KURL3RIGzKf
--HG--
extra : rebase_source : dc79d3f6aa4e91a12cab0e26d5fc0a3e15afa833
extra : source : 2eceb30625acd8cfadda0baa6326a7e9fd07dece
Checks like this are what configure is for.
In addition to moving the check, we also validate topobjdir as well.
MozReview-Commit-ID: 9sVNQJsAnjO
--HG--
extra : rebase_source : 688961fffca5922c7186c0d39182de7220f7dbe3
extra : source : d9a4ea9bc34a1e0c710469fc0a556ed624ea387b
I suspect these were related to MOZ_BUILD_PROJECTS, which has been
removed.
MozReview-Commit-ID: DvwwTrczQWv
--HG--
extra : rebase_source : e4bb52e966bc78eb2843c499436c523c43a8a7f3
extra : source : 5b5b2a1c0af35d36cedeed50c94efb9d84a8a251
Previously, client.mk made the decision of whether to perform a PGO
build. This required passing around MOZ_PGO and invoking a separate
make target if this variable was set.
In this commit, we move this logic to Makefile.in. We employ a special
mechanism in rules.mk to override the default make target so
`make` evaluates "profiledbuild" if MOZ_PGO is set. This also
required using an explicit target for $(MAKE) invocations inside
the "profiledbuild" rule to avoid infinite recursion.
MozReview-Commit-ID: 8sHiVspMisM
--HG--
extra : rebase_source : 76c8a0c592015802f2b9ad52fe2001012a4611f6
extra : source : 24e28b37fbc0ad55ee449699e09c62042af41b55
Before, the default "build" target either invoked "realbuild" or
"profiledbuild" depending on whether MOZ_PGO was set. Now that
PGO is handled by Makefile.in, we only have "realbuild" and there
is no need to have a recursive invocation of client.mk. So we
merge "realbuild" into "build."
MozReview-Commit-ID: IIX8iAXlsD1
--HG--
extra : rebase_source : b9eb0753b8b214d56f9bee44350899c2dae46969
extra : source : 7d58979b22f8539bd8931e580745999f4ab17887
client.mk is going away. This commit moves the PGO build target out of
it.
We move the code to Makefile.in. This likely isn't its final
home, as we'll want to eliminate Makefile.in at some point. However,
it is easier to move things to the root Makefile.in than other
alternatives. Keep in mind that the logic needs access to primitives
like BUILDSTATUS and variables from mk_add_options, which the
Makefile.in mechanism has easy access to.
MozReview-Commit-ID: 4C43lBhZz4l
--HG--
extra : rebase_source : 0ff1e95dcf8cb0082b019713db0cb7f47f0bccb8
extra : source : 0d2083b8edc43fc13c77f71199e0fcaddb32fcfb
Mike Hommey
Mitchell Hentges
Cosmin Sabou
Narcis Beleuzu
Ricky Stewart
Bogdan Tara
Chris Manchester
Ted Mielczarek
Mike Shal
Nika Layzell
Gregory Szorc