Граф коммитов

63 Коммитов

Автор SHA1 Сообщение Дата
Mike Shal 56ba0730bc Bug 1580899 - Remove MOZ_PGO from mozconfigs; r=glandium
Setting this flag still attempts to build with 1-tier PGO, unless we
detect that 3-tier PGO is enabled. Since we should be using 3-tier PGO
everywhere, the flag should be redundant in our automation mozconfigs.
It can still be used locally in a mozconfig to do a 1-tier PGO build.

Depends on D46069

Differential Revision: https://phabricator.services.mozilla.com/D46070

--HG--
extra : moz-landing-system : lando
2019-09-18 18:29:00 +00:00
Wander Lairson Costa 1e4f81ec22 Bug 1572236: Support sccache in Google Storage r=chmanchester,dustin
With the migration from AWS to GCP, we also need to migrate sccache
buckets from S3 to Google Storage.

The problem is how we deal with regions, since there isn't an exact
correspondence on the region names between the two cloud providers.

To make the transition smoother, docker-worker (and soon generic-worker)
provides a new environment variable called TASKCLUSTER_WORKER_LOCATION,
with information about the cloud provider the task is running on. Using
this new variable, we configure sccache to use the corresponding storage
service of the cloud provider where the task runs.

The bucket names in Google Storage are shorter because GCS imposes a
limit of 30 characteres for the names.

Ref: https://github.com/taskcluster/taskcluster-rfcs/pull/148/files

Differential Revision: https://phabricator.services.mozilla.com/D44845

--HG--
extra : moz-landing-system : lando
2019-09-09 16:51:42 +00:00
Dustin J. Mitchell af8483e1ae Bug 1562686 - use AWS_IAM_CREDENTIALS_URL for all S3 sccache invocations r=chmanchester
Differential Revision: https://phabricator.services.mozilla.com/D41454

--HG--
extra : moz-landing-system : lando
2019-08-23 12:39:25 +00:00
Razvan Maries 3d74e341f9 Backed out 2 changesets (bug 1562686) as per glandium's request. CLOSED TREE
Backed out changeset 0ce37eda652a (bug 1562686)
Backed out changeset fe7b9445e1d3 (bug 1562686)
2019-08-22 02:43:16 +03:00
Dustin J. Mitchell 13f9bc563c Bug 1562686 - use AWS_IAM_CREDENTIALS_URL for all S3 sccache invocations r=chmanchester
Differential Revision: https://phabricator.services.mozilla.com/D41454

--HG--
extra : moz-landing-system : lando
2019-08-20 14:03:12 +00:00
Razvan Maries cff855aa9c Backed out changeset 91eca815c9fc (bug 1562686) for build bustages. CLOSED TREE 2019-08-16 22:38:19 +03:00
Dustin J. Mitchell ad85abed68 Bug 1562686 - use AWS_IAM_CREDENTIALS_URL for all S3 sccache invocations r=chmanchester
Differential Revision: https://phabricator.services.mozilla.com/D41454

--HG--
extra : moz-landing-system : lando
2019-08-16 17:17:02 +00:00
Mike Hommey c173540215 Bug 1573435 - Use toolchain fetches for all remaining toolchain uses. r=nalexander
The remaining uses all need adjustements to in-tree mozconfigs, so they
all need to be done at once.

However, to make things slightly more intelligible, we do this in two
steps. This is step 1: we modify the use_toolchain transform to take care of
the transformation, while keeping the task definitions intact, so that
we only deal with mozconfig and build script adjustements here.

Differential Revision: https://phabricator.services.mozilla.com/D41890
2019-08-15 11:21:52 +09:00
Chris Manchester a18c4d2cbd Bug 1553339 - Abandon "2" in name of sccache toolchain artifacts. r=nalexander
Differential Revision: https://phabricator.services.mozilla.com/D32218

--HG--
extra : moz-landing-system : lando
2019-05-22 21:06:16 +00:00
Rob Thijssen c00716af37 Bug 1543026 - enable sccache on gcp r=wcosta
this change comprises the in-tree changes required to make use of sccache in gcp.
specifically:
- a gcp metadata lookup for availability-zone is added to mozconfig, enabling a build to determine its regional gcp sccache bucket
- the sccache cargo build command is modified to include the gcs feature when the environment contains gcs configuration

note that further changes are required on infra to support sccache use. the required changes already [exist](https://github.com/mozilla-releng/OpenCloudConfig/commit/1d515dc) and are enabled for gcp windows infra, including:
- a json credential file on the build instance filesystem, containing credentials valid for the appropriate scm level bucket for the gcp region
- an `SCCACHE_GCS_KEY_PATH` env variable containing the path to the json credential file
- an `SCCACHE_GCS_RW_MODE` env variable containg the text `READ_WRITE`
- sccache buckets must exist for each region and scm levels 1 & 3
- credentials for scm level 1 buckets **must not** be valid for scm level 3 buckets

on gcp systems which do not contain credential files and the above mentioned env variables (eg gecko-[1-3]-b-linux), sccache should fail gracefully without breaking builds.

Differential Revision: https://phabricator.services.mozilla.com/D29622

--HG--
extra : moz-landing-system : lando
2019-05-02 12:56:20 +00:00
Ted Mielczarek dacb17a7c0 bug 1506848 - remove some no-longer-necessary bits from mozconfig.cache. r=nalexander,glandium
Differential Revision: https://phabricator.services.mozilla.com/D18265

--HG--
extra : moz-landing-system : lando
2019-02-01 11:22:16 +00:00
Ted Mielczarek 8a4288a193 bug 1506848 - switch Windows debug flags from -Zi to -Z7. r=nalexander
We default to -Zi for Windows debug flags, which is for generating debug info
into a PDB file, but clang-cl doesn't actually implement that so -Zi is
an alias for -Z7 in clang-cl:
http://clang.llvm.org/docs/UsersManual.html#id9

sccache has special handling for -Zi because multiple compiles writing
to the same PDB file is not cacheable, so we've always overridden -Zi with -Z7
in CI when using sccache.

Given that everyone should be using clang-cl nowadays and MSVC will no longer
be supported soon, this patch changes the default and removes some PDB
file name flag setting from rules.mk, as well as the no-longer-necessary
overrides from mozconfig.cache.

Differential Revision: https://phabricator.services.mozilla.com/D18264

--HG--
extra : moz-landing-system : lando
2019-02-01 11:12:06 +00:00
Ted Mielczarek e11506c714 bug 1490325 - write sccache log directly to artifact directory, make logs slightly more verbose. r=mshal
This patch makes it so we write the sccache log directly to the artifact
directory, so that it will be uploaded even if the build fails. It also makes
the log slightly more verbose. Both of these should help with diagnosing
sccache failures in CI.

The sccache log will no longer be explicitly gzip compressed, but some
Taskcluster client implementations will store it as gzip compressed.

Differential Revision: https://phabricator.services.mozilla.com/D6187

--HG--
extra : moz-landing-system : lando
2018-09-18 20:02:17 +00:00
Ted Mielczarek 201bd292e7 Bug 1384557 - move _DEPEND_CFLAGS+CL_INCLUDES_PREFIX to toolchain.configure, ignore {CC,CXX}_WRAPPER when using sccache; r=glandium
Currently mozconfig.cache overrides a few build options for sccache.
This patch moves them into toolchain.configure so that the build system
will set them properly when sccache is in use.  Additionally,
{CC,CXX}_WRAPPER are set in config.mk, so just avoid setting them when
sccache is in use.
2018-09-14 12:12:34 -04:00
Mike Hommey 335daba18f Bug 1490218 - Disable sccache on PGO builds r=dmajor
Currently, sccache doesn't work at all with clang-cl (PGO or not, bug
1476604), and newer versions of sccache that do the right thing for PGO
flags with clang have other problems (see bug 1484897).

The benefit of sccache on PGO+LTO builds is however limited (the
profile-use half of the build doesn't benefit at all, and since it does
LTO as well, it's the most time-consuming part of the build), so in
order not to further block switching Linux builds to clang, let's just
disable sccache on PGO builds for now (the earlier we can switch to
clang, the more time it leaves to catch any problem that causes).

Differential Revision: https://phabricator.services.mozilla.com/D5509

--HG--
extra : moz-landing-system : lando
2018-09-11 22:44:09 +00:00
Chris AtLee 600b5b46ac Bug 1237182: get rid of oauth.txt r=mtabara
Differential Revision: https://phabricator.services.mozilla.com/D1444

--HG--
extra : rebase_source : dad4f2102dc1e74b681a765169eae691724f8b61
2018-05-25 17:43:15 -04:00
Chris AtLee 8e5587759f Bug 1237182: Get rid of buildprops.json r=tomprince,sfraser
Differential Revision: https://phabricator.services.mozilla.com/D1443

--HG--
extra : rebase_source : 1683b76377e27fdaa5292e1781573ddc4e61afa8
2018-05-25 17:35:43 -04:00
Mike Hommey aa8197cc80 Bug 1426788 - Don't fall back to ccache when sccache is not enabled. r=gps
ccache is not beneficial on taskcluster, don't try to use it when
sccache is not enabled for some reason.

--HG--
extra : rebase_source : a17fe88eb92072935cb86f9ada4205863cfc8f85
2017-12-22 10:08:40 +09:00
Gregory Szorc 67d153aef0 Bug 1415971 - Inline sccache.mk into client.mk; r=nalexander
sccache.mk is the only thing that uses MOZ_PREFLIGHT_ALL and
MOZ_POSTFLIGHT_ALL. We're trying to kill client.mk and these
variables are next on the chopping block.

This commit essentially moves the logic of sccache.mk inline into
client.mk. Initially, I wanted to move the management of the
sccache daemon to Python as part of what `mach build` invokes.
However, the sccache daemon needs access to the make jobserver.
Since we don't have an active make process in `mach`, this is
obviously problematic. The sccache daemon is also used by
configure, which is launched from client.mk. So the best we
can do right now is move the sccache daemon logic into client.mk.

As part of the port, we pass the path to the sccache binary via
an environment variable. This feels slightly better than hardcoding
the path that automation uses.

MozReview-Commit-ID: zcOYR4I1OG

--HG--
extra : rebase_source : 305a237dc9f5bd96e4aa83d491ac2498fe3c3ba0
2017-11-09 14:59:06 -08:00
Masatoshi Kimura 383e90a03b Backed out changeset 6555a2a899a0 (bug 1384557) due to causing bug 1414060. r=backout
MozReview-Commit-ID: FV44JQSDXPQ

--HG--
extra : amend_source : 8c05281fad889edd723ab58786d0bf16f7d325b0
2017-11-07 20:39:00 +09:00
Ted Mielczarek 54b6236e29 bug 1384557 - move _DEPEND_CFLAGS+CL_INCLUDES_PREFIX to toolchain.configure, ignore {CC,CXX}_WRAPPER when using sccache r=glandium
Currently mozconfig.cache overrides a few build options for sccache. This
patch moves them into toolchain.configure so that the build system will
set them properly when sccache is in use. Additionally, {CC,CXX}_WRAPPER
are set in config.mk, so just avoid setting them when sccache is in use.

MozReview-Commit-ID: FYlVKRI8OiN

--HG--
extra : rebase_source : 00715beb5fbd2c11311dec43809bd1febab56a11
extra : intermediate-source : 0f2b1b75b83737378d882a3c3e0d8dfb4efecd1f
extra : source : a8032ae9cb2ad1c4574c6ac6f5c2778863cd71e0
2017-01-10 16:52:27 -05:00
Mike Hommey eaa93380ca Bug 1382502 - Enable sccache on PGO builds. r=ted
Note this will only enable it on try, autoland and mozilla-inbound,
which are the only branches where sccache are enabled at the moment.
Enabling on more builds (or not) is the subject of bug 1373460.

Also note that bug 1181040, that ensured PGO builds weren't using
sccache mentions that back then, link times went up when using sccache
(with -Z7) vs. without, but that was presumably with MSVC 2013. Try
suggests link times are the same using sccache now (still using -Z7,
pending bug 1318370).

--HG--
extra : rebase_source : 9f9d87714f3c841b790eb7d692ea0968b1755b70
2017-07-20 11:31:14 +09:00
Mike Hommey f751ce9e63 Bug 1382502 - Include mozconfig.cache in most cases. r=ted
We want most builds to be actually using sccache, so we include
mozconfig.cache from mozconfig.common. However, since the --with-ccache
configure option doesn't exist on non-compile jobs (e.g. artifact
builds), we move to using the CCACHE environment variable instead, which
allows us to unset it in mozconfig.no-compile.

And since mozconfig.no-compile is always included where no_sccache is
set, we can remove that variable.

--HG--
extra : rebase_source : a8c743de1fd7a3c0fbc53f7c233df36585897767
2017-07-20 11:07:53 +09:00
Ted Mielczarek 60fab48ceb bug 1350093 - fix sccache configuration to handle changes in the format of TASKCLUSTER_WORKER_GROUP. r=froydnj
The TASKCLUSTER_WORKER_GROUP environment variable used to contain the full
AWS availability zone, but a recent docker-worker change changed it to
be simply the AWS region, which broke sccache in taskcluster because we
were using it as part of the S3 bucket name.

MozReview-Commit-ID: 1KsfWpB4PoY

--HG--
extra : rebase_source : bdc61f180bf079eb0ad2cdbbd25e3e3a0deb62e6
2017-04-06 12:03:14 -04:00
Mike Hommey a1d07f91d8 Bug 1343713 - Avoid mozconfig failure when sccache is not there. r=ted
The build will fail during configure anyways in that case, not silently
fallback to compiling without it.

--HG--
extra : rebase_source : 8f12f64da6a635e63339a988f27ada893f567712
2017-03-01 11:06:40 +09:00
Phil Ringnalda 53ccf8158e Backed out 5 changesets (bug 1318370) for mozilla-central Windows static-analysis bustage
Backed out changeset 196f9ddbf7aa (bug 1318370)
Backed out changeset 72f6eace33a1 (bug 1318370)
Backed out changeset c0c0e10ed613 (bug 1318370)
Backed out changeset a8032ae9cb2a (bug 1318370)
Backed out changeset 0848b3201c40 (bug 1318370)
2017-02-21 21:22:01 -08:00
Ted Mielczarek c841558f8e bug 1318370 - remove SCCACHE_NAMESERVER from mozconfig.cache, unused since we switched to sccache2. r=glandium
MozReview-Commit-ID: 1ktLJiIY9jh

--HG--
extra : rebase_source : 7516f1c845656093818d6350a0927c47e56a429d
2017-02-17 14:31:20 -05:00
Ted Mielczarek 0ba51f3683 bug 1318370 - move _DEPEND_CFLAGS+CL_INCLUDES_PREFIX to toolchain.configure, ignore {CC,CXX}_WRAPPER when using sccache r=glandium
Currently mozconfig.cache overrides a few build options for sccache. This
patch moves them into toolchain.configure so that the build system will
set them properly when sccache is in use. Additionally, {CC,CXX}_WRAPPER
are set in config.mk, so just avoid setting them when sccache is in use.

MozReview-Commit-ID: FYlVKRI8OiN

--HG--
extra : rebase_source : cc7e4346869b98a52840c101824044abc236637f
2017-01-10 16:52:27 -05:00
Ted Mielczarek 5d759be63f bug 1318370 - stop using -Z7 for MSVC compilation with sccache. r=glandium
We've been using -Z7 to make MSVC emit CodeView debug info directly in
the object files instead of in PDB files because sccache can't cache
PDB files if the same one is written to by multiple compiles, which
was previously happening in our build system, and was hard to override
in the NSS build. Now that those are fixed this should work fine.

This adds a bit to the compile rule to remove the PDB file before
compilation, since sccache won't cache a compilation if the PDB file
it is supposed to generate already exists (for the aforementioned reason).

MozReview-Commit-ID: rFlX0XfTGw

--HG--
extra : rebase_source : 8f991ce72115537466f5720c20dc53083f3b2b35
2016-12-14 14:58:26 -05:00
Ted Mielczarek 1c52e9500b bug 1269355 - Enable sccache for taskcluster linux32/64 builds. r=dustin,gps
MozReview-Commit-ID: LGqpNDb7uHN

--HG--
extra : rebase_source : dfd3d0c7208178d7a44cd49a84cd1ead194e8095
2016-11-10 13:02:54 -05:00
Ted Mielczarek fa343243d0 bug 1286934 - Switch to using sccache2. r=gps
This patch does a few things:
1) Change all the in-tree tooltool manifests to contain sccache2 instead of the existing Python sccache
2) Change mozconfig.cache to point at sccache.
3) Lightly tweak the --with-cccache configure option to support sccache, and detect whether we're using ccache or sccache and set an option appropriately.
4) Add a MOZ_SCCACHE_VERBOSE_STATS option, and add a target in the top-level Makefile to make sccache spit out its stats at the end of the build. This is useful to see the cache hits/errors until we get something better.
5) Add MOZ_USING_SCCACHE to the build telemetry. Not that I think it will be immediately useful, but for future use.

MozReview-Commit-ID: 9lrdLwNj5Bm

--HG--
extra : rebase_source : d323457df10d0ee0ac5811940e518d9422a7e070
2016-08-25 18:39:57 -04:00
Gregory Szorc abb5076523 Bug 1315330 - Enable sccache for autoland; r=ted
This should have been done when autoland came into existence.

MozReview-Commit-ID: HaWNq41mDvn

--HG--
extra : rebase_source : 1f8905a45388a147970b65d6d7e226600cf8afc2
2016-11-04 11:49:20 -07:00
Gregory Szorc 0739e582b1 Bug 1315330 - Remove references to fx-team; r=ted
fx-team is read only. We shouldn't be performing any more builds. The
configuration can be removed.

MozReview-Commit-ID: 3cbEnnOKwaZ

--HG--
extra : rebase_source : 1e3b69721f0615327c3882767f0b90ee10bce1cf
2016-11-04 11:47:51 -07:00
Rob Thijssen 2678725c30 Bug 1304998 - enable tc win pgo; r=dustin
MozReview-Commit-ID: 8X9aj19Kp13

--HG--
extra : rebase_source : 34817db2922fa27a4aa36e719dad11f598ad3f2d
2016-09-29 07:49:30 +01:00
Rob Thijssen 2671fd099b Bug 1285506 - include eu-central-1 in tc sccache bucket regions; r=glandium
MozReview-Commit-ID: 801dRLIwGpq

--HG--
extra : transplant_source : %AD%07%F8rm%27%FC%A00%82%60%FD%7EL%8F%9B%DCf%7F%DA
2016-07-25 18:58:23 +01:00
Rob Thijssen 24932bb5c1 Bug 1278990 - Configure sccache for taskcluster ec2 builders. r=mshal 2016-07-11 12:47:09 +01:00
Wes Kocher b8d55438df Backed out changeset cf971ca7f132 (bug 1278990) for likely nightly bustage a=backout CLOSED TREE
--HG--
extra : amend_source : 885e0942249b31cba5b3123caad9d83c3f555c6c
2016-07-08 11:59:12 -07:00
Rob Thijssen c23ccee534 Bug 1278990 - configure sccache for taskcluster ec2 builders; r=glandium
MozReview-Commit-ID: GiPGAaFjZ5t

--HG--
extra : transplant_source : d%0DM%A4%B1%C2%B9K%C2%04%99%07%83%3B%85%E125%F0%93
2016-07-05 16:09:13 +01:00
Vlad Ciobancai eba5401caa Bug 1265799 - Disable b2g-inbound. r=Callek
MozReview-Commit-ID: GYzymyiPz1D
2016-04-25 06:40:55 -04:00
Mike Hommey 78ec73cc58 Bug 1259381 - Don't add --with-ccache in mozconfigs doing --disable-compile-environment or --enable-artifact-builds. r=chmanchester
While moving the ccache flag, we're going to make it disappear when the
compiler environment is not enabled, so adding --with-ccache will fail.
2016-03-30 07:07:44 +09:00
Mike Hommey d50fca7424 Bug 1229613 - Properly disable sccache on PGO builds. r=mshal
Sccache was enabled mechanically by the switch to EC2 Windows instances, while
it is not intended to be used on PGO builds.

This happened because the fix for bug 1181040 disabled sccache for PGO builds
where MOZ_PGO is set through mozconfig (in which case MOZ_PGO_IS_SET is set)
*while* ignoring the case where MOZ_PGO is set through the environment (in
which case MOZ_PGO_IS_SET is, unconveniently, *not* set). The latter is what
Windows PGO builds do.
2015-12-03 13:34:02 +09:00
Mike Hommey cb7ad8ebc8 Bug 1188766 - Avoid exporting UPLOAD_EXTRA_FILES from mozconfig. r=mshal
The way UPLOAD_EXTRA_FILES is currently exported makes the files added
to the list added in each directory that is being traversed recursively
because of the += in .mozconfig.mk, combined with "export".

The easy way out is to remove the export altogether, but being unsure of
the side effects of putting all non exported mk_add_options variables
in .mozconfig.mk, limit this to UPLOAD_EXTRA_FILES.
2015-08-11 07:22:31 +09:00
Mike Hommey 09e6660ab4 Bug 1181040 - Always disable sccache when doing PGO builds. r=gps 2015-08-07 06:51:56 +09:00
Mike Hommey 84878777a1 Bug 1181040 - Include mozconfig.cache after mozconfig.common.override. r=mshal
Enable sccache on windows b2g builds at the same time to simplify things.
2015-07-11 06:26:20 +09:00
Mike Hommey eedb419830 Bug 1042432 - Change how UPLOAD_EXTRA_FILES is passed down from mozconfig.cache to the build system. r=mshal 2014-08-14 07:15:35 +09:00
Mike Hommey 7dc1a15761 Bug 1026841 - Switch in-house builds to us-west-1 bucket for sccache. r=mshal 2014-06-19 09:06:14 +09:00
Mike Hommey 2dade204e9 Bug 1013060 - Stop setting SCCACHE_NO_HTTPS. r=mshal 2014-06-13 22:36:45 +09:00
Mike Hommey 6f0db214f5 Bug 1009981 - Switch non-windows in-house builds to us-west-2 bucket for sccache. r=mshal 2014-05-15 07:14:59 +09:00
Mike Hommey fdfb59dd9b Bug 1009359 - Don't replace UPLOAD_EXTRA_FILES in build/mozconfig.cache. r=mshal 2014-05-15 07:14:37 +09:00
Mike Hommey d0874c3fec Bug 1007991 - Enable sccache on mozilla-inbound and fx-team builds on AWS. r=mshal 2014-05-15 07:14:37 +09:00