Also adjust 'basestring' usage in some places.
Differential Revision: https://phabricator.services.mozilla.com/D28102
--HG--
extra : rebase_source : f0fecd6b040336b0cf2f055e266fe37eea5a9a48
extra : intermediate-source : 712718250f0ba5c286c1fb6b6c7f370b713c7225
extra : source : 6b24ce2da518d8158e13627adb11c9fcad19c92a
shellutil.py needed special handling since it is invoked earlier than the
vendor directory adding we do, so it does not have six available.
Differential Revision: https://phabricator.services.mozilla.com/D28096
--HG--
extra : rebase_source : c5c611d280dc821747c9db736e4943c2cc421557
extra : intermediate-source : 30fb268b41fdc45937a22ab446a8805fb099c1f0
extra : source : d102292b6b5b812e7df165116410488d18042bd3
Lint python/mozbuild/{mozbuild,mozpack}.
This makes sure we use byte strings (type(b'')) in many places.
This also has a few places where we know we want unicode, by enforcing it.
This code also has a few places where we call into windows API's which expect either all args to be unicode,
or all args to be bytestrings (not-unicode) so make sure those calls align.
This is the most risky of the stack, since there are some mach commands and codepaths that are neither excercised in automation
nor excercised in tests.
Differential Revision: https://phabricator.services.mozilla.com/D26645
--HG--
extra : rebase_source : 1ee367576bede0318a2db05644693d4f440a250a
extra : intermediate-source : 2516c350cba6f27eb204f0087728721abbb0e085
extra : source : 6351a29c5765c13c39ef6186f9e5e911e00420e7
When an added file is not under an app or addon directory, its base
directory is ''. The code added in bug 1539355 would skip doing its
thing in that case, which it shouldn't.
Also revert the workaround from bug 1525762.
We have some convoluted packaging logic to automatically package any directory
with a manifest.json file as an XPI. Unfortunately, that logic also applies to
extensions that are supposed to be part of omni.ja.
It would be nice to have a cleaner way to filter out any resources which we've
already flagged to be part of omni.ja, but this patch takes the simpler
approach of just including anything that's in a modules/ directory, which is
whitelisted for omni.ja elsewhere.
--HG--
extra : source : d30967b60a03f8f4286c26f14342d50e03a59f9d
extra : amend_source : 2ddab7bd7ce2fb666d5335bf9941328d9dd7d66f
Practically speaking, it makes no difference, as only the calls to
add_base from tests weren't already ordered.
This allows a simpler approach for next change.
Differential Revision: https://phabricator.services.mozilla.com/D25036
--HG--
extra : moz-landing-system : lando
This demonstrates how addons positioned under a resource directory (a
directory that would normally go in omni.ja) is currently treated.
Differential Revision: https://phabricator.services.mozilla.com/D25035
--HG--
extra : moz-landing-system : lando
Bug 1533425 makes Gecko try to load from $ARCH/greprefs.js, etc on
Android. This patch teaches the packager to put preferences into
those architecture-specific locations for that code to find.
Differential Revision: https://phabricator.services.mozilla.com/D24984
--HG--
extra : moz-landing-system : lando
The jar log is used for optimization of the packaged jar files according
to their usage patterns during a profile run. The current content of the
file currently come with 2 caveats:
- it contains entries for jar archives that aren't relevant to
packaging, which is not a problem in itself, but see below.
- it contains full paths for jar archives that may not correspond to the
location of the packaged directory (on e.g. Android, where the build
almost certainly doesn't happen in the same directory on the host as
Fennec runs in the emulator/on the device).
The current JarLog code does somehow handle the various ways paths are
currently presented, but it's clearly missing code to map the paths in
the log to packaged paths. Instead of requiring manual work and extra
build options to handle this mapping, and considering the caveats above,
it's just simpler to log archive paths as if they were relative to the
packaged application directory in a build, and use that during
packaging.
Depends on D21655
Differential Revision: https://phabricator.services.mozilla.com/D21656
--HG--
extra : moz-landing-system : lando
The jar log is used for optimization of the packaged jar files according
to their usage patterns during a profile run. The current content of the
file currently come with 2 caveats:
- it contains entries for jar archives that aren't relevant to
packaging, which is not a problem in itself, but see below.
- it contains full paths for jar archives that may not correspond to the
location of the packaged directory (on e.g. Android, where the build
almost certainly doesn't happen in the same directory on the host as
Fennec runs in the emulator/on the device).
The current JarLog code does somehow handle the various ways paths are
currently presented, but it's clearly missing code to map the paths in
the log to packaged paths. Instead of requiring manual work and extra
build options to handle this mapping, and considering the caveats above,
it's just simpler to log archive paths as if they were relative to the
packaged application directory in a build, and use that during
packaging.
Depends on D21655
Differential Revision: https://phabricator.services.mozilla.com/D21656
--HG--
extra : moz-landing-system : lando
Before this change linting python/mozbuild brought about many errors. Some of this errors could be fixed using |mach lint --fix|. The remaining errors where fixed in this bug.
Differential Revision: https://phabricator.services.mozilla.com/D20138
--HG--
extra : rebase_source : 8bd3d622d2221b981f08b8a080c1198b002cdc49
extra : amend_source : 3f6a70c1b7104a7c2d83e11fe911942771e331ea
Fennec has a value of OMNIJAR_NAME that contains a directory, contrary
to other platforms, and relies in post-packaging, pre-unpacking steps to
accommodate with the difference.
With this change, we just make the packaging and unpacking steps aware
of this setup, and make allow them to pack/unpack resources in foo/
under foo/$OMNIJAR_NAME, whether $OMNIJAR_NAME is a file name or a path.
This will, further down the road, allow the packager code to handle jar
logs from PGO instrumentation without munging them.
Differential Revision: https://phabricator.services.mozilla.com/D21654
--HG--
extra : moz-landing-system : lando
Optimizing jars without preloading/reordering data only moves the
jar central directory to the beginning of the file, which, without
preloading information, is not very useful. Let's just stop doing it if
there's not going to be preloading/reordering information at all.
Differential Revision: https://phabricator.services.mozilla.com/D21170
--HG--
extra : moz-landing-system : lando
On CI, Windows builds start from different directories on every build,
except when sccache is enabled. This affects many build types, such as
l10n repacks, and the preprocessor likes to put full paths in its
output, which means it includes those different directories, making the
builds non reproducible.
This changes the preprocessor to replace the source and object
directories with generic strings.
Differential Revision: https://phabricator.services.mozilla.com/D20421
--HG--
extra : moz-landing-system : lando
All directories are part of the langpack that is being merged in, but
when the langpack includes the english dictionary, it is not handled
at the same time as other dictionaries, because it is also part of the
original application.
Instead of trying to catch all places where a dictionary might be added
to the final repack, we wrap the formatter so that it tracks all of them
wherever they're added from, and updates the built_in_addons.json file
accordingly.
Differential Revision: https://phabricator.services.mozilla.com/D16785
--HG--
extra : moz-landing-system : lando
By doing this in the packager, it makes it easier to incorporate the
strip and XZ compress logic into the local Gradle build process.
To that end, this patch makes XZ compression a little more explicit in
package-manifest.in and lifts the logic next to the existing logic for
stripping. Since we only want to XZ compress assets/ (and not libs/),
we need a new flag.
Differential Revision: https://phabricator.services.mozilla.com/D7314
--HG--
extra : moz-landing-system : lando
By doing this in the packager, it makes it easier to incorporate the
strip and XZ compress logic into the local Gradle build process.
To that end, this patch makes XZ compression a little more explicit in
package-manifest.in and lifts the logic next to the existing logic for
stripping. Since we only want to XZ compress assets/ (and not libs/),
we need a new flag.
Differential Revision: https://phabricator.services.mozilla.com/D7314
--HG--
extra : moz-landing-system : lando
We don't actually ship XPT files anymore, but it's still useful for the
packager code to handle old Firefox versions. But for that, we don't
really need the complexity of "linking" XPT files in a single unit per
directory. We can just as well keep the XPT files intact, as long as we
retain individual `interfaces` manifest entries for each.
And since those entries used to be all merged into one, we now instead
group them all together in manifests (which also happens to make it
easier on unit test changes).
Differential Revision: https://phabricator.services.mozilla.com/D5221
--HG--
extra : moz-landing-system : lando
This will make sure that when running |mach python-test --python 3| locally,
we only run the tests that also run in CI with python 3 (and therefore pass
presumably).
MozReview-Commit-ID: 3OBr9yLSlSq
--HG--
extra : rebase_source : 456340d0ecdddf1078f2b5b4ebb1eddf3813b26a
This is slightly ugly, but is unfortunately necessary due to do the nature of
l10n repacks. Hopefully this can go away once we move to bundling lancpack
add-ons rather than repacking in the future.
MozReview-Commit-ID: JZUblVsEbZI
--HG--
extra : rebase_source : 60c9ced2184a52f52c7f2a8820021b14b1a66abf
There's a bug in InstallManifest::add_entries_from: some of the
manifest entries bake their destination into both the manifest key and
the manifest value, and add_entries_from with a non-empty "base"
parameter to prepend to the destination only updates the manifest key
and not the value.
This bug causes |mach watch| to fail to _read_ the unified manifest
that aggregates all of the build manifests relevant to |mach watch|
that |mach build-backend -b FasterMake| successfully _wrote_, because
the manifest keys are validated against the manifest values written to
disk.
I see no way to address this other than to manually reach into the
manifest values and patch the internal destination parameter, which
this patch does.
MozReview-Commit-ID: FVyRB41NnHa
--HG--
extra : rebase_source : 23eb18ddc0452955539ce2e7a6d7bbfd083c940c
When repackaging, we want to take all localization resources from the source package,
and add all localization resources from the repackage target locale.
This patch makes packager.l10n scan for `localization` directories in l10n_finder and
add them.
MozReview-Commit-ID: CRLP3bOAyDx
--HG--
extra : rebase_source : 3b1bc098aaed2481a1cc06fc86d0865e654a1b6e
This ensures we use forward slashes, even if Mercurial emits
backslashes (which it can do on Windows).
MozReview-Commit-ID: 2dnWAEvytwn
--HG--
extra : rebase_source : 9ebb454bc9ad11b3eba334e412685e529573a0a1
These tests weren't running in automation because hglib wasn't
available. An upcoming commit will vendor hglib. This exposed that the
tests can fail if ui.username isn't set.
In this commit, we introduce a helper function to obtain an hglib
client with ui.username set. We also convert tests to use the context
manager form of the client so resources are cleaned up immediately
without relying on refcounting or garbage collection.
MozReview-Commit-ID: HRSBDlYgqpC
--HG--
extra : rebase_source : 83deb56a0c2efefa883d6df104cd67194a811907
The install manifest with the .track files uses os.path.exists() to
determine if a previously tracked file is no longer installed and needs
to be removed from the system. However, exists() returns False for
broken symlinks, so as far as the manifest is concerned, there is no
file in the filesystem that needs to be removed. We should use lexists()
so we know that the broken symlink still exists in the system so
that it can be removed when the install manifest is processed.
MozReview-Commit-ID: 6v7CYOKzjGs
--HG--
extra : rebase_source : 8aeeef59e644613f34c8458bd30a83d8299585ea
This passes `python3 -mcompileall`. Changes:
* use `0o` prefix for octal literals
* print as a function
* except .. as
* use six.reraise to replace a multi-argument raise statement
* use six.string_types and six.moves.configparser
* remove uses of `L` suffix for long integers
MozReview-Commit-ID: KLaYRNHGpay
--HG--
extra : rebase_source : 6ca1b5447cd28eff8d9f2805add6a0f07e8b4c63
Bug 1355661 added support for brotli streams in "jar" files handled by
Gecko, and bug 1355671 made us build the `bro` command line utility that
allows to compress and decompress brotli streams.
This change uses the `bro` command line utility in the packager so that
it can create and handle "jar" files using brotli streams.
However, the `bro` command line utility is not available to l10n
repacks. As, at the moment, we're only hoping that the outcome of using
brotli will be good, we avoid doing all the work to make those work and
just hook things enough to enable brotli, while ensuring l10n repacks
don't break. This involves forcing some files to be deflated, and to
disable some optimizations from the packager.
Things will need to be figured out more properly if the experiment
proves brotli to be worthwhile.
--HG--
extra : rebase_source : a2e0cff67dcaed465fd441ed5d2a7de94b6351c5
This patch introduces a new registry for localization resources to replace
ChromeRegistry. It uses asynchronous I/O and iterators to generate
permutations of possible sources of language resources for use in the new
Localization API.
In the initial form the API handles packages resources and has API for
interacting with the AddonsManager which will report language packs.
MozReview-Commit-ID: LfERDYMROh
--HG--
extra : rebase_source : 68a664c2c59a82b4dfcae66542c315a00ddae565
This patch introduces a new registry for localization resources to replace
ChromeRegistry. It uses asynchronous I/O and iterators to generate
permutations of possible sources of language resources for use in the new
Localization API.
In the initial form the API handles packages resources and has API for
interacting with the AddonsManager which will report language packs.
MozReview-Commit-ID: LfERDYMROh
--HG--
extra : rebase_source : a6e5b790142e5afb1ce750478190e5ad87012f8d
There's a natural follow-on that I haven't time to explore right now:
I want the faster make backend to also write a "unified chrome
manifest" that maps outputs
(browser/chrome/browser/content/browser/ext-utils.js) to chrome:// or
resource:// URLs (chrome://content/browser/ext-utils.js or similar).
MozReview-Commit-ID: LDQmm8KD57I
--HG--
extra : rebase_source : 13dbeac2fbfa78741be3718fd5305a8ae0d698a8
extra : source : 2af5df7741147769a51da78770c308b370a0cded
This is necessary because the existing manifests don't expose full
dependency information. I needed to avoid the existing dependency
files because those code paths need to know the output destination of
the manifest in order to parse the Make dependency files; trying to
adapt this system is more complicated than just preprocessing each
file to extract dependency information directly.
MozReview-Commit-ID: 5m0SEqmhJMM
--HG--
extra : rebase_source : 1ff6a1a51bc76efa78eb564cd8e572777dace0f6
Also removes InstallManifestNoSymlinks which can be more simply expressed by
passing link_policy='copy' to InstallManifest.populate_registry.
MozReview-Commit-ID: Bkjc2hIub4A
--HG--
extra : rebase_source : df05080fef3baf7e6c60ff9a468e71f0b2c67462
The change to l10n packager from bug 780562 worked in practice because
no chrome category had exclusively manifest entries with flags, which
we're changing in this bug.
It turns out this was only due to a missing change in the patch for bug
780562.
--HG--
extra : rebase_source : 9f782e115f97063c97f165ed95eb4beeb72f86d0
The change to l10n packager from bug 780562 worked in practice because
no chrome category had exclusively manifest entries with flags, which
we're changing in this bug.
It turns out this was only due to a missing change in the patch for bug
780562.
--HG--
extra : rebase_source : 3c8c31c37d8fb48bb99b1758bcd8ef5f32c71fe0
This allows us to write files coming from a finder or other source
that isn't directly the filesystem.
MozReview-Commit-ID: KhPSD0JYzsQ
--HG--
extra : rebase_source : fea376f642e20e8c7723506fd4a73e3f8ac5d0e5
extra : intermediate-source : a872303fd08497bbde0e3b4cea09a88a4182810e
extra : source : ae8bce278626bc84914063f93292ac5e825eec36
This allows us to write files coming from a finder or other source
that isn't directly the filesystem.
MozReview-Commit-ID: KhPSD0JYzsQ
--HG--
extra : rebase_source : 24db84974b54a714ba82dfad7ff68fd1a5bf656a
extra : source : ae8bce278626bc84914063f93292ac5e825eec36
Some manifest entries (e.g. skin or locale) have an attached identifier,
and there can be different entries with different identifiers for the
same chrome name. The change from bug 1366169 would consider those as
errors, while they are the expected configuration.
--HG--
extra : rebase_source : ceb08da909121a2ac0a2cdaba7970e4594dde09f
This allows us to write files coming from a finder or other source
that isn't directly the filesystem.
MozReview-Commit-ID: KhPSD0JYzsQ
--HG--
extra : rebase_source : 10d494fc910982c3e34f4744592edca906d3a85d
As described in changeset c94e87a18096, chrome manifest entries are
reordered and don't necessarily appear in the order they have in jar.mn.
And in some cases, the order does matter: when entries with flags are
followed by entries with more broad flags or no flags at all. Nobody
should be writing entries in that order on purpose, so error out during
packaging when we detect the pattern.
--HG--
extra : rebase_source : d9617bbcbd8560503c532a13c10c8afb0fd49411
This is the initial support of 'mach repackage', which can take an
existing tarball and create a DMG on either an OSX host or on a Linux
host with cross-OSX tools. Configure is needed in order to find the
tools necessary to create the DMG. On a Linux cross-compiled environment
with tooltool, this can be as simple as:
export MKFSHFS=$topsrcdir/hfsplus-tools/newfs_hfs
export DMG_TOOL=$topsrcdir/dmg/dmg
export HFS_TOOL=$topsrcdir/dmg/hfsplus
ac_add_options --disable-compile-environment
MozReview-Commit-ID: 6t2rlXpwUvu
--HG--
extra : rebase_source : 39f2f9ecac9d7e5af197f1b5dea70cd307acf488
This removes the UNIFY_DIST and UNIFIED_BUILD variables, as well as the
--unify flag from the packager and UnifiedBuildFinder from mozpack. As a
result the STAGEPATH variable is never defined anymore, so its uses can
be removed as well.
test_unify.py is currently the only mozbuild/mozpack test that fails
without running configure first, and there isn't much point in fixing
tests for things that we don't actually use anymore.
MozReview-Commit-ID: F5q1FPW3Did
--HG--
extra : rebase_source : cadbd237f51c23ea1983135294521d628d16f0df
CLOSED TREE
Backed out changeset 158233bce738 (bug 1197325)
Backed out changeset b5ac3fa0bbe7 (bug 1197325)
Backed out changeset 55a8ad127517 (bug 1197325)
Back when the class was written, for the packaging code, it made sense
that the default was True. But now that it's used all over the place,
and that the vast majority of uses are with find_executables=False, it
makes more sense for that to be the default.
--HG--
extra : rebase_source : ff813735fc0d53093f348f20eb77ee03e9b09d4e
And make it an error not to give it. While the default is True, we do
pass a value of False wherever it makes sense, which happens to be, in
most places.
This is an intermediate step to flip the default from True to False,
ensuring that we don't unwantedly switch some calls to False.
--HG--
extra : rebase_source : 432e03f032fef378af482581685583262e5d2661
Make the symlinks in the dist directory relative instead of absolute.
MozReview-Commit-ID: HS7KL4JwSbV
--HG--
extra : rebase_source : 5dca673cc17423d47e6707d8800f7ee9693a9c48
This deprecates PYTHON_UNIT_TESTS and replaces it with PYTHON_UNITTEST_MANIFESTS.
In the build system, this means python unittests will be treated the same as all
other test suites that use manifestparser. New manifests called 'python.ini' have
been created for all test directories containing python unittests.
MozReview-Commit-ID: IBHG7Thif2D
--HG--
extra : rebase_source : 11a92a2bc544d067946bbd774975140147458caa
As of Python 3, decimal notations of octal values for permission modes
are no longer permitted and will result in a `SyntaxError` exception
(`invalid token`).
Using the proper octal notation which is also Python 2.7 compatible will
fix this issue.
--HG--
extra : rebase_source : 2e897c51f04ad0ee69071f84b98df224f3af72d3
Mingw python has a different os.path setup from native python, and has
os.sep and os.altsep reversed. In that case, the normsep function was
doing the wrong thing, leading to all sorts of problems.
While fixing this, also ensure the corresponding unit test covers this
peculiarity, even when running under the native win32 python.
--HG--
extra : rebase_source : 8fb18e0d4dc669c1d7e069f73fc44c22d419d43c
I have a need to create tar archives deterministically
and reproducibly. Since we already have similar functionality in
mozpack for producting zip/jar archives, I figured it made sense for
this functionality to live in mozpack.
I made the functionality as simple as possible: we only accept
files from the filesystem and the set of files must be known in
advance. No class to hold/buffer state: just a simple function
that takes a mapping of files and writes to a stream.
MozReview-Commit-ID: If0NTcA7wpc
--HG--
extra : rebase_source : 9cbea36347ba2840dd5bff9dffefd994a73a0725
This will be needed to teach artifact builds to extract files from
omni.ja files whose content is loaded into memory (from a tar
archive).
MozReview-Commit-ID: LH2HkKx5Zj3
--HG--
extra : rebase_source : 7f6b176f0ef9fc87889151d1d02da62de8a455d8
extra : source : 266928b5a7615fa054c70adf0f649cbb3f085e8d
Generally speaking, the configuration needs forward-slashes in paths.
We might as well make it hard(er) to set configuration items with
backslash separators on Windows by exposing mozpath.* functions in place
of os.path functions. The downside is that functions explicitly
importing os will still get the real os.path functions.
As previous measurements have shown, creating/appending files
on Windows/NTFS is slow because the CloseHandle() Win32 API takes
1-3ms to complete. This is apparently due to a fundamental issue
with NTFS extents. A way to work around this slowness is to use
multiple threads for I/O so file closing doesn't block execution
as much.
This commit updates the file copier to use a thread pool of 4
threads when processing file copies. Additional threads appear
to have diminishing returns.
On my i7-6700K, this reduces the time for processing the tests install
manifest (24,572 files) on Windows from ~22.0s to ~12.5s in the best
case.
Using the thread pool globally resulted in a performance regression
on Linux. Given the performance sensitivity of manifest copying,
I thought it best to implement a slightly redundant non-Windows
branch to preserve performance. For the record, that same machine
running Linux is capable of processing nearly the same install
manifest (24,616 files) in ~2.2s in the best case.
MozReview-Commit-ID: B9LbKaOoO1u
--HG--
extra : rebase_source : e9fee3861a70e1da9d18448f8435f4bd3e28c647
This is two straightforward optimizations in FileCopier: avoiding a redundant iteration
over the directory structure to find destination files (which includes a
call to normpath) and avoiding redundant calls to determine directories to preserve
when remove_unaccounted is not specified (which include a call to dirname).
Running a no-op install of _tests with this patch results in a reduction of about
25,000 calls to normpath and remove about 220,000 calls to dirname, resulting in
an overall speedup of 10-20%.
--HG--
extra : commitid : 8nyTo489q8X
Future improvements to process_install_manifest's --track option will require
adding data in the tracking dump that uses an install manifest form, and I don't
want e.g. switching branches or bisection to require to clobber in order to do the
right thing, so this change future-proofs the install manifest reader.
The default behavior for a FileCopier's copy is to remove all the files and
directories in the destination that aren't in its registry.
The remove_unaccounted argument can be passed as False to disable this
behavior.
This change adds another possibility, where remove_unaccounted may be a
FileRegistry, in which case only the files in that registry are removed.
This allows to e.g. only remove files that were copied from a previous
FileCopier.copy, leaving aside files that were in the destination for some
other reason.
Future improvements to process_install_manifest's --track option will require
adding data in the tracking dump that uses an install manifest form, and I don't
want e.g. switching branches or bisection to require to clobber in order to do the
right thing, so this change future-proofs the install manifest reader.
The default behavior for a FileCopier's copy is to remove all the files and
directories in the destination that aren't in its registry.
The remove_unaccounted argument can be passed as False to disable this
behavior.
This change adds another possibility, where remove_unaccounted may be a
FileRegistry, in which case only the files in that registry are removed.
This allows to e.g. only remove files that were copied from a previous
FileCopier.copy, leaving aside files that were in the destination for some
other reason.
FlatFormatter, JarFormatter and OmniJarFormatter all, in some way, deal
with different pieces of the package being handled differently.
Instead of each of them dealing with their different pieces in some subtly
different way, instead, introduce a new base package formatter class that
will handle it for all of them.
Use this new PiecemealFormatter for the FlatFormatter.
Justin Wood
Mike Hommey
Kris Maglione
Noemi Erli
Nick Alexander
Gurzau Raul
kerlyn
Marco Castelluccio
Harry Twyford
Daniel Varga
Ciure Andrei
Andrew Swan
Andrew Halberstadt
assiya
Tim Nguyen
Tom Prince
Masatoshi Kimura
Sebastian Hengst
Zibi Braniecki
Gregory Szorc
Mike Shal
Dustin J. Mitchell
Chung-Sheng Fu
Wes Kocher
Swapnesh Kumar Sahoo
Alex Gaynor
Philip Chimento
Ryan VanderMeulen
Chris AtLee
Carsten "Tomcat" Book
Phil Ringnalda
Jack Bates
aleth
Yen Chi Hsuan
Henrik Skupin
Chris Manchester
Elias Probst
Ms2ger
Myk Melez
Gijs Kruitbosch