Граф коммитов

372 Коммитов

Автор SHA1 Сообщение Дата
Simon Giesecke 971b645fe3 Bug 1660470 - Add missing include directives/forward declarations. r=nika
Differential Revision: https://phabricator.services.mozilla.com/D87865
2020-11-23 16:21:38 +00:00
Ricky Stewart 02a7b4ebdf Bug 1654103: Standardize on Black for Python code in `mozilla-central`.
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.

5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-26 18:34:53 +00:00
Bogdan Tara da1098d4aa Backed out 10 changesets (bug 1654103, bug 1672023, bug 1518999) for PanZoomControllerTest.touchEventForResult gv-junit failures CLOSED TREE
Backed out changeset ff3fb0b4a512 (bug 1672023)
Backed out changeset e7834b600201 (bug 1654103)
Backed out changeset 807893ca8069 (bug 1518999)
Backed out changeset 13e6b92440e9 (bug 1518999)
Backed out changeset 8b2ac5a6c98a (bug 1518999)
Backed out changeset 575748295752 (bug 1518999)
Backed out changeset 65f07ce7b39b (bug 1518999)
Backed out changeset 4bb80556158d (bug 1518999)
Backed out changeset 8ac8461d7bd7 (bug 1518999)
Backed out changeset e8ba13ee17f5 (bug 1518999)
2020-10-24 03:36:18 +03:00
Ricky Stewart c0cea3b0fa Bug 1654103: Standardize on Black for Python code in `mozilla-central`. r=remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers,devtools-backward-compat-reviewers,jgilbert,preferences-reviewers,sylvestre,maja_zf,webcompat-reviewers,denschub,ntim,whimboo,sparky
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Make some ad-hoc manual updates to `testing/marionette/client/setup.py`, `testing/marionette/harness/setup.py`, and `testing/firefox-ui/harness/setup.py`, which have hard-coded regexes that break after the reformat.

5. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-23 20:40:42 +00:00
Dorel Luca 1ff59cb7a3 Backed out changeset 7558c8821a07 (bug 1654103) for multiple failures. CLOSED TREE 2020-10-22 03:51:06 +03:00
Ricky Stewart 50762dacab Bug 1654103: Standardize on Black for Python code in `mozilla-central`. r=remote-protocol-reviewers,marionette-reviewers,webdriver-reviewers,perftest-reviewers,devtools-backward-compat-reviewers,jgilbert,preferences-reviewers,sylvestre,maja_zf,webcompat-reviewers,denschub,ntim,whimboo,sparky
Allow-list all Python code in tree for use with the black linter, and re-format all code in-tree accordingly.

To produce this patch I did all of the following:

1. Make changes to tools/lint/black.yml to remove include: stanza and update list of source extensions.

2. Run ./mach lint --linter black --fix

3. Make some ad-hoc manual updates to python/mozbuild/mozbuild/test/configure/test_configure.py -- it has some hard-coded line numbers that the reformat breaks.

4. Add a set of exclusions to black.yml. These will be deleted in a follow-up bug (1672023).

# ignore-this-changeset

Differential Revision: https://phabricator.services.mozilla.com/D94045
2020-10-21 21:27:27 +00:00
Doug Thayer 025f5fb9f0 Bug 1656261 - Back out all recent StartupCache work r=RyanVM
This backs out all work from bug 1627075 as well as all of its
descendents. There were a few conflicts when backing this out but
overall it was pretty clean, so I would say it's a fairly mild
level of risk. Historically Nathan Froyd has reviewed these patches,
but he is no longer at Mozilla, and no one else is particularly
familiar with the code, so I am passing this off to RyanVM who has
at least been familiar with the history of the bug.

Differential Revision: https://phabricator.services.mozilla.com/D90096
2020-09-14 17:00:53 +00:00
Doug Thayer 2b194eb9d0 Bug 1656261 - Ensure we clean up after WriteToDisk r=froydnj
Given all of the MOZ_TRYs in WriteToDisk, it's entirely possible that
we occasionally exit it before writing everything out. WriteToDisk
additionally adds guards to try to ensure that we do not call it twice,
and that we do not try to read things out of the file via the memmapped
buffer after it has written to it. The latter should not generally be
a problem, since we write to a temp file and move that over at the end.
However, writing twice could be a problem as we modify our values'
mOffsets to keep track of their offsets within the resulting file.

Accordingly, this change simply moves the cleanup and guards into an
RAII helper to ensure they survive all of the early returns.

Differential Revision: https://phabricator.services.mozilla.com/D89253
2020-09-09 13:42:27 +00:00
Doug Thayer 741f066b9a Bug 1656261 - Don't set scache mData until decompression done r=froydnj
So far I have not found a way for this issue to cause bug 1656261, but
it's definitely plausible that there's something there. If we hit a
EXCEPTION_IN_PAGE_ERROR in GetBuffer, then we will exit the
DecompressEntry method with the entry's mData being set. In any
subsequent calls to GetBuffer for that key, we will return the
uninitialized memory. If we can hit this path with chrome.manifest,
then we would end up reading garbage from it, and could conclude that
xul.css is missing.

Whether this is the culprit or not, this is a problem that needs
addressing.

Differential Revision: https://phabricator.services.mozilla.com/D87663
2020-08-21 13:20:19 +00:00
Doug Thayer 5f41a2f565 Bug 1658732 - Replace mStartupFinished with TryLock in StartupCache r=froydnj
To be honest, it's still a mystery why we observed a regression in
sessionrestore_no_auto_restore in bug 1658732. The regression won't reproduce
on profiled runs, and the bad recordings happen before the supposedly offending
code ever actually runs. It feels most likely that it is a more or less random
confluence of factors causing a regression; however, 33% is too large of a
number to ignore.

The changes in this patch do not seem to yield the same regression, and they
are arguably more correct anyway. Instead of simply turning off the cache after
startup is finished, we simply avoid blocking waiting for the write from inside
GetBuffer. This way, if the write is not getting in the way of GetBuffer, we
can still benefit from a cached version of whatever it is we're looking for.

Differential Revision: https://phabricator.services.mozilla.com/D87221
2020-08-20 18:37:21 +00:00
Doug Thayer 973cdcbc2e Bug 1656974 - Turn off StartupCache when startup is finished r=froydnj
I don't think this will fully resolve the shutdown hangs associated
with this bug. The underlying problem is that writing the startup
cache during shutdown can take a long time. This seems like A Bad Thing,
and I have a mild leaning toward the thought that we shouldn't write the
startup cache at all during shutdown. However, I do think that trading
shutdown time regressions for startup time wins is probably good, so
I don't feel super comfortable making that change.

However, this change at least fixes the problem where we A) block the main
thread waiting for the write to finish, and B) are more likely to hit a
shutdown hang, since we concentrate all of the hang time in one phase,
rather than it being spread across the shutdown timeline. This is likely
the major change that the regressing bug introduced.

The unfortunate consequence is that if we try to GetBuffer during shutdown,
we will now read out of the omnijar. However, it should overall be better
since we'll be waiting on a (hopefully) small number of small reads rather
than a large write.

Differential Revision: https://phabricator.services.mozilla.com/D86054
2020-08-07 19:42:56 +00:00
Simon Giesecke 1e02318b49 Bug 1653335 - Replace MakeSpan uses by constructor calls. r=froydnj
Differential Revision: https://phabricator.services.mozilla.com/D83817
2020-08-07 07:49:47 +00:00
Mihai Alexandru Michis b549c013c7 Backed out changeset 25e500b64866 (bug 1656974) for causing failures in TestStartupCache
CLOSED TREE
2020-08-07 02:37:59 +03:00
Doug Thayer d10827791a Bug 1656974 - Turn off StartupCache when startup is finished r=froydnj
I don't think this will fully resolve the shutdown hangs associated
with this bug. The underlying problem is that writing the startup
cache during shutdown can take a long time. This seems like A Bad Thing,
and I have a mild leaning toward the thought that we shouldn't write the
startup cache at all during shutdown. However, I do think that trading
shutdown time regressions for startup time wins is probably good, so
I don't feel super comfortable making that change.

However, this change at least fixes the problem where we A) block the main
thread waiting for the write to finish, and B) are more likely to hit a
shutdown hang, since we concentrate all of the hang time in one phase,
rather than it being spread across the shutdown timeline. This is likely
the major change that the regressing bug introduced.

The unfortunate consequence is that if we try to GetBuffer during shutdown,
we will now read out of the omnijar. However, it should overall be better
since we'll be waiting on a (hopefully) small number of small reads rather
than a large write.

Differential Revision: https://phabricator.services.mozilla.com/D86054
2020-08-06 16:18:13 +00:00
Doug Thayer 249401cc24 Bug 1651941 - Free unreferenced child StartupCache entries on send r=froydnj
This is similar to the first patch in the stack, but instead of freeing the
entries on write, we free them when we send them to the parent to be written.

Differential Revision: https://phabricator.services.mozilla.com/D83401
2020-07-22 20:31:31 +00:00
Doug Thayer a538eb7804 Bug 1651941 - Avoid unnecessary empty StartupCaches r=froydnj
Prior to this patch, we were sending a boolean from InitContentChild (which
creates our StartupCache IPC actors) indicating whether we wanted to collect
new entries from a given process or not. This was so that we wouldn't accept
PutBuffer requests in these processes, since collecting them in one process
would be enough, and we don't want to waste memory. However, we actually
want the cache to be available before we can even get that IPC constructor
to the child process, so there's a window where we accept new entries
no matter what. This patch changes this by sending a boolean argument via
the command line indicating that we want to disable the Startupcache in this
process entirely. We send this when we didn't load a StartupCache off disk,
as this should be the only circumstance in which we're actually collecting
a substantial number of entries in content processes.

Differential Revision: https://phabricator.services.mozilla.com/D83400
2020-07-22 20:31:21 +00:00
Doug Thayer 6bc1e34006 Bug 1651941 - Free unreferenced StartupCache entries on write r=froydnj
On first run, when the cache is empty, we collect a lot of freshly allocated
buffers for StartupCache entries. These entries are generally not requested
again after being put in the StartupCache, so they are generally safe to free
(as ownership of them was given to us by contract in PutBuffer.) This tends
to free up >10MB.

Differential Revision: https://phabricator.services.mozilla.com/D83399
2020-07-22 20:30:41 +00:00
Doug Thayer c4dd1705c3 Bug 1651941 - Use shared buffer for Content Process StartupCache keys r=froydnj
Keys can take up a nontrivial chunk of memory. This change ensures that the
memory cost is not per-process. In order to do this we had to switch from
nsCStrings to MaybeOwnedCharPtrs pointing to null-terminated char sequences.
This meant fiddling around with the code that loads / stores to disk and
making it a little bit uglier, rather than adding another method to the
IOBuffers classes to support our weird maybe-shared string type.

Differential Revision: https://phabricator.services.mozilla.com/D83398
2020-07-22 20:30:24 +00:00
Nika Layzell 22a65a237e Bug 1650163 - Part 1: Switch native remoteType values to nsCString, r=farre,geckoview-reviewers,agi
Differential Revision: https://phabricator.services.mozilla.com/D82104
2020-07-08 20:15:59 +00:00
Doug Thayer fcbcc674d2 Bug 1627075 - OMT and OMP StartupCache access r=froydnj
The overall goal of this patch is to make the StartupCache accessible anywhere.
There's two main pieces to that equation:

1. Allowing it to be accessed off main thread, which means modifying the
   mutex usage to ensure that all data accessed from non-main threads is
   protected.
2. Allowing it to be accessed out of the chrome process, which means passing
   a handle to a shared cache buffer down to child processes.

Number 1 is somewhat fiddly, but it's all generally straightforward work. I'll
hope that the comments and the code are sufficient to explain what's going on
there.

Number 2 has some decisions to be made:
- The first decision was to pass a handle to a frozen chunk of memory down to
  all child processes, rather than passing a handle to an actual file. There's
  two reasons for this: 1) since we want to compress the underlying file on
  disk, giving that file to child processes would mean they have to decompress
  it themselves, eating CPU time. 2) since they would have to decompress it
  themselves, they would have to allocate the memory for the decompressed
  buffers, meaning they cannot all simply share one big decompressed buffer.

  - The drawback of this decision is that we have to load and decompress the
    buffer up front, before we spawn any child processes. We attempt to
    mitigate this by keeping track of all the entries that child processes
    access, and only including those in the frozen decompressed shared buffer.

  - We base our implementation of this approach off of the shared preferences
    implementation. Hopefully I got all of the pieces to fit together
    correctly. They seem to work in local testing and on try, but I think
    they require a set of experienced eyes looking carefully at them.

- Another decision was whether to send the handles to the buffers over IPC or
  via command line. We went with the command line approach, because the startup
  cache would need to be accessed very early on in order to ensure we do not
  read from any omnijars, and we could not make that work via IPC.

  - Unfortunately this means adding another hard-coded FD, similar to
    kPrefMapFileDescriptor. It seems like at the very least we need to rope all
    of these together into one place, but I think that should be filed as a
    follow-up?

Lastly, because this patch is a bit of a monster to review - first, thank you
for looking at it, and second, the reason we're invested in this is because we
saw a >10% improvement in cold startup times on reference hardware, with a p
value less than 0.01. It's still not abundantly clear how reference hardware
numbers translate to numbers on release, and they certainly don't translate
well to Nightly numbers, but it's enough to convince me that it's worth some
effort.

Depends on D78584

Differential Revision: https://phabricator.services.mozilla.com/D77635
2020-07-08 02:46:11 +00:00
Doug Thayer 00ed10f38d Bug 1627075 - Init StartupCache before Omnijar r=froydnj
We need to be able to init StartupCache before the Omnijar in order to cache
all of the Omnijar contents we access. This patch implements that.

Depends on D77632

Differential Revision: https://phabricator.services.mozilla.com/D77633
2020-07-08 02:43:30 +00:00
Doug Thayer d9fd460e11 Bug 1627075 - Build Omnijar file list from startup cache r=froydnj
We would like to be able to defer opening the omnijar files until after startup
if the StartupCache has already been populated. Opening the omnijar files takes
a nontrivial time, at least on Windows, and almost everything in the omnijar
should be fairly compressible, and thus makes sense to live in the StartupCache.
See the last patch in this series for a little more discussion on numbers, but
tl;dr: we saw a 12% improvement in time to about:home being finished on reference
hardware with these changes together with the changes from the descendant patches.

Differential Revision: https://phabricator.services.mozilla.com/D77632
2020-07-08 02:43:02 +00:00
Narcis Beleuzu a182c015f5 Backed out 6 changesets (bug 1627075) for bustages on startupcache/StartupCache.cpp . CLOSED TREE
Backed out changeset 21605186687e (bug 1627075)
Backed out changeset e29b15980da2 (bug 1627075)
Backed out changeset eb5265addd5e (bug 1627075)
Backed out changeset dfd71f4ecb81 (bug 1627075)
Backed out changeset 13ecd68b3c0d (bug 1627075)
Backed out changeset 333d035afe92 (bug 1627075)
2020-07-07 23:30:48 +03:00
Doug Thayer 4d6b5a1f91 Bug 1627075 - OMT and OMP StartupCache access r=froydnj
The overall goal of this patch is to make the StartupCache accessible anywhere.
There's two main pieces to that equation:

1. Allowing it to be accessed off main thread, which means modifying the
   mutex usage to ensure that all data accessed from non-main threads is
   protected.
2. Allowing it to be accessed out of the chrome process, which means passing
   a handle to a shared cache buffer down to child processes.

Number 1 is somewhat fiddly, but it's all generally straightforward work. I'll
hope that the comments and the code are sufficient to explain what's going on
there.

Number 2 has some decisions to be made:
- The first decision was to pass a handle to a frozen chunk of memory down to
  all child processes, rather than passing a handle to an actual file. There's
  two reasons for this: 1) since we want to compress the underlying file on
  disk, giving that file to child processes would mean they have to decompress
  it themselves, eating CPU time. 2) since they would have to decompress it
  themselves, they would have to allocate the memory for the decompressed
  buffers, meaning they cannot all simply share one big decompressed buffer.

  - The drawback of this decision is that we have to load and decompress the
    buffer up front, before we spawn any child processes. We attempt to
    mitigate this by keeping track of all the entries that child processes
    access, and only including those in the frozen decompressed shared buffer.

  - We base our implementation of this approach off of the shared preferences
    implementation. Hopefully I got all of the pieces to fit together
    correctly. They seem to work in local testing and on try, but I think
    they require a set of experienced eyes looking carefully at them.

- Another decision was whether to send the handles to the buffers over IPC or
  via command line. We went with the command line approach, because the startup
  cache would need to be accessed very early on in order to ensure we do not
  read from any omnijars, and we could not make that work via IPC.

  - Unfortunately this means adding another hard-coded FD, similar to
    kPrefMapFileDescriptor. It seems like at the very least we need to rope all
    of these together into one place, but I think that should be filed as a
    follow-up?

Lastly, because this patch is a bit of a monster to review - first, thank you
for looking at it, and second, the reason we're invested in this is because we
saw a >10% improvement in cold startup times on reference hardware, with a p
value less than 0.01. It's still not abundantly clear how reference hardware
numbers translate to numbers on release, and they certainly don't translate
well to Nightly numbers, but it's enough to convince me that it's worth some
effort.

Depends on D78584

Differential Revision: https://phabricator.services.mozilla.com/D77635
2020-07-07 17:03:28 +00:00
Doug Thayer fafe31f134 Bug 1627075 - Init StartupCache before Omnijar r=froydnj
We need to be able to init StartupCache before the Omnijar in order to cache
all of the Omnijar contents we access. This patch implements that.

Depends on D77632

Differential Revision: https://phabricator.services.mozilla.com/D77633
2020-07-07 17:02:33 +00:00
Doug Thayer 7915db05b2 Bug 1627075 - Build Omnijar file list from startup cache r=froydnj
We would like to be able to defer opening the omnijar files until after startup
if the StartupCache has already been populated. Opening the omnijar files takes
a nontrivial time, at least on Windows, and almost everything in the omnijar
should be fairly compressible, and thus makes sense to live in the StartupCache.
See the last patch in this series for a little more discussion on numbers, but
tl;dr: we saw a 12% improvement in time to about:home being finished on reference
hardware with these changes together with the changes from the descendant patches.

Differential Revision: https://phabricator.services.mozilla.com/D77632
2020-07-07 17:02:30 +00:00
Mihai Alexandru Michis 87cb0ad6fa Backed out 6 changesets (bug 1627075) for causing bustages in StartupCache.cpp
CLOSED TREE

Backed out changeset fc144caf5d06 (bug 1627075)
Backed out changeset a345e05df151 (bug 1627075)
Backed out changeset 288a67aed661 (bug 1627075)
Backed out changeset 2cb021a493d8 (bug 1627075)
Backed out changeset 920398d1c3d3 (bug 1627075)
Backed out changeset ebdcd96a9d20 (bug 1627075)
2020-07-07 08:47:14 +03:00
Doug Thayer 5ff30b60fa Bug 1627075 - OMT and OMP StartupCache access r=froydnj
The overall goal of this patch is to make the StartupCache accessible anywhere.
There's two main pieces to that equation:

1. Allowing it to be accessed off main thread, which means modifying the
   mutex usage to ensure that all data accessed from non-main threads is
   protected.
2. Allowing it to be accessed out of the chrome process, which means passing
   a handle to a shared cache buffer down to child processes.

Number 1 is somewhat fiddly, but it's all generally straightforward work. I'll
hope that the comments and the code are sufficient to explain what's going on
there.

Number 2 has some decisions to be made:
- The first decision was to pass a handle to a frozen chunk of memory down to
  all child processes, rather than passing a handle to an actual file. There's
  two reasons for this: 1) since we want to compress the underlying file on
  disk, giving that file to child processes would mean they have to decompress
  it themselves, eating CPU time. 2) since they would have to decompress it
  themselves, they would have to allocate the memory for the decompressed
  buffers, meaning they cannot all simply share one big decompressed buffer.

  - The drawback of this decision is that we have to load and decompress the
    buffer up front, before we spawn any child processes. We attempt to
    mitigate this by keeping track of all the entries that child processes
    access, and only including those in the frozen decompressed shared buffer.

  - We base our implementation of this approach off of the shared preferences
    implementation. Hopefully I got all of the pieces to fit together
    correctly. They seem to work in local testing and on try, but I think
    they require a set of experienced eyes looking carefully at them.

- Another decision was whether to send the handles to the buffers over IPC or
  via command line. We went with the command line approach, because the startup
  cache would need to be accessed very early on in order to ensure we do not
  read from any omnijars, and we could not make that work via IPC.

  - Unfortunately this means adding another hard-coded FD, similar to
    kPrefMapFileDescriptor. It seems like at the very least we need to rope all
    of these together into one place, but I think that should be filed as a
    follow-up?

Lastly, because this patch is a bit of a monster to review - first, thank you
for looking at it, and second, the reason we're invested in this is because we
saw a >10% improvement in cold startup times on reference hardware, with a p
value less than 0.01. It's still not abundantly clear how reference hardware
numbers translate to numbers on release, and they certainly don't translate
well to Nightly numbers, but it's enough to convince me that it's worth some
effort.

Depends on D78584

Differential Revision: https://phabricator.services.mozilla.com/D77635
2020-07-07 04:35:08 +00:00
Doug Thayer 7dcd89a08d Bug 1627075 - Init StartupCache before Omnijar r=froydnj
We need to be able to init StartupCache before the Omnijar in order to cache
all of the Omnijar contents we access. This patch implements that.

Depends on D77632

Differential Revision: https://phabricator.services.mozilla.com/D77633
2020-07-07 04:34:07 +00:00
Doug Thayer 5b755bb7f3 Bug 1627075 - Build Omnijar file list from startup cache r=froydnj
We would like to be able to defer opening the omnijar files until after startup
if the StartupCache has already been populated. Opening the omnijar files takes
a nontrivial time, at least on Windows, and almost everything in the omnijar
should be fairly compressible, and thus makes sense to live in the StartupCache.
See the last patch in this series for a little more discussion on numbers, but
tl;dr: we saw a 12% improvement in time to about:home being finished on reference
hardware with these changes together with the changes from the descendant patches.

Differential Revision: https://phabricator.services.mozilla.com/D77632
2020-07-07 04:33:49 +00:00
Noemi Erli b13f2bcb47 Backed out 7 changesets (bug 1627075) for causing @nsZipArchive crashes CLOSED TREE
Backed out changeset 9705b2759d45 (bug 1627075)
Backed out changeset 699212a443c3 (bug 1627075)
Backed out changeset 7ae4df10749c (bug 1627075)
Backed out changeset ece9a4223349 (bug 1627075)
Backed out changeset 6c4eedaa0d04 (bug 1627075)
Backed out changeset f5106898239f (bug 1627075)
Backed out changeset b6029c7c0016 (bug 1627075)
2020-07-02 14:05:53 +03:00
Sebastian Hengst f85d3e690a Bug 1627075 - Remove unused 'start' variable from startupcache/StartupCache.cpp. a=Bb-fix CLOSED TREE 2020-07-02 11:29:15 +02:00
Doug Thayer a493fefece Bug 1627075 - OMT and OMP StartupCache access r=froydnj
The overall goal of this patch is to make the StartupCache accessible anywhere.
There's two main pieces to that equation:

1. Allowing it to be accessed off main thread, which means modifying the
   mutex usage to ensure that all data accessed from non-main threads is
   protected.
2. Allowing it to be accessed out of the chrome process, which means passing
   a handle to a shared cache buffer down to child processes.

Number 1 is somewhat fiddly, but it's all generally straightforward work. I'll
hope that the comments and the code are sufficient to explain what's going on
there.

Number 2 has some decisions to be made:
- The first decision was to pass a handle to a frozen chunk of memory down to
  all child processes, rather than passing a handle to an actual file. There's
  two reasons for this: 1) since we want to compress the underlying file on
  disk, giving that file to child processes would mean they have to decompress
  it themselves, eating CPU time. 2) since they would have to decompress it
  themselves, they would have to allocate the memory for the decompressed
  buffers, meaning they cannot all simply share one big decompressed buffer.

  - The drawback of this decision is that we have to load and decompress the
    buffer up front, before we spawn any child processes. We attempt to
    mitigate this by keeping track of all the entries that child processes
    access, and only including those in the frozen decompressed shared buffer.

  - We base our implementation of this approach off of the shared preferences
    implementation. Hopefully I got all of the pieces to fit together
    correctly. They seem to work in local testing and on try, but I think
    they require a set of experienced eyes looking carefully at them.

- Another decision was whether to send the handles to the buffers over IPC or
  via command line. We went with the command line approach, because the startup
  cache would need to be accessed very early on in order to ensure we do not
  read from any omnijars, and we could not make that work via IPC.

  - Unfortunately this means adding another hard-coded FD, similar to
    kPrefMapFileDescriptor. It seems like at the very least we need to rope all
    of these together into one place, but I think that should be filed as a
    follow-up?

Lastly, because this patch is a bit of a monster to review - first, thank you
for looking at it, and second, the reason we're invested in this is because we
saw a >10% improvement in cold startup times on reference hardware, with a p
value less than 0.01. It's still not abundantly clear how reference hardware
numbers translate to numbers on release, and they certainly don't translate
well to Nightly numbers, but it's enough to convince me that it's worth some
effort.

Depends on D78584

Differential Revision: https://phabricator.services.mozilla.com/D77635
2020-07-02 03:39:46 +00:00
Doug Thayer a336fc378c Bug 1627075 - Init StartupCache before Omnijar r=froydnj
We need to be able to init StartupCache before the Omnijar in order to cache
all of the Omnijar contents we access. This patch implements that.

Depends on D77632

Differential Revision: https://phabricator.services.mozilla.com/D77633
2020-07-02 02:49:49 +00:00
Doug Thayer 10f4af3f14 Bug 1627075 - Build Omnijar file list from startup cache r=froydnj
We would like to be able to defer opening the omnijar files until after startup
if the StartupCache has already been populated. Opening the omnijar files takes
a nontrivial time, at least on Windows, and almost everything in the omnijar
should be fairly compressible, and thus makes sense to live in the StartupCache.
See the last patch in this series for a little more discussion on numbers, but
tl;dr: we saw a 12% improvement in time to about:home being finished on reference
hardware with these changes together with the changes from the descendant patches.

Differential Revision: https://phabricator.services.mozilla.com/D77632
2020-07-02 02:49:31 +00:00
Mihai Alexandru Michis bab20702a8 Backed out 6 changesets (bug 1627075) for causing failures regarding startupcache.
CLOSED TREE

Backed out changeset cf23b456ba12 (bug 1627075)
Backed out changeset b07887474f51 (bug 1627075)
Backed out changeset 65c0e6790a33 (bug 1627075)
Backed out changeset 6cd31f17a931 (bug 1627075)
Backed out changeset 0f0d1bd2a8ac (bug 1627075)
Backed out changeset 763a5a7b43a0 (bug 1627075)
2020-07-01 22:16:28 +03:00
Doug Thayer 42ac8f4294 Bug 1627075 - OMT and OMP StartupCache access r=froydnj
The overall goal of this patch is to make the StartupCache accessible anywhere.
There's two main pieces to that equation:

1. Allowing it to be accessed off main thread, which means modifying the
   mutex usage to ensure that all data accessed from non-main threads is
   protected.
2. Allowing it to be accessed out of the chrome process, which means passing
   a handle to a shared cache buffer down to child processes.

Number 1 is somewhat fiddly, but it's all generally straightforward work. I'll
hope that the comments and the code are sufficient to explain what's going on
there.

Number 2 has some decisions to be made:
- The first decision was to pass a handle to a frozen chunk of memory down to
  all child processes, rather than passing a handle to an actual file. There's
  two reasons for this: 1) since we want to compress the underlying file on
  disk, giving that file to child processes would mean they have to decompress
  it themselves, eating CPU time. 2) since they would have to decompress it
  themselves, they would have to allocate the memory for the decompressed
  buffers, meaning they cannot all simply share one big decompressed buffer.

  - The drawback of this decision is that we have to load and decompress the
    buffer up front, before we spawn any child processes. We attempt to
    mitigate this by keeping track of all the entries that child processes
    access, and only including those in the frozen decompressed shared buffer.

  - We base our implementation of this approach off of the shared preferences
    implementation. Hopefully I got all of the pieces to fit together
    correctly. They seem to work in local testing and on try, but I think
    they require a set of experienced eyes looking carefully at them.

- Another decision was whether to send the handles to the buffers over IPC or
  via command line. We went with the command line approach, because the startup
  cache would need to be accessed very early on in order to ensure we do not
  read from any omnijars, and we could not make that work via IPC.

  - Unfortunately this means adding another hard-coded FD, similar to
    kPrefMapFileDescriptor. It seems like at the very least we need to rope all
    of these together into one place, but I think that should be filed as a
    follow-up?

Lastly, because this patch is a bit of a monster to review - first, thank you
for looking at it, and second, the reason we're invested in this is because we
saw a >10% improvement in cold startup times on reference hardware, with a p
value less than 0.01. It's still not abundantly clear how reference hardware
numbers translate to numbers on release, and they certainly don't translate
well to Nightly numbers, but it's enough to convince me that it's worth some
effort.

Depends on D78584

Differential Revision: https://phabricator.services.mozilla.com/D77635
2020-07-01 17:55:38 +00:00
Doug Thayer e1810ad0e8 Bug 1627075 - Init StartupCache before Omnijar r=froydnj
We need to be able to init StartupCache before the Omnijar in order to cache
all of the Omnijar contents we access. This patch implements that.

Depends on D77632

Differential Revision: https://phabricator.services.mozilla.com/D77633
2020-07-01 17:07:29 +00:00
Doug Thayer b19fd0dabb Bug 1627075 - Build Omnijar file list from startup cache r=froydnj
We would like to be able to defer opening the omnijar files until after startup
if the StartupCache has already been populated. Opening the omnijar files takes
a nontrivial time, at least on Windows, and almost everything in the omnijar
should be fairly compressible, and thus makes sense to live in the StartupCache.
See the last patch in this series for a little more discussion on numbers, but
tl;dr: we saw a 12% improvement in time to about:home being finished on reference
hardware with these changes together with the changes from the descendant patches.

Differential Revision: https://phabricator.services.mozilla.com/D77632
2020-07-01 17:07:17 +00:00
Simon Giesecke 9364b353d4 Bug 1648010 - Remove NS_NAMED_LITERAL_CSTRING and NS_NAMED_LITERAL_STRING macros. r=froydnj
Differential Revision: https://phabricator.services.mozilla.com/D80631
2020-07-01 08:42:31 +00:00
Simon Giesecke cd8b8939b9 Bug 1648010 - Replace uses of NS_LITERAL_STRING/NS_LITERAL_CSTRING macros by _ns literals. r=geckoview-reviewers,jgilbert,agi,hsivonen,froydnj
Differential Revision: https://phabricator.services.mozilla.com/D80860
2020-07-01 08:29:29 +00:00
Gijs Kruitbosch 380e9f04f1 Bug 1631991 - stop wasting minutes of infra time waiting for the startup cache in gtests, r=dthayer
Differential Revision: https://phabricator.services.mozilla.com/D80765
2020-06-24 16:00:32 +00:00
Gijs Kruitbosch a9867db924 Bug 1647378 - stop using mStartupWriteInitiated in the gtest for startupcache, r=dthayer
Differential Revision: https://phabricator.services.mozilla.com/D80764
2020-06-24 02:38:28 +00:00
Nicholas Nethercote a8f5f49b8a Bug 1645982 - Rename some service getters in `Services.py` to better match the types. r=froydnj
Differential Revision: https://phabricator.services.mozilla.com/D79791
2020-06-16 23:32:21 +00:00
Doug Thayer 285706e532 Bug 1636244 - Relax the StartupCache telemetry test r=froydnj
Differential Revision: https://phabricator.services.mozilla.com/D79899
2020-06-16 21:34:28 +00:00
Gijs Kruitbosch af2df0c8a7 Bug 1614795 - use the background task queue for startupcache writes, r=dthayer,decoder
Prior to this patch, the startupcache created its own mWriteThread off which it
wrote to disk. It's initialized by MaybeSpawnWriteThread, which got called
at shutdown, to do the shutdown write if there was any reason to do so, and
from a timer that is re-initialized after every addition to the startup cache,
to run 60s after the last change to the cache.

It then joined that write thread on the main thread (in other words, blocks
on that off-main-thread write completing from the main thread) when:
- xpcom-shutdown fired
- the startupcache itself gets destroyed
- someone calls any of:
  * HasEntry
  * GetBuffer
  * PutBuffer
  * InvalidateCache

This patch removes the separate write thread, and instead dispatches a task to
the background task queue, indicating it can block. The task is started in
the same circumstances where we previously used to write (timer from the last
PutBuffer call, and shutdown if necessary).

To ensure it cannot be trying to use the data it writes out (mTable) from
the other thread while that data changes on the main thread, we use a mutex.
The task locks the mutex before starting, and unlocks when finished.
Enumerating the cases that we used to block on joining the thread:

In terms of application shutdown, we expect the background task queue to
either finish the write task, or fail to run it if it hasn't started it yet.
In the FastStartup case, we check if a write was necessary; if so, we
attempt to gain the lock without waiting. If we're successful, the write has
not yet started, and we instead run the write on the main thread. Otherwise,
we retry gaining the lock, blocking this time, thus guaranteeing the
off-the-main-thread write completes.

The task keeps a reference to the startupcache object, so it cannot be
destroyed while the task is pending.

Because the write does not modify `mTable`, and neither does `HasEntry`,
we do not need to do anything there.

In the `GetBuffer` case, we do not modify the table unless we have to read
the entry off disk (memmapped into `mCacheData`). This can only happen if
`mCacheData.initialized()` returns true, and we specifically call
`mCacheData.reset()` before firing off the write task to avoid this.
`mCacheData` is only re-initialized if someone calls `LoadArchive()`,
which can only happen from `Init()` (which is guaranteed not to run
again because this is a singleton), or `InvalidateCache()`, where we lock
the mutex (see below). So this is safe - but we assert on the lock to try
and avoid people breaking this chain of assumptions in the future.

When `PutBuffer` is called, we try to lock the mutex - but if locking fails
(ie the background thread is writing), we simply fail to store the entry
in the startupcache. In practice, this should be rare - it'd happen if
new calls to PutBuffer happen while writing during shutdown (when really,
we don't care) or when it's been 60 seconds since the last PutBuffer so
we started writing the startupcache.

When InvalidateCache is called, we lock the mutex - we shouldn't try to
write while invalidating, or invalidate while writing. This may be slow,
but in practice nothing should call `InvalidateCache` except developer
restarts or the `-purgecaches` commandline flag, so it shouldn't
matter a great deal.

Differential Revision: https://phabricator.services.mozilla.com/D70413

--HG--
extra : moz-landing-system : lando
2020-04-15 20:43:44 +00:00
Tooru Fujisawa 8c0843c04c Bug 1432749 - Part 1: Add DefaultHasher for nsTString. r=smaug
Differential Revision: https://phabricator.services.mozilla.com/D67836

--HG--
extra : moz-landing-system : lando
2020-04-14 06:00:05 +00:00
Narcis Beleuzu 1783003a34 Backed out 3 changesets (bug 1624150, bug 1432749) for wpt failures at open-features-tokenization-top-left.html. CLOSED TREE
Backed out changeset ff061c8d9da9 (bug 1624150)
Backed out changeset 7e96d4acf317 (bug 1432749)
Backed out changeset a95f77732a1c (bug 1432749)
2020-04-13 22:51:48 +03:00
Tooru Fujisawa c2e8f7014c Bug 1432749 - Part 1: Add DefaultHasher for nsTString. r=smaug
Differential Revision: https://phabricator.services.mozilla.com/D67836

--HG--
extra : moz-landing-system : lando
2020-04-13 15:42:06 +00:00
Doug Thayer c5939cab4c Bug 1595596 - Use MMAP_FAULT_HANDLER in StartupCache r=aklotz
Please double check that I am using this correctly. I believe we are
seeing the crash in the linked bug because we are not handling hardware
faults when reading from the memory mapped file. This patch just wraps
all accesses in the MMAP_FAULT_HANDLER_ macros.

Depends on D53042

Differential Revision: https://phabricator.services.mozilla.com/D53043

--HG--
rename : modules/libjar/MmapFaultHandler.cpp => mozglue/misc/MmapFaultHandler.cpp
rename : modules/libjar/MmapFaultHandler.h => mozglue/misc/MmapFaultHandler.h
extra : moz-landing-system : lando
2020-04-10 21:16:15 +00:00