When a local or remote item changed, we'd potentially scan three tables
(with an expensive `LEFT JOIN`!) to check if anything changed...then
scan the same tables again to build the local and remote trees. This
check was originally meant to avoid unnecessary merges. However, the
bottleneck isn't merging now; it's reading from the database.
Since the merger has been rewritten in Rust, is synchronous, doesn't
keep a transaction open for the entire merge (see the
`total_sync_changes` check), and only emits ops for items that actually
changed, it's more efficient to build and merge optimistically, and
bail before applying if nothing changed.
This commit also moves `validateLocalRoots` into Rust.
Differential Revision: https://phabricator.services.mozilla.com/D41690
--HG--
extra : moz-landing-system : lando
This commit introduces a new `Watchdog` class that signals an abort,
either after a delay or at shutdown, and wires up the buffered engine
to use it.
Differential Revision: https://phabricator.services.mozilla.com/D41311
--HG--
extra : moz-landing-system : lando
This commit reduces the number of database writes and table scans
needed to merge synced bookmarks.
* Remove `fetchNew{Local, Remote}Contents`. Fetching the tree already
scans the table, so we can piggyback on it to fetch content info for
deduping.
* Store completion ops in temp tables to only update changed parts of
the local tree, and remove the `mergeStates` table and views.
* Replace the `itemsToMerge` view with an indexed `itemsToApply` temp
table.
* Replace the `updateGuidsAndSyncFlags` trigger with a `changeGuidOps`
table and a `changeGuids` trigger.
* Replace the `updateLocalItems` trigger with an `apply_remote_items`
function to bulk upsert new and updated items.
* Replace the `structureToMerge` view with an
`applyNewLocalStructureOps` table that holds parents and positions
for moved items, and an `applyNewLocalStructure` trigger to update
them.
* Remove tombstones for revived items, update change counters, and flag
mirror items as merged directly in `update_local_items_in_places`,
instead of indirecting through temp tables.
* Don't mark items flagged for reupload as merged, since we'll write
them back to the mirror after upload.
* Use a scalar subquery instead of a join in the `localTags` view to
look up the tags root ID.
* Replace `relatedIdsToReupload` with a `Store::prepare` method that
flags all bookmarks with keyword-URL mismatches for reupload.
* Trigger frecency updates for origins once, not for every item.
* Remove two extra scans on `itemsAdded` and `itemsChanged` when
recording observer notifications for changed keywords.
* Notify all `bookmark-added` listeners in a single batch.
This also fixes some edge cases:
* Update root positions correctly after deleting a non-syncable root
or item.
* Keyword-URL mismatches may reupload more items than before, but now
ensure that all bookmarks with the same URL have the same keyword.
* Only set items with deduped GUIDs to `SYNC_STATUS_NORMAL` after
merging.
* Bump the last modified time for modified items only.
Differential Revision: https://phabricator.services.mozilla.com/D39889
--HG--
extra : moz-landing-system : lando
Specifically, a "control pref" for a pref must already exist locally, or
a new preference, `services.sync.prefs.dangerously_allow_arbitrary` must
be set to true.
This also removes a few preferences from the set we sync by default based
due to potential harm which can be caused syncing inappropriate values.
Differential Revision: https://phabricator.services.mozilla.com/D29775
--HG--
extra : moz-landing-system : lando
This is a test-only change to make the existing synced prefs tests more
explicit about values being checked and moves a couple of prefs around to make
things a bit clearer, which should make the test changes in part 2 clearer.
Differential Revision: https://phabricator.services.mozilla.com/D29774
--HG--
extra : moz-landing-system : lando
Before this commit, we used event telemetry to record timing and counts
for the different steps of a bookmark merge (fetching the local tree,
new local contents, remote tree, and new remote contents; merging;
applying; fetching outgoing records; and notifying observers).
This has several limitations. We need to store all numbers as strings,
include a "flow ID" to tag events from the same merge, and collect
failure reasons twice. We also can't correlate these events to the
existing engine telemetry, meaning we can't see other stats for that
engine, or for the entire sync. Finally, we need to run separate
queries on these events for analysis, instead of extending our
existing engine dashboards.
This approach also feels like an abuse of event telemetry, so this
commit adds a "steps" field for each engine in the Sync ping. Each step
has a name, time taken, and additional named counts, like the number of
items in the tree, or merged structure stats.
Currently, only the `buffered-bookmarks` engine records these steps.
Differential Revision: https://phabricator.services.mozilla.com/D33411
--HG--
extra : moz-landing-system : lando
This commit introduces a `mozISyncedBookmarksMirrorProgressListener`
interface for capturing telemetry and updating shutdown blocker state
after each step of the merge, instead of waiting until the end. This
also means we can also record events for interrupted and failed merges,
and pass validation data through to the Sync ping.
Shutdown hang crash reports now have a `steps` field, indicating the
sequence of completed steps and when they were recorded. If the last
step in the hang report is `fetchLocalTree`, we know the merger is
blocked on `fetchNewLocalContents`. If the last step is
`fetchNewLocalContents`, the merger is stuck at `fetchRemoteTree`,
since that's the next step after `fetchNewLocalContents`.
This commit also implements `Driver::record_telemetry_event` to
dispatch progress callback runnables to the main thread.
Differential Revision: https://phabricator.services.mozilla.com/D31950
--HG--
extra : moz-landing-system : lando
Edouard Oger
Mark Hammond
Bogdan Tara
ffxbld
Sylvestre Ledru
Ciure Andrei
Moritz Birghan
Mark Banner
Lina Cambridge
Nicholas Nethercote
Jared Wein
Luca Greco
Narcis Beleuzu
Jonathan Kingston
Victor Porof
Marco Bonardo
Boris Zbarsky
Sebastian Hengst
Matthew Noorenberghe
Coroiu Cristina
shindli
Cosmin Sabou
Michael Kaply
Gurzau Raul
Jonas Allmann