Instead of using the list of FxA devices from the Sync clients engine,
we now fetch the list of Send Tab devices from FxA. This works like
this:
* `FxAccountsDevice#getDeviceList` has been split up into
`recentDeviceList` and `refreshDeviceList`.
* `recentDeviceList` synchronously returns the last fetched list, so
that consumers like Send Tab can use it right away.
* `refreshDeviceList` is asynchronous, and refreshes the last fetched
list. Refreshes are limited to once every minute by default, matching
the minimum sync interval (Send Tab passes the `ignoreCached` option
to override the limit if the user clicks the "refresh" button).
Concurrent calls to `refreshDeviceList` are also serialized, to
ensure the list is only fetched once.
* The list is flagged as stale when a device is connected or
disconnected. It's still kept around, but the next call to
`refreshDeviceList` will fetch a new list from the server.
* The Send Tab UI refreshes FxA devices in the background. Matching FxA
devices to Sync client records is best effort; we don't do it if Sync
isn't configured or hasn't run yet. This only impacts the fallback
case if the target doesn't support FxA commands.
Differential Revision: https://phabricator.services.mozilla.com/D47521
--HG--
extra : moz-landing-system : lando
Most of these tests have been disabled for a long time; they run well
in the current test environment.
Differential Revision: https://phabricator.services.mozilla.com/D46642
--HG--
extra : moz-landing-system : lando
When a local or remote item changed, we'd potentially scan three tables
(with an expensive `LEFT JOIN`!) to check if anything changed...then
scan the same tables again to build the local and remote trees. This
check was originally meant to avoid unnecessary merges. However, the
bottleneck isn't merging now; it's reading from the database.
Since the merger has been rewritten in Rust, is synchronous, doesn't
keep a transaction open for the entire merge (see the
`total_sync_changes` check), and only emits ops for items that actually
changed, it's more efficient to build and merge optimistically, and
bail before applying if nothing changed.
This commit also moves `validateLocalRoots` into Rust.
Differential Revision: https://phabricator.services.mozilla.com/D41690
--HG--
extra : moz-landing-system : lando
This commit introduces a new `Watchdog` class that signals an abort,
either after a delay or at shutdown, and wires up the buffered engine
to use it.
Differential Revision: https://phabricator.services.mozilla.com/D41311
--HG--
extra : moz-landing-system : lando
This commit reduces the number of database writes and table scans
needed to merge synced bookmarks.
* Remove `fetchNew{Local, Remote}Contents`. Fetching the tree already
scans the table, so we can piggyback on it to fetch content info for
deduping.
* Store completion ops in temp tables to only update changed parts of
the local tree, and remove the `mergeStates` table and views.
* Replace the `itemsToMerge` view with an indexed `itemsToApply` temp
table.
* Replace the `updateGuidsAndSyncFlags` trigger with a `changeGuidOps`
table and a `changeGuids` trigger.
* Replace the `updateLocalItems` trigger with an `apply_remote_items`
function to bulk upsert new and updated items.
* Replace the `structureToMerge` view with an
`applyNewLocalStructureOps` table that holds parents and positions
for moved items, and an `applyNewLocalStructure` trigger to update
them.
* Remove tombstones for revived items, update change counters, and flag
mirror items as merged directly in `update_local_items_in_places`,
instead of indirecting through temp tables.
* Don't mark items flagged for reupload as merged, since we'll write
them back to the mirror after upload.
* Use a scalar subquery instead of a join in the `localTags` view to
look up the tags root ID.
* Replace `relatedIdsToReupload` with a `Store::prepare` method that
flags all bookmarks with keyword-URL mismatches for reupload.
* Trigger frecency updates for origins once, not for every item.
* Remove two extra scans on `itemsAdded` and `itemsChanged` when
recording observer notifications for changed keywords.
* Notify all `bookmark-added` listeners in a single batch.
This also fixes some edge cases:
* Update root positions correctly after deleting a non-syncable root
or item.
* Keyword-URL mismatches may reupload more items than before, but now
ensure that all bookmarks with the same URL have the same keyword.
* Only set items with deduped GUIDs to `SYNC_STATUS_NORMAL` after
merging.
* Bump the last modified time for modified items only.
Differential Revision: https://phabricator.services.mozilla.com/D39889
--HG--
extra : moz-landing-system : lando
Specifically, a "control pref" for a pref must already exist locally, or
a new preference, `services.sync.prefs.dangerously_allow_arbitrary` must
be set to true.
This also removes a few preferences from the set we sync by default based
due to potential harm which can be caused syncing inappropriate values.
Differential Revision: https://phabricator.services.mozilla.com/D29775
--HG--
extra : moz-landing-system : lando
This is a test-only change to make the existing synced prefs tests more
explicit about values being checked and moves a couple of prefs around to make
things a bit clearer, which should make the test changes in part 2 clearer.
Differential Revision: https://phabricator.services.mozilla.com/D29774
--HG--
extra : moz-landing-system : lando
Before this commit, we used event telemetry to record timing and counts
for the different steps of a bookmark merge (fetching the local tree,
new local contents, remote tree, and new remote contents; merging;
applying; fetching outgoing records; and notifying observers).
This has several limitations. We need to store all numbers as strings,
include a "flow ID" to tag events from the same merge, and collect
failure reasons twice. We also can't correlate these events to the
existing engine telemetry, meaning we can't see other stats for that
engine, or for the entire sync. Finally, we need to run separate
queries on these events for analysis, instead of extending our
existing engine dashboards.
This approach also feels like an abuse of event telemetry, so this
commit adds a "steps" field for each engine in the Sync ping. Each step
has a name, time taken, and additional named counts, like the number of
items in the tree, or merged structure stats.
Currently, only the `buffered-bookmarks` engine records these steps.
Differential Revision: https://phabricator.services.mozilla.com/D33411
--HG--
extra : moz-landing-system : lando
The number of Sync users on these channels is low compared to Release,
so we can do this without a gradual rollout. This also lets more users
test the new bookmark sync engine without manually flipping the pref.
Differential Revision: https://phabricator.services.mozilla.com/D29859
--HG--
extra : moz-landing-system : lando
Mark Hammond
Moritz Birghan
rfkelly
Coroiu Cristina
Thom Chiovoloni
lougenia
Cosmin Sabou
Lina Cambridge
Edouard Oger
Geoff Brown
Bogdan Tara
Ciure Andrei
Mark Banner
Luca Greco
Victor Porof
Marco Bonardo
Sebastian Hengst
Matthew Noorenberghe
Jonas Allmann
Boris Zbarsky