To get this fix: https://github.com/mozilla/application-services/pull/3235
Updated as follows:
```
sed -i 's/e8d7530319fa6c20d9de78d031c9398630eca3cd/61dcc364ac0d6d0816ab88a494bbf20d824b009b/g' services/fxaccounts/rust-bridge/firefox-accounts-bridge/Cargo.toml services/sync/golden_gate/Cargo.toml toolkit/components/extensions/storage/webext_storage_bridge/Cargo.toml toolkit/components/glean/Cargo.toml toolkit/library/rust/shared/Cargo.toml
./mach vendor rust
```
Verified by running the new regression test that I added in the bug:
```
./mach test toolkit/components/extensions/test/xpcshell/test_ext_storage_{local,sync,sync_kinto}.js
```
Differential Revision: https://phabricator.services.mozilla.com/D79628
Specifically:
> Log warning: The log 'Services.Common.RESTRequest' is configured to use the preference 'services.common.log.logger.rest.request' - you must adjust the level by setting this preference, not by using the level setter
This is because the hawkrequest module attempts to explicitly set the level
from the preference value, but these days Log.jsm manages that itself.
Also skips obtaining the log itself until it's needed, as it almost never
actually is.
Differential Revision: https://phabricator.services.mozilla.com/D79633
This new method fetches pending synced changes from the extension
storage store, and passes them to `storage.onChanged` listeners.
This allows extensions that listen for these events to know when a
sync happened, which Kinto supported as well.
To guard against misuse, this method is implemented on a separate
`mozISyncedExtensionStorageArea` interface. To avoid multiple
inheritance (if `mozI{Synced, Configurable}ExtensionStorageArea` both
inherited from `mozIExtensionStorageArea`, which base method is called
when?), both of these internal-ish interfaces now inherit from
`nsISupports` instead.
Finally, because `fetchPendingSyncChanges` can return changes for
multiple extensions, `mozIExtensionStorageListener.onChanged` now takes
the affected extension ID as its first argument.
Differential Revision: https://phabricator.services.mozilla.com/D76976
This new method fetches pending synced changes from the extension
storage store, and passes them to `storage.onChanged` listeners.
This allows extensions that listen for these events to know when a
sync happened, which Kinto supported as well.
To guard against misuse, this method is implemented on a separate
`mozISyncedExtensionStorageArea` interface. To avoid multiple
inheritance (if `mozI{Synced, Configurable}ExtensionStorageArea` both
inherited from `mozIExtensionStorageArea`, which base method is called
when?), both of these internal-ish interfaces now inherit from
`nsISupports` instead.
Finally, because `fetchPendingSyncChanges` can return changes for
multiple extensions, `mozIExtensionStorageListener.onChanged` now takes
the affected extension ID as its first argument.
Differential Revision: https://phabricator.services.mozilla.com/D76976
This commit does the following:
- Feature / Optimization: Check the dump before the cache, instead of
the reverse. The dump is expected to match the requested attachment in
the common case, and checking it first helps with ensuring that the
expected (packaged dump) is used when available.
- Optimization: Defer reading the cached attachment until it's needed.
- Refactor / Feature: Treat a missing `.meta.json` file as a sign that
the attachment dump does not exist, rather than an error.
Previously, if an attachment cannot be downloaded from the network,
that error would be replaced with a generic `DownloadError` (from the
missing `.meta.json` file). This is mostly relevant for telemetry.
- Refactor / Maintainability: Create helper to manage lazy access to the
record and attachment, to ensure that the record and attachment is
only read on demand, and at most once.
- Refactor / Readability: Move the common return value generation logic
to the helper as `getResult`, to avoid the verbose duplication of the
logic. Now the return value fits in one line instead of 5-6 lines.
- Fix test: Rename filename-of-dump.meta.json and fix test expectation
to ensure that the test checks the absence of the file content,
rather than the absence of the meta data file.
Differential Revision: https://phabricator.services.mozilla.com/D76962
In practice, the cache of the attachment downloader can become corrupt
and unusable when IndexedDB breaks. The implementation correctly handled
this case, but there were no tests that verified that it did.
This patch adds test coverage for the scenario of a broken cache,
to ensure that the implementation continues to behave in a sane way.
Differential Revision: https://phabricator.services.mozilla.com/D76961
The crypto API does most of its work on the background thread. There is
no benefit in posting the buffer to a worker thread.
Differential Revision: https://phabricator.services.mozilla.com/D76960
This implements reading the list from remote settings. We only read it at startup if necessary, or on add-on installation.
We do not check for updates - if something is removed, we'll wait until next startup before processing it.
Also adds lots of tests for canOverride as this seems a critical part to get right.
Differential Revision: https://phabricator.services.mozilla.com/D76473
By default, toggling the add-ons engine also toggles extension storage.
However, it's also possible to enable extension storage without add-ons
by setting an override pref. If the override pref is set, we'll treat
extension storage as an engine that can be toggled independently. If
it's not set, we'll ignore attempts to toggle the engine separately.
Differential Revision: https://phabricator.services.mozilla.com/D76635
This implements reading the list from remote settings. We only read it at startup if necessary, or on add-on installation.
We do not check for updates - if something is removed, we'll wait until next startup before processing it.
Also adds lots of tests for canOverride as this seems a critical part to get right.
Differential Revision: https://phabricator.services.mozilla.com/D76473
Alternative engines are registered using the lowercased version of
the keys in the modules object. But `extension-storage` is hyphenated,
so we need to use `Extension-Storage` (not `ExtensionStorage`) as the
key name, to match the name of the engine and its collection.
Without the hyphen, we'll register the alternative as
`extensionstorage`, so it'll never be used because everything else
expects the engine to be called `extension-storage`.
Differential Revision: https://phabricator.services.mozilla.com/D76355
This commit fixes two issues: moves `trackRemainingChanges` to the
engine, since that's where it defined (not on the store), and skips
setting `modified` if all records have been successfully uploaded.
Differential Revision: https://phabricator.services.mozilla.com/D76270
This matches how the `Dispatch(already_AddRefed<nsIRunnable>)`
overloads work in C++: `Dispatch` takes ownership of the runnable, and
leaks it if dispatch fails—because the thread manager is shutting down,
for instance. This avoids a race where a runnable can be released on
either the owning or target thread.
Rust doesn't allow arbitrary `Self` types yet (see
rust-lang/rust#44874), so we need to change `dispatch` and
`dispatch_with_options` to be associated methods.
Differential Revision: https://phabricator.services.mozilla.com/D75858
The tracker base class currently does two things: bump the score in
response to observer notifications, and store a list of changed IDs.
The bookmarks, form autofill, and now bridged trackers need to hack
around this to opt out of persistence, since they handle change
tracking in the storage layer.
This commit keeps the score logic in `Tracker`, but moves all the
persistence code into an intermediate `LegacyTracker` class, and
changes all engines that need persistence to inherit from it.
`ignoreAll` is more interesting. We want new-style stores to emit
observer notifications with change sources, so that the tracker knows
to ignore changes made by Sync. Ignoring all observer notifications
during a sync is a blunter version of this. But, not every new store
supports change sources, so we reimplement `ignoreAll` manually for
ones that don't.
Differential Revision: https://phabricator.services.mozilla.com/D74374
To use a single transaction for `importJSONDump`, this commit:
- changes IDBHelpers' `executeIDB` method to take either a string or array
pointing to `objectStore`s that the caller wants to use.
- uses that from RemoteSettingsWorker to start a single transaction using both
the `records` and the `timestamps` store
- updates `bulkOperationHelper` to take an optional `completion` callback, in
addition to the rejection callback, to be called when all the bulk
operations are complete
- uses that optional argument from RemoteSettingsWorker's `importDumpIDB`
(the actual implementation of IDB access from `importJSONDump`) to first
bulk-import the actual records, and then update the timestamp stored for
that remote settings collection.
Then to abort that single transaction, this commit:
- stores pending transactions in a set, similar to what Database.jsm already
does, and removes items from that set when the `promise` from `executeIDB`
either resolves or rejects.
- adds a `prepareShutdown` action on the RemoteSettingsWorker's `Agent` class,
to be called by the jsm side of the worker manager when shutdown happens.
When called, it iterates over the pending transactions and aborts all of
them.
This also sets a `gShutdown` flag.
- ensures that where code `await`s in the middle of an operation, it stops
(throws) immediately if `gShutdown` has been set.
- adds a test to test_shutdown_handling.js to verify that this mechanism now
stops pending import tasks in the worker.
Finally, as a driveby, fixes an oversight in test_remote_settings_worker.js
where the second `.get()` call wasn't actually testing whether the
`importJSONDump` call in the worker had succeeded, because if the collection
was empty it would do the import itself, which I realized when I used similar
code in the shutdown test...
Differential Revision: https://phabricator.services.mozilla.com/D74315
Raw Cr.ERROR don't get stack information, same as throwing JS literals instead
of `new Error()`s.
This was done automatically with a new eslint rule that will be introduced in
the next commit. One instance of a raw Cr.ERROR was not replaced since it is
used in a test that specifically checks the preservation of raw Cr values in
XPCJS. The rule will be disabled for that instance.
Differential Revision: https://phabricator.services.mozilla.com/D28073
This commit adds syncing support to the `StorageSyncArea` class, via
the Golden Gate library.
It also changes the `BridgedEngine` trait: `initialize` and `finalize`
haven't been useful in practice, since that's managed by the storage
service, and the `LazyStore` takes care of setting up the storage
connection on first use. But, what we do need is a way to signal a
sync is starting, so that the engine can set up temp tables. That's
handled by the new `sync_started`.
Finally, this commit changes `BridgedEngine::set_uploaded` to take a
`sync15_traits::Guid` instead of a `String`.
Differential Revision: https://phabricator.services.mozilla.com/D73415
This commit splits `CryptoWrapper` into a base `RawCryptoWrapper`
class, which only handles encryption and decryption without
parsing the cleartext's contents, and the existing `CryptoWrapper`
class, which works like before.
Our bridged engine subclasses `RawCryptoWrapper`, and
implements some methods to convert records to and from envelopes.
Envelopes are a concept we introduced in `sync15_traits` to pass
along metadata from the BSO wrapper (like the modified time from the
server, and ID, to ensure they match) in addition to the cleartext.
This lets us reuse `sync15_traits::Payload` to parse record payloads
in Rust, and avoids parsing the cleartext in JS, only to stringify it
again when we pass it to the bridged Rust engine.
Differential Revision: https://phabricator.services.mozilla.com/D73581
This commit adds a `mozIInterruptible` implementation to
`StorageSyncArea`, and changes `LazyStore` to get an a-s interrupt
handle and interrupt pending operations.
Differential Revision: https://phabricator.services.mozilla.com/D73414
This commit removes the `nsICancelable` return values from all
`mozIBridgedSyncEngine` methods, and replaces them with a
`mozIInterruptible` interface that can be implemented by store
classes that support interrupting.
The `nsICancelable` pattern was intended to make each operation
interruptible, without affecting the others. But we can't guarantee
that with SQLite, because it only has a way to interrupt all
running statements on a connection, not specific ones. Further,
this pattern doesn't match what we currently do in a-s, where we
create an internal "interrupt scope" for each operation, and hand
out an "interrupt handle" for interrupting all in-progress
operations.
Storage classes like `StorageSyncArea` can opt in to interruption
by implementing `mozIInterruptible`. It's a separate interface to
protect against accidental misuse: because it interrupts all
statements on the connection, it might lose writes if the current
operation is a `set`, for example. But it's useful for testing and
debugging, so we still expose it.
This commit also changes Golden Gate ferries to hold weak references to
the `BridgedEngine`, so that they don't block teardown.
Differential Revision: https://phabricator.services.mozilla.com/D73413
These support the new implementation of the addon blocklist (bug 1620621),
which is a more space-efficient way to represent the blocklist.
A comparison of file size was given in D73159. In short, 913KB for the
old JSON-based blocklist (addons.json), 64KB for the new one.
In practice, addons.json is 273KB compressed.
The new blocklist (addons-mlbf.bin) does not compress that well since it
is already an efficiently packed binary format.
Differential Revision: https://phabricator.services.mozilla.com/D73438
This test uses the Kinto client to access the RemoteSettings database.
However, due to the database version bump from D72416, the kinto client
is no longer compatible with the RemoteSettings database.
This will be fixed in bug 1634203.
Differential Revision: https://phabricator.services.mozilla.com/D73160
The MLBF (addons-mlbf.bin) itself is 64 KB.
Together with the metadata, this is 65 KB.
In contrast, the current JSON-based dump (addons.json) is 913 KB.
Differential Revision: https://phabricator.services.mozilla.com/D73159
With this piece, it is now possible for RemoteSettings clients to always
have a valid attachment.
This adds the client APIs that could support bug 1542177.
Differential Revision: https://phabricator.services.mozilla.com/D72417
The current RemoteSettings API has two methods for downloading:
- `download(record)` to download an attachment to disk, with a filename
given by the record. The file is read and returned before downloading.
A new download attempt overwrites the file, with no recovery mechanism
if the download fails.
- `downloadAsBytes(record)` to download an attachment in memory.
The `download` method does have a cache, but it is only useful for
reducing bandwidth usage, not for availability of data. Moreover, if its
associated record is removed from the collection, then callers do not
have a way to delete the file since its identifier (filename) originates
from the record.
The `downloadAsBytes` method does not have this file tracking issue
since it does not persist the data, but it forces callers to implement
their own attachment storing mechanism.
This commit adds the `useCache` option to the `download()` method to
enable callers to use an IndexedDB-based cache instead of the
filesystem. The following options unlock significant features:
- `fallbackToCache` - If the requested attachment is not available, the
last known attachment is returned.
- `fallbackToDump` - If the requested attachment is not available, nor
cached, then the attachment (dump) that is packaged with the client is
returned instead. This is implemented in the next commit (D72417).
The original record is cached along with the attachment, to allow
callers to use the file (identified by the given `attachmentId`) and
its metadata, even when the original record has been removed from the
collection.
This is particularly useful for scenarios where one wants to keep a file
(and metadata) up to date via RemoteSettings, without having to develop
a separate storage and synchronization mechanism.
The deprecation of the old behavior will be handled in bug 1634127.
Differential Revision: https://phabricator.services.mozilla.com/D72416
This test uses the Kinto client to access the RemoteSettings database.
However, due to the database version bump from D72416, the kinto client
is no longer compatible with the RemoteSettings database.
This will be fixed in bug 1634203.
Differential Revision: https://phabricator.services.mozilla.com/D73160
The MLBF (addons-mlbf.bin) itself is 64 KB.
Together with the metadata, this is 65 KB.
In contrast, the current JSON-based dump (addons.json) is 913 KB.
Differential Revision: https://phabricator.services.mozilla.com/D73159
With this piece, it is now possible for RemoteSettings clients to always
have a valid attachment.
This adds the client APIs that could support bug 1542177.
Differential Revision: https://phabricator.services.mozilla.com/D72417
The current RemoteSettings API has two methods for downloading:
- `download(record)` to download an attachment to disk, with a filename
given by the record. The file is read and returned before downloading.
A new download attempt overwrites the file, with no recovery mechanism
if the download fails.
- `downloadAsBytes(record)` to download an attachment in memory.
The `download` method does have a cache, but it is only useful for
reducing bandwidth usage, not for availability of data. Moreover, if its
associated record is removed from the collection, then callers do not
have a way to delete the file since its identifier (filename) originates
from the record.
The `downloadAsBytes` method does not have this file tracking issue
since it does not persist the data, but it forces callers to implement
their own attachment storing mechanism.
This commit adds the `useCache` option to the `download()` method to
enable callers to use an IndexedDB-based cache instead of the
filesystem. The following options unlock significant features:
- `fallbackToCache` - If the requested attachment is not available, the
last known attachment is returned.
- `fallbackToDump` - If the requested attachment is not available, nor
cached, then the attachment (dump) that is packaged with the client is
returned instead. This is implemented in the next commit (D72417).
The original record is cached along with the attachment, to allow
callers to use the file (identified by the given `attachmentId`) and
its metadata, even when the original record has been removed from the
collection.
This is particularly useful for scenarios where one wants to keep a file
(and metadata) up to date via RemoteSettings, without having to develop
a separate storage and synchronization mechanism.
The deprecation of the old behavior will be handled in bug 1634127.
Differential Revision: https://phabricator.services.mozilla.com/D72416
This adds abort handlers to Database.jsm and RemoteSettingsWorker.js's
indexedDB transactions, so we handle transaction aborts appropriately
and do not leave consumers waiting forever.
It also adds explicit error passing to places where we continue operating on a
store via a live transaction after control flow passes back to executeIDB,
because we use the `onsuccess` handler of earlier IDBRequests to run more
requests in the transaction.
In this case, in theory exceptions get handled by IndexedDB by invoking the
abort handler on the transaction (which we do not have right now...), but as a
belt-and-braces approach, it's probably better to do this explicitly.
Differential Revision: https://phabricator.services.mozilla.com/D71326
These two tests use the Kinto client to access the RemoteSettings
database. However, due to the database version bump from D72416,
the kinto client is no longer compatible with the RemoteSettings
database.
This will be fixed in bug 1634203.
Differential Revision: https://phabricator.services.mozilla.com/D73160
The MLBF (addons-mlbf.bin) itself is 64 KB.
Together with the metadata, this is 65 KB.
In contrast, the current JSON-based dump (addons.json) is 913 KB.
Differential Revision: https://phabricator.services.mozilla.com/D73159
With this piece, it is now possible for RemoteSettings clients to always
have a valid attachment.
This adds the client APIs that could support bug 1542177.
Differential Revision: https://phabricator.services.mozilla.com/D72417
The current RemoteSettings API has two methods for downloading:
- `download(record)` to download an attachment to disk, with a filename
given by the record. The file is read and returned before downloading.
A new download attempt overwrites the file, with no recovery mechanism
if the download fails.
- `downloadAsBytes(record)` to download an attachment in memory.
The `download` method does have a cache, but it is only useful for
reducing bandwidth usage, not for availability of data. Moreover, if its
associated record is removed from the collection, then callers do not
have a way to delete the file since its identifier (filename) originates
from the record.
The `downloadAsBytes` method does not have this file tracking issue
since it does not persist the data, but it forces callers to implement
their own attachment storing mechanism.
This commit adds the `useCache` option to the `download()` method to
enable callers to use an IndexedDB-based cache instead of the
filesystem. The following options unlock significant features:
- `fallbackToCache` - If the requested attachment is not available, the
last known attachment is returned.
- `fallbackToDump` - If the requested attachment is not available, nor
cached, then the attachment (dump) that is packaged with the client is
returned instead. This is implemented in the next commit (D72417).
The original record is cached along with the attachment, to allow
callers to use the file (identified by the given `attachmentId`) and
its metadata, even when the original record has been removed from the
collection.
This is particularly useful for scenarios where one wants to keep a file
(and metadata) up to date via RemoteSettings, without having to develop
a separate storage and synchronization mechanism.
The deprecation of the old behavior will be handled in bug 1634127.
Differential Revision: https://phabricator.services.mozilla.com/D72416
This commit wires up `StorageSyncArea::teardown()` to call the new
`webext_storage::Store::close()` method.
It also changes `teardown` to drop the `LazyStore`, and thus close its
database connection, on the main thread if dispatching to the task
queue fails. Dispatch should only fail at shutdown, and putting the
owned reference back only adds indirection, since the `StorageSyncArea`
will still drop its `LazyStore` later in shutdown.
Finally, it includes an xpcshell test fix for
https://github.com/mozilla/application-services/pull/3050.
Differential Revision: https://phabricator.services.mozilla.com/D72992
Now that `BridgedEngine` has been moved to `sync15_traits`, we can
remove `golden_gate_traits` from the tree, and change Golden Gate to
depend on `sync15_traits` directly.
This commit also adds a Cargo feature, `services_sync`, which reflects
the `MOZ_SERVICES_SYNC` config option. In the future, we'll use this
feature to gate implementations of `mozIBridgedSyncEngine`.
Differential Revision: https://phabricator.services.mozilla.com/D72784
Csoregi Natalia
Mathieu Leplatre
Mark Hammond
Cosmin Sabou
Mihai Alexandru Michis
Vlad Filippov
ffxbld
Rob Wu
Thom Chiovoloni
Lina Cambridge
Dorel Luca
Mozilla Releng Treescript
Mark Banner
Bogdan Tara
Ryan Kelly
Sid Stamm
Florian Quèze
Noemi Erli
Edouard Oger
Michael Kaply
Vijay Budhram
Gijs Kruitbosch
Andrea Marchesini
Coroiu Cristina
Ian Moody
Razvan Maries
Stefan Hindli