It was observed that FHR was sending invalid JSON payloads to the
server. Specifically, JSON payloads contained invalid Unicode strings.
Investigation revealed that the culprint was CommonUtils.convertString()
silently swallowing high bytes. When the Bagheera client went to gzip
the JSON payload, the input buffer into gzip was missing high bytes.
This patch changes the bagheera client to UTF-8 encode strings before
gzip, thus ensuring all data is preserved. A corresponding change was
also added to the mock bagheera server implementation.
Alternatively, we could have changed CommonUtils.convertString() to
be high byte aware. However, many consumers rely on this function.
This patch is written with the intent of being uplifted and the change
performed is targeted at the specific problem.
Tests for Unicode preserving behavior have been added to both the
generic Bagheera client and to FHR. The latter test is arguably
not necessary, but peace of mind is a good thing, especially with
FHR.
See also bug 915850.
--HG--
extra : rebase_source : 4efddea7767c2e5f8cf19df247c3aba07c40eec6
extra : amend_source : ae3b6d89efa54fc9ed1794404476622946ad4b22
The data reporting notification was over-complicated. It wasn't
displayed for +24hr after first run and it had a weird, non-required
policy around what constituted acceptance of the policy.
The notification is now shown shortly after first startup.
The logic around "notification accepted" has been greatly simplified by
rolling it into "notification shown." Where we once were checking
whether the notification has been "accepted," we now check whether it
has been displayed. The overly complicated logic around the implicit
acceptance of the policy has also been removed.
The end result is the code for managing the state of the notification is
greatly simplified.
If recording FHR data during uploading raised an exception, it could
potentially abort the upload. This would appear to Mozilla as clients
that suddenly stopped using Firefox.
This patch adds explicit exception trapping around event record to
ensure this doesn't happen.
--HG--
extra : rebase_source : 7cd207b08a4f62be55093c71cb56e28832fd39d8
extra : amend_source : 9144ecea16a013370fffa5c2e833af4ec528ef5b
The v20140527.01 update hotfix deposited a JSON file in the profile
directory containing state. This file was purposefully not deleted
during hotfix uninstall so its contents could be later reported in
FHR.
This patch adds that reporting to FHR.
Currently, we only report a subset of the available fields. The
remaining "public" fields could be added easily enough. For now, it is
important that we capture the important ones.
--HG--
extra : rebase_source : 925d6ea04118296108327dc843323b150b87b208
The data reporting notification was over-complicated. It wasn't
displayed for +24hr after first run and it had a weird, non-required
policy around what constituted acceptance of the policy.
The notification is now shown shortly after first startup.
The logic around "notification accepted" has been greatly simplified by
rolling it into "notification shown." Where we once were checking
whether the notification has been "accepted," we now check whether it
has been displayed. The overly complicated logic around the implicit
acceptance of the policy has also been removed.
The end result is the code for managing the state of the notification is
greatly simplified.
--HG--
extra : rebase_source : 808efdf1edd103552f6aa10b5c4309b64e514773
extra : amend_source : e4252e6a850a348d1b5aca733121dd07cbc6a70a
extra : histedit_source : 10ec20a07677674a8c9a705a3ffb4dc46a22b890%2Ca9442934d5964f16e9ad1101b786b4d094ac228d
The upcoming build system patches don't support hypthens in path names.
Changing this for that reason is kind of silly, but it's the easiest
way. Besides, nothing else uses hyphens in directory names.
--HG--
extra : rebase_source : 42dda2b1f16a3c0bfe17397a70092362e400530f
This introduces a new crash event type for submissions and handles
them in the Crash Manager. These crash submissions will also be counted
in FHR.
--HG--
extra : rebase_source : c37fc86cecc86aea0afdd8cfa7b31295aad3d6f8
This introduces a new crash event type for submissions and handles
them in the Crash Manager. These crash submissions will also be counted
in FHR.
--HG--
extra : rebase_source : 41fb43f9588885033e49dccfaf9351ef805c476b
Up to this point, Firefox Health Report has generated and submitted a
random UUID with each upload. Generated UUIDs were stored on the client.
During upload, the client asked the server to delete all old UUIDs.
Well-behaving clients thus left at most one record/ID on the server.
Unfortunately, clients in the wild have not been behaving properly. We
are seeing multiple documents on the server that appear to come from the
same client. Clients are uploading new records but failing to delete the
old ones. These old, undeleted "orphan" records are severely impacting
the ability to derive useful knowledge from FHR data because it is
difficult, resource intensive, and error prone to filter the records on
the server. This is undermining the ability for FHR data to be put to
good use.
This patch introduces a persistent client identifier. When the client is
initialized, it generates a random UUID. That UUID is persisted to the
profile and sent as part of every upload.
For privacy reasons, if a client opts out of data submission, the client
ID will be reset as soon as all remote data has been deleted.
We still issue and send upload IDs. They exist mostly for forensics
purposes so we may log client behavior and more accurately determine
what exactly misbehaving, orphan-producing clients are doing.
It is worth noting that this persistent client identifier will not solve
all problems of branching and orphaned records. For example, profile
copying will result in multiple clients sharing a client identifier. A
"client ID version" field has been added to facilitate an upgrade path
towards client IDs with different generation semantics.
--HG--
extra : rebase_source : b761daab39fb07b6ab8883819d68bf53462314a0
This patch changes Firefox Health Report to pull crash counts from the
newly-implemented crash manager. The crash manager is now the canonical
source of crash data, so all the code for reading crash dump files from
disk has been removed.
This regresses the collection capability of Firefox Health Report by
removing plugin crashes and hangs from the reported values. This
regression is intended to be temporary and a follow-up bug will be
created to add plugin process event recording to the crash events
system.
This regression has been justified because the old crash reporting
mechanism was severely flawed and wasn't sending accurate data (not all
crashes were being saved to dumps and some dumps would be deleted).
--HG--
extra : amend_source : d54bb3bf9f740508cb4eda286536ce16a4c3bee5
This patch changes Firefox Health Report to pull crash counts from the
newly-implemented crash manager. The crash manager is now the canonical
source of crash data, so all the code for reading crash dump files from
disk has been removed.
This regresses the collection capability of Firefox Health Report by
removing plugin crashes and hangs from the reported values. This
regression is intended to be temporary and a follow-up bug will be
created to add plugin process event recording to the crash events
system.
This regression has been justified because the old crash reporting
mechanism was severely flawed and wasn't sending accurate data (not all
crashes were being saved to dumps and some dumps would be deleted).
--HG--
extra : rebase_source : 7442a712d06b4d875010e78d4dd4f5cc179c8dfc
Metrics for Sync have been added to Firefox Health Report. If Sync is
not configured, we'll report that fact and the supported and preferred
Sync protocols (1.1 or 1.5).
If Sync is configured, we report the daily counts of sync attempts and
how many are successful vs errored. We also report daily counts of the
device types attached to the account.
--HG--
extra : rebase_source : 77170b323706a85cbe1542ac993ebdc1dba3b505
extra : amend_source : 7802e80b4fc94937fbe3f67505b447bfb048732d
Previously, it was technically possible for the FHR client to have
multiple simultaneous uploads. While this should never occur in
well-behaving systems, server logs have indicated that this behavior
might be occurring.
This patch adds a lock around uploading to ensure only 1 upload
operation may be active at a given time.
To measure the impact of this change in the wild, we added a counter
that increments whenever a held lock is encountered.
--HG--
extra : rebase_source : f0bb5065a3618cd335b6b6f31e4e68850f31f151
Previously, we weren't measuring whether the machine was 64-bit
properly. This change allows us to report actual CPU architecture (at
least on Windows).
In the future, we may wish to properly report actual CPU architecture on
all operating systems.
--HG--
extra : rebase_source : 73b5673e870e959a33172ec22fee8c31db6e0e53
extra : amend_source : 64d2e23be07aa87bc5c150b6e613e6226c848ec8
FHR docs were extracted from /services/docs into their own compartment.
This arguably should have been done originally.
Documentation on the payload format has been imported from
https://github.com/mozilla-services/docs. Those docs were created in a
time before we had in-tree docs and this tree is a more logical home for
them.
--HG--
rename : services/docs/healthreport.rst => services/healthreport/docs/index.rst
extra : rebase_source : 4e2fd62cc98ea2620f580e34d032cb39bafb5108
extra : amend_source : bd6309cc9d53aba1b085067960bf20a94125c5b0
Benjamin Smedberg
Carsten "Tomcat" Book
Richard Newman
Andrew Halberstadt
Birunthan Mohanathas
Ed Morley
Bill McCloskey
Gregory Szorc
Nathan Froyd
Georg Fritzsche
Drew Willcoxon
Alexandre Poirot
Wes Kocher
Steven MacLeod
Asaf Romano
Felipe Gomes
David Rajchenbach-Teller
Mike de Boer
Ryan VanderMeulen
Blair McBride
Stefan Mirea
Stefan Mirea
Mike Hommey
Joey Armstrong
Gavin Sharp
Mike Connor
Shane Caraveo