In Telemetry and elsewhere we typically use add-on GUIDs to uniquely represent specific add-ons. Often times a GUID is ambiguous, revealing little to no information about the add-on. This script allows a user to quickly get add-on names, e10s compatibility, versions, weekly downloads, categories, etc. from AMO with just an add-on GUID. See the Appendix for an example JSON blob displaying all possible fields. Aside from easily acquiring meta data for add-ons, this example shows the various fields the user can access not accessible via telemetry at the moment.
+The example below is a simplification of the script used to generate the arewee10syet.com page. For more details please see the AMO API doc.
+import pandas as pd +import os +import requests +import json +import urllib +import sys +
# for manual editting of missing or incorrect add-on names +fixups = { + 'testpilot@labs.mozilla.com': 'Test Pilot (old one)', + '{20a82645-c095-46ed-80e3-08825760534b}': 'Microsoft .NET framework assistant', +} + +def process_amo(result): + """ + Selects and processes specific fields from the dict, + result, and returns new dict + """ + try: + name = result['name']['en-US'] + except KeyError: + name = result['slug'] + return { + 'name': name, + 'url': result['url'], + 'guid': result['guid'], + 'e10s_status': result['e10s'], + 'avg_daily_users': result['average_daily_users'], + 'categories': ','.join(result['categories']['firefox']), + 'weekly_downloads': result['weekly_downloads'], + 'ratings': result['ratings'] + } + +def amo(guid, raw=False): + """ + Make AMO API call to request data for a given add-on guid + + Return raw data if raw=True, which returns the full + json returned from AMO as a python dict, otherwise call + process_amo() to only return fields of interest + (specified in process_amo()) + """ + addon_url = AMO_SERVER + '/api/v3/addons/addon/{}/'.format(guid) + compat_url = AMO_SERVER + '/api/v3/addons/addon/{}/feature_compatibility/'.format(guid) + + result = {} + print "Fetching Data for:", guid + for url in (addon_url, compat_url): + res = requests.get(url) + if res.status_code != 200: + return { + 'name': fixups.get( + guid, '{} error fetching data from AMO'.format(res.status_code)), + 'guid': guid + } + res.raise_for_status() + res_json = res.json() + result.update(res_json) + if raw: + return result + return process_amo(result) + +def reorder_list(lst, move_to_front): + """ + Reorganizes the list <lst> such that the elements in + <move_to_front> appear at the beginning, in the order they appear in + <move_to_front>, returning a new list + """ + result = lst[:] + for elem in move_to_front[::-1]: + assert elem in lst, "'{}' is not in the list".format(elem) + result = [result.pop(result.index(elem))] + result + return result +
Instantiate amo server object to be used by the above functions
+AMO_SERVER = os.getenv('AMO_SERVER', 'https://addons.mozilla.org') +
As an example, we can call the amo() function for a list of 10 add-on GUIDs formatting them into a pandas DF.
addon_guids = \ +['easyscreenshot@mozillaonline.com', + 'firebug@software.joehewitt.com', + 'firefox@ghostery.com', + 'uBlock0@raymondhill.net', + '{20a82645-c095-46ed-80e3-08825760534b}', + '{73a6fe31-595d-460b-a920-fcc0f8843232}', + '{DDC359D1-844A-42a7-9AA1-88A850A938A8}', + '{b9db16a4-6edc-47ec-a1f4-b86292ed211d}', + '{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}', + '{e4a8a97b-f2ed-450b-b12d-ee082ba24781}'] + +df = pd.DataFrame([amo(i) for i in addon_guids]) + +# move guid and name to front of DF +df = df[reorder_list(list(df), move_to_front=['guid', 'name'])] +df +
Fetching Data for: easyscreenshot@mozillaonline.com
+Fetching Data for: firebug@software.joehewitt.com
+Fetching Data for: firefox@ghostery.com
+Fetching Data for: uBlock0@raymondhill.net
+Fetching Data for: {20a82645-c095-46ed-80e3-08825760534b}
+Fetching Data for: {73a6fe31-595d-460b-a920-fcc0f8843232}
+Fetching Data for: {DDC359D1-844A-42a7-9AA1-88A850A938A8}
+Fetching Data for: {b9db16a4-6edc-47ec-a1f4-b86292ed211d}
+Fetching Data for: {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}
+Fetching Data for: {e4a8a97b-f2ed-450b-b12d-ee082ba24781}
+| + | guid | +name | +avg_daily_users | +categories | +e10s_status | +ratings | +url | +weekly_downloads | +
|---|---|---|---|---|---|---|---|---|
| 0 | +easyscreenshot@mozillaonline.com | +Easy Screenshot | +2472392.0 | +photos-music-videos | +unknown | +{u'count': 121, u'average': 3.9587} | +https://addons.mozilla.org/en-US/firefox/addon... | +58682.0 | +
| 1 | +firebug@software.joehewitt.com | +Firebug | +1895612.0 | +web-development | +compatible | +{u'count': 1930, u'average': 4.4813} | +https://addons.mozilla.org/en-US/firefox/addon... | +103304.0 | +
| 2 | +firefox@ghostery.com | +Ghostery | +1359377.0 | +privacy-security,web-development | +compatible-webextension | +{u'count': 1342, u'average': 4.5745} | +https://addons.mozilla.org/en-US/firefox/addon... | +56074.0 | +
| 3 | +uBlock0@raymondhill.net | +uBlock Origin | +2825755.0 | +privacy-security | +compatible | +{u'count': 813, u'average': 4.6347} | +https://addons.mozilla.org/en-US/firefox/addon... | +464652.0 | +
| 4 | +{20a82645-c095-46ed-80e3-08825760534b} | +Microsoft .NET framework assistant | +NaN | +NaN | +NaN | +NaN | +NaN | +NaN | +
| 5 | +{73a6fe31-595d-460b-a920-fcc0f8843232} | +NoScript Security Suite | +2134660.0 | +privacy-security,web-development | +compatible | +{u'count': 1620, u'average': 4.7068} | +https://addons.mozilla.org/en-US/firefox/addon... | +66640.0 | +
| 6 | +{DDC359D1-844A-42a7-9AA1-88A850A938A8} | +DownThemAll! | +1185122.0 | +download-management | +compatible | +{u'count': 1830, u'average': 4.459} | +https://addons.mozilla.org/en-US/firefox/addon... | +71914.0 | +
| 7 | +{b9db16a4-6edc-47ec-a1f4-b86292ed211d} | +Video DownloadHelper | +4431752.0 | +download-management | +compatible | +{u'count': 7547, u'average': 4.2242} | +https://addons.mozilla.org/en-US/firefox/addon... | +273598.0 | +
| 8 | +{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} | +Adblock Plus | +18902366.0 | +privacy-security | +compatible | +{u'count': 4947, u'average': 4.8737} | +https://addons.mozilla.org/en-US/firefox/addon... | +744265.0 | +
| 9 | +{e4a8a97b-f2ed-450b-b12d-ee082ba24781} | +Greasemonkey | +1162583.0 | +other | +compatible | +{u'count': 1018, u'average': 4.2613} | +https://addons.mozilla.org/en-US/firefox/addon... | +63610.0 | +
There you have it! Please look at the Appendix for the possible fields obtainable through AMO.
+The function process_amo() uses prespecified fields. Here you can take a look at a number of the available fields and make necessary edits.
# request data for a single add-on guid +result = amo(addon_guids[1], raw=True) +
Fetching Data for: firebug@software.joehewitt.com +
result
+{u'authors': [{u'id': 9265,
+ u'name': u'Joe Hewitt',
+ u'url': u'https://addons.mozilla.org/en-US/firefox/user/joe-hewitt/'},
+ {u'id': 857086,
+ u'name': u'Jan Odvarko',
+ u'url': u'https://addons.mozilla.org/en-US/firefox/user/jan-odvarko/'},
+ {u'id': 95117,
+ u'name': u'robcee',
+ u'url': u'https://addons.mozilla.org/en-US/firefox/user/robcee/'},
+ {u'id': 4957771,
+ u'name': u'Firebug Working Group',
+ u'url': u'https://addons.mozilla.org/en-US/firefox/user/firebugworkinggroup/'}],
+ u'average_daily_users': 1895612,
+ u'categories': {u'firefox': [u'web-development']},
+ u'current_beta_version': {u'compatibility': {u'firefox': {u'max': u'52.0',
+ u'min': u'30.0a1'}},
+ u'edit_url': u'https://addons.mozilla.org/en-US/developers/addon/firebug/versions/1951656',
+ u'files': [{u'created': u'2016-10-11T11:26:53Z',
+ u'hash': u'sha256:512522fd0036e4daa267f9d57d14eaf31be2913391b12e468ccc89c07b8b48eb',
+ u'id': 518451,
+ u'platform': u'all',
+ u'size': 2617049,
+ u'status': u'beta',
+ u'url': u'https://addons.mozilla.org/firefox/downloads/file/518451/firebug-2.0.18b1-fx.xpi?src='}],
+ u'id': 1951656,
+ u'license': {u'id': 18,
+ u'name': {u'bg': u'BSD \u041b\u0438\u0446\u0435\u043d\u0437',
+ u'ca': u'Llic\xe8ncia BSD',
+ u'cs': u'BSD licence',
+ u'da': u'BSD-licens',
+ u'de': u'BSD-Lizenz',
+ u'el': u'\u0386\u03b4\u03b5\u03b9\u03b1 BSD',
+ u'en-US': u'BSD License',
+ u'es': u'Licencia BSD',
+ u'eu': u'BSD Lizentzia',
+ u'fa': u'\u0645\u062c\u0648\u0632 BSD',
+ u'fr': u'Licence BSD',
+ u'ga-IE': u'Cead\xfanas BSD',
+ u'hu': u'BSD licenc',
+ u'id': u'Lisensi BSD',
+ u'it': u'Licenza BSD',
+ u'nl': u'BSD-licentie',
+ u'pt-PT': u'Licen\xe7a BSD',
+ u'ru': u'\u041b\u0438\u0446\u0435\u043d\u0437\u0438\u044f BSD',
+ u'sk': u'Licencia BSD',
+ u'sq': u'Leje BSD',
+ u'sr': u'\u0411\u0421\u0414 \u043b\u0438\u0446\u0435\u043d\u0446\u0430',
+ u'sv-SE': u'BSD-licens',
+ u'vi': u'Gi\u1ea5y ph\xe9p BSD',
+ u'zh-CN': u'BSD \u6388\u6743'},
+ u'url': u'http://www.opensource.org/licenses/bsd-license.php'},
+ u'reviewed': None,
+ u'url': u'https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/2.0.18b1',
+ u'version': u'2.0.18b1'},
+ u'current_version': {u'compatibility': {u'firefox': {u'max': u'52.0',
+ u'min': u'30.0a1'}},
+ u'edit_url': u'https://addons.mozilla.org/en-US/developers/addon/firebug/versions/1949588',
+ u'files': [{u'created': u'2016-10-07T12:02:17Z',
+ u'hash': u'sha256:7cd395e1a79f24b25856f73abaa2f1ca45b9a12dd83276e3b6b6bcb0f25c2944',
+ u'id': 516537,
+ u'platform': u'all',
+ u'size': 2617046,
+ u'status': u'public',
+ u'url': u'https://addons.mozilla.org/firefox/downloads/file/516537/firebug-2.0.18-fx.xpi?src='}],
+ u'id': 1949588,
+ u'license': {u'id': 18,
+ u'name': {u'bg': u'BSD \u041b\u0438\u0446\u0435\u043d\u0437',
+ u'ca': u'Llic\xe8ncia BSD',
+ u'cs': u'BSD licence',
+ u'da': u'BSD-licens',
+ u'de': u'BSD-Lizenz',
+ u'el': u'\u0386\u03b4\u03b5\u03b9\u03b1 BSD',
+ u'en-US': u'BSD License',
+ u'es': u'Licencia BSD',
+ u'eu': u'BSD Lizentzia',
+ u'fa': u'\u0645\u062c\u0648\u0632 BSD',
+ u'fr': u'Licence BSD',
+ u'ga-IE': u'Cead\xfanas BSD',
+ u'hu': u'BSD licenc',
+ u'id': u'Lisensi BSD',
+ u'it': u'Licenza BSD',
+ u'nl': u'BSD-licentie',
+ u'pt-PT': u'Licen\xe7a BSD',
+ u'ru': u'\u041b\u0438\u0446\u0435\u043d\u0437\u0438\u044f BSD',
+ u'sk': u'Licencia BSD',
+ u'sq': u'Leje BSD',
+ u'sr': u'\u0411\u0421\u0414 \u043b\u0438\u0446\u0435\u043d\u0446\u0430',
+ u'sv-SE': u'BSD-licens',
+ u'vi': u'Gi\u1ea5y ph\xe9p BSD',
+ u'zh-CN': u'BSD \u6388\u6743'},
+ u'url': u'http://www.opensource.org/licenses/bsd-license.php'},
+ u'reviewed': None,
+ u'url': u'https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/2.0.18',
+ u'version': u'2.0.18'},
+ u'default_locale': u'en-US',
+ u'description': {u'en-US': u'<a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/fc4997ab3399c96f4fddde69c127a1a7ac4d471f70a11827f5965b04dd3d60a0/https%3A//twitter.com/%23%21/firebugnews">Follow Firebug news on Twitter</a>\n\nCompatibility table:\n\n<ul><li><strong>Firefox 3.6</strong> with <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/?page=1#version-1.7.3">Firebug 1.7.3</a></li><li><strong>Firefox 4.0</strong> with <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/?page=1#version-1.7.3">Firebug 1.7.3</a></li><li><strong>Firefox 5.0</strong> with <strong>Firebug 1.8.2</strong> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/?page=1#version-1.7.3">Firebug 1.7.3</a>)</li><li><strong>Firefox 6.0</strong> with <strong>Firebug 1.8.2</strong> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a>)</li><li><strong>Firefox 7.0</strong> with <strong>Firebug 1.8.2</strong> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a>)</li><li><strong>Firefox 8.0</strong> with <strong>Firebug 1.8.3</strong> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a>)</li><li><strong>Firefox 9.0</strong> with <strong>Firebug 1.8.4</strong> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a>)</li><li><strong>Firefox 10.0</strong> with <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a></li><li><strong>Firefox 11.0</strong> with <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a></li><li><strong>Firefox 12.0</strong> with <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a></li><li><strong>Firefox 13.0</strong> with <b>Firebug 1.10</b> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a>)</li><li><strong>Firefox 14.0</strong> with <b>Firebug 1.10</b></li><li><strong>Firefox 15.0</strong> with <b>Firebug 1.10</b></li><li><strong>Firefox 16.0</strong> with <b>Firebug 1.10</b></li><li><strong>Firefox 17.0</strong> with <b>Firebug 1.11</b> (and also Firebug 1.10)</li><li><strong>Firefox 18.0</strong> with <b>Firebug 1.11</b></li><li><strong>Firefox 19.0</strong> with <b>Firebug 1.11</b></li><li><strong>Firefox 20.0</strong> with <b>Firebug 1.11</b></li><li><strong>Firefox 21.0</strong> with <b>Firebug 1.11</b></li><li><strong>Firefox 22.0</strong> with <b>Firebug 1.11</b></li><li><strong>Firefox 23.0</strong> with <b>Firebug 1.12</b> (and also Firebug 1.11)</li><li><strong>Firefox 24.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 25.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 26.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 27.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 28.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 29.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 30.0</strong> with <b>Firebug 2.0</b> (and also Firebug 1.12)</li><li><strong>Firefox 31.0</strong> with <b>Firebug 2.0</b> (and also Firebug 1.12)</li><li><strong>Firefox 32.0</strong> with <b>Firebug 2.0</b> (and also Firebug 1.12)</li><li><strong>Firefox 33 - 46</strong> with <b>Firebug 2.0</b></li></ul>\nFirebug integrates with Firefox to put a wealth of development tools at your fingertips while you browse. You can edit, debug, and monitor CSS, HTML, and JavaScript live in any web page.\n\nVisit the Firebug website for documentation and screen shots: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>\nAsk questions on Firebug newsgroup: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/2398490bb3575c96d4cf9054e4f11236717aa6b9f353f5ada85dd8c74ed0dbbe/https%3A//groups.google.com/forum/%23%21forum/firebug">https://groups.google.com/forum/#!forum/firebug</a>\nReport an issue: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/82ba7e7d7600d78965967de98579334986246b827dcc5c8a0bcf0ba036ef62dd/http%3A//code.google.com/p/fbug/issues/list">http://code.google.com/p/fbug/issues/list</a>\n\n<i>Please note that bug reports or feature requests in comments on this page won\'t be tracked. Use the issue tracker or Firebug newsgroup instead thanks!</i>',
+ u'ja': u'Firebug \u306f\u3001Web \u30da\u30fc\u30b8\u3092\u95b2\u89a7\u4e2d\u306b\u30af\u30ea\u30c3\u30af\u4e00\u3064\u3067\u4f7f\u3048\u308b\u8c4a\u5bcc\u306a\u958b\u767a\u30c4\u30fc\u30eb\u3092 Firefox \u306b\u7d71\u5408\u3057\u307e\u3059\u3002\u3042\u306a\u305f\u306f\u3042\u3089\u3086\u308b Web \u30da\u30fc\u30b8\u306e CSS\u3001HTML\u3001\u53ca\u3073 JavaScript \u3092\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u306b\u7de8\u96c6\u3001\u30c7\u30d0\u30c3\u30b0\u3001\u307e\u305f\u306f\u30e2\u30cb\u30bf\u3059\u308b\u3053\u3068\u304c\u51fa\u6765\u307e\u3059\u3002',
+ u'pl': u'Firebug integruje si\u0119 z Firefoksem, by doda\u0107 bogactwo narz\u0119dzi programistycznych dost\u0119pnych b\u0142yskawicznie podczas u\u017cywania przegl\u0105darki. Mo\u017cna edytowa\u0107, analizowa\u0107 kod oraz monitorowa\u0107 CSS, HTML i JavaScript bezpo\u015brednio na dowolnej stronie internetowej\u2026\n\nAby zapozna\u0107 si\u0119 z dokumentacj\u0105, zrzutami ekranu lub skorzysta\u0107 z forum dyskusyjnego, przejd\u017a na stron\u0119: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>',
+ u'pt-PT': u'O Firebug integra-se com o Firefox para colocar um conjunto de ferramentas de desenvolvimento nas suas m\xe3os enquanto navega. Pode editar, depurar e monitorizar CSS, HTML e JavaScript, em tempo real e em qualquer p\xe1gina Web\u2026\n\nVisite o s\xedtio do Firebug para obter documenta\xe7\xe3o, screen shots, e f\xf3runs de discuss\xe3o: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>',
+ u'ro': u'Firebug \xee\u021bi ofer\u0103 \xeen Firefox o gr\u0103mad\u0103 de unelte de dezvoltare \xeen timp ce navighezi. Po\u021bi edita, depana, monitoriza CSS-ul, HTML \u0219i codul JavaScript \xeen timp real \xeen orice pagin\u0103.\n\nViziteaz\u0103 saitul Firebug pentru documenta\u021bie, capturi de ecran \u0219i forumuri de discu\u021bii: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>',
+ u'ru': u'Firebug \u0438\u043d\u0442\u0435\u0433\u0440\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u0432 Firefox \u0434\u043b\u044f \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u043f\u0440\u0438\u043d\u0435\u0441\u0442\u0438 \u0438\u0437\u043e\u0431\u0438\u043b\u0438\u0435 \u0441\u0440\u0435\u0434\u0441\u0442\u0432 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u043d\u0430 \u043a\u043e\u043d\u0447\u0438\u043a\u0438 \u0412\u0430\u0448\u0438\u0445 \u043f\u0430\u043b\u044c\u0446\u0435\u0432, \u0432 \u0442\u043e \u0432\u0440\u0435\u043c\u044f \u043a\u0430\u043a \u0412\u044b \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0443\u0435\u0442\u0435 \u043f\u043e \u0441\u0435\u0442\u0438. \u0412\u044b \u043c\u043e\u0436\u0435\u0442\u0435 \u0440\u0435\u0434\u0430\u043a\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c, \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0442\u044c \u043e\u0442\u043b\u0430\u0434\u043a\u0443 \u0438 \u043f\u0440\u043e\u0441\u043c\u0430\u0442\u0440\u0438\u0432\u0430\u0442\u044c CSS, HTML \u0438 JavaScript \u0432 \u0440\u0435\u0436\u0438\u043c\u0435 \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438 \u043d\u0430 \u043b\u044e\u0431\u043e\u0439 \u0441\u0442\u0440\u0430\u043d\u0438\u0446\u0435 \u0432 \u0441\u0435\u0442\u0438...\n\n\u041f\u043e\u0441\u0435\u0442\u0438\u0442\u0435 \u0441\u0430\u0439\u0442 Firebug - \u0442\u0430\u043c \u0412\u044b \u043d\u0430\u0439\u0434\u0451\u0442\u0435 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u044e, \u0441\u043d\u0438\u043c\u043a\u0438 \u044d\u043a\u0440\u0430\u043d\u0430 \u0438 \u0444\u043e\u0440\u0443\u043c\u044b \u0434\u043b\u044f \u043e\u0431\u0441\u0443\u0436\u0434\u0435\u043d\u0438\u044f: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>',
+ u'vi': u'Firebug t\xedch h\u1ee3p v\xe0o Firefox v\xf4 s\u1ed1 c\xf4ng c\u1ee5 ph\xe1t tri\u1ec3n ngay tr\u01b0\u1edbc \u0111\u1ea7u ng\xf3n tay c\u1ee7a b\u1ea1n. B\u1ea1n c\xf3 th\u1ec3 ch\u1ec9nh s\u1eeda, g\u1ee1 l\u1ed7i, v\xe0 theo d\xf5i CSS, HTML v\xe0 JavaScript tr\u1ef1c ti\u1ebfp tr\xean b\u1ea5t k\xec trang web n\xe0o.\n\nV\xe0o trang web Firebug \u0111\u1ec3 xem t\xe0i li\u1ec7u, \u1ea3nh ch\u1ee5p m\xe0n h\xecnh, v\xe0 di\u1ec5n \u0111\xe0n th\u1ea3o lu\u1eadn: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>',
+ u'zh-CN': u'Firebug \u4e3a\u4f60\u7684 Firefox \u96c6\u6210\u4e86\u6d4f\u89c8\u7f51\u9875\u7684\u540c\u65f6\u968f\u624b\u53ef\u5f97\u7684\u4e30\u5bcc\u5f00\u53d1\u5de5\u5177\u3002\u4f60\u53ef\u4ee5\u5bf9\u4efb\u4f55\u7f51\u9875\u7684 CSS\u3001HTML \u548c JavaScript \u8fdb\u884c\u5b9e\u65f6\u7f16\u8f91\u3001\u8c03\u8bd5\u548c\u76d1\u63a7\u3002\\n\\n\u8bbf\u95ee Firebug \u7f51\u7ad9\u6765\u67e5\u770b\u6587\u6863\u3001\u622a\u5c4f\u4ee5\u53ca\u8ba8\u8bba\u7ec4\uff08\u82f1\u8bed\uff09\uff1a<a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>'},
+ u'e10s': u'compatible',
+ u'edit_url': u'https://addons.mozilla.org/en-US/developers/addon/firebug/edit',
+ u'guid': u'firebug@software.joehewitt.com',
+ u'has_eula': False,
+ u'has_privacy_policy': False,
+ u'homepage': {u'en-US': u'http://getfirebug.com/',
+ u'ja': u'http://getfirebug.com/jp'},
+ u'icon_url': u'https://addons.cdn.mozilla.net/user-media/addon_icons/1/1843-64.png?modified=1476141618',
+ u'id': 1843,
+ u'is_disabled': False,
+ u'is_experimental': False,
+ u'is_source_public': True,
+ u'last_updated': u'2016-10-10T22:39:32Z',
+ u'name': {u'en-US': u'Firebug', u'ja': u'Firebug', u'pt-BR': u'Firebug'},
+ u'previews': [{u'caption': {u'cs': u'',
+ u'en-US': u'Command line and its auto-completion.'},
+ u'id': 52710,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52710.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52710.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'Console panel with an example error logs.'},
+ u'id': 52711,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52711.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52711.png?modified=1295251100'},
+ {u'caption': {u'cs': u'', u'en-US': u'CSS panel with inline editor.'},
+ u'id': 52712,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52712.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52712.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'DOM panel displays structure of the current page.'},
+ u'id': 52713,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52713.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52713.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'HTML panel shows markup of the current page.'},
+ u'id': 52714,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52714.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52714.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'Layout panel reveals layout of selected elements.'},
+ u'id': 52715,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52715.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52715.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'Net panel monitors HTTP communication.'},
+ u'id': 52716,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52716.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52716.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'Script panel allows to explore and debug JavaScript on the current page.'},
+ u'id': 52717,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52717.png?modified=1295251101',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52717.png?modified=1295251101'}],
+ u'public_stats': True,
+ u'ratings': {u'average': 4.4813, u'count': 1930},
+ u'review_url': u'https://addons.mozilla.org/en-US/editors/review/1843',
+ u'slug': u'firebug',
+ u'status': u'public',
+ u'summary': {u'en-US': u'Firebug integrates with Firefox to put a wealth of development tools at your fingertips while you browse. You can edit, debug, and monitor CSS, HTML, and JavaScript live in any web page...',
+ u'ja': u'Firebug \u306f\u3001Web \u30da\u30fc\u30b8\u3092\u95b2\u89a7\u4e2d\u306b\u30af\u30ea\u30c3\u30af\u4e00\u3064\u3067\u4f7f\u3048\u308b\u8c4a\u5bcc\u306a\u958b\u767a\u30c4\u30fc\u30eb\u3092 Firefox \u306b\u7d71\u5408\u3057\u307e\u3059\u3002\u3042\u306a\u305f\u306f\u3042\u3089\u3086\u308b Web \u30da\u30fc\u30b8\u306e CSS\u3001HTML\u3001\u53ca\u3073 JavaScript \u3092\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u306b\u7de8\u96c6\u3001\u30c7\u30d0\u30c3\u30b0\u3001\u307e\u305f\u306f\u30e2\u30cb\u30bf\u3059\u308b\u3053\u3068\u304c\u51fa\u6765\u307e\u3059\u3002',
+ u'pl': u'Firebug dodaje do Firefoksa bogactwo narz\u0119dzi programistycznych. Mo\u017cna edytowa\u0107, analizowa\u0107 kod oraz monitorowa\u0107 CSS, HTML i JavaScript bezpo\u015brednio na dowolnej stronie internetowej\u2026\n\nFirebug 1.4 dzia\u0142a z Firefoksem 3.0 i nowszymi wersjami.',
+ u'pt-PT': u'O Firebug integra-se com o Firefox para colocar um conjunto de ferramentas de desenvolvimento nas suas m\xe3os enquanto navega. Pode editar, depurar e monitorizar CSS, HTML e JavaScript, em tempo real e em qualquer p\xe1gina Web\u2026\n\nFb1.4 req o Firefox 3.0+',
+ u'ro': u'Firebug \xee\u021bi ofer\u0103 \xeen Firefox o gr\u0103mad\u0103 de unelte de dezvoltare \xeen timp ce navighezi. Po\u021bi edita, depana, monitoriza CSS, HTML \u0219i JavaScript \xeen timp real \xeen orice pagin\u0103...\n\nFirebug 1.4 necesit\u0103 Firefox 3.0 sau o versiune mai recent\u0103.',
+ u'ru': u'Firebug \u0438\u043d\u0442\u0435\u0433\u0440\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u0432 Firefox \u0434\u043b\u044f \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u043f\u0440\u0438\u043d\u0435\u0441\u0442\u0438 \u0438\u0437\u043e\u0431\u0438\u043b\u0438\u0435 \u0441\u0440\u0435\u0434\u0441\u0442\u0432 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u043d\u0430 \u043a\u043e\u043d\u0447\u0438\u043a\u0438 \u0412\u0430\u0448\u0438\u0445 \u043f\u0430\u043b\u044c\u0446\u0435\u0432, \u0432 \u0442\u043e \u0432\u0440\u0435\u043c\u044f \u043a\u0430\u043a \u0412\u044b \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0443\u0435\u0442\u0435 \u043f\u043e \u0441\u0435\u0442\u0438.\n\n\u0414\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b Firebug 1.4 \u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f Firefox 3.0 \u0438\u043b\u0438 \u0432\u044b\u0448\u0435.',
+ u'vi': u'Firebug t\xedch h\u1ee3p v\xe0o Firefox v\xf4 s\u1ed1 c\xf4ng c\u1ee5 ph\xe1t tri\u1ec3n ngay tr\u01b0\u1edbc \u0111\u1ea7u ng\xf3n tay c\u1ee7a b\u1ea1n. B\u1ea1n c\xf3 th\u1ec3 ch\u1ec9nh s\u1eeda, g\u1ee1 l\u1ed7i, v\xe0 theo d\xf5i CSS, HTML v\xe0 JavaScript tr\u1ef1c ti\u1ebfp tr\xean b\u1ea5t k\xec trang web n\xe0o...\n\nFirebug 1.4 y\xeau c\u1ea7u Firefox 3.0 ho\u1eb7c cao h\u01a1n.',
+ u'zh-CN': u'Firebug \u4e3a\u4f60\u7684 Firefox \u96c6\u6210\u4e86\u6d4f\u89c8\u7f51\u9875\u7684\u540c\u65f6\u968f\u624b\u53ef\u5f97\u7684\u4e30\u5bcc\u5f00\u53d1\u5de5\u5177\u3002\u4f60\u53ef\u4ee5\u5bf9\u4efb\u4f55\u7f51\u9875\u7684 CSS\u3001HTML \u548c JavaScript \u8fdb\u884c\u5b9e\u65f6\u7f16\u8f91\u3001\u8c03\u8bd5\u548c\u76d1\u63a7\u2026\\n\\nFirebug 1.4 \u4ec5\u652f\u6301 Firefox 3.0 \u6216\u66f4\u9ad8\u7248\u672c\u3002'},
+ u'support_email': None,
+ u'support_url': {u'en-US': u'http://getfirebug.com'},
+ u'tags': [u'ads',
+ u'ads filter refinement',
+ u'console',
+ u'css',
+ u'debugging',
+ u'developer',
+ u'development',
+ u'dom',
+ u'firebug',
+ u'html',
+ u'javascript',
+ u'js',
+ u'logging',
+ u'minacoda',
+ u'network',
+ u'performance',
+ u'profile',
+ u'restartless',
+ u'web',
+ u'xpath'],
+ u'type': u'extension',
+ u'url': u'https://addons.mozilla.org/en-US/firefox/addon/firebug/',
+ u'weekly_downloads': 103304}
+In Telemetry and elsewhere we typically use add-on GUIDs to uniquely represent specific add-ons. Often times a GUID is ambiguous, revealing little to no information about the add-on. This script allows a user to quickly get add-on names, e10s compatibility, versions, weekly downloads, categories, etc. from AMO with just an add-on GUID. See the Appendix for an example JSON blob displaying all possible fields. Aside from easily acquiring meta data for add-ons, this example shows the various fields the user can access not accessible via telemetry at the moment.
+The example below is a simplification of the script used to generate the arewee10syet.com page. For more details please see the AMO API doc.
+import pandas as pd +import os +import requests +import json +import urllib +import sys +
# for manual editting of missing or incorrect add-on names +fixups = { + 'testpilot@labs.mozilla.com': 'Test Pilot (old one)', + '{20a82645-c095-46ed-80e3-08825760534b}': 'Microsoft .NET framework assistant', +} + +def process_amo(result): + """ + Selects and processes specific fields from the dict, + result, and returns new dict + """ + try: + name = result['name']['en-US'] + except KeyError: + name = result['slug'] + return { + 'name': name, + 'url': result['url'], + 'guid': result['guid'], + 'e10s_status': result['e10s'], + 'avg_daily_users': result['average_daily_users'], + 'categories': ','.join(result['categories']['firefox']), + 'weekly_downloads': result['weekly_downloads'], + 'ratings': result['ratings'] + } + +def amo(guid, raw=False): + """ + Make AMO API call to request data for a given add-on guid + + Return raw data if raw=True, which returns the full + json returned from AMO as a python dict, otherwise call + process_amo() to only return fields of interest + (specified in process_amo()) + """ + addon_url = AMO_SERVER + '/api/v3/addons/addon/{}/'.format(guid) + compat_url = AMO_SERVER + '/api/v3/addons/addon/{}/feature_compatibility/'.format(guid) + + result = {} + print "Fetching Data for:", guid + for url in (addon_url, compat_url): + res = requests.get(url) + if res.status_code != 200: + return { + 'name': fixups.get( + guid, '{} error fetching data from AMO'.format(res.status_code)), + 'guid': guid + } + res.raise_for_status() + res_json = res.json() + result.update(res_json) + if raw: + return result + return process_amo(result) + +def reorder_list(lst, move_to_front): + """ + Reorganizes the list <lst> such that the elements in + <move_to_front> appear at the beginning, in the order they appear in + <move_to_front>, returning a new list + """ + result = lst[:] + for elem in move_to_front[::-1]: + assert elem in lst, "'{}' is not in the list".format(elem) + result = [result.pop(result.index(elem))] + result + return result +
Instantiate amo server object to be used by the above functions
+AMO_SERVER = os.getenv('AMO_SERVER', 'https://addons.mozilla.org') +
As an example, we can call the amo() function for a list of 10 add-on GUIDs formatting them into a pandas DF.
addon_guids = \ +['easyscreenshot@mozillaonline.com', + 'firebug@software.joehewitt.com', + 'firefox@ghostery.com', + 'uBlock0@raymondhill.net', + '{20a82645-c095-46ed-80e3-08825760534b}', + '{73a6fe31-595d-460b-a920-fcc0f8843232}', + '{DDC359D1-844A-42a7-9AA1-88A850A938A8}', + '{b9db16a4-6edc-47ec-a1f4-b86292ed211d}', + '{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}', + '{e4a8a97b-f2ed-450b-b12d-ee082ba24781}'] + +df = pd.DataFrame([amo(i) for i in addon_guids]) + +# move guid and name to front of DF +df = df[reorder_list(list(df), move_to_front=['guid', 'name'])] +df +
+ Fetching Data for: easyscreenshot@mozillaonline.com
+Fetching Data for: firebug@software.joehewitt.com
+Fetching Data for: firefox@ghostery.com
+Fetching Data for: uBlock0@raymondhill.net
+Fetching Data for: {20a82645-c095-46ed-80e3-08825760534b}
+Fetching Data for: {73a6fe31-595d-460b-a920-fcc0f8843232}
+Fetching Data for: {DDC359D1-844A-42a7-9AA1-88A850A938A8}
+Fetching Data for: {b9db16a4-6edc-47ec-a1f4-b86292ed211d}
+Fetching Data for: {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}
+Fetching Data for: {e4a8a97b-f2ed-450b-b12d-ee082ba24781}
+| + | guid | +name | +avg_daily_users | +categories | +e10s_status | +ratings | +url | +weekly_downloads | +
|---|---|---|---|---|---|---|---|---|
| 0 | +easyscreenshot@mozillaonline.com | +Easy Screenshot | +2472392.0 | +photos-music-videos | +unknown | +{u'count': 121, u'average': 3.9587} | +https://addons.mozilla.org/en-US/firefox/addon... | +58682.0 | +
| 1 | +firebug@software.joehewitt.com | +Firebug | +1895612.0 | +web-development | +compatible | +{u'count': 1930, u'average': 4.4813} | +https://addons.mozilla.org/en-US/firefox/addon... | +103304.0 | +
| 2 | +firefox@ghostery.com | +Ghostery | +1359377.0 | +privacy-security,web-development | +compatible-webextension | +{u'count': 1342, u'average': 4.5745} | +https://addons.mozilla.org/en-US/firefox/addon... | +56074.0 | +
| 3 | +uBlock0@raymondhill.net | +uBlock Origin | +2825755.0 | +privacy-security | +compatible | +{u'count': 813, u'average': 4.6347} | +https://addons.mozilla.org/en-US/firefox/addon... | +464652.0 | +
| 4 | +{20a82645-c095-46ed-80e3-08825760534b} | +Microsoft .NET framework assistant | +NaN | +NaN | +NaN | +NaN | +NaN | +NaN | +
| 5 | +{73a6fe31-595d-460b-a920-fcc0f8843232} | +NoScript Security Suite | +2134660.0 | +privacy-security,web-development | +compatible | +{u'count': 1620, u'average': 4.7068} | +https://addons.mozilla.org/en-US/firefox/addon... | +66640.0 | +
| 6 | +{DDC359D1-844A-42a7-9AA1-88A850A938A8} | +DownThemAll! | +1185122.0 | +download-management | +compatible | +{u'count': 1830, u'average': 4.459} | +https://addons.mozilla.org/en-US/firefox/addon... | +71914.0 | +
| 7 | +{b9db16a4-6edc-47ec-a1f4-b86292ed211d} | +Video DownloadHelper | +4431752.0 | +download-management | +compatible | +{u'count': 7547, u'average': 4.2242} | +https://addons.mozilla.org/en-US/firefox/addon... | +273598.0 | +
| 8 | +{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} | +Adblock Plus | +18902366.0 | +privacy-security | +compatible | +{u'count': 4947, u'average': 4.8737} | +https://addons.mozilla.org/en-US/firefox/addon... | +744265.0 | +
| 9 | +{e4a8a97b-f2ed-450b-b12d-ee082ba24781} | +Greasemonkey | +1162583.0 | +other | +compatible | +{u'count': 1018, u'average': 4.2613} | +https://addons.mozilla.org/en-US/firefox/addon... | +63610.0 | +
There you have it! Please look at the Appendix for the possible fields obtainable through AMO.
+The function process_amo() uses prespecified fields. Here you can take a look at a number of the available fields and make necessary edits.
# request data for a single add-on guid +result = amo(addon_guids[1], raw=True) +
Fetching Data for: firebug@software.joehewitt.com +
result
+{u'authors': [{u'id': 9265,
+ u'name': u'Joe Hewitt',
+ u'url': u'https://addons.mozilla.org/en-US/firefox/user/joe-hewitt/'},
+ {u'id': 857086,
+ u'name': u'Jan Odvarko',
+ u'url': u'https://addons.mozilla.org/en-US/firefox/user/jan-odvarko/'},
+ {u'id': 95117,
+ u'name': u'robcee',
+ u'url': u'https://addons.mozilla.org/en-US/firefox/user/robcee/'},
+ {u'id': 4957771,
+ u'name': u'Firebug Working Group',
+ u'url': u'https://addons.mozilla.org/en-US/firefox/user/firebugworkinggroup/'}],
+ u'average_daily_users': 1895612,
+ u'categories': {u'firefox': [u'web-development']},
+ u'current_beta_version': {u'compatibility': {u'firefox': {u'max': u'52.0',
+ u'min': u'30.0a1'}},
+ u'edit_url': u'https://addons.mozilla.org/en-US/developers/addon/firebug/versions/1951656',
+ u'files': [{u'created': u'2016-10-11T11:26:53Z',
+ u'hash': u'sha256:512522fd0036e4daa267f9d57d14eaf31be2913391b12e468ccc89c07b8b48eb',
+ u'id': 518451,
+ u'platform': u'all',
+ u'size': 2617049,
+ u'status': u'beta',
+ u'url': u'https://addons.mozilla.org/firefox/downloads/file/518451/firebug-2.0.18b1-fx.xpi?src='}],
+ u'id': 1951656,
+ u'license': {u'id': 18,
+ u'name': {u'bg': u'BSD \u041b\u0438\u0446\u0435\u043d\u0437',
+ u'ca': u'Llic\xe8ncia BSD',
+ u'cs': u'BSD licence',
+ u'da': u'BSD-licens',
+ u'de': u'BSD-Lizenz',
+ u'el': u'\u0386\u03b4\u03b5\u03b9\u03b1 BSD',
+ u'en-US': u'BSD License',
+ u'es': u'Licencia BSD',
+ u'eu': u'BSD Lizentzia',
+ u'fa': u'\u0645\u062c\u0648\u0632 BSD',
+ u'fr': u'Licence BSD',
+ u'ga-IE': u'Cead\xfanas BSD',
+ u'hu': u'BSD licenc',
+ u'id': u'Lisensi BSD',
+ u'it': u'Licenza BSD',
+ u'nl': u'BSD-licentie',
+ u'pt-PT': u'Licen\xe7a BSD',
+ u'ru': u'\u041b\u0438\u0446\u0435\u043d\u0437\u0438\u044f BSD',
+ u'sk': u'Licencia BSD',
+ u'sq': u'Leje BSD',
+ u'sr': u'\u0411\u0421\u0414 \u043b\u0438\u0446\u0435\u043d\u0446\u0430',
+ u'sv-SE': u'BSD-licens',
+ u'vi': u'Gi\u1ea5y ph\xe9p BSD',
+ u'zh-CN': u'BSD \u6388\u6743'},
+ u'url': u'http://www.opensource.org/licenses/bsd-license.php'},
+ u'reviewed': None,
+ u'url': u'https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/2.0.18b1',
+ u'version': u'2.0.18b1'},
+ u'current_version': {u'compatibility': {u'firefox': {u'max': u'52.0',
+ u'min': u'30.0a1'}},
+ u'edit_url': u'https://addons.mozilla.org/en-US/developers/addon/firebug/versions/1949588',
+ u'files': [{u'created': u'2016-10-07T12:02:17Z',
+ u'hash': u'sha256:7cd395e1a79f24b25856f73abaa2f1ca45b9a12dd83276e3b6b6bcb0f25c2944',
+ u'id': 516537,
+ u'platform': u'all',
+ u'size': 2617046,
+ u'status': u'public',
+ u'url': u'https://addons.mozilla.org/firefox/downloads/file/516537/firebug-2.0.18-fx.xpi?src='}],
+ u'id': 1949588,
+ u'license': {u'id': 18,
+ u'name': {u'bg': u'BSD \u041b\u0438\u0446\u0435\u043d\u0437',
+ u'ca': u'Llic\xe8ncia BSD',
+ u'cs': u'BSD licence',
+ u'da': u'BSD-licens',
+ u'de': u'BSD-Lizenz',
+ u'el': u'\u0386\u03b4\u03b5\u03b9\u03b1 BSD',
+ u'en-US': u'BSD License',
+ u'es': u'Licencia BSD',
+ u'eu': u'BSD Lizentzia',
+ u'fa': u'\u0645\u062c\u0648\u0632 BSD',
+ u'fr': u'Licence BSD',
+ u'ga-IE': u'Cead\xfanas BSD',
+ u'hu': u'BSD licenc',
+ u'id': u'Lisensi BSD',
+ u'it': u'Licenza BSD',
+ u'nl': u'BSD-licentie',
+ u'pt-PT': u'Licen\xe7a BSD',
+ u'ru': u'\u041b\u0438\u0446\u0435\u043d\u0437\u0438\u044f BSD',
+ u'sk': u'Licencia BSD',
+ u'sq': u'Leje BSD',
+ u'sr': u'\u0411\u0421\u0414 \u043b\u0438\u0446\u0435\u043d\u0446\u0430',
+ u'sv-SE': u'BSD-licens',
+ u'vi': u'Gi\u1ea5y ph\xe9p BSD',
+ u'zh-CN': u'BSD \u6388\u6743'},
+ u'url': u'http://www.opensource.org/licenses/bsd-license.php'},
+ u'reviewed': None,
+ u'url': u'https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/2.0.18',
+ u'version': u'2.0.18'},
+ u'default_locale': u'en-US',
+ u'description': {u'en-US': u'<a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/fc4997ab3399c96f4fddde69c127a1a7ac4d471f70a11827f5965b04dd3d60a0/https%3A//twitter.com/%23%21/firebugnews">Follow Firebug news on Twitter</a>\n\nCompatibility table:\n\n<ul><li><strong>Firefox 3.6</strong> with <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/?page=1#version-1.7.3">Firebug 1.7.3</a></li><li><strong>Firefox 4.0</strong> with <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/?page=1#version-1.7.3">Firebug 1.7.3</a></li><li><strong>Firefox 5.0</strong> with <strong>Firebug 1.8.2</strong> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/?page=1#version-1.7.3">Firebug 1.7.3</a>)</li><li><strong>Firefox 6.0</strong> with <strong>Firebug 1.8.2</strong> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a>)</li><li><strong>Firefox 7.0</strong> with <strong>Firebug 1.8.2</strong> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a>)</li><li><strong>Firefox 8.0</strong> with <strong>Firebug 1.8.3</strong> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a>)</li><li><strong>Firefox 9.0</strong> with <strong>Firebug 1.8.4</strong> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a>)</li><li><strong>Firefox 10.0</strong> with <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a></li><li><strong>Firefox 11.0</strong> with <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a></li><li><strong>Firefox 12.0</strong> with <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a></li><li><strong>Firefox 13.0</strong> with <b>Firebug 1.10</b> (and also <a rel="nofollow" href="https://addons.mozilla.org/en-US/firefox/addon/firebug/versions/">Firebug 1.9</a>)</li><li><strong>Firefox 14.0</strong> with <b>Firebug 1.10</b></li><li><strong>Firefox 15.0</strong> with <b>Firebug 1.10</b></li><li><strong>Firefox 16.0</strong> with <b>Firebug 1.10</b></li><li><strong>Firefox 17.0</strong> with <b>Firebug 1.11</b> (and also Firebug 1.10)</li><li><strong>Firefox 18.0</strong> with <b>Firebug 1.11</b></li><li><strong>Firefox 19.0</strong> with <b>Firebug 1.11</b></li><li><strong>Firefox 20.0</strong> with <b>Firebug 1.11</b></li><li><strong>Firefox 21.0</strong> with <b>Firebug 1.11</b></li><li><strong>Firefox 22.0</strong> with <b>Firebug 1.11</b></li><li><strong>Firefox 23.0</strong> with <b>Firebug 1.12</b> (and also Firebug 1.11)</li><li><strong>Firefox 24.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 25.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 26.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 27.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 28.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 29.0</strong> with <b>Firebug 1.12</b></li><li><strong>Firefox 30.0</strong> with <b>Firebug 2.0</b> (and also Firebug 1.12)</li><li><strong>Firefox 31.0</strong> with <b>Firebug 2.0</b> (and also Firebug 1.12)</li><li><strong>Firefox 32.0</strong> with <b>Firebug 2.0</b> (and also Firebug 1.12)</li><li><strong>Firefox 33 - 46</strong> with <b>Firebug 2.0</b></li></ul>\nFirebug integrates with Firefox to put a wealth of development tools at your fingertips while you browse. You can edit, debug, and monitor CSS, HTML, and JavaScript live in any web page.\n\nVisit the Firebug website for documentation and screen shots: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>\nAsk questions on Firebug newsgroup: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/2398490bb3575c96d4cf9054e4f11236717aa6b9f353f5ada85dd8c74ed0dbbe/https%3A//groups.google.com/forum/%23%21forum/firebug">https://groups.google.com/forum/#!forum/firebug</a>\nReport an issue: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/82ba7e7d7600d78965967de98579334986246b827dcc5c8a0bcf0ba036ef62dd/http%3A//code.google.com/p/fbug/issues/list">http://code.google.com/p/fbug/issues/list</a>\n\n<i>Please note that bug reports or feature requests in comments on this page won\'t be tracked. Use the issue tracker or Firebug newsgroup instead thanks!</i>',
+ u'ja': u'Firebug \u306f\u3001Web \u30da\u30fc\u30b8\u3092\u95b2\u89a7\u4e2d\u306b\u30af\u30ea\u30c3\u30af\u4e00\u3064\u3067\u4f7f\u3048\u308b\u8c4a\u5bcc\u306a\u958b\u767a\u30c4\u30fc\u30eb\u3092 Firefox \u306b\u7d71\u5408\u3057\u307e\u3059\u3002\u3042\u306a\u305f\u306f\u3042\u3089\u3086\u308b Web \u30da\u30fc\u30b8\u306e CSS\u3001HTML\u3001\u53ca\u3073 JavaScript \u3092\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u306b\u7de8\u96c6\u3001\u30c7\u30d0\u30c3\u30b0\u3001\u307e\u305f\u306f\u30e2\u30cb\u30bf\u3059\u308b\u3053\u3068\u304c\u51fa\u6765\u307e\u3059\u3002',
+ u'pl': u'Firebug integruje si\u0119 z Firefoksem, by doda\u0107 bogactwo narz\u0119dzi programistycznych dost\u0119pnych b\u0142yskawicznie podczas u\u017cywania przegl\u0105darki. Mo\u017cna edytowa\u0107, analizowa\u0107 kod oraz monitorowa\u0107 CSS, HTML i JavaScript bezpo\u015brednio na dowolnej stronie internetowej\u2026\n\nAby zapozna\u0107 si\u0119 z dokumentacj\u0105, zrzutami ekranu lub skorzysta\u0107 z forum dyskusyjnego, przejd\u017a na stron\u0119: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>',
+ u'pt-PT': u'O Firebug integra-se com o Firefox para colocar um conjunto de ferramentas de desenvolvimento nas suas m\xe3os enquanto navega. Pode editar, depurar e monitorizar CSS, HTML e JavaScript, em tempo real e em qualquer p\xe1gina Web\u2026\n\nVisite o s\xedtio do Firebug para obter documenta\xe7\xe3o, screen shots, e f\xf3runs de discuss\xe3o: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>',
+ u'ro': u'Firebug \xee\u021bi ofer\u0103 \xeen Firefox o gr\u0103mad\u0103 de unelte de dezvoltare \xeen timp ce navighezi. Po\u021bi edita, depana, monitoriza CSS-ul, HTML \u0219i codul JavaScript \xeen timp real \xeen orice pagin\u0103.\n\nViziteaz\u0103 saitul Firebug pentru documenta\u021bie, capturi de ecran \u0219i forumuri de discu\u021bii: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>',
+ u'ru': u'Firebug \u0438\u043d\u0442\u0435\u0433\u0440\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u0432 Firefox \u0434\u043b\u044f \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u043f\u0440\u0438\u043d\u0435\u0441\u0442\u0438 \u0438\u0437\u043e\u0431\u0438\u043b\u0438\u0435 \u0441\u0440\u0435\u0434\u0441\u0442\u0432 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u043d\u0430 \u043a\u043e\u043d\u0447\u0438\u043a\u0438 \u0412\u0430\u0448\u0438\u0445 \u043f\u0430\u043b\u044c\u0446\u0435\u0432, \u0432 \u0442\u043e \u0432\u0440\u0435\u043c\u044f \u043a\u0430\u043a \u0412\u044b \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0443\u0435\u0442\u0435 \u043f\u043e \u0441\u0435\u0442\u0438. \u0412\u044b \u043c\u043e\u0436\u0435\u0442\u0435 \u0440\u0435\u0434\u0430\u043a\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c, \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0442\u044c \u043e\u0442\u043b\u0430\u0434\u043a\u0443 \u0438 \u043f\u0440\u043e\u0441\u043c\u0430\u0442\u0440\u0438\u0432\u0430\u0442\u044c CSS, HTML \u0438 JavaScript \u0432 \u0440\u0435\u0436\u0438\u043c\u0435 \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0433\u043e \u0432\u0440\u0435\u043c\u0435\u043d\u0438 \u043d\u0430 \u043b\u044e\u0431\u043e\u0439 \u0441\u0442\u0440\u0430\u043d\u0438\u0446\u0435 \u0432 \u0441\u0435\u0442\u0438...\n\n\u041f\u043e\u0441\u0435\u0442\u0438\u0442\u0435 \u0441\u0430\u0439\u0442 Firebug - \u0442\u0430\u043c \u0412\u044b \u043d\u0430\u0439\u0434\u0451\u0442\u0435 \u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u044e, \u0441\u043d\u0438\u043c\u043a\u0438 \u044d\u043a\u0440\u0430\u043d\u0430 \u0438 \u0444\u043e\u0440\u0443\u043c\u044b \u0434\u043b\u044f \u043e\u0431\u0441\u0443\u0436\u0434\u0435\u043d\u0438\u044f: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>',
+ u'vi': u'Firebug t\xedch h\u1ee3p v\xe0o Firefox v\xf4 s\u1ed1 c\xf4ng c\u1ee5 ph\xe1t tri\u1ec3n ngay tr\u01b0\u1edbc \u0111\u1ea7u ng\xf3n tay c\u1ee7a b\u1ea1n. B\u1ea1n c\xf3 th\u1ec3 ch\u1ec9nh s\u1eeda, g\u1ee1 l\u1ed7i, v\xe0 theo d\xf5i CSS, HTML v\xe0 JavaScript tr\u1ef1c ti\u1ebfp tr\xean b\u1ea5t k\xec trang web n\xe0o.\n\nV\xe0o trang web Firebug \u0111\u1ec3 xem t\xe0i li\u1ec7u, \u1ea3nh ch\u1ee5p m\xe0n h\xecnh, v\xe0 di\u1ec5n \u0111\xe0n th\u1ea3o lu\u1eadn: <a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>',
+ u'zh-CN': u'Firebug \u4e3a\u4f60\u7684 Firefox \u96c6\u6210\u4e86\u6d4f\u89c8\u7f51\u9875\u7684\u540c\u65f6\u968f\u624b\u53ef\u5f97\u7684\u4e30\u5bcc\u5f00\u53d1\u5de5\u5177\u3002\u4f60\u53ef\u4ee5\u5bf9\u4efb\u4f55\u7f51\u9875\u7684 CSS\u3001HTML \u548c JavaScript \u8fdb\u884c\u5b9e\u65f6\u7f16\u8f91\u3001\u8c03\u8bd5\u548c\u76d1\u63a7\u3002\\n\\n\u8bbf\u95ee Firebug \u7f51\u7ad9\u6765\u67e5\u770b\u6587\u6863\u3001\u622a\u5c4f\u4ee5\u53ca\u8ba8\u8bba\u7ec4\uff08\u82f1\u8bed\uff09\uff1a<a rel="nofollow" href="https://outgoing.prod.mozaws.net/v1/835dbec6bd77fa1341bf5841171bc90e8a5d4069419f083a25b7f81f9fb254cb/http%3A//getfirebug.com">http://getfirebug.com</a>'},
+ u'e10s': u'compatible',
+ u'edit_url': u'https://addons.mozilla.org/en-US/developers/addon/firebug/edit',
+ u'guid': u'firebug@software.joehewitt.com',
+ u'has_eula': False,
+ u'has_privacy_policy': False,
+ u'homepage': {u'en-US': u'http://getfirebug.com/',
+ u'ja': u'http://getfirebug.com/jp'},
+ u'icon_url': u'https://addons.cdn.mozilla.net/user-media/addon_icons/1/1843-64.png?modified=1476141618',
+ u'id': 1843,
+ u'is_disabled': False,
+ u'is_experimental': False,
+ u'is_source_public': True,
+ u'last_updated': u'2016-10-10T22:39:32Z',
+ u'name': {u'en-US': u'Firebug', u'ja': u'Firebug', u'pt-BR': u'Firebug'},
+ u'previews': [{u'caption': {u'cs': u'',
+ u'en-US': u'Command line and its auto-completion.'},
+ u'id': 52710,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52710.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52710.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'Console panel with an example error logs.'},
+ u'id': 52711,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52711.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52711.png?modified=1295251100'},
+ {u'caption': {u'cs': u'', u'en-US': u'CSS panel with inline editor.'},
+ u'id': 52712,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52712.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52712.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'DOM panel displays structure of the current page.'},
+ u'id': 52713,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52713.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52713.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'HTML panel shows markup of the current page.'},
+ u'id': 52714,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52714.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52714.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'Layout panel reveals layout of selected elements.'},
+ u'id': 52715,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52715.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52715.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'Net panel monitors HTTP communication.'},
+ u'id': 52716,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52716.png?modified=1295251100',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52716.png?modified=1295251100'},
+ {u'caption': {u'cs': u'',
+ u'en-US': u'Script panel allows to explore and debug JavaScript on the current page.'},
+ u'id': 52717,
+ u'image_url': u'https://addons.cdn.mozilla.net/user-media/previews/full/52/52717.png?modified=1295251101',
+ u'thumbnail_url': u'https://addons.cdn.mozilla.net/user-media/previews/thumbs/52/52717.png?modified=1295251101'}],
+ u'public_stats': True,
+ u'ratings': {u'average': 4.4813, u'count': 1930},
+ u'review_url': u'https://addons.mozilla.org/en-US/editors/review/1843',
+ u'slug': u'firebug',
+ u'status': u'public',
+ u'summary': {u'en-US': u'Firebug integrates with Firefox to put a wealth of development tools at your fingertips while you browse. You can edit, debug, and monitor CSS, HTML, and JavaScript live in any web page...',
+ u'ja': u'Firebug \u306f\u3001Web \u30da\u30fc\u30b8\u3092\u95b2\u89a7\u4e2d\u306b\u30af\u30ea\u30c3\u30af\u4e00\u3064\u3067\u4f7f\u3048\u308b\u8c4a\u5bcc\u306a\u958b\u767a\u30c4\u30fc\u30eb\u3092 Firefox \u306b\u7d71\u5408\u3057\u307e\u3059\u3002\u3042\u306a\u305f\u306f\u3042\u3089\u3086\u308b Web \u30da\u30fc\u30b8\u306e CSS\u3001HTML\u3001\u53ca\u3073 JavaScript \u3092\u30ea\u30a2\u30eb\u30bf\u30a4\u30e0\u306b\u7de8\u96c6\u3001\u30c7\u30d0\u30c3\u30b0\u3001\u307e\u305f\u306f\u30e2\u30cb\u30bf\u3059\u308b\u3053\u3068\u304c\u51fa\u6765\u307e\u3059\u3002',
+ u'pl': u'Firebug dodaje do Firefoksa bogactwo narz\u0119dzi programistycznych. Mo\u017cna edytowa\u0107, analizowa\u0107 kod oraz monitorowa\u0107 CSS, HTML i JavaScript bezpo\u015brednio na dowolnej stronie internetowej\u2026\n\nFirebug 1.4 dzia\u0142a z Firefoksem 3.0 i nowszymi wersjami.',
+ u'pt-PT': u'O Firebug integra-se com o Firefox para colocar um conjunto de ferramentas de desenvolvimento nas suas m\xe3os enquanto navega. Pode editar, depurar e monitorizar CSS, HTML e JavaScript, em tempo real e em qualquer p\xe1gina Web\u2026\n\nFb1.4 req o Firefox 3.0+',
+ u'ro': u'Firebug \xee\u021bi ofer\u0103 \xeen Firefox o gr\u0103mad\u0103 de unelte de dezvoltare \xeen timp ce navighezi. Po\u021bi edita, depana, monitoriza CSS, HTML \u0219i JavaScript \xeen timp real \xeen orice pagin\u0103...\n\nFirebug 1.4 necesit\u0103 Firefox 3.0 sau o versiune mai recent\u0103.',
+ u'ru': u'Firebug \u0438\u043d\u0442\u0435\u0433\u0440\u0438\u0440\u0443\u0435\u0442\u0441\u044f \u0432 Firefox \u0434\u043b\u044f \u0442\u043e\u0433\u043e, \u0447\u0442\u043e\u0431\u044b \u043f\u0440\u0438\u043d\u0435\u0441\u0442\u0438 \u0438\u0437\u043e\u0431\u0438\u043b\u0438\u0435 \u0441\u0440\u0435\u0434\u0441\u0442\u0432 \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u043d\u0430 \u043a\u043e\u043d\u0447\u0438\u043a\u0438 \u0412\u0430\u0448\u0438\u0445 \u043f\u0430\u043b\u044c\u0446\u0435\u0432, \u0432 \u0442\u043e \u0432\u0440\u0435\u043c\u044f \u043a\u0430\u043a \u0412\u044b \u043f\u0443\u0442\u0435\u0448\u0435\u0441\u0442\u0432\u0443\u0435\u0442\u0435 \u043f\u043e \u0441\u0435\u0442\u0438.\n\n\u0414\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b Firebug 1.4 \u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f Firefox 3.0 \u0438\u043b\u0438 \u0432\u044b\u0448\u0435.',
+ u'vi': u'Firebug t\xedch h\u1ee3p v\xe0o Firefox v\xf4 s\u1ed1 c\xf4ng c\u1ee5 ph\xe1t tri\u1ec3n ngay tr\u01b0\u1edbc \u0111\u1ea7u ng\xf3n tay c\u1ee7a b\u1ea1n. B\u1ea1n c\xf3 th\u1ec3 ch\u1ec9nh s\u1eeda, g\u1ee1 l\u1ed7i, v\xe0 theo d\xf5i CSS, HTML v\xe0 JavaScript tr\u1ef1c ti\u1ebfp tr\xean b\u1ea5t k\xec trang web n\xe0o...\n\nFirebug 1.4 y\xeau c\u1ea7u Firefox 3.0 ho\u1eb7c cao h\u01a1n.',
+ u'zh-CN': u'Firebug \u4e3a\u4f60\u7684 Firefox \u96c6\u6210\u4e86\u6d4f\u89c8\u7f51\u9875\u7684\u540c\u65f6\u968f\u624b\u53ef\u5f97\u7684\u4e30\u5bcc\u5f00\u53d1\u5de5\u5177\u3002\u4f60\u53ef\u4ee5\u5bf9\u4efb\u4f55\u7f51\u9875\u7684 CSS\u3001HTML \u548c JavaScript \u8fdb\u884c\u5b9e\u65f6\u7f16\u8f91\u3001\u8c03\u8bd5\u548c\u76d1\u63a7\u2026\\n\\nFirebug 1.4 \u4ec5\u652f\u6301 Firefox 3.0 \u6216\u66f4\u9ad8\u7248\u672c\u3002'},
+ u'support_email': None,
+ u'support_url': {u'en-US': u'http://getfirebug.com'},
+ u'tags': [u'ads',
+ u'ads filter refinement',
+ u'console',
+ u'css',
+ u'debugging',
+ u'developer',
+ u'development',
+ u'dom',
+ u'firebug',
+ u'html',
+ u'javascript',
+ u'js',
+ u'logging',
+ u'minacoda',
+ u'network',
+ u'performance',
+ u'profile',
+ u'restartless',
+ u'web',
+ u'xpath'],
+ u'type': u'extension',
+ u'url': u'https://addons.mozilla.org/en-US/firefox/addon/firebug/',
+ u'weekly_downloads': 103304}
+Some OKRs for 2017 can be feasibly collected via the addons and main_summary tables. These tables are huge and aren’t appropriate to query directly via re:dash. This script condenses these tables so that the result contains the least data possible to track the following OKRs:
These OKRs, in addition to other add-on metrics, are tracked via the Add-on OKRs Dashboard in re:dash.
+from __future__ import division +import pyspark.sql.functions as fun +import pyspark.sql.types as st +import math +import os +import datetime as dt + +sc.setLogLevel("INFO") +
def optimize_repartition(df, record_size, partition_size=280): + ''' + Repartitions a spark DataFrame <df> so that each partition is + ~ <partition_size>MB, defaulting to 280MB. record_size must be + estimated beforehand--i.e. write the dataframe to s3, get the size + in bytes and divide by df.count(). + + Returns repartitioned dataframe if a repartition is necessary. + ''' + total_records = df.count() + print "-- Found {} records".format(total_records), + + #convert megabytes to bytes + partition_size *= 1000000 + + records_per_partition = partition_size / record_size + num_partitions = int(math.ceil(total_records / records_per_partition)) + + if num_partitions != df.rdd.getNumPartitions(): + print "-- Repartitioning with {} partitions".format(num_partitions) + df = df.repartition(num_partitions) + return df + +def get_env_date(): + ''' + Returns environment date if it exists. + otherwise returns yesterday's date + ''' + yesterday = dt.datetime.strftime(dt.datetime.utcnow() - dt.timedelta(1), "%Y%m%d") + return os.environ.get('date', yesterday) + +def get_dest(bucket, prefix): + ''' + Uses environment bucket if it exists. + Otherwises uses the bucket passed as a parameter + ''' + bucket = os.environ.get('bucket', bucket) + return '/'.join([bucket, prefix]) + + +# I use -1 and 1 because it allows me to segment users +# into three groups for two different cases: +# +# **Case 1**: +# Users that have only foreign-installed add-ons, only self-installed add-ons, +# or a combination. Applying `boot_to_int()` on a `foreign_install` boolean, +# I can sum the resulting field grouped by `client_id` and `submission_date_s3` +# to identify these groups as 1, -1, and 0 respectively. +# +# **Case 2**: Users that have the default theme, a custom theme, +# or changed their theme (from default to custom or visa versa) on a given day: +# Applying `boot_to_int()` on a `has_custom_theme` boolean, I can sum the +# resulting field grouped by `client_id` and `submission_date_s3` +# to identify these groups as -1, 1, and 0 respectively. +bool_to_int = fun.udf(lambda x: 1 if x == True else -1, st.IntegerType()) +
Unless specified in the environment, the target date is yesterday, and the bucket used is passed as a string to get_dest()
target_date = get_env_date() +dest = get_dest(bucket="telemetry-parquet", prefix="addons/agg/v1") +
addons and main_summary for yesterday (unless specified in the environment)addons = sqlContext.read.parquet("s3://telemetry-parquet/addons/v2") +addons = addons.filter(addons.submission_date_s3 == target_date) \ + .filter(addons.is_system == False) \ + .filter(addons.user_disabled == False) \ + .filter(addons.app_disabled == False) \ + +ms = sqlContext.read.option('mergeSchema', 'true')\ + .parquet('s3://telemetry-parquet/main_summary/v4') +ms = ms.filter(ms.submission_date_s3 == target_date) +
These are the aggregations / joins that we don’t want to do in re:dash.
+user_typehas_custom_theme.%%time + +default_theme_id = "{972ce4c6-7e08-4474-a285-3208198ce6fd}" + + +# count of distinct client submission_date, channel and install type +count_by_client_day = addons\ + .select(['client_id', 'submission_date_s3', 'normalized_channel', + 'foreign_install', 'addon_id'])\ + .distinct()\ + .groupBy(['client_id', 'submission_date_s3','foreign_install', 'normalized_channel'])\ + .count() + +# count of clients that have only foreign_installed, only self_installed and both +# per day/channel +user_types = count_by_client_day\ + .select(['client_id', 'submission_date_s3', 'normalized_channel', + bool_to_int('foreign_install').alias('user_type')])\ + .groupBy(['client_id', 'submission_date_s3', 'normalized_channel'])\ + .sum('user_type')\ + .withColumnRenamed('sum(user_type)', 'user_type') + +count_by_client_day = count_by_client_day.join(user_types, + on=['client_id', 'submission_date_s3', 'normalized_channel']) + + +# does a client have a custom theme? +# aggregate distinct values on a day/channel, since a client could have +# changed from default to custom +ms_has_theme = ms.select(\ + ms.client_id, ms.normalized_channel, bool_to_int(ms.active_theme.addon_id != default_theme_id).alias('has_custom_theme'))\ + .distinct()\ + .groupBy(['client_id', 'normalized_channel']).sum('has_custom_theme') \ + .withColumnRenamed('sum(has_custom_theme)', 'has_custom_theme') + + +# client_id, profile_creation_date and the earliest +# install day for an addon +ms_install_days = ms\ + .select(['client_id', 'profile_creation_date', + fun.explode('active_addons').alias('addons')])\ + .groupBy(['client_id', 'profile_creation_date'])\ + .agg(fun.min("addons.install_day").alias('min_install_day')) + + +# combine data +current = count_by_client_day\ + .join(ms_install_days, on='client_id', how='left')\ + .join(ms_has_theme, on=['client_id', 'normalized_channel'], how='left')\ + .drop('submission_date_s3') + + +# add total number of distinct clients per day/channel +# and total number of distinct clients with a custom theme per day/channel +# Note that we could see the same client on multiple channels +# so downstream analysis should be done within channel +n_clients = ms.select(['client_id', 'normalized_channel']).distinct()\ + .groupby('normalized_channel').count()\ + .withColumnRenamed('count', 'n_clients') + +n_custom_themes = ms_has_theme\ + .filter(ms_has_theme.has_custom_theme >= 0)\ + .select(['client_id', 'normalized_channel']).distinct()\ + .groupby('normalized_channel').count()\ + .withColumnRenamed('count', 'n_custom_theme_clients') + +current = current.join(n_custom_themes, on='normalized_channel')\ + .join(n_clients, on='normalized_channel') + +current = current.withColumn('n_clients', current.n_clients.cast(st.IntegerType()))\ + .withColumn('n_custom_theme_clients', current.n_custom_theme_clients.cast(st.IntegerType())) + +# repartition data +current = optimize_repartition(current, record_size=39) + +# write to s3 +current.write.format("parquet")\ + .save('s3://' + dest + '/submission_date_s3={}'.format(target_date), mode='overwrite') +
current.printSchema() +
Some OKRs for 2017 can be feasibly collected via the addons and main_summary tables. These tables are huge and aren’t appropriate to query directly via re:dash. This script condenses these tables so that the result contains the least data possible to track the following OKRs:
These OKRs, in addition to other add-on metrics, are tracked via the Add-on OKRs Dashboard in re:dash.
+from __future__ import division +import pyspark.sql.functions as fun +import pyspark.sql.types as st +import math +import os +import datetime as dt + +sc.setLogLevel("INFO") +
def optimize_repartition(df, record_size, partition_size=280): + ''' + Repartitions a spark DataFrame <df> so that each partition is + ~ <partition_size>MB, defaulting to 280MB. record_size must be + estimated beforehand--i.e. write the dataframe to s3, get the size + in bytes and divide by df.count(). + + Returns repartitioned dataframe if a repartition is necessary. + ''' + total_records = df.count() + print "-- Found {} records".format(total_records), + + #convert megabytes to bytes + partition_size *= 1000000 + + records_per_partition = partition_size / record_size + num_partitions = int(math.ceil(total_records / records_per_partition)) + + if num_partitions != df.rdd.getNumPartitions(): + print "-- Repartitioning with {} partitions".format(num_partitions) + df = df.repartition(num_partitions) + return df + +def get_env_date(): + ''' + Returns environment date if it exists. + otherwise returns yesterday's date + ''' + yesterday = dt.datetime.strftime(dt.datetime.utcnow() - dt.timedelta(1), "%Y%m%d") + return os.environ.get('date', yesterday) + +def get_dest(bucket, prefix): + ''' + Uses environment bucket if it exists. + Otherwises uses the bucket passed as a parameter + ''' + bucket = os.environ.get('bucket', bucket) + return '/'.join([bucket, prefix]) + + +# I use -1 and 1 because it allows me to segment users +# into three groups for two different cases: +# +# **Case 1**: +# Users that have only foreign-installed add-ons, only self-installed add-ons, +# or a combination. Applying `boot_to_int()` on a `foreign_install` boolean, +# I can sum the resulting field grouped by `client_id` and `submission_date_s3` +# to identify these groups as 1, -1, and 0 respectively. +# +# **Case 2**: Users that have the default theme, a custom theme, +# or changed their theme (from default to custom or visa versa) on a given day: +# Applying `boot_to_int()` on a `has_custom_theme` boolean, I can sum the +# resulting field grouped by `client_id` and `submission_date_s3` +# to identify these groups as -1, 1, and 0 respectively. +bool_to_int = fun.udf(lambda x: 1 if x == True else -1, st.IntegerType()) +
Unless specified in the environment, the target date is yesterday, and the bucket used is passed as a string to get_dest()
target_date = get_env_date() +dest = get_dest(bucket="telemetry-parquet", prefix="addons/agg/v1") +
addons and main_summary for yesterday (unless specified in the environment)addons = sqlContext.read.parquet("s3://telemetry-parquet/addons/v2") +addons = addons.filter(addons.submission_date_s3 == target_date) \ + .filter(addons.is_system == False) \ + .filter(addons.user_disabled == False) \ + .filter(addons.app_disabled == False) \ + +ms = sqlContext.read.option('mergeSchema', 'true')\ + .parquet('s3://telemetry-parquet/main_summary/v4') +ms = ms.filter(ms.submission_date_s3 == target_date) +
These are the aggregations / joins that we don’t want to do in re:dash.
+user_typehas_custom_theme.%%time + +default_theme_id = "{972ce4c6-7e08-4474-a285-3208198ce6fd}" + + +# count of distinct client submission_date, channel and install type +count_by_client_day = addons\ + .select(['client_id', 'submission_date_s3', 'normalized_channel', + 'foreign_install', 'addon_id'])\ + .distinct()\ + .groupBy(['client_id', 'submission_date_s3','foreign_install', 'normalized_channel'])\ + .count() + +# count of clients that have only foreign_installed, only self_installed and both +# per day/channel +user_types = count_by_client_day\ + .select(['client_id', 'submission_date_s3', 'normalized_channel', + bool_to_int('foreign_install').alias('user_type')])\ + .groupBy(['client_id', 'submission_date_s3', 'normalized_channel'])\ + .sum('user_type')\ + .withColumnRenamed('sum(user_type)', 'user_type') + +count_by_client_day = count_by_client_day.join(user_types, + on=['client_id', 'submission_date_s3', 'normalized_channel']) + + +# does a client have a custom theme? +# aggregate distinct values on a day/channel, since a client could have +# changed from default to custom +ms_has_theme = ms.select(\ + ms.client_id, ms.normalized_channel, bool_to_int(ms.active_theme.addon_id != default_theme_id).alias('has_custom_theme'))\ + .distinct()\ + .groupBy(['client_id', 'normalized_channel']).sum('has_custom_theme') \ + .withColumnRenamed('sum(has_custom_theme)', 'has_custom_theme') + + +# client_id, profile_creation_date and the earliest +# install day for an addon +ms_install_days = ms\ + .select(['client_id', 'profile_creation_date', + fun.explode('active_addons').alias('addons')])\ + .groupBy(['client_id', 'profile_creation_date'])\ + .agg(fun.min("addons.install_day").alias('min_install_day')) + + +# combine data +current = count_by_client_day\ + .join(ms_install_days, on='client_id', how='left')\ + .join(ms_has_theme, on=['client_id', 'normalized_channel'], how='left')\ + .drop('submission_date_s3') + + +# add total number of distinct clients per day/channel +# and total number of distinct clients with a custom theme per day/channel +# Note that we could see the same client on multiple channels +# so downstream analysis should be done within channel +n_clients = ms.select(['client_id', 'normalized_channel']).distinct()\ + .groupby('normalized_channel').count()\ + .withColumnRenamed('count', 'n_clients') + +n_custom_themes = ms_has_theme\ + .filter(ms_has_theme.has_custom_theme >= 0)\ + .select(['client_id', 'normalized_channel']).distinct()\ + .groupby('normalized_channel').count()\ + .withColumnRenamed('count', 'n_custom_theme_clients') + +current = current.join(n_custom_themes, on='normalized_channel')\ + .join(n_clients, on='normalized_channel') + +current = current.withColumn('n_clients', current.n_clients.cast(st.IntegerType()))\ + .withColumn('n_custom_theme_clients', current.n_custom_theme_clients.cast(st.IntegerType())) + +# repartition data +current = optimize_repartition(current, record_size=39) + +# write to s3 +current.write.format("parquet")\ + .save('s3://' + dest + '/submission_date_s3={}'.format(target_date), mode='overwrite') +
current.printSchema() +
In bug 1380880 :billm found that keyed histograms recorded on different processes would be aggregated together if their keys matched.
+How often does this happen in practice? How long has this been happening?
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +
The whole child-process client aggregation thing was introduced by bug 1218576 back in September of 2016 for Firefox 52. So that’s the earliest this could have started.
+pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appVersion=lambda x: x.startswith("52")) \ + .where(appUpdateChannel="nightly") \ + .records(sc, sample=0.1) +
fetching 13254.61440MB in 54449 files... +
def set_of_hgram_key_tuples(payload): + return set((kh_name, key) for (kh_name, v) in payload['keyedHistograms'].items() for key in v.keys()) + +def get_problem_combos(aping): + parent_tuples = set_of_hgram_key_tuples(aping['payload']) + child_tuples = [set_of_hgram_key_tuples(pp) for (process_name, pp) in aping['payload'].get('processes', {}).items() if 'keyedHistograms' in pp] + problem_combos = set.intersection(*(child_tuples + [parent_tuples])) if len(child_tuples) else set() + return problem_combos +
problem_combos = pings.flatMap(get_problem_combos) +
problem_combos.cache() +
PythonRDD[15] at RDD at PythonRDD.scala:48 +
Alright, let’s get a list of the most commonly-seen histograms:
+sorted(problem_combos.map(lambda c: (c[0], 1)).countByKey().iteritems(), key=lambda x: x[1], reverse=True) +
[(u'IPC_MESSAGE_SIZE', 396905), + (u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', 72248), + (u'SYNC_WORKER_OPERATION', 47653), + (u'MESSAGE_MANAGER_MESSAGE_SIZE2', 35884), + (u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', 13846), + (u'MEDIA_CODEC_USED', 1030), + (u'CANVAS_WEBGL_FAILURE_ID', 289), + (u'VIDEO_INFERRED_DECODE_SUSPEND_PERCENTAGE', 288), + (u'VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE', 288), + (u'VIDEO_INTER_KEYFRAME_MAX_MS', 208), + (u'CANVAS_WEBGL_ACCL_FAILURE_ID', 183), + (u'JS_TELEMETRY_ADDON_EXCEPTIONS', 150), + (u'VIDEO_SUSPEND_RECOVERY_TIME_MS', 117), + (u'VIDEO_INTER_KEYFRAME_AVERAGE_MS', 111), + (u'PRINT_DIALOG_OPENED_COUNT', 4), + (u'PRINT_COUNT', 2)] +
More verbosely, what are the 20 most-commonly-seen histogram,key pairs:
+sorted(problem_combos.map(lambda c: (c, 1)).countByKey().iteritems(), key=lambda x: x[1], reverse=True)[:20] +
[((u'IPC_MESSAGE_SIZE', u'PLayerTransaction::Msg_Update'), 185499), + ((u'IPC_MESSAGE_SIZE', u'PBrowser::Msg_AsyncMessage'), 133954), + ((u'IPC_MESSAGE_SIZE', u'PLayerTransaction::Msg_UpdateNoSwap'), 64489), + ((u'SYNC_WORKER_OPERATION', u'WorkerCheckAPIExposureOnMainThread'), 41428), + ((u'MESSAGE_MANAGER_MESSAGE_SIZE2', u'SessionStore:update'), 24408), + ((u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', u'Shockwave Flash23.0.0.185'), 21854), + ((u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', u'Shockwave Flash23.0.0.205'), 18713), + ((u'IPC_MESSAGE_SIZE', u'PContent::Msg_AsyncMessage'), 12066), + ((u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', u'Shockwave Flash23.0.0.162'), 11700), + ((u'MESSAGE_MANAGER_MESSAGE_SIZE2', u'sdk/remote/process/message'), 7776), + ((u'SYNC_WORKER_OPERATION', u'XHR'), 5866), + ((u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', u'Shockwave Flash23.0.0.207'), 4580), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'flb,r'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'dl,flb'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'dl'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'flb'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'r'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'dl,r'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'dl,flb,r'), 1978), + ((u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', u'Shockwave Flash22.0.0.209'), 1642)] +
From earlier we note that IPC_MESSAGE_SIZE/PLayerTransaction::Msg_Update is the most common “present on multiple processes” combination.
To see if we’ve had this problem the whole time, how many pings have these messages in both parent and content, and whose histograms have identical sums?
+def relevant_ping(p): + parent = p.get('payload', {}).get('keyedHistograms', {}).get('IPC_MESSAGE_SIZE', {}).get('PLayerTransaction::Msg_Update') + content = p.get('payload', {}).get('processes', {}).get('content', {}).get('keyedHistograms', {}).get('IPC_MESSAGE_SIZE', {}).get('PLayerTransaction::Msg_Update') + return parent is not None and content is not None and parent['sum'] == content['sum'] + +relevant_pings = pings.filter(relevant_ping) +
relevant_pings.count() +
149126 +
Yup, it appears as though we’ve had this problem since nightly/52.
+modern_pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(submissionDate="20170716") \ + .records(sc, sample=0.01) +
fetching 7012.25715MB in 1970 files... +
modern_combos = modern_pings.flatMap(get_problem_combos) +
modern_combos.cache() +
PythonRDD[51] at RDD at PythonRDD.scala:48 +
sorted(modern_combos.map(lambda c: (c[0], 1)).countByKey().iteritems(), key=lambda x: x[1], reverse=True) +
[(u'NOTIFY_OBSERVERS_LATENCY_MS', 72463), + (u'DOM_SCRIPT_SRC_ENCODING', 33021), + (u'FX_SESSION_RESTORE_CONTENT_COLLECT_DATA_MS', 30709), + (u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', 11613), + (u'IPC_WRITE_MAIN_THREAD_LATENCY_MS', 11186), + (u'MAIN_THREAD_RUNNABLE_MS', 7872), + (u'IPC_READ_MAIN_THREAD_LATENCY_MS', 6646), + (u'SYNC_WORKER_OPERATION', 5614), + (u'IPC_SYNC_RECEIVE_MS', 4227), + (u'IPC_MESSAGE_SIZE', 3514), + (u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', 2377), + (u'IPC_SYNC_MESSAGE_MANAGER_LATENCY_MS', 902), + (u'IPC_SYNC_MAIN_LATENCY_MS', 833), + (u'IDLE_RUNNABLE_BUDGET_OVERUSE_MS', 701), + (u'MESSAGE_MANAGER_MESSAGE_SIZE2', 615), + (u'FX_TAB_REMOTE_NAVIGATION_DELAY_MS', 433), + (u'CANVAS_WEBGL_FAILURE_ID', 138), + (u'CANVAS_WEBGL_ACCL_FAILURE_ID', 110), + (u'MEDIA_CODEC_USED', 20), + (u'IPC_SYNC_LATENCY_MS', 9), + (u'VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE', 8), + (u'VIDEO_INFERRED_DECODE_SUSPEND_PERCENTAGE', 8), + (u'PRINT_DIALOG_OPENED_COUNT', 2)] +
sorted(modern_combos.map(lambda c: (c, 1)).countByKey().iteritems(), key=lambda x: x[1], reverse=True)[:20] +
[((u'DOM_SCRIPT_SRC_ENCODING', u'UTF-8'), 16824), + ((u'DOM_SCRIPT_SRC_ENCODING', u'windows-1252'), 16165), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'cycle-collector-begin'), 13727), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'garbage-collection-statistics'), 13150), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'cycle-collector-forget-skippable'), + 12719), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'inner-window-destroyed'), 8619), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'tab-content-frameloader-created'), 7924), + ((u'FX_SESSION_RESTORE_CONTENT_COLLECT_DATA_MS', u'historychange'), 7537), + ((u'FX_SESSION_RESTORE_CONTENT_COLLECT_DATA_MS', u'pageStyle'), 7390), + ((u'FX_SESSION_RESTORE_CONTENT_COLLECT_DATA_MS', u'scroll'), 7389), + ((u'FX_SESSION_RESTORE_CONTENT_COLLECT_DATA_MS', u'storage'), 7380), + ((u'IPC_WRITE_MAIN_THREAD_LATENCY_MS', u'PLayerTransaction::Msg_Update'), + 6284), + ((u'SYNC_WORKER_OPERATION', u'WorkerCheckAPIExposureOnMainThread'), 4926), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'content-document-global-created'), 4486), + ((u'IPC_SYNC_RECEIVE_MS', u'???'), 4227), + ((u'IPC_READ_MAIN_THREAD_LATENCY_MS', u'PCompositorBridge::Msg_DidComposite'), + 3523), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'document-element-inserted'), 3498), + ((u'IPC_WRITE_MAIN_THREAD_LATENCY_MS', + u'PCompositorBridge::Msg_PTextureConstructor'), + 2231), + ((u'IPC_MESSAGE_SIZE', u'PBrowser::Msg_AsyncMessage'), 2083), + ((u'IPC_READ_MAIN_THREAD_LATENCY_MS', u'PBrowser::Msg_AsyncMessage'), 2031)] +
The behaviour still exists, though this suggests that plugins and ipc messages are now less common. Instead we see more latency probes.
+In bug 1380880 :billm found that keyed histograms recorded on different processes would be aggregated together if their keys matched.
+How often does this happen in practice? How long has this been happening?
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +
The whole child-process client aggregation thing was introduced by bug 1218576 back in September of 2016 for Firefox 52. So that’s the earliest this could have started.
+pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appVersion=lambda x: x.startswith("52")) \ + .where(appUpdateChannel="nightly") \ + .records(sc, sample=0.1) +
fetching 13254.61440MB in 54449 files... +
def set_of_hgram_key_tuples(payload): + return set((kh_name, key) for (kh_name, v) in payload['keyedHistograms'].items() for key in v.keys()) + +def get_problem_combos(aping): + parent_tuples = set_of_hgram_key_tuples(aping['payload']) + child_tuples = [set_of_hgram_key_tuples(pp) for (process_name, pp) in aping['payload'].get('processes', {}).items() if 'keyedHistograms' in pp] + problem_combos = set.intersection(*(child_tuples + [parent_tuples])) if len(child_tuples) else set() + return problem_combos +
problem_combos = pings.flatMap(get_problem_combos) +
problem_combos.cache() +
PythonRDD[15] at RDD at PythonRDD.scala:48 +
Alright, let’s get a list of the most commonly-seen histograms:
+sorted(problem_combos.map(lambda c: (c[0], 1)).countByKey().iteritems(), key=lambda x: x[1], reverse=True) +
[(u'IPC_MESSAGE_SIZE', 396905), + (u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', 72248), + (u'SYNC_WORKER_OPERATION', 47653), + (u'MESSAGE_MANAGER_MESSAGE_SIZE2', 35884), + (u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', 13846), + (u'MEDIA_CODEC_USED', 1030), + (u'CANVAS_WEBGL_FAILURE_ID', 289), + (u'VIDEO_INFERRED_DECODE_SUSPEND_PERCENTAGE', 288), + (u'VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE', 288), + (u'VIDEO_INTER_KEYFRAME_MAX_MS', 208), + (u'CANVAS_WEBGL_ACCL_FAILURE_ID', 183), + (u'JS_TELEMETRY_ADDON_EXCEPTIONS', 150), + (u'VIDEO_SUSPEND_RECOVERY_TIME_MS', 117), + (u'VIDEO_INTER_KEYFRAME_AVERAGE_MS', 111), + (u'PRINT_DIALOG_OPENED_COUNT', 4), + (u'PRINT_COUNT', 2)] +
More verbosely, what are the 20 most-commonly-seen histogram,key pairs:
+sorted(problem_combos.map(lambda c: (c, 1)).countByKey().iteritems(), key=lambda x: x[1], reverse=True)[:20] +
[((u'IPC_MESSAGE_SIZE', u'PLayerTransaction::Msg_Update'), 185499), + ((u'IPC_MESSAGE_SIZE', u'PBrowser::Msg_AsyncMessage'), 133954), + ((u'IPC_MESSAGE_SIZE', u'PLayerTransaction::Msg_UpdateNoSwap'), 64489), + ((u'SYNC_WORKER_OPERATION', u'WorkerCheckAPIExposureOnMainThread'), 41428), + ((u'MESSAGE_MANAGER_MESSAGE_SIZE2', u'SessionStore:update'), 24408), + ((u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', u'Shockwave Flash23.0.0.185'), 21854), + ((u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', u'Shockwave Flash23.0.0.205'), 18713), + ((u'IPC_MESSAGE_SIZE', u'PContent::Msg_AsyncMessage'), 12066), + ((u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', u'Shockwave Flash23.0.0.162'), 11700), + ((u'MESSAGE_MANAGER_MESSAGE_SIZE2', u'sdk/remote/process/message'), 7776), + ((u'SYNC_WORKER_OPERATION', u'XHR'), 5866), + ((u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', u'Shockwave Flash23.0.0.207'), 4580), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'flb,r'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'dl,flb'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'dl'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'flb'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'r'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'dl,r'), 1978), + ((u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', u'dl,flb,r'), 1978), + ((u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', u'Shockwave Flash22.0.0.209'), 1642)] +
From earlier we note that IPC_MESSAGE_SIZE/PLayerTransaction::Msg_Update is the most common “present on multiple processes” combination.
To see if we’ve had this problem the whole time, how many pings have these messages in both parent and content, and whose histograms have identical sums?
+def relevant_ping(p): + parent = p.get('payload', {}).get('keyedHistograms', {}).get('IPC_MESSAGE_SIZE', {}).get('PLayerTransaction::Msg_Update') + content = p.get('payload', {}).get('processes', {}).get('content', {}).get('keyedHistograms', {}).get('IPC_MESSAGE_SIZE', {}).get('PLayerTransaction::Msg_Update') + return parent is not None and content is not None and parent['sum'] == content['sum'] + +relevant_pings = pings.filter(relevant_ping) +
relevant_pings.count() +
149126 +
Yup, it appears as though we’ve had this problem since nightly/52.
+modern_pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(submissionDate="20170716") \ + .records(sc, sample=0.01) +
fetching 7012.25715MB in 1970 files... +
modern_combos = modern_pings.flatMap(get_problem_combos) +
modern_combos.cache() +
PythonRDD[51] at RDD at PythonRDD.scala:48 +
sorted(modern_combos.map(lambda c: (c[0], 1)).countByKey().iteritems(), key=lambda x: x[1], reverse=True) +
[(u'NOTIFY_OBSERVERS_LATENCY_MS', 72463), + (u'DOM_SCRIPT_SRC_ENCODING', 33021), + (u'FX_SESSION_RESTORE_CONTENT_COLLECT_DATA_MS', 30709), + (u'CONTENT_LARGE_PAINT_PHASE_WEIGHT', 11613), + (u'IPC_WRITE_MAIN_THREAD_LATENCY_MS', 11186), + (u'MAIN_THREAD_RUNNABLE_MS', 7872), + (u'IPC_READ_MAIN_THREAD_LATENCY_MS', 6646), + (u'SYNC_WORKER_OPERATION', 5614), + (u'IPC_SYNC_RECEIVE_MS', 4227), + (u'IPC_MESSAGE_SIZE', 3514), + (u'BLOCKED_ON_PLUGIN_MODULE_INIT_MS', 2377), + (u'IPC_SYNC_MESSAGE_MANAGER_LATENCY_MS', 902), + (u'IPC_SYNC_MAIN_LATENCY_MS', 833), + (u'IDLE_RUNNABLE_BUDGET_OVERUSE_MS', 701), + (u'MESSAGE_MANAGER_MESSAGE_SIZE2', 615), + (u'FX_TAB_REMOTE_NAVIGATION_DELAY_MS', 433), + (u'CANVAS_WEBGL_FAILURE_ID', 138), + (u'CANVAS_WEBGL_ACCL_FAILURE_ID', 110), + (u'MEDIA_CODEC_USED', 20), + (u'IPC_SYNC_LATENCY_MS', 9), + (u'VIDEO_HIDDEN_PLAY_TIME_PERCENTAGE', 8), + (u'VIDEO_INFERRED_DECODE_SUSPEND_PERCENTAGE', 8), + (u'PRINT_DIALOG_OPENED_COUNT', 2)] +
sorted(modern_combos.map(lambda c: (c, 1)).countByKey().iteritems(), key=lambda x: x[1], reverse=True)[:20] +
[((u'DOM_SCRIPT_SRC_ENCODING', u'UTF-8'), 16824), + ((u'DOM_SCRIPT_SRC_ENCODING', u'windows-1252'), 16165), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'cycle-collector-begin'), 13727), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'garbage-collection-statistics'), 13150), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'cycle-collector-forget-skippable'), + 12719), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'inner-window-destroyed'), 8619), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'tab-content-frameloader-created'), 7924), + ((u'FX_SESSION_RESTORE_CONTENT_COLLECT_DATA_MS', u'historychange'), 7537), + ((u'FX_SESSION_RESTORE_CONTENT_COLLECT_DATA_MS', u'pageStyle'), 7390), + ((u'FX_SESSION_RESTORE_CONTENT_COLLECT_DATA_MS', u'scroll'), 7389), + ((u'FX_SESSION_RESTORE_CONTENT_COLLECT_DATA_MS', u'storage'), 7380), + ((u'IPC_WRITE_MAIN_THREAD_LATENCY_MS', u'PLayerTransaction::Msg_Update'), + 6284), + ((u'SYNC_WORKER_OPERATION', u'WorkerCheckAPIExposureOnMainThread'), 4926), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'content-document-global-created'), 4486), + ((u'IPC_SYNC_RECEIVE_MS', u'???'), 4227), + ((u'IPC_READ_MAIN_THREAD_LATENCY_MS', u'PCompositorBridge::Msg_DidComposite'), + 3523), + ((u'NOTIFY_OBSERVERS_LATENCY_MS', u'document-element-inserted'), 3498), + ((u'IPC_WRITE_MAIN_THREAD_LATENCY_MS', + u'PCompositorBridge::Msg_PTextureConstructor'), + 2231), + ((u'IPC_MESSAGE_SIZE', u'PBrowser::Msg_AsyncMessage'), 2083), + ((u'IPC_READ_MAIN_THREAD_LATENCY_MS', u'PBrowser::Msg_AsyncMessage'), 2031)] +
The behaviour still exists, though this suggests that plugins and ipc messages are now less common. Instead we see more latency probes.
+import datetime as dt +import os +import pandas as pd +import operator +import ujson as json +from pyspark.sql.types import * + +from moztelemetry import get_pings, get_pings_properties, get_one_ping_per_client + +%pylab inline +
Take the set of pings, make sure we have actual clientIds and remove duplicate pings.
+def safe_str(obj): + """ return the byte string representation of obj """ + if obj is None: + return unicode("") + return unicode(obj) + +def dedupe_pings(rdd): + return rdd.filter(lambda p: p["meta/clientId"] is not None)\ + .map(lambda p: (p["meta/documentId"], p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) + +def dedupe_addons(rdd): + return rdd.map(lambda p: (p[0] + safe_str(p[2]) + safe_str(p[3]), p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) +
We’re going to dump each event from the pings. Do a little empty data sanitization so we don’t get NoneType errors during the dump. We create a JSON array of active experiments as part of the dump.
+def clean(s): + try: + s = s.decode("ascii").strip() + return s if len(s) > 0 else None + except: + return None + +def transform(ping): + output = [] + + # These should not be None since we filter those out & ingestion process adds the data + clientId = ping["meta/clientId"] + submissionDate = dt.datetime.strptime(ping["meta/submissionDate"], "%Y%m%d") + + addonset = {} + addons = ping["environment/addons/activeAddons"] + if addons is not None: + for addon, desc in addons.iteritems(): + name = clean(desc.get("name", None)) + if name is not None: + addonset[name] = 1 + + persona = ping["environment/addons/persona"] + + if len(addonset) > 0 or persona is not None: + addonarray = None + if len(addonset) > 0: + addonarray = json.dumps(addonset.keys()) + output.append([clientId, submissionDate, addonarray, persona]) + + return output +
Create a set of events from “saved-session” UI telemetry. Output the data to CSV or Parquet.
+This script is designed to loop over a range of days and output a single day for the given channels. Use explicit date ranges for backfilling, or now() - ‘1day’ for automated runs.
+channels = ["nightly", "aurora", "beta", "release"] + +batch_date = os.environ.get('date') +if batch_date: + start = end = dt.datetime.strptime(batch_date, '%Y%m%d') +else: + start = start = dt.datetime.now() - dt.timedelta(1) + +day = start +while day <= end: + for channel in channels: + print "\nchannel: " + channel + ", date: " + day.strftime("%Y%m%d") + + pings = get_pings(sc, app="Fennec", channel=channel, + submission_date=(day.strftime("%Y%m%d"), day.strftime("%Y%m%d")), + build_id=("20100101000000", "99999999999999"), + fraction=1) + + subset = get_pings_properties(pings, ["meta/clientId", + "meta/documentId", + "meta/submissionDate", + "environment/addons/activeAddons", + "environment/addons/persona"]) + + subset = dedupe_pings(subset) + print subset.first() + + rawAddons = subset.flatMap(transform) + print "\nrawAddons count: " + str(rawAddons.count()) + print rawAddons.first() + + uniqueAddons = dedupe_addons(rawAddons) + print "\nuniqueAddons count: " + str(uniqueAddons.count()) + print uniqueAddons.first() + + s3_output = "s3n://net-mozaws-prod-us-west-2-pipeline-analysis/mobile/android_addons" + s3_output += "/v1/channel=" + channel + "/submission=" + day.strftime("%Y%m%d") + schema = StructType([ + StructField("clientid", StringType(), False), + StructField("submissiondate", TimestampType(), False), + StructField("addons", StringType(), True), + StructField("lwt", StringType(), True) + ]) + grouped = sqlContext.createDataFrame(uniqueAddons, schema) + grouped.coalesce(1).write.parquet(s3_output, mode="overwrite") + + day += dt.timedelta(1) +
import datetime as dt +import os +import pandas as pd +import operator +import ujson as json +from pyspark.sql.types import * + +from moztelemetry import get_pings, get_pings_properties, get_one_ping_per_client + +%pylab inline +
Take the set of pings, make sure we have actual clientIds and remove duplicate pings.
+def safe_str(obj): + """ return the byte string representation of obj """ + if obj is None: + return unicode("") + return unicode(obj) + +def dedupe_pings(rdd): + return rdd.filter(lambda p: p["meta/clientId"] is not None)\ + .map(lambda p: (p["meta/documentId"], p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) + +def dedupe_addons(rdd): + return rdd.map(lambda p: (p[0] + safe_str(p[2]) + safe_str(p[3]), p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) +
We’re going to dump each event from the pings. Do a little empty data sanitization so we don’t get NoneType errors during the dump. We create a JSON array of active experiments as part of the dump.
+def clean(s): + try: + s = s.decode("ascii").strip() + return s if len(s) > 0 else None + except: + return None + +def transform(ping): + output = [] + + # These should not be None since we filter those out & ingestion process adds the data + clientId = ping["meta/clientId"] + submissionDate = dt.datetime.strptime(ping["meta/submissionDate"], "%Y%m%d") + + addonset = {} + addons = ping["environment/addons/activeAddons"] + if addons is not None: + for addon, desc in addons.iteritems(): + name = clean(desc.get("name", None)) + if name is not None: + addonset[name] = 1 + + persona = ping["environment/addons/persona"] + + if len(addonset) > 0 or persona is not None: + addonarray = None + if len(addonset) > 0: + addonarray = json.dumps(addonset.keys()) + output.append([clientId, submissionDate, addonarray, persona]) + + return output +
Create a set of events from “saved-session” UI telemetry. Output the data to CSV or Parquet.
+This script is designed to loop over a range of days and output a single day for the given channels. Use explicit date ranges for backfilling, or now() - ‘1day’ for automated runs.
+channels = ["nightly", "aurora", "beta", "release"] + +batch_date = os.environ.get('date') +if batch_date: + start = end = dt.datetime.strptime(batch_date, '%Y%m%d') +else: + start = start = dt.datetime.now() - dt.timedelta(1) + +day = start +while day <= end: + for channel in channels: + print "\nchannel: " + channel + ", date: " + day.strftime("%Y%m%d") + + pings = get_pings(sc, app="Fennec", channel=channel, + submission_date=(day.strftime("%Y%m%d"), day.strftime("%Y%m%d")), + build_id=("20100101000000", "99999999999999"), + fraction=1) + + subset = get_pings_properties(pings, ["meta/clientId", + "meta/documentId", + "meta/submissionDate", + "environment/addons/activeAddons", + "environment/addons/persona"]) + + subset = dedupe_pings(subset) + print subset.first() + + rawAddons = subset.flatMap(transform) + print "\nrawAddons count: " + str(rawAddons.count()) + print rawAddons.first() + + uniqueAddons = dedupe_addons(rawAddons) + print "\nuniqueAddons count: " + str(uniqueAddons.count()) + print uniqueAddons.first() + + s3_output = "s3n://net-mozaws-prod-us-west-2-pipeline-analysis/mobile/android_addons" + s3_output += "/v1/channel=" + channel + "/submission=" + day.strftime("%Y%m%d") + schema = StructType([ + StructField("clientid", StringType(), False), + StructField("submissiondate", TimestampType(), False), + StructField("addons", StringType(), True), + StructField("lwt", StringType(), True) + ]) + grouped = sqlContext.createDataFrame(uniqueAddons, schema) + grouped.coalesce(1).write.parquet(s3_output, mode="overwrite") + + day += dt.timedelta(1) +
import datetime as dt +import os +import pandas as pd +import ujson as json +from pyspark.sql.types import * + +from moztelemetry import get_pings, get_pings_properties + +%pylab inline +
Take the set of pings, make sure we have actual clientIds and remove duplicate pings. We collect each unique ping.
+def dedupe_pings(rdd): + return rdd.filter(lambda p: p["meta/clientId"] is not None)\ + .map(lambda p: (p["meta/documentId"], p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) +
Transform and sanitize the pings into arrays.
+# bug 1362659 - int values exceeded signed 32 bit range +MAX_INT = (2**31)-1 + +def transform(ping): + # Should not be None since we filter those out. + clientId = ping["meta/clientId"] + + profileDate = None + profileDaynum = ping["environment/profile/creationDate"] + if profileDaynum is not None: + try: + # Bad data could push profileDaynum > 32767 (size of a C int) and throw exception + profileDate = dt.datetime(1970, 1, 1) + dt.timedelta(int(profileDaynum)) + except: + profileDate = None + + # Create date should already be in ISO format + creationDate = ping["creationDate"] + if creationDate is not None: + # This is only accurate because we know the creation date is always in 'Z' (zulu) time. + creationDate = dt.datetime.strptime(ping["creationDate"], "%Y-%m-%dT%H:%M:%S.%fZ") + + # Added via the ingestion process so should not be None. + submissionDate = dt.datetime.strptime(ping["meta/submissionDate"], "%Y%m%d") + + appVersion = ping["application/version"] + osVersion = ping["environment/system/os/version"] + if osVersion is not None: + osVersion = int(osVersion) if int(osVersion) <= MAX_INT else None + + locale = ping["environment/settings/locale"] + + # Truncate to 32 characters + defaultSearch = ping["environment/settings/defaultSearchEngine"] + if defaultSearch is not None: + defaultSearch = defaultSearch[0:32] + + # Build up the device string, truncating like we do in 'core' ping. + device = ping["environment/system/device/manufacturer"] + model = ping["environment/system/device/model"] + if device is not None and model is not None: + device = device[0:12] + "-" + model[0:19] + + xpcomABI = ping["application/xpcomAbi"] + arch = "arm" + if xpcomABI is not None and "x86" in xpcomABI: + arch = "x86" + + # Bug 1337896 + as_topsites_loader_time = ping["payload/histograms/FENNEC_ACTIVITY_STREAM_TOPSITES_LOADER_TIME_MS"] + topsites_loader_time = ping["payload/histograms/FENNEC_TOPSITES_LOADER_TIME_MS"] + + if as_topsites_loader_time is not None: + as_topsites_loader_time = map(int, as_topsites_loader_time.tolist()) + if any([v > MAX_INT for v in as_topsites_loader_time]): + as_topsites_loader_time = None + + if topsites_loader_time is not None: + topsites_loader_time = map(int, topsites_loader_time.tolist()) + if any([v > MAX_INT for v in topsites_loader_time]): + topsites_loader_time = None + + return [clientId, + profileDate, + submissionDate, + creationDate, + appVersion, + osVersion, + locale, + defaultSearch, + device, + arch, + as_topsites_loader_time, + topsites_loader_time] +
Create a set of pings from “saved-session” to build a set of core client data. Output the data to CSV or Parquet.
+This script is designed to loop over a range of days and output a single day for the given channels. Use explicit date ranges for backfilling, or now() - ‘1day’ for automated runs.
+channels = ["nightly", "aurora", "beta", "release"] + +batch_date = os.environ.get('date') +if batch_date: + start = end = dt.datetime.strptime(batch_date, '%Y%m%d') +else: + start = end = dt.datetime.now() - dt.timedelta(1) + +day = start +while day <= end: + for channel in channels: + print "\nchannel: " + channel + ", date: " + day.strftime("%Y%m%d") + + pings = get_pings(sc, app="Fennec", channel=channel, + submission_date=(day.strftime("%Y%m%d"), day.strftime("%Y%m%d")), + build_id=("20100101000000", "99999999999999"), + fraction=1) + + subset = get_pings_properties(pings, ["meta/clientId", + "meta/documentId", + "meta/submissionDate", + "creationDate", + "application/version", + "environment/system/os/version", + "environment/profile/creationDate", + "environment/settings/locale", + "environment/settings/defaultSearchEngine", + "environment/system/device/model", + "environment/system/device/manufacturer", + "application/xpcomAbi", + "payload/histograms/FENNEC_ACTIVITY_STREAM_TOPSITES_LOADER_TIME_MS", + "payload/histograms/FENNEC_TOPSITES_LOADER_TIME_MS"]) + + subset = dedupe_pings(subset) + transformed = subset.map(transform) + + s3_output = "s3n://net-mozaws-prod-us-west-2-pipeline-analysis/mobile/android_clients" + s3_output += "/v2/channel=" + channel + "/submission=" + day.strftime("%Y%m%d") + schema = StructType([ + StructField("clientid", StringType(), False), + StructField("profiledate", TimestampType(), True), + StructField("submissiondate", TimestampType(), False), + StructField("creationdate", TimestampType(), True), + StructField("appversion", StringType(), True), + StructField("osversion", IntegerType(), True), + StructField("locale", StringType(), True), + StructField("defaultsearch", StringType(), True), + StructField("device", StringType(), True), + StructField("arch", StringType(), True), + StructField("fennec_activity_stream_topsites_loader_time_ms", + ArrayType(IntegerType()), + True + ), + StructField("fennec_topsites_loader_time_ms", + ArrayType(IntegerType()), + True + ) + ]) + grouped = sqlContext.createDataFrame(transformed, schema) + grouped.coalesce(1).write.parquet(s3_output, mode="overwrite") + + day += dt.timedelta(1) +
import datetime as dt +import os +import pandas as pd +import ujson as json +from pyspark.sql.types import * + +from moztelemetry import get_pings, get_pings_properties + +%pylab inline +
Take the set of pings, make sure we have actual clientIds and remove duplicate pings. We collect each unique ping.
+def dedupe_pings(rdd): + return rdd.filter(lambda p: p["meta/clientId"] is not None)\ + .map(lambda p: (p["meta/documentId"], p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) +
Transform and sanitize the pings into arrays.
+# bug 1362659 - int values exceeded signed 32 bit range +MAX_INT = (2**31)-1 + +def transform(ping): + # Should not be None since we filter those out. + clientId = ping["meta/clientId"] + + profileDate = None + profileDaynum = ping["environment/profile/creationDate"] + if profileDaynum is not None: + try: + # Bad data could push profileDaynum > 32767 (size of a C int) and throw exception + profileDate = dt.datetime(1970, 1, 1) + dt.timedelta(int(profileDaynum)) + except: + profileDate = None + + # Create date should already be in ISO format + creationDate = ping["creationDate"] + if creationDate is not None: + # This is only accurate because we know the creation date is always in 'Z' (zulu) time. + creationDate = dt.datetime.strptime(ping["creationDate"], "%Y-%m-%dT%H:%M:%S.%fZ") + + # Added via the ingestion process so should not be None. + submissionDate = dt.datetime.strptime(ping["meta/submissionDate"], "%Y%m%d") + + appVersion = ping["application/version"] + osVersion = ping["environment/system/os/version"] + if osVersion is not None: + osVersion = int(osVersion) if int(osVersion) <= MAX_INT else None + + locale = ping["environment/settings/locale"] + + # Truncate to 32 characters + defaultSearch = ping["environment/settings/defaultSearchEngine"] + if defaultSearch is not None: + defaultSearch = defaultSearch[0:32] + + # Build up the device string, truncating like we do in 'core' ping. + device = ping["environment/system/device/manufacturer"] + model = ping["environment/system/device/model"] + if device is not None and model is not None: + device = device[0:12] + "-" + model[0:19] + + xpcomABI = ping["application/xpcomAbi"] + arch = "arm" + if xpcomABI is not None and "x86" in xpcomABI: + arch = "x86" + + # Bug 1337896 + as_topsites_loader_time = ping["payload/histograms/FENNEC_ACTIVITY_STREAM_TOPSITES_LOADER_TIME_MS"] + topsites_loader_time = ping["payload/histograms/FENNEC_TOPSITES_LOADER_TIME_MS"] + + if as_topsites_loader_time is not None: + as_topsites_loader_time = map(int, as_topsites_loader_time.tolist()) + if any([v > MAX_INT for v in as_topsites_loader_time]): + as_topsites_loader_time = None + + if topsites_loader_time is not None: + topsites_loader_time = map(int, topsites_loader_time.tolist()) + if any([v > MAX_INT for v in topsites_loader_time]): + topsites_loader_time = None + + return [clientId, + profileDate, + submissionDate, + creationDate, + appVersion, + osVersion, + locale, + defaultSearch, + device, + arch, + as_topsites_loader_time, + topsites_loader_time] +
Create a set of pings from “saved-session” to build a set of core client data. Output the data to CSV or Parquet.
+This script is designed to loop over a range of days and output a single day for the given channels. Use explicit date ranges for backfilling, or now() - ‘1day’ for automated runs.
+channels = ["nightly", "aurora", "beta", "release"] + +batch_date = os.environ.get('date') +if batch_date: + start = end = dt.datetime.strptime(batch_date, '%Y%m%d') +else: + start = end = dt.datetime.now() - dt.timedelta(1) + +day = start +while day <= end: + for channel in channels: + print "\nchannel: " + channel + ", date: " + day.strftime("%Y%m%d") + + pings = get_pings(sc, app="Fennec", channel=channel, + submission_date=(day.strftime("%Y%m%d"), day.strftime("%Y%m%d")), + build_id=("20100101000000", "99999999999999"), + fraction=1) + + subset = get_pings_properties(pings, ["meta/clientId", + "meta/documentId", + "meta/submissionDate", + "creationDate", + "application/version", + "environment/system/os/version", + "environment/profile/creationDate", + "environment/settings/locale", + "environment/settings/defaultSearchEngine", + "environment/system/device/model", + "environment/system/device/manufacturer", + "application/xpcomAbi", + "payload/histograms/FENNEC_ACTIVITY_STREAM_TOPSITES_LOADER_TIME_MS", + "payload/histograms/FENNEC_TOPSITES_LOADER_TIME_MS"]) + + subset = dedupe_pings(subset) + transformed = subset.map(transform) + + s3_output = "s3n://net-mozaws-prod-us-west-2-pipeline-analysis/mobile/android_clients" + s3_output += "/v2/channel=" + channel + "/submission=" + day.strftime("%Y%m%d") + schema = StructType([ + StructField("clientid", StringType(), False), + StructField("profiledate", TimestampType(), True), + StructField("submissiondate", TimestampType(), False), + StructField("creationdate", TimestampType(), True), + StructField("appversion", StringType(), True), + StructField("osversion", IntegerType(), True), + StructField("locale", StringType(), True), + StructField("defaultsearch", StringType(), True), + StructField("device", StringType(), True), + StructField("arch", StringType(), True), + StructField("fennec_activity_stream_topsites_loader_time_ms", + ArrayType(IntegerType()), + True + ), + StructField("fennec_topsites_loader_time_ms", + ArrayType(IntegerType()), + True + ) + ]) + grouped = sqlContext.createDataFrame(transformed, schema) + grouped.coalesce(1).write.parquet(s3_output, mode="overwrite") + + day += dt.timedelta(1) +
import datetime as dt +import os +import pandas as pd +import ujson as json +from pyspark.sql.types import * + +from moztelemetry import get_pings, get_pings_properties, get_one_ping_per_client + +%pylab inline +
Take the set of pings, make sure we have actual clientIds and remove duplicate pings.
+def dedupe_pings(rdd): + return rdd.filter(lambda p: p["meta/clientId"] is not None)\ + .map(lambda p: (p["meta/documentId"], p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) +
We’re going to dump each event from the pings. Do a little empty data sanitization so we don’t get NoneType errors during the dump. We create a JSON array of active experiments as part of the dump.
+def safe_str(obj): + """ return the byte string representation of obj """ + if obj is None: + return unicode("") + return unicode(obj) + +def transform(ping): + output = [] + + # These should not be None since we filter those out & ingestion process adds the data + clientId = ping["meta/clientId"] + submissionDate = dt.datetime.strptime(ping["meta/submissionDate"], "%Y%m%d") + + events = ping["payload/UIMeasurements"] + if events and isinstance(events, list): + for event in events: + if isinstance(event, dict) and "type" in event and event["type"] == "event": + if "timestamp" not in event or "action" not in event or "method" not in event or "sessions" not in event: + continue + + # Verify timestamp is a long, otherwise ignore the event + timestamp = None + try: + timestamp = long(event["timestamp"]) + except: + continue + + # Force all fields to strings + action = safe_str(event["action"]) + method = safe_str(event["method"]) + + # The extras is an optional field + extras = unicode("") + if "extras" in event and safe_str(event["extras"]) is not None: + extras = safe_str(event["extras"]) + + sessions = set() + experiments = [] + + try: + for session in event["sessions"]: + if "experiment.1:" in session: + experiments.append(safe_str(session[13:])) + else: + sessions.add(safe_str(session)) + except TypeError: + pass + + output.append([clientId, submissionDate, timestamp, action, method, extras, json.dumps(list(sessions)), json.dumps(experiments)]) + + return output +
The data can have duplicate events, due to a bug in the data collection that was fixed (bug 1246973). We still need to de-dupe the events. Because pings can be archived on device and submitted on later days, we can’t assume dupes only happen on the same submission day. We don’t use submission date when de-duping.
+def dedupe_events(rdd): + return rdd.map(lambda p: (p[0] + safe_str(p[2]) + p[3] + p[4], p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) +
Create a set of events from “saved-session” UI telemetry. Output the data to CSV or Parquet.
+This script is designed to loop over a range of days and output a single day for the given channels. Use explicit date ranges for backfilling, or now() - ‘1day’ for automated runs.
+channels = ["nightly", "aurora", "beta", "release"] + +batch_date = os.environ.get('date') +if batch_date: + start = end = dt.datetime.strptime(batch_date, '%Y%m%d') +else: + start = start = dt.datetime.now() - dt.timedelta(1) + +day = start +while day <= end: + for channel in channels: + print "\nchannel: " + channel + ", date: " + day.strftime("%Y%m%d") + + pings = get_pings(sc, app="Fennec", channel=channel, + submission_date=(day.strftime("%Y%m%d"), day.strftime("%Y%m%d")), + build_id=("20100101000000", "99999999999999"), + fraction=1) + + subset = get_pings_properties(pings, ["meta/clientId", + "meta/documentId", + "meta/submissionDate", + "payload/UIMeasurements"]) + + subset = dedupe_pings(subset) + print subset.first() + + rawEvents = subset.flatMap(transform) + print "\nRaw count: " + str(rawEvents.count()) + print rawEvents.first() + + uniqueEvents = dedupe_events(rawEvents) + print "\nUnique count: " + str(uniqueEvents.count()) + print uniqueEvents.first() + + s3_output = "s3n://net-mozaws-prod-us-west-2-pipeline-analysis/mobile/android_events" + s3_output += "/v1/channel=" + channel + "/submission=" + day.strftime("%Y%m%d") + schema = StructType([ + StructField("clientid", StringType(), False), + StructField("submissiondate", TimestampType(), False), + StructField("ts", LongType(), True), + StructField("action", StringType(), True), + StructField("method", StringType(), True), + StructField("extras", StringType(), True), + StructField("sessions", StringType(), True), + StructField("experiments", StringType(), True) + ]) + grouped = sqlContext.createDataFrame(uniqueEvents, schema) + grouped.coalesce(1).write.parquet(s3_output, mode="overwrite") + + day += dt.timedelta(1) +
import datetime as dt +import os +import pandas as pd +import ujson as json +from pyspark.sql.types import * + +from moztelemetry import get_pings, get_pings_properties, get_one_ping_per_client + +%pylab inline +
Take the set of pings, make sure we have actual clientIds and remove duplicate pings.
+def dedupe_pings(rdd): + return rdd.filter(lambda p: p["meta/clientId"] is not None)\ + .map(lambda p: (p["meta/documentId"], p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) +
We’re going to dump each event from the pings. Do a little empty data sanitization so we don’t get NoneType errors during the dump. We create a JSON array of active experiments as part of the dump.
+def safe_str(obj): + """ return the byte string representation of obj """ + if obj is None: + return unicode("") + return unicode(obj) + +def transform(ping): + output = [] + + # These should not be None since we filter those out & ingestion process adds the data + clientId = ping["meta/clientId"] + submissionDate = dt.datetime.strptime(ping["meta/submissionDate"], "%Y%m%d") + + events = ping["payload/UIMeasurements"] + if events and isinstance(events, list): + for event in events: + if isinstance(event, dict) and "type" in event and event["type"] == "event": + if "timestamp" not in event or "action" not in event or "method" not in event or "sessions" not in event: + continue + + # Verify timestamp is a long, otherwise ignore the event + timestamp = None + try: + timestamp = long(event["timestamp"]) + except: + continue + + # Force all fields to strings + action = safe_str(event["action"]) + method = safe_str(event["method"]) + + # The extras is an optional field + extras = unicode("") + if "extras" in event and safe_str(event["extras"]) is not None: + extras = safe_str(event["extras"]) + + sessions = set() + experiments = [] + + try: + for session in event["sessions"]: + if "experiment.1:" in session: + experiments.append(safe_str(session[13:])) + else: + sessions.add(safe_str(session)) + except TypeError: + pass + + output.append([clientId, submissionDate, timestamp, action, method, extras, json.dumps(list(sessions)), json.dumps(experiments)]) + + return output +
The data can have duplicate events, due to a bug in the data collection that was fixed (bug 1246973). We still need to de-dupe the events. Because pings can be archived on device and submitted on later days, we can’t assume dupes only happen on the same submission day. We don’t use submission date when de-duping.
+def dedupe_events(rdd): + return rdd.map(lambda p: (p[0] + safe_str(p[2]) + p[3] + p[4], p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) +
Create a set of events from “saved-session” UI telemetry. Output the data to CSV or Parquet.
+This script is designed to loop over a range of days and output a single day for the given channels. Use explicit date ranges for backfilling, or now() - ‘1day’ for automated runs.
+channels = ["nightly", "aurora", "beta", "release"] + +batch_date = os.environ.get('date') +if batch_date: + start = end = dt.datetime.strptime(batch_date, '%Y%m%d') +else: + start = start = dt.datetime.now() - dt.timedelta(1) + +day = start +while day <= end: + for channel in channels: + print "\nchannel: " + channel + ", date: " + day.strftime("%Y%m%d") + + pings = get_pings(sc, app="Fennec", channel=channel, + submission_date=(day.strftime("%Y%m%d"), day.strftime("%Y%m%d")), + build_id=("20100101000000", "99999999999999"), + fraction=1) + + subset = get_pings_properties(pings, ["meta/clientId", + "meta/documentId", + "meta/submissionDate", + "payload/UIMeasurements"]) + + subset = dedupe_pings(subset) + print subset.first() + + rawEvents = subset.flatMap(transform) + print "\nRaw count: " + str(rawEvents.count()) + print rawEvents.first() + + uniqueEvents = dedupe_events(rawEvents) + print "\nUnique count: " + str(uniqueEvents.count()) + print uniqueEvents.first() + + s3_output = "s3n://net-mozaws-prod-us-west-2-pipeline-analysis/mobile/android_events" + s3_output += "/v1/channel=" + channel + "/submission=" + day.strftime("%Y%m%d") + schema = StructType([ + StructField("clientid", StringType(), False), + StructField("submissiondate", TimestampType(), False), + StructField("ts", LongType(), True), + StructField("action", StringType(), True), + StructField("method", StringType(), True), + StructField("extras", StringType(), True), + StructField("sessions", StringType(), True), + StructField("experiments", StringType(), True) + ]) + grouped = sqlContext.createDataFrame(uniqueEvents, schema) + grouped.coalesce(1).write.parquet(s3_output, mode="overwrite") + + day += dt.timedelta(1) +
This script turns the parquet dataset generated by churn notebook into csv files.
+import boto3 +import botocore +import gzip + +from boto3.s3.transfer import S3Transfer +from datetime import datetime, timedelta +from pyspark.sql import functions as F + + +def csv(f): + return ",".join([unicode(a) for a in f]) + +def fmt(d, date_format="%Y%m%d"): + return datetime.strftime(d, date_format) + +def collect_and_upload_csv(df, filename, upload_config): + """ Collect the dataframe into a csv file and upload to target locations. """ + client = boto3.client('s3', 'us-west-2') + transfer = S3Transfer(client) + + print("{}: Writing output to {}".format(datetime.utcnow(), filename)) + + # Write the file out as gzipped csv + with gzip.open(filename, 'wb') as fout: + fout.write(",".join(df.columns) + "\n") + print("{}: Wrote header to {}".format(datetime.utcnow(), filename)) + records = df.rdd.collect() + for r in records: + try: + fout.write(csv(r)) + fout.write("\n") + except UnicodeEncodeError as e: + print("{}: Error writing line: {} // {}".format(datetime.utcnow(), e, r)) + print("{}: finished writing lines".format(datetime.utcnow())) + + # upload files to s3 + try: + for config in upload_config: + print("{}: Uploading to {} at s3://{}/{}/{}".format( + datetime.utcnow(), config["name"], config["bucket"], + config["prefix"], filename)) + + s3_path = "{}/{}".format(config["prefix"], filename) + transfer.upload_file(filename, config["bucket"], s3_path, + extra_args={'ACL': 'bucket-owner-full-control'}) + except botocore.exceptions.ClientError as e: + print("File for {} already exists, skipping upload: {}".format(filename, e)) + + +def marginalize_dataframe(df, attributes, aggregates): + """ Reduce the granularity of the dataset to the original set of attributes. + The original set of attributes can be found on commit 2de3ef1 of mozilla-reports. """ + + return df.groupby(attributes).agg(*[F.sum(x).alias(x) for x in aggregates]) + + +def convert_week(config, week_start=None): + """ Convert a given retention period from parquet to csv. """ + df = spark.read.parquet(config["source"]) + + # find the latest start date based on the dataset if not provided + if not week_start: + start_dates = df.select("week_start").distinct().collect() + week_start = sorted(start_dates)[-1].week_start + + # find the week end for the filename + week_end = fmt(datetime.strptime(week_start, "%Y%m%d") + timedelta(6)) + + print("Running for the week of {} to {}".format(week_start, week_end)) + + # find the target subset of data + df = df.where(df.week_start == week_start) + + # marginalize the dataframe to the original attributes and upload to s3 + initial_attributes = ['channel', 'geo', 'is_funnelcake', + 'acquisition_period', 'start_version', 'sync_usage', + 'current_version', 'current_week', 'is_active'] + initial_aggregates = ['n_profiles', 'usage_hours', 'sum_squared_usage_hours'] + + upload_df = marginalize_dataframe(df, initial_attributes, initial_aggregates) + filename = "churn-{}-{}.by_activity.csv.gz".format(week_start, week_end) + collect_and_upload_csv(upload_df, filename, config["uploads"]) + + # Bug 1355988 + # The size of the data explodes significantly with extra dimensions and is too + # large to fit into the driver memory. We can write directly to s3 from a + # dataframe. + bucket = config['search_cohort']['bucket'] + prefix = config['search_cohort']['prefix'] + location = "s3://{}/{}/week_start={}".format(bucket, prefix, week_start) + + print("Saving additional search cohort churn data to {}".format(location)) + + search_attributes = [ + 'source', 'medium', 'campaign', 'content', + 'distribution_id', 'default_search_engine', 'locale' + ] + attributes = initial_attributes + search_attributes + upload_df = marginalize_dataframe(df, attributes, initial_aggregates) + upload_df.write.csv(location, header=True, mode='overwrite', compression='gzip') + + print("Sucessfully finished churn_to_csv") +
def assert_valid_config(config): + """ Assert that the configuration looks correct. """ + # This could be replaced with python schema's + assert set(["source", "uploads", "search_cohort"]).issubset(config.keys()) + assert set(["bucket", "prefix"]).issubset(config['search_cohort'].keys()) + for entry in config["uploads"]: + assert set(["name", "bucket", "prefix"]).issubset(entry.keys()) +
from moztelemetry.standards import snap_to_beginning_of_week +from os import environ + +config = { + "source": "s3://telemetry-parquet/churn/v2", + "uploads": [ + { + "name": "Pipeline-Analysis", + "bucket": "net-mozaws-prod-us-west-2-pipeline-analysis", + "prefix": "mreid/churn" + }, + { + "name": "Dashboard", + "bucket": "net-mozaws-prod-metrics-data", + "prefix": "telemetry-churn" + } + ], + "search_cohort": { + "bucket": "net-mozaws-prod-us-west-2-pipeline-analysis", + "prefix": "amiyaguchi/churn_csv" + } +} +assert_valid_config(config) + +# Set to True to overwrite the configuration with debugging route +if False: + config["uploads"] = [ + { + "name": "Testing", + "bucket": "net-mozaws-prod-us-west-2-pipeline-analysis", + "prefix": "amiyaguchi/churn_csv_testing" + } + ] + config['search_cohort'] = { + "bucket": "net-mozaws-prod-us-west-2-pipeline-analysis", + "prefix": "amiyaguchi/churn_csv_testing" + } + assert_valid_config(config) + + +# check for a date, in the case of a backfill +env_date = environ.get('date') +week_start = None +if env_date: + # Churn waits 10 days for pings to be sent from the client + week_start_date = snap_to_beginning_of_week( + datetime.strptime(env_date, "%Y%m%d") - timedelta(10), + "Sunday") + week_start = fmt(week_start_date) + +convert_week(config, week_start) +
This script turns the parquet dataset generated by churn notebook into csv files.
+import boto3 +import botocore +import gzip + +from boto3.s3.transfer import S3Transfer +from datetime import datetime, timedelta +from pyspark.sql import functions as F + + +def csv(f): + return ",".join([unicode(a) for a in f]) + +def fmt(d, date_format="%Y%m%d"): + return datetime.strftime(d, date_format) + +def collect_and_upload_csv(df, filename, upload_config): + """ Collect the dataframe into a csv file and upload to target locations. """ + client = boto3.client('s3', 'us-west-2') + transfer = S3Transfer(client) + + print("{}: Writing output to {}".format(datetime.utcnow(), filename)) + + # Write the file out as gzipped csv + with gzip.open(filename, 'wb') as fout: + fout.write(",".join(df.columns) + "\n") + print("{}: Wrote header to {}".format(datetime.utcnow(), filename)) + records = df.rdd.collect() + for r in records: + try: + fout.write(csv(r)) + fout.write("\n") + except UnicodeEncodeError as e: + print("{}: Error writing line: {} // {}".format(datetime.utcnow(), e, r)) + print("{}: finished writing lines".format(datetime.utcnow())) + + # upload files to s3 + try: + for config in upload_config: + print("{}: Uploading to {} at s3://{}/{}/{}".format( + datetime.utcnow(), config["name"], config["bucket"], + config["prefix"], filename)) + + s3_path = "{}/{}".format(config["prefix"], filename) + transfer.upload_file(filename, config["bucket"], s3_path, + extra_args={'ACL': 'bucket-owner-full-control'}) + except botocore.exceptions.ClientError as e: + print("File for {} already exists, skipping upload: {}".format(filename, e)) + + +def marginalize_dataframe(df, attributes, aggregates): + """ Reduce the granularity of the dataset to the original set of attributes. + The original set of attributes can be found on commit 2de3ef1 of mozilla-reports. """ + + return df.groupby(attributes).agg(*[F.sum(x).alias(x) for x in aggregates]) + + +def convert_week(config, week_start=None): + """ Convert a given retention period from parquet to csv. """ + df = spark.read.parquet(config["source"]) + + # find the latest start date based on the dataset if not provided + if not week_start: + start_dates = df.select("week_start").distinct().collect() + week_start = sorted(start_dates)[-1].week_start + + # find the week end for the filename + week_end = fmt(datetime.strptime(week_start, "%Y%m%d") + timedelta(6)) + + print("Running for the week of {} to {}".format(week_start, week_end)) + + # find the target subset of data + df = df.where(df.week_start == week_start) + + # marginalize the dataframe to the original attributes and upload to s3 + initial_attributes = ['channel', 'geo', 'is_funnelcake', + 'acquisition_period', 'start_version', 'sync_usage', + 'current_version', 'current_week', 'is_active'] + initial_aggregates = ['n_profiles', 'usage_hours', 'sum_squared_usage_hours'] + + upload_df = marginalize_dataframe(df, initial_attributes, initial_aggregates) + filename = "churn-{}-{}.by_activity.csv.gz".format(week_start, week_end) + collect_and_upload_csv(upload_df, filename, config["uploads"]) + + # Bug 1355988 + # The size of the data explodes significantly with extra dimensions and is too + # large to fit into the driver memory. We can write directly to s3 from a + # dataframe. + bucket = config['search_cohort']['bucket'] + prefix = config['search_cohort']['prefix'] + location = "s3://{}/{}/week_start={}".format(bucket, prefix, week_start) + + print("Saving additional search cohort churn data to {}".format(location)) + + search_attributes = [ + 'source', 'medium', 'campaign', 'content', + 'distribution_id', 'default_search_engine', 'locale' + ] + attributes = initial_attributes + search_attributes + upload_df = marginalize_dataframe(df, attributes, initial_aggregates) + upload_df.write.csv(location, header=True, mode='overwrite', compression='gzip') + + print("Sucessfully finished churn_to_csv") +
def assert_valid_config(config): + """ Assert that the configuration looks correct. """ + # This could be replaced with python schema's + assert set(["source", "uploads", "search_cohort"]).issubset(config.keys()) + assert set(["bucket", "prefix"]).issubset(config['search_cohort'].keys()) + for entry in config["uploads"]: + assert set(["name", "bucket", "prefix"]).issubset(entry.keys()) +
from moztelemetry.standards import snap_to_beginning_of_week +from os import environ + +config = { + "source": "s3://telemetry-parquet/churn/v2", + "uploads": [ + { + "name": "Pipeline-Analysis", + "bucket": "net-mozaws-prod-us-west-2-pipeline-analysis", + "prefix": "mreid/churn" + }, + { + "name": "Dashboard", + "bucket": "net-mozaws-prod-metrics-data", + "prefix": "telemetry-churn" + } + ], + "search_cohort": { + "bucket": "net-mozaws-prod-us-west-2-pipeline-analysis", + "prefix": "amiyaguchi/churn_csv" + } +} +assert_valid_config(config) + +# Set to True to overwrite the configuration with debugging route +if False: + config["uploads"] = [ + { + "name": "Testing", + "bucket": "net-mozaws-prod-us-west-2-pipeline-analysis", + "prefix": "amiyaguchi/churn_csv_testing" + } + ] + config['search_cohort'] = { + "bucket": "net-mozaws-prod-us-west-2-pipeline-analysis", + "prefix": "amiyaguchi/churn_csv_testing" + } + assert_valid_config(config) + + +# check for a date, in the case of a backfill +env_date = environ.get('date') +week_start = None +if env_date: + # Churn waits 10 days for pings to be sent from the client + week_start_date = snap_to_beginning_of_week( + datetime.strptime(env_date, "%Y%m%d") - timedelta(10), + "Sunday") + week_start = fmt(week_start_date) + +convert_week(config, week_start) +
# %load ~/cliqz_ping_pipeline/transform.py +import ujson as json +from datetime import * +import pandas as pd +from pyspark.sql.types import * +from pyspark.sql.functions import split +import base64 +from Crypto.Cipher import AES + +from moztelemetry import get_pings_properties +from moztelemetry.dataset import Dataset + +class ColumnConfig: + def __init__(self, name, path, cleaning_func, struct_type): + self.name = name + self.path = path + self.cleaning_func = cleaning_func + self.struct_type = struct_type + +class DataFrameConfig: + def __init__(self, col_configs, ping_filter): + self.columns = [ColumnConfig(*col) for col in col_configs] + self.ping_filter = ping_filter + + def toStructType(self): + return StructType(map( + lambda col: StructField(col.name, col.struct_type, True), + self.columns)) + + def get_names(self): + return map(lambda col: col.name, self.columns) + + def get_paths(self): + return map(lambda col: col.path, self.columns) + + +def pings_to_df(sqlContext, pings, data_frame_config): + """Performs simple data pipelining on raw pings + + Arguments: + data_frame_config: a list of tuples of the form: + (name, path, cleaning_func, column_type) + """ + filtered_pings = get_pings_properties(pings, data_frame_config.get_paths())\ + .filter(data_frame_config.ping_filter) + + return config_to_df(sqlContext, filtered_pings, data_frame_config) + +def config_to_df(sqlContext, raw_data, data_frame_config): + """Performs simple data pipelining on raw pings + + Arguments: + data_frame_config: a list of tuples of the form: + (name, path, cleaning_func, column_type) + """ + def build_cell(ping, column_config): + """Takes a json ping and a column config and returns a cleaned cell""" + raw_value = ping[column_config.path] + func = column_config.cleaning_func + if func is not None: + return func(raw_value) + else: + return raw_value + + def ping_to_row(ping): + return [build_cell(ping, col) for col in data_frame_config.columns] + + return sqlContext.createDataFrame( + raw_data.map(ping_to_row).collect(), + schema = data_frame_config.toStructType()) +
def save_df(df, name, date_partition, partitions=1): + if date_partition is not None: + partition_str = "/submission_date={day}".format(day=date_partition) + else: + partition_str="" + + # TODO: this name should include the experiment name + path_fmt = "s3n://telemetry-parquet/harter/containers_{name}/v1{partition_str}" + path = path_fmt.format(name=name, partition_str=partition_str) + df.repartition(partitions).write.mode("overwrite").parquet(path) + +def __main__(sc, sqlContext, day=None, save=True): + if day is None: + # Set day to yesterday + day = (date.today() - timedelta(1)).strftime("%Y%m%d") + + get_doctype_pings = lambda docType: Dataset.from_source("telemetry") \ + .where(docType=docType) \ + .where(submissionDate=day) \ + .where(appName="Firefox") \ + .records(sc) + + testpilottest_df = pings_to_df( + sqlContext, + get_doctype_pings("testpilottest"), + DataFrameConfig( + [ + ("uuid", "payload/payload/uuid", None, StringType()), + ("userContextId", "payload/payload/userContextId", None, LongType()), + ("clickedContainerTabCount", "payload/payload/clickedContainerTabCount", None, LongType()), + ("eventSource", "payload/payload/eventSource", None, StringType()), + ("event", "payload/payload/event", None, StringType()), + ("hiddenContainersCount", "payload/payload/hiddenContainersCount", None, LongType()), + ("shownContainersCount", "payload/payload/shownContainersCount", None, LongType()), + ("totalContainersCount", "payload/payload/totalContainersCount", None, LongType()), + ("totalContainerTabsCount", "payload/payload/totalContainerTabsCount", None, LongType()), + ("totalNonContainerTabsCount", "payload/payload/totalNonContainerTabsCount", None, LongType()), + ("test", "payload/test", None, StringType()), + ], + lambda ping: ping['payload/test'] == "@testpilot-containers" + ) + ) + + if save: + save_df(testpilottest_df, "testpilottest", day, partitions=1) + + return testpilottest_df +
tpt = __main__(sc, sqlContext) +
tpt.take(2) +
+
# %load ~/cliqz_ping_pipeline/transform.py +import ujson as json +from datetime import * +import pandas as pd +from pyspark.sql.types import * +from pyspark.sql.functions import split +import base64 +from Crypto.Cipher import AES + +from moztelemetry import get_pings_properties +from moztelemetry.dataset import Dataset + +class ColumnConfig: + def __init__(self, name, path, cleaning_func, struct_type): + self.name = name + self.path = path + self.cleaning_func = cleaning_func + self.struct_type = struct_type + +class DataFrameConfig: + def __init__(self, col_configs, ping_filter): + self.columns = [ColumnConfig(*col) for col in col_configs] + self.ping_filter = ping_filter + + def toStructType(self): + return StructType(map( + lambda col: StructField(col.name, col.struct_type, True), + self.columns)) + + def get_names(self): + return map(lambda col: col.name, self.columns) + + def get_paths(self): + return map(lambda col: col.path, self.columns) + + +def pings_to_df(sqlContext, pings, data_frame_config): + """Performs simple data pipelining on raw pings + + Arguments: + data_frame_config: a list of tuples of the form: + (name, path, cleaning_func, column_type) + """ + filtered_pings = get_pings_properties(pings, data_frame_config.get_paths())\ + .filter(data_frame_config.ping_filter) + + return config_to_df(sqlContext, filtered_pings, data_frame_config) + +def config_to_df(sqlContext, raw_data, data_frame_config): + """Performs simple data pipelining on raw pings + + Arguments: + data_frame_config: a list of tuples of the form: + (name, path, cleaning_func, column_type) + """ + def build_cell(ping, column_config): + """Takes a json ping and a column config and returns a cleaned cell""" + raw_value = ping[column_config.path] + func = column_config.cleaning_func + if func is not None: + return func(raw_value) + else: + return raw_value + + def ping_to_row(ping): + return [build_cell(ping, col) for col in data_frame_config.columns] + + return sqlContext.createDataFrame( + raw_data.map(ping_to_row).collect(), + schema = data_frame_config.toStructType()) +
def save_df(df, name, date_partition, partitions=1): + if date_partition is not None: + partition_str = "/submission_date={day}".format(day=date_partition) + else: + partition_str="" + + # TODO: this name should include the experiment name + path_fmt = "s3n://telemetry-parquet/harter/containers_{name}/v1{partition_str}" + path = path_fmt.format(name=name, partition_str=partition_str) + df.repartition(partitions).write.mode("overwrite").parquet(path) + +def __main__(sc, sqlContext, day=None, save=True): + if day is None: + # Set day to yesterday + day = (date.today() - timedelta(1)).strftime("%Y%m%d") + + get_doctype_pings = lambda docType: Dataset.from_source("telemetry") \ + .where(docType=docType) \ + .where(submissionDate=day) \ + .where(appName="Firefox") \ + .records(sc) + + testpilottest_df = pings_to_df( + sqlContext, + get_doctype_pings("testpilottest"), + DataFrameConfig( + [ + ("uuid", "payload/payload/uuid", None, StringType()), + ("userContextId", "payload/payload/userContextId", None, LongType()), + ("clickedContainerTabCount", "payload/payload/clickedContainerTabCount", None, LongType()), + ("eventSource", "payload/payload/eventSource", None, StringType()), + ("event", "payload/payload/event", None, StringType()), + ("hiddenContainersCount", "payload/payload/hiddenContainersCount", None, LongType()), + ("shownContainersCount", "payload/payload/shownContainersCount", None, LongType()), + ("totalContainersCount", "payload/payload/totalContainersCount", None, LongType()), + ("totalContainerTabsCount", "payload/payload/totalContainerTabsCount", None, LongType()), + ("totalNonContainerTabsCount", "payload/payload/totalNonContainerTabsCount", None, LongType()), + ("test", "payload/test", None, StringType()), + ], + lambda ping: ping['payload/test'] == "@testpilot-containers" + ) + ) + + if save: + save_df(testpilottest_df, "testpilottest", day, partitions=1) + + return testpilottest_df +
tpt = __main__(sc, sqlContext) +
tpt.take(2) +
+
from datetime import datetime as dt, timedelta, date +import moztelemetry +from os import environ + +# get the desired target date from the environment, or run +# on 'yesterday' by default. +yesterday = dt.strftime(dt.utcnow() - timedelta(1), "%Y%m%d") +target_date = environ.get('date', yesterday) +
from moztelemetry.dataset import Dataset + +sample_rate = environ.get('sample', 1) +pings = Dataset.from_source("telemetry-experiments") \ + .where(submissionDate=target_date) \ + .where(docType="main") \ + .records(sc, sample=sample_rate) \ + .filter(lambda x: x.get("environment", {}).get("build", {}).get("applicationName") == "Firefox") +
from moztelemetry import get_pings_properties + +subset = get_pings_properties(pings, { + "appUpdateChannel": "meta/appUpdateChannel", + "log": "payload/log", + "activeExperiment": "environment/addons/activeExperiment/id", + "activeExperimentBranch": "environment/addons/activeExperiment/branch" +}) +
from collections import defaultdict +from copy import deepcopy + +### Setup data structures and constants ### + +ALLOWED_ENTRY_TYPES = ('EXPERIMENT_ACTIVATION', 'EXPERIMENT_TERMINATION') + +experiment = { + 'EXPERIMENT_ACTIVATION': defaultdict(int), + 'active': defaultdict(int), + 'EXPERIMENT_TERMINATION': defaultdict(int) +} + +channel = { + 'errors': [], + 'experiments': {} +} + +def get_empty_channel(): + return deepcopy(channel) +
import gzip +import ujson +import requests + +# This is a json object with {Date => {channel: count}}. It is created +# by the main_channel_counts plugin, and may be inaccurate if the ec2 +# box crashed, but only for the day of the crash. If it crashes, the +# previous data will be lost. +COUNTS_JSON_URI = "https://pipeline-cep.prod.mozaws.net/dashboard_output/analysis.frank.main_channel_counts.counts.json" + +### Aggregation functions, Spark job, output file creation ### + +def channel_ping_agg(channel_agg, ping): + """Aggregate a channel with a ping""" + try: + for item in (ping.get("log") or []): + if item[0] in ALLOWED_ENTRY_TYPES: + entry, _, reason, exp_id = item[:4] + data = item[4:] + if exp_id not in channel_agg['experiments']: + channel_agg['experiments'][exp_id] = deepcopy(experiment) + channel_agg['experiments'][exp_id][entry][tuple([reason] + data)] += 1 + + exp_id = ping.get("activeExperiment") + branch = ping.get("activeExperimentBranch") + if exp_id is not None and branch is not None: + if exp_id not in channel_agg['experiments']: + channel_agg['experiments'][exp_id] = deepcopy(experiment) + channel_agg['experiments'][exp_id]['active'][branch] += 1 + except Exception as e: + channel_agg['errors'].append('{}: {}'.format(e.__class__, str(e))) + + return channel_agg + +def channel_channel_agg(channel_agg_1, channel_agg_2): + """Aggregate a channel with a channel""" + channel_agg_1['errors'] += channel_agg_2['errors'] + + for exp_id, exp in channel_agg_2['experiments'].iteritems(): + if exp_id not in channel_agg_1['experiments']: + channel_agg_1['experiments'][exp_id] = deepcopy(experiment) + for entry, exp_activities in exp.iteritems(): + for exp_activity, counts in exp_activities.iteritems(): + channel_agg_1['experiments'][exp_id][entry][exp_activity] += counts + + return channel_agg_1 + +def get_channel_or_other(ping): + channel = ping.get("appUpdateChannel") + if channel in ("release", "nightly", "beta", "aurora"): + return channel + return "OTHER" + +def aggregate_pings(pings): + """Get the channel experiments from an rdd of pings""" + return pings\ + .map(lambda x: (get_channel_or_other(x), x))\ + .aggregateByKey(get_empty_channel(), channel_ping_agg, channel_channel_agg) + + +def add_counts(result): + """Add counts from a running CEP""" + counts = requests.get(COUNTS_JSON_URI).json() + + for cname, channel in result: + channel['total'] = counts.get(target_date, {}).get(cname, None) + + return result + +def write_aggregate(agg, date, filename_prefix='experiments'): + filenames = [] + + for cname, channel in agg: + d = { + "total": channel['total'], + "experiments": {} + } + for exp_id, experiment in channel['experiments'].iteritems(): + d["experiments"][exp_id] = { + "active": experiment['active'], + "activations": experiment['EXPERIMENT_ACTIVATION'].items(), + "terminations": experiment['EXPERIMENT_TERMINATION'].items() + } + + filename = "{}{}-{}.json.gz".format(filename_prefix, date, cname) + filenames.append(filename) + + with gzip.open(filename, "wb") as fd: + ujson.dump(d, fd) + + return filenames +
### Setup Test Pings ### + +def make_ping(ae, aeb, chan, log): + return {'activeExperiment': ae, + 'activeExperimentBranch': aeb, + 'appUpdateChannel': chan, + 'log': log} + +NUM_ACTIVATIONS = 5 +NUM_ACTIVES = 7 +NUM_TERMINATIONS = 3 +TOTAL = NUM_ACTIVATIONS + NUM_ACTIVES + NUM_TERMINATIONS + +_channel, exp_id, the_date = 'release', 'tls13-compat-ff51@experiments.mozilla.org', '20140101' +branch, reason, data = 'branch', 'REJECTED', ['minBuildId'] +log = [17786, reason, exp_id] + data + +pings = [make_ping(exp_id, branch, _channel, []) + for i in xrange(NUM_ACTIVES)] +\ + [make_ping(exp_id, branch, _channel, [['EXPERIMENT_ACTIVATION'] + log]) + for i in xrange(NUM_ACTIVATIONS)] +\ + [make_ping(exp_id, branch, _channel, [['EXPERIMENT_TERMINATION'] + log]) + for i in xrange(NUM_TERMINATIONS)] + +### Setup expected result aggregate ### + +def channels_agg_assert(channels, counts=1): + #Should just be the channel we provided + assert channels.viewkeys() == set([_channel]), 'Incorrect channels: ' + ','.join(channels.keys()) + + #just check this one channel now + release = channels[_channel] + assert len(release['errors']) == 0, 'Had Errors: ' + ','.join(release['errors']) + + #now check experiment totals + assert release['experiments'][exp_id]['EXPERIMENT_ACTIVATION'][tuple([reason] + data)] == NUM_ACTIVATIONS * counts,\ + 'Expected ' + str(NUM_ACTIVATIONS * counts) + \ + ', Got ' + str(release['experiments'][exp_id]['EXPERIMENT_ACTIVATION'][tuple([reason] + data)]) + assert release['experiments'][exp_id]['EXPERIMENT_TERMINATION'][tuple([reason] + data)] == NUM_TERMINATIONS * counts,\ + 'Expected ' + str(NUM_TERMINATIONS * counts) + \ + ', Got ' + str(release['experiments'][exp_id]['EXPERIMENT_TERMINATION'][tuple([reason] + data)]) + + #`active` is counted for both just active, and for activations and terminations above + assert release['experiments'][exp_id]['active'][branch] == TOTAL * counts,\ + 'Expected ' + str(TOTAL * counts) +\ + 'Got ' + str(release['experiments'][exp_id]['active'][branch]) + + +### Test non-spark - easier debugging ### + +channel_1, channel_2 = get_empty_channel(), get_empty_channel() +for ping in pings: + channel_1 = channel_ping_agg(channel_1, ping) + channel_2 = channel_ping_agg(channel_2, ping) + +# no actual key-value reduce, so just have to add the channel as key +res_chan = ((_channel, channel_channel_agg(channel_1, channel_2)),) +res_chan = add_counts(res_chan) + +# we've agggregated over the pings twice, so counts=2 +channels_agg_assert({channel: agg for channel, agg in res_chan}, counts=2) + +write_aggregate(res_chan, the_date, filename_prefix="nonspark_test") + + +#### Test Spark ### +res = aggregate_pings(sc.parallelize(pings)).collect() +res = add_counts(res) + +channels = {channel: agg for channel, agg in res} + +channels_agg_assert(channels, counts=1) + +write_aggregate(res, the_date, filename_prefix="spark_test") +
['spark_test20140101-release.json.gz'] +
### Run on actual data - use CEP to get counts ### + +result = aggregate_pings(subset).collect() +result = add_counts(result) +
### Upload target day's data files ### + +import boto3 +import botocore +from boto3.s3.transfer import S3Transfer + +output_files = write_aggregate(result, target_date) + +data_bucket = "telemetry-public-analysis-2" +s3path = "experiments/data" +gz_csv_args = {'ContentEncoding': 'gzip', 'ContentType': 'text/csv'} + +client = boto3.client('s3', 'us-west-2') +transfer = S3Transfer(client) + +for output_file in output_files: + transfer.upload_file( + output_file, + data_bucket, + "{}/{}".format(s3path, output_file), + extra_args=gz_csv_args + ) +
from datetime import datetime as dt, timedelta, date +import moztelemetry +from os import environ + +# get the desired target date from the environment, or run +# on 'yesterday' by default. +yesterday = dt.strftime(dt.utcnow() - timedelta(1), "%Y%m%d") +target_date = environ.get('date', yesterday) +
from moztelemetry.dataset import Dataset + +sample_rate = environ.get('sample', 1) +pings = Dataset.from_source("telemetry-experiments") \ + .where(submissionDate=target_date) \ + .where(docType="main") \ + .records(sc, sample=sample_rate) \ + .filter(lambda x: x.get("environment", {}).get("build", {}).get("applicationName") == "Firefox") +
from moztelemetry import get_pings_properties + +subset = get_pings_properties(pings, { + "appUpdateChannel": "meta/appUpdateChannel", + "log": "payload/log", + "activeExperiment": "environment/addons/activeExperiment/id", + "activeExperimentBranch": "environment/addons/activeExperiment/branch" +}) +
from collections import defaultdict +from copy import deepcopy + +### Setup data structures and constants ### + +ALLOWED_ENTRY_TYPES = ('EXPERIMENT_ACTIVATION', 'EXPERIMENT_TERMINATION') + +experiment = { + 'EXPERIMENT_ACTIVATION': defaultdict(int), + 'active': defaultdict(int), + 'EXPERIMENT_TERMINATION': defaultdict(int) +} + +channel = { + 'errors': [], + 'experiments': {} +} + +def get_empty_channel(): + return deepcopy(channel) +
import gzip +import ujson +import requests + +# This is a json object with {Date => {channel: count}}. It is created +# by the main_channel_counts plugin, and may be inaccurate if the ec2 +# box crashed, but only for the day of the crash. If it crashes, the +# previous data will be lost. +COUNTS_JSON_URI = "https://pipeline-cep.prod.mozaws.net/dashboard_output/analysis.frank.main_channel_counts.counts.json" + +### Aggregation functions, Spark job, output file creation ### + +def channel_ping_agg(channel_agg, ping): + """Aggregate a channel with a ping""" + try: + for item in (ping.get("log") or []): + if item[0] in ALLOWED_ENTRY_TYPES: + entry, _, reason, exp_id = item[:4] + data = item[4:] + if exp_id not in channel_agg['experiments']: + channel_agg['experiments'][exp_id] = deepcopy(experiment) + channel_agg['experiments'][exp_id][entry][tuple([reason] + data)] += 1 + + exp_id = ping.get("activeExperiment") + branch = ping.get("activeExperimentBranch") + if exp_id is not None and branch is not None: + if exp_id not in channel_agg['experiments']: + channel_agg['experiments'][exp_id] = deepcopy(experiment) + channel_agg['experiments'][exp_id]['active'][branch] += 1 + except Exception as e: + channel_agg['errors'].append('{}: {}'.format(e.__class__, str(e))) + + return channel_agg + +def channel_channel_agg(channel_agg_1, channel_agg_2): + """Aggregate a channel with a channel""" + channel_agg_1['errors'] += channel_agg_2['errors'] + + for exp_id, exp in channel_agg_2['experiments'].iteritems(): + if exp_id not in channel_agg_1['experiments']: + channel_agg_1['experiments'][exp_id] = deepcopy(experiment) + for entry, exp_activities in exp.iteritems(): + for exp_activity, counts in exp_activities.iteritems(): + channel_agg_1['experiments'][exp_id][entry][exp_activity] += counts + + return channel_agg_1 + +def get_channel_or_other(ping): + channel = ping.get("appUpdateChannel") + if channel in ("release", "nightly", "beta", "aurora"): + return channel + return "OTHER" + +def aggregate_pings(pings): + """Get the channel experiments from an rdd of pings""" + return pings\ + .map(lambda x: (get_channel_or_other(x), x))\ + .aggregateByKey(get_empty_channel(), channel_ping_agg, channel_channel_agg) + + +def add_counts(result): + """Add counts from a running CEP""" + counts = requests.get(COUNTS_JSON_URI).json() + + for cname, channel in result: + channel['total'] = counts.get(target_date, {}).get(cname, None) + + return result + +def write_aggregate(agg, date, filename_prefix='experiments'): + filenames = [] + + for cname, channel in agg: + d = { + "total": channel['total'], + "experiments": {} + } + for exp_id, experiment in channel['experiments'].iteritems(): + d["experiments"][exp_id] = { + "active": experiment['active'], + "activations": experiment['EXPERIMENT_ACTIVATION'].items(), + "terminations": experiment['EXPERIMENT_TERMINATION'].items() + } + + filename = "{}{}-{}.json.gz".format(filename_prefix, date, cname) + filenames.append(filename) + + with gzip.open(filename, "wb") as fd: + ujson.dump(d, fd) + + return filenames +
### Setup Test Pings ### + +def make_ping(ae, aeb, chan, log): + return {'activeExperiment': ae, + 'activeExperimentBranch': aeb, + 'appUpdateChannel': chan, + 'log': log} + +NUM_ACTIVATIONS = 5 +NUM_ACTIVES = 7 +NUM_TERMINATIONS = 3 +TOTAL = NUM_ACTIVATIONS + NUM_ACTIVES + NUM_TERMINATIONS + +_channel, exp_id, the_date = 'release', 'tls13-compat-ff51@experiments.mozilla.org', '20140101' +branch, reason, data = 'branch', 'REJECTED', ['minBuildId'] +log = [17786, reason, exp_id] + data + +pings = [make_ping(exp_id, branch, _channel, []) + for i in xrange(NUM_ACTIVES)] +\ + [make_ping(exp_id, branch, _channel, [['EXPERIMENT_ACTIVATION'] + log]) + for i in xrange(NUM_ACTIVATIONS)] +\ + [make_ping(exp_id, branch, _channel, [['EXPERIMENT_TERMINATION'] + log]) + for i in xrange(NUM_TERMINATIONS)] + +### Setup expected result aggregate ### + +def channels_agg_assert(channels, counts=1): + #Should just be the channel we provided + assert channels.viewkeys() == set([_channel]), 'Incorrect channels: ' + ','.join(channels.keys()) + + #just check this one channel now + release = channels[_channel] + assert len(release['errors']) == 0, 'Had Errors: ' + ','.join(release['errors']) + + #now check experiment totals + assert release['experiments'][exp_id]['EXPERIMENT_ACTIVATION'][tuple([reason] + data)] == NUM_ACTIVATIONS * counts,\ + 'Expected ' + str(NUM_ACTIVATIONS * counts) + \ + ', Got ' + str(release['experiments'][exp_id]['EXPERIMENT_ACTIVATION'][tuple([reason] + data)]) + assert release['experiments'][exp_id]['EXPERIMENT_TERMINATION'][tuple([reason] + data)] == NUM_TERMINATIONS * counts,\ + 'Expected ' + str(NUM_TERMINATIONS * counts) + \ + ', Got ' + str(release['experiments'][exp_id]['EXPERIMENT_TERMINATION'][tuple([reason] + data)]) + + #`active` is counted for both just active, and for activations and terminations above + assert release['experiments'][exp_id]['active'][branch] == TOTAL * counts,\ + 'Expected ' + str(TOTAL * counts) +\ + 'Got ' + str(release['experiments'][exp_id]['active'][branch]) + + +### Test non-spark - easier debugging ### + +channel_1, channel_2 = get_empty_channel(), get_empty_channel() +for ping in pings: + channel_1 = channel_ping_agg(channel_1, ping) + channel_2 = channel_ping_agg(channel_2, ping) + +# no actual key-value reduce, so just have to add the channel as key +res_chan = ((_channel, channel_channel_agg(channel_1, channel_2)),) +res_chan = add_counts(res_chan) + +# we've agggregated over the pings twice, so counts=2 +channels_agg_assert({channel: agg for channel, agg in res_chan}, counts=2) + +write_aggregate(res_chan, the_date, filename_prefix="nonspark_test") + + +#### Test Spark ### +res = aggregate_pings(sc.parallelize(pings)).collect() +res = add_counts(res) + +channels = {channel: agg for channel, agg in res} + +channels_agg_assert(channels, counts=1) + +write_aggregate(res, the_date, filename_prefix="spark_test") +
['spark_test20140101-release.json.gz'] +
### Run on actual data - use CEP to get counts ### + +result = aggregate_pings(subset).collect() +result = add_counts(result) +
### Upload target day's data files ### + +import boto3 +import botocore +from boto3.s3.transfer import S3Transfer + +output_files = write_aggregate(result, target_date) + +data_bucket = "telemetry-public-analysis-2" +s3path = "experiments/data" +gz_csv_args = {'ContentEncoding': 'gzip', 'ContentType': 'text/csv'} + +client = boto3.client('s3', 'us-west-2') +transfer = S3Transfer(client) + +for output_file in output_files: + transfer.upload_file( + output_file, + data_bucket, + "{}/{}".format(s3path, output_file), + extra_args=gz_csv_args + ) +
import os +import datetime as dt +import pandas as pd +import ujson as json +from pyspark.sql.types import * + +from moztelemetry import get_pings, get_pings_properties + +%pylab inline +
+
Take the set of pings, make sure we have actual clientIds and remove duplicate pings. We collect each unique ping.
+def dedupe_pings(rdd): + return rdd.filter(lambda p: p["meta/clientId"] is not None)\ + .map(lambda p: (p["meta/documentId"], p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) +
Transform and sanitize the pings into arrays.
+def transform(ping): + # Should not be None since we filter those out. + clientId = ping["meta/clientId"] + + # Added via the ingestion process so should not be None. + submissionDate = dt.datetime.strptime(ping["meta/submissionDate"], "%Y%m%d") + geoCountry = ping["meta/geoCountry"] + + profileDate = None + profileDaynum = ping["profileDate"] + if profileDaynum is not None: + try: + # Bad data could push profileDaynum > 32767 (size of a C int) and throw exception + profileDate = dt.datetime(1970, 1, 1) + dt.timedelta(int(profileDaynum)) + except: + profileDate = None + + # Create date can be an improper string (~.03% of the time, so ignore) + # Year can be < 2000 (~.005% of the time, so ignore) + try: + # Create date should already be in ISO format + creationDate = ping["created"] + if creationDate is not None: + # This is only accurate because we know the creation date is always in 'Z' (zulu) time. + creationDate = dt.datetime.strptime(ping["created"], "%Y-%m-%d") + if creationDate.year < 2000: + creationDate = None + except ValueError: + creationDate = None + + appVersion = ping["meta/appVersion"] + buildId = ping["meta/appBuildId"] + locale = ping["locale"] + os = ping["os"] + osVersion = ping["osversion"] + device = ping["device"] + arch = ping["arch"] + defaultSearch = ping["defaultSearch"] + distributionId = ping["distributionId"] + + experiments = ping["experiments"] + if experiments is None: + experiments = [] + + #bug 1315028 + defaultNewTabExperience = ping["defaultNewTabExperience"] + defaultMailClient = ping["defaultMailClient"] + + #bug 1307419 + searches = ping["searches"] + durations = ping["durations"] + sessions = ping["sessions"] + + return [clientId, submissionDate, creationDate, profileDate, geoCountry, locale, os, + osVersion, buildId, appVersion, device, arch, defaultSearch, distributionId, + json.dumps(experiments), defaultNewTabExperience, defaultMailClient, searches, + durations, sessions] +
Create a set of pings from “core” to build a set of core client data. Output the data to CSV or Parquet.
+This script is designed to loop over a range of days and output a single day for the given channels. Use explicit date ranges for backfilling, or now() - ‘1day’ for automated runs.
+channels = ["nightly", "aurora", "beta", "release"] + +batch_date = os.environ.get('date') +if batch_date: + start = end = dt.datetime.strptime(batch_date, '%Y%m%d') +else: + start = dt.datetime.now() - dt.timedelta(1) + end = dt.datetime.now() - dt.timedelta(1) + + + +day = start +while day <= end: + for channel in channels: + print "\nchannel: " + channel + ", date: " + day.strftime("%Y%m%d") + + kwargs = dict( + doc_type="core", + submission_date=(day.strftime("%Y%m%d"), day.strftime("%Y%m%d")), + channel=channel, + app="Fennec", + fraction=1 + ) + + # Grab all available source_version pings + pings = get_pings(sc, source_version="*", **kwargs) + + subset = get_pings_properties(pings, ["meta/clientId", + "meta/documentId", + "meta/submissionDate", + "meta/appVersion", + "meta/appBuildId", + "meta/geoCountry", + "locale", + "os", + "osversion", + "device", + "arch", + "profileDate", + "created", + "defaultSearch", + "distributionId", + "experiments", + "defaultNewTabExperience", + "defaultMailClient", + "searches", + "durations", + "sessions"]) + + subset = dedupe_pings(subset) + print "\nDe-duped pings:" + str(subset.count()) + print subset.first() + + transformed = subset.map(transform) + print "\nTransformed pings:" + str(transformed.count()) + print transformed.first() + + s3_output = "s3n://net-mozaws-prod-us-west-2-pipeline-analysis/mobile/mobile_clients" + s3_output += "/v2/channel=" + channel + "/submission=" + day.strftime("%Y%m%d") + schema = StructType([ + StructField("clientid", StringType(), False), + StructField("submissiondate", TimestampType(), False), + StructField("creationdate", TimestampType(), True), + StructField("profiledate", TimestampType(), True), + StructField("geocountry", StringType(), True), + StructField("locale", StringType(), True), + StructField("os", StringType(), True), + StructField("osversion", StringType(), True), + StructField("buildid", StringType(), True), + StructField("appversion", StringType(), True), + StructField("device", StringType(), True), + StructField("arch", StringType(), True), + StructField("defaultsearch", StringType(), True), + StructField("distributionid", StringType(), True), + StructField("experiments", StringType(), True), + StructField("default_new_tab_experience", StringType(), True), + StructField("default_mail_client", StringType(), True), + StructField("searches", StringType(), True), + StructField("durations", StringType(), True), + StructField("sessions", StringType(), True) + ]) + # Make parquet parition file size large, but not too large for s3 to handle + coalesce = 1 + if channel == "release": + coalesce = 4 + grouped = sqlContext.createDataFrame(transformed, schema) + grouped.coalesce(coalesce).write.mode('overwrite').parquet(s3_output) + + day += dt.timedelta(1) +
import os +import datetime as dt +import pandas as pd +import ujson as json +from pyspark.sql.types import * + +from moztelemetry import get_pings, get_pings_properties + +%pylab inline +
+
Take the set of pings, make sure we have actual clientIds and remove duplicate pings. We collect each unique ping.
+def dedupe_pings(rdd): + return rdd.filter(lambda p: p["meta/clientId"] is not None)\ + .map(lambda p: (p["meta/documentId"], p))\ + .reduceByKey(lambda x, y: x)\ + .map(lambda x: x[1]) +
Transform and sanitize the pings into arrays.
+def transform(ping): + # Should not be None since we filter those out. + clientId = ping["meta/clientId"] + + # Added via the ingestion process so should not be None. + submissionDate = dt.datetime.strptime(ping["meta/submissionDate"], "%Y%m%d") + geoCountry = ping["meta/geoCountry"] + + profileDate = None + profileDaynum = ping["profileDate"] + if profileDaynum is not None: + try: + # Bad data could push profileDaynum > 32767 (size of a C int) and throw exception + profileDate = dt.datetime(1970, 1, 1) + dt.timedelta(int(profileDaynum)) + except: + profileDate = None + + # Create date can be an improper string (~.03% of the time, so ignore) + # Year can be < 2000 (~.005% of the time, so ignore) + try: + # Create date should already be in ISO format + creationDate = ping["created"] + if creationDate is not None: + # This is only accurate because we know the creation date is always in 'Z' (zulu) time. + creationDate = dt.datetime.strptime(ping["created"], "%Y-%m-%d") + if creationDate.year < 2000: + creationDate = None + except ValueError: + creationDate = None + + appVersion = ping["meta/appVersion"] + buildId = ping["meta/appBuildId"] + locale = ping["locale"] + os = ping["os"] + osVersion = ping["osversion"] + device = ping["device"] + arch = ping["arch"] + defaultSearch = ping["defaultSearch"] + distributionId = ping["distributionId"] + + experiments = ping["experiments"] + if experiments is None: + experiments = [] + + #bug 1315028 + defaultNewTabExperience = ping["defaultNewTabExperience"] + defaultMailClient = ping["defaultMailClient"] + + #bug 1307419 + searches = ping["searches"] + durations = ping["durations"] + sessions = ping["sessions"] + + return [clientId, submissionDate, creationDate, profileDate, geoCountry, locale, os, + osVersion, buildId, appVersion, device, arch, defaultSearch, distributionId, + json.dumps(experiments), defaultNewTabExperience, defaultMailClient, searches, + durations, sessions] +
Create a set of pings from “core” to build a set of core client data. Output the data to CSV or Parquet.
+This script is designed to loop over a range of days and output a single day for the given channels. Use explicit date ranges for backfilling, or now() - ‘1day’ for automated runs.
+channels = ["nightly", "aurora", "beta", "release"] + +batch_date = os.environ.get('date') +if batch_date: + start = end = dt.datetime.strptime(batch_date, '%Y%m%d') +else: + start = dt.datetime.now() - dt.timedelta(1) + end = dt.datetime.now() - dt.timedelta(1) + + + +day = start +while day <= end: + for channel in channels: + print "\nchannel: " + channel + ", date: " + day.strftime("%Y%m%d") + + kwargs = dict( + doc_type="core", + submission_date=(day.strftime("%Y%m%d"), day.strftime("%Y%m%d")), + channel=channel, + app="Fennec", + fraction=1 + ) + + # Grab all available source_version pings + pings = get_pings(sc, source_version="*", **kwargs) + + subset = get_pings_properties(pings, ["meta/clientId", + "meta/documentId", + "meta/submissionDate", + "meta/appVersion", + "meta/appBuildId", + "meta/geoCountry", + "locale", + "os", + "osversion", + "device", + "arch", + "profileDate", + "created", + "defaultSearch", + "distributionId", + "experiments", + "defaultNewTabExperience", + "defaultMailClient", + "searches", + "durations", + "sessions"]) + + subset = dedupe_pings(subset) + print "\nDe-duped pings:" + str(subset.count()) + print subset.first() + + transformed = subset.map(transform) + print "\nTransformed pings:" + str(transformed.count()) + print transformed.first() + + s3_output = "s3n://net-mozaws-prod-us-west-2-pipeline-analysis/mobile/mobile_clients" + s3_output += "/v2/channel=" + channel + "/submission=" + day.strftime("%Y%m%d") + schema = StructType([ + StructField("clientid", StringType(), False), + StructField("submissiondate", TimestampType(), False), + StructField("creationdate", TimestampType(), True), + StructField("profiledate", TimestampType(), True), + StructField("geocountry", StringType(), True), + StructField("locale", StringType(), True), + StructField("os", StringType(), True), + StructField("osversion", StringType(), True), + StructField("buildid", StringType(), True), + StructField("appversion", StringType(), True), + StructField("device", StringType(), True), + StructField("arch", StringType(), True), + StructField("defaultsearch", StringType(), True), + StructField("distributionid", StringType(), True), + StructField("experiments", StringType(), True), + StructField("default_new_tab_experience", StringType(), True), + StructField("default_mail_client", StringType(), True), + StructField("searches", StringType(), True), + StructField("durations", StringType(), True), + StructField("sessions", StringType(), True) + ]) + # Make parquet parition file size large, but not too large for s3 to handle + coalesce = 1 + if channel == "release": + coalesce = 4 + grouped = sqlContext.createDataFrame(transformed, schema) + grouped.coalesce(coalesce).write.mode('overwrite').parquet(s3_output) + + day += dt.timedelta(1) +
Read, convert, and store sync log data to Parquet form per bug 1291340.
+Conversion code is ported from the smt repo.
+from datetime import datetime as dt, timedelta, date +from moztelemetry.dataset import Dataset +from os import environ + +# Determine run parameters +source_bucket = 'net-mozaws-prod-us-west-2-pipeline-analysis' +dest_bucket = source_bucket +dest_s3_prefix = "s3://{}/mreid".format(dest_bucket) + +if "bucket" in os.environ: + dest_bucket = environ["bucket"] + dest_s3_prefix = "s3://{}".format(dest_bucket) + +yesterday = dt.strftime(dt.utcnow() - timedelta(1), "%Y%m%d") + +# Default to running for "yesterday" unless we've been given a +# specific date via the environment. +target_day = environ.get("date", yesterday) +print "Running import for {}".format(target_day) +
The sync data on S3 is stored in framed heka format, and is read using the Dataset API.
# Read the source data +schema = [] +target_prefix = 'sync-metrics/data' +sync = Dataset(source_bucket, schema, prefix=target_prefix) + +# The sync data on S3 does not have a proper "date" dimension, but the date is encoded +# in the key names themselves. +# Fetch the summaries and filter the list to the target day. +summary_prefix = "{}/{}".format(target_prefix, target_day) +sync_summaries = [ s for s in sync.summaries(sc) if s['key'].startswith(summary_prefix) ] +
The standard heka decoder assumes (based on Telemetry data) that all fields whose names have a . in them contain nested json strings. This is not true for sync log messages, which have fields such as syncstorage.storage.sql.db.execute with simple scalar values.
import ssl +from moztelemetry.heka.message_parser import unpack + +# Custom decoder for sync messages since we can have scalar fields with dots in their names. +def sync_decoder(message): + try: + for record, total_bytes in unpack(message): + result = {} + result["meta"] = { + "Timestamp": record.message.timestamp, + "Type": record.message.type, + "Hostname": record.message.hostname, + } + for field in record.message.fields: + name = field.name + value = field.value_string + if field.value_type == 1: + # TODO: handle bytes in a way that doesn't cause problems with JSON + # value = field.value_bytes + continue + elif field.value_type == 2: + value = field.value_integer + elif field.value_type == 3: + value = field.value_double + elif field.value_type == 4: + value = field.value_bool + + result[name] = value[0] if len(value) else "" + + yield result + + except ssl.SSLError: + pass # https://github.com/boto/boto/issues/2830 + +sync_records = sync.records(sc, decode=sync_decoder, summaries=sync_summaries) +
# What do the records look like? + +# Example heka message: +#Timestamp: 2016-10-28 15:11:45.98653696 -0300 ADT +#Type: mozsvc.metrics +#Hostname: ip-172-31-39-11 +#Pid: 11383 +#UUID: 155866c8-cc58-4048-a58c-6226c620fc57 +#Logger: Sync-1_5 +#Payload: +#EnvVersion: 1 +#Severity: 7 +#Fields: [name:"remoteAddressChain" representation:"" value_string:"" value_string:"" +# name:"path" value_string:"https://host/ver/somenum/storage/tabs" +# name:"fxa_uid" value_string:"some_id" +# name:"user_agent_version" value_type:DOUBLE value_double:49 +# name:"user_agent_os" value_string:"Windows 7" +# name:"device_id" value_string:"some_device_id" +# name:"method" value_string:"POST" +# name:"user_agent_browser" value_string:"Firefox" +# name:"name" value_string:"mozsvc.metrics" +# name:"request_time" value_type:DOUBLE value_double:0.003030061721801758 +# name:"code" value_type:DOUBLE value_double:200 +# ] + +# Example record: +#sync_records.first() + +# {u'code': 200.0, +# u'device_id': u'some_device_id', +# u'fxa_uid': u'some_id', +# 'meta': {'Hostname': u'ip-172-31-39-11', +# 'Timestamp': 1477678305976742912L, +# 'Type': u'mozsvc.metrics'}, +# u'method': u'GET', +# u'name': u'mozsvc.metrics', +# u'path': u'https://host/ver/somenum/storage/crypto/keys', +# u'remoteAddressChain': u'', +# u'request_time': 0.017612934112548828, +# u'syncstorage.storage.sql.db.execute': 0.014925241470336914, +# u'syncstorage.storage.sql.pool.get': 5.221366882324219e-05, +# u'user_agent_browser': u'Firefox', +# u'user_agent_os': u'Windows 7', +# u'user_agent_version': 49.0} +
# Convert data. Code ported from https://github.com/dannycoates/smt +import re +import hashlib +import math +from pyspark.sql import Row + +def sha_prefix(v): + h = hashlib.sha256() + h.update(v) + return h.hexdigest()[0:32] + +path_uid = re.compile("(\d+)\/storage\/") +path_bucket = re.compile("\d+\/storage\/(\w+)") + +def getUid(path): + if path is None: + return None + match = re.search(path_uid, path) + if match is not None: + uid = match.group(1) + return sha_prefix(uid) + return None + +def deriveDeviceId(uid, agent): + if uid is None: + return None + return sha_prefix("{}{}".format(uid, agent)) + +SyncRow = Row("uid", "s_uid", "dev", "s_dev", "ts", "method", "code", + "bucket", "t", "ua_browser", "ua_version", "ua_os", "host") + +def convert(msg): + bmatch = re.search(path_bucket, msg.get("path", "")) + if bmatch is None: + return None + bucket = bmatch.group(1) + + uid = msg.get("fxa_uid") + synth_uid = getUid(msg.get("path")) + dev = msg.get("device_id") + synth_dev = deriveDeviceId(synth_uid, + "{}{}{}".format( + msg.get("user_agent_browser", ""), + msg.get("user_agent_version", ""), + msg.get("user_agent_os", "")) + ) + + code = 200 + # support modern mozlog's use of errno for http status + errno = msg.get("errno") + if errno is not None: + if errno == 0: # success + code = 200 + else: + code = errno + else: + code = msg.get("code") + if code is not None: + code = int(code) + + t = msg.get("t", 0) + if t == 0: + t = math.floor(msg.get("request_time", 0) * 1000) + if t is None: + t = 0 + + converted = SyncRow( + (uid or synth_uid), + synth_uid, + (dev or synth_dev), + synth_dev, + msg.get("meta").get("Timestamp"), + msg.get("method"), + code, + bucket, + t, + msg.get("user_agent_browser"), + msg.get("user_agent_version"), + msg.get("user_agent_os"), + msg.get("meta").get("Hostname"), + ) + return converted +
converted = sync_records.map(lambda x: convert(x)) +
converted = converted.filter(lambda x: x is not None) +
from pyspark.sql import SQLContext +sync_df = sqlContext.createDataFrame(converted) +sync_df.printSchema() +
root + |-- uid: string (nullable = true) + |-- s_uid: string (nullable = true) + |-- dev: string (nullable = true) + |-- s_dev: string (nullable = true) + |-- ts: long (nullable = true) + |-- method: string (nullable = true) + |-- code: long (nullable = true) + |-- bucket: string (nullable = true) + |-- t: double (nullable = true) + |-- ua_browser: string (nullable = true) + |-- ua_version: double (nullable = true) + |-- ua_os: string (nullable = true) + |-- host: string (nullable = true) +
# Determine if we need to repartition. +# A record is something like 112 bytes, so figure out how many partitions +# we need to end up with reasonably-sized files. +records_per_partition = 2500000 +total_records = sync_df.count() +print "Found {} sync records".format(total_records) +
import math +num_partitions = int(math.ceil(float(total_records) / records_per_partition)) + +if num_partitions != sync_df.rdd.getNumPartitions(): + print "Repartitioning with {} partitions".format(num_partitions) + sync_df = sync_df.repartition(num_partitions) + +# Store data +sync_log_s3path = "{}/sync_log/v1/day={}".format(dest_s3_prefix, target_day) +sync_df.write.parquet(sync_log_s3path, mode="overwrite") +
# Transform, compute and store rollups +sync_df.registerTempTable("sync") +sql_transform = ''' + select + uid, + dev, + ts, + t, + case + when substring(ua_os,0,7) in ('iPad', 'iPod', 'iPhone') then 'ios' + when substring(ua_os,0,7) = 'Android' then 'android' + when substring(ua_os,0,7) = 'Windows' then 'windows' + when substring(ua_os,0,7) = 'Macinto' then 'mac' + when substring(ua_os,0,7) = 'Linux' then 'linux' + when ua_os is null then 'unknown' + else 'other' + end as ua_os, + ua_browser, + ua_version, + case method when 'POST' then 1 end as posts, + case method when 'GET' then 1 end as gets, + case method when 'PUT' then 1 end as puts, + case method when 'DELETE' then 1 end as dels, + case when code < 300 then 1 end as aoks, + case when code > 399 and code < 500 then 1 end as oops, + case when code > 499 and code < 999 then 1 end as fups, + case when bucket = 'clients' and method = 'GET' then 1 end as r_clients, + case when bucket = 'crypto' and method = 'GET' then 1 end as r_crypto, + case when bucket = 'forms' and method = 'GET' then 1 end as r_forms, + case when bucket = 'history' and method = 'GET' then 1 end as r_history, + case when bucket = 'keys' and method = 'GET' then 1 end as r_keys, + case when bucket = 'meta' and method = 'GET' then 1 end as r_meta, + case when bucket = 'bookmarks' and method = 'GET' then 1 end as r_bookmarks, + case when bucket = 'prefs' and method = 'GET' then 1 end as r_prefs, + case when bucket = 'tabs' and method = 'GET' then 1 end as r_tabs, + case when bucket = 'passwords' and method = 'GET' then 1 end as r_passwords, + case when bucket = 'addons' and method = 'GET' then 1 end as r_addons, + case when bucket = 'clients' and method = 'POST' then 1 end as w_clients, + case when bucket = 'crypto' and method = 'POST' then 1 end as w_crypto, + case when bucket = 'forms' and method = 'POST' then 1 end as w_forms, + case when bucket = 'history' and method = 'POST' then 1 end as w_history, + case when bucket = 'keys' and method = 'POST' then 1 end as w_keys, + case when bucket = 'meta' and method = 'POST' then 1 end as w_meta, + case when bucket = 'bookmarks' and method = 'POST' then 1 end as w_bookmarks, + case when bucket = 'prefs' and method = 'POST' then 1 end as w_prefs, + case when bucket = 'tabs' and method = 'POST' then 1 end as w_tabs, + case when bucket = 'passwords' and method = 'POST' then 1 end as w_passwords, + case when bucket = 'addons' and method = 'POST' then 1 end as w_addons + from sync +''' + +transformed = sqlContext.sql(sql_transform) +
transformed.registerTempTable("tx") + +sql_device_activity = ''' + select + uid, + dev, + max(ua_os) as ua_os, + max(ua_browser) as ua_browser, + max(ua_version) as ua_version, + min(t) as min_t, + max(t) as max_t, + sum(posts) as posts, + sum(gets) as gets, + sum(puts) as puts, + sum(dels) as dels, + sum(aoks) as aoks, + sum(oops) as oops, + sum(fups) as fups, + sum(r_clients) as r_clients, + sum(r_crypto) as r_crypto, + sum(r_forms) as r_forms, + sum(r_history) as r_history, + sum(r_keys) as r_keys, + sum(r_meta) as r_meta, + sum(r_bookmarks) as r_bookmarks, + sum(r_prefs) as r_prefs, + sum(r_tabs) as r_tabs, + sum(r_passwords) as r_passwords, + sum(r_addons) as r_addons, + sum(w_clients) as w_clients, + sum(w_crypto) as w_crypto, + sum(w_forms) as w_forms, + sum(w_history) as w_history, + sum(w_keys) as w_keys, + sum(w_meta) as w_meta, + sum(w_bookmarks) as w_bookmarks, + sum(w_prefs) as w_prefs, + sum(w_tabs) as w_tabs, + sum(w_passwords) as w_passwords, + sum(w_addons) as w_addons + from tx group by uid, dev +''' +rolled_up = sqlContext.sql(sql_device_activity) +
# Store device activity rollups +sync_log_device_activity_s3base = "{}/sync_log_device_activity/v1".format(dest_s3_prefix) +sync_log_device_activity_s3path = "{}/day={}".format(sync_log_device_activity_s3base, target_day) + +# TODO: Do we need to repartition? +rolled_up.repartition(5).write.parquet(sync_log_device_activity_s3path, mode="overwrite") +
def compute_device_counts(device_activity, target_day): + device_activity.registerTempTable("device_activity") + df = "%Y%m%d" + last_week_date = dt.strptime(target_day, df) - timedelta(7) + last_week = dt.strftime(last_week_date, df) + sql_device_counts = """ + select + uid, + count(distinct dev) as devs + from + (select + uid, + dev + from device_activity + where uid in + (select distinct(uid) from device_activity where day = '{}') + and day > '{}' + and day <= '{}') + group by uid + """.format(target_day, last_week, target_day) + + return sqlContext.sql(sql_device_counts) +
# Compute and store device counts + +# Re-read device activity data from S3 so we can look at historic info +device_activity = sqlContext.read.parquet(sync_log_device_activity_s3base) + +device_counts = compute_device_counts(device_activity, target_day) + +sync_log_device_counts_s3path = "{}/sync_log_device_counts/v1/day={}".format(dest_s3_prefix, target_day) +device_counts.repartition(1).write.parquet(sync_log_device_counts_s3path, mode="overwrite") +
Read, convert, and store sync log data to Parquet form per bug 1291340.
+Conversion code is ported from the smt repo.
+from datetime import datetime as dt, timedelta, date +from moztelemetry.dataset import Dataset +from os import environ + +# Determine run parameters +source_bucket = 'net-mozaws-prod-us-west-2-pipeline-analysis' +dest_bucket = source_bucket +dest_s3_prefix = "s3://{}/mreid".format(dest_bucket) + +if "bucket" in os.environ: + dest_bucket = environ["bucket"] + dest_s3_prefix = "s3://{}".format(dest_bucket) + +yesterday = dt.strftime(dt.utcnow() - timedelta(1), "%Y%m%d") + +# Default to running for "yesterday" unless we've been given a +# specific date via the environment. +target_day = environ.get("date", yesterday) +print "Running import for {}".format(target_day) +
The sync data on S3 is stored in framed heka format, and is read using the Dataset API.
# Read the source data +schema = [] +target_prefix = 'sync-metrics/data' +sync = Dataset(source_bucket, schema, prefix=target_prefix) + +# The sync data on S3 does not have a proper "date" dimension, but the date is encoded +# in the key names themselves. +# Fetch the summaries and filter the list to the target day. +summary_prefix = "{}/{}".format(target_prefix, target_day) +sync_summaries = [ s for s in sync.summaries(sc) if s['key'].startswith(summary_prefix) ] +
The standard heka decoder assumes (based on Telemetry data) that all fields whose names have a . in them contain nested json strings. This is not true for sync log messages, which have fields such as syncstorage.storage.sql.db.execute with simple scalar values.
import ssl +from moztelemetry.heka.message_parser import unpack + +# Custom decoder for sync messages since we can have scalar fields with dots in their names. +def sync_decoder(message): + try: + for record, total_bytes in unpack(message): + result = {} + result["meta"] = { + "Timestamp": record.message.timestamp, + "Type": record.message.type, + "Hostname": record.message.hostname, + } + for field in record.message.fields: + name = field.name + value = field.value_string + if field.value_type == 1: + # TODO: handle bytes in a way that doesn't cause problems with JSON + # value = field.value_bytes + continue + elif field.value_type == 2: + value = field.value_integer + elif field.value_type == 3: + value = field.value_double + elif field.value_type == 4: + value = field.value_bool + + result[name] = value[0] if len(value) else "" + + yield result + + except ssl.SSLError: + pass # https://github.com/boto/boto/issues/2830 + +sync_records = sync.records(sc, decode=sync_decoder, summaries=sync_summaries) +
# What do the records look like? + +# Example heka message: +#Timestamp: 2016-10-28 15:11:45.98653696 -0300 ADT +#Type: mozsvc.metrics +#Hostname: ip-172-31-39-11 +#Pid: 11383 +#UUID: 155866c8-cc58-4048-a58c-6226c620fc57 +#Logger: Sync-1_5 +#Payload: +#EnvVersion: 1 +#Severity: 7 +#Fields: [name:"remoteAddressChain" representation:"" value_string:"" value_string:"" +# name:"path" value_string:"https://host/ver/somenum/storage/tabs" +# name:"fxa_uid" value_string:"some_id" +# name:"user_agent_version" value_type:DOUBLE value_double:49 +# name:"user_agent_os" value_string:"Windows 7" +# name:"device_id" value_string:"some_device_id" +# name:"method" value_string:"POST" +# name:"user_agent_browser" value_string:"Firefox" +# name:"name" value_string:"mozsvc.metrics" +# name:"request_time" value_type:DOUBLE value_double:0.003030061721801758 +# name:"code" value_type:DOUBLE value_double:200 +# ] + +# Example record: +#sync_records.first() + +# {u'code': 200.0, +# u'device_id': u'some_device_id', +# u'fxa_uid': u'some_id', +# 'meta': {'Hostname': u'ip-172-31-39-11', +# 'Timestamp': 1477678305976742912L, +# 'Type': u'mozsvc.metrics'}, +# u'method': u'GET', +# u'name': u'mozsvc.metrics', +# u'path': u'https://host/ver/somenum/storage/crypto/keys', +# u'remoteAddressChain': u'', +# u'request_time': 0.017612934112548828, +# u'syncstorage.storage.sql.db.execute': 0.014925241470336914, +# u'syncstorage.storage.sql.pool.get': 5.221366882324219e-05, +# u'user_agent_browser': u'Firefox', +# u'user_agent_os': u'Windows 7', +# u'user_agent_version': 49.0} +
# Convert data. Code ported from https://github.com/dannycoates/smt +import re +import hashlib +import math +from pyspark.sql import Row + +def sha_prefix(v): + h = hashlib.sha256() + h.update(v) + return h.hexdigest()[0:32] + +path_uid = re.compile("(\d+)\/storage\/") +path_bucket = re.compile("\d+\/storage\/(\w+)") + +def getUid(path): + if path is None: + return None + match = re.search(path_uid, path) + if match is not None: + uid = match.group(1) + return sha_prefix(uid) + return None + +def deriveDeviceId(uid, agent): + if uid is None: + return None + return sha_prefix("{}{}".format(uid, agent)) + +SyncRow = Row("uid", "s_uid", "dev", "s_dev", "ts", "method", "code", + "bucket", "t", "ua_browser", "ua_version", "ua_os", "host") + +def convert(msg): + bmatch = re.search(path_bucket, msg.get("path", "")) + if bmatch is None: + return None + bucket = bmatch.group(1) + + uid = msg.get("fxa_uid") + synth_uid = getUid(msg.get("path")) + dev = msg.get("device_id") + synth_dev = deriveDeviceId(synth_uid, + "{}{}{}".format( + msg.get("user_agent_browser", ""), + msg.get("user_agent_version", ""), + msg.get("user_agent_os", "")) + ) + + code = 200 + # support modern mozlog's use of errno for http status + errno = msg.get("errno") + if errno is not None: + if errno == 0: # success + code = 200 + else: + code = errno + else: + code = msg.get("code") + if code is not None: + code = int(code) + + t = msg.get("t", 0) + if t == 0: + t = math.floor(msg.get("request_time", 0) * 1000) + if t is None: + t = 0 + + converted = SyncRow( + (uid or synth_uid), + synth_uid, + (dev or synth_dev), + synth_dev, + msg.get("meta").get("Timestamp"), + msg.get("method"), + code, + bucket, + t, + msg.get("user_agent_browser"), + msg.get("user_agent_version"), + msg.get("user_agent_os"), + msg.get("meta").get("Hostname"), + ) + return converted +
converted = sync_records.map(lambda x: convert(x)) +
converted = converted.filter(lambda x: x is not None) +
from pyspark.sql import SQLContext +sync_df = sqlContext.createDataFrame(converted) +sync_df.printSchema() +
root + |-- uid: string (nullable = true) + |-- s_uid: string (nullable = true) + |-- dev: string (nullable = true) + |-- s_dev: string (nullable = true) + |-- ts: long (nullable = true) + |-- method: string (nullable = true) + |-- code: long (nullable = true) + |-- bucket: string (nullable = true) + |-- t: double (nullable = true) + |-- ua_browser: string (nullable = true) + |-- ua_version: double (nullable = true) + |-- ua_os: string (nullable = true) + |-- host: string (nullable = true) +
# Determine if we need to repartition. +# A record is something like 112 bytes, so figure out how many partitions +# we need to end up with reasonably-sized files. +records_per_partition = 2500000 +total_records = sync_df.count() +print "Found {} sync records".format(total_records) +
import math +num_partitions = int(math.ceil(float(total_records) / records_per_partition)) + +if num_partitions != sync_df.rdd.getNumPartitions(): + print "Repartitioning with {} partitions".format(num_partitions) + sync_df = sync_df.repartition(num_partitions) + +# Store data +sync_log_s3path = "{}/sync_log/v1/day={}".format(dest_s3_prefix, target_day) +sync_df.write.parquet(sync_log_s3path, mode="overwrite") +
# Transform, compute and store rollups +sync_df.registerTempTable("sync") +sql_transform = ''' + select + uid, + dev, + ts, + t, + case + when substring(ua_os,0,7) in ('iPad', 'iPod', 'iPhone') then 'ios' + when substring(ua_os,0,7) = 'Android' then 'android' + when substring(ua_os,0,7) = 'Windows' then 'windows' + when substring(ua_os,0,7) = 'Macinto' then 'mac' + when substring(ua_os,0,7) = 'Linux' then 'linux' + when ua_os is null then 'unknown' + else 'other' + end as ua_os, + ua_browser, + ua_version, + case method when 'POST' then 1 end as posts, + case method when 'GET' then 1 end as gets, + case method when 'PUT' then 1 end as puts, + case method when 'DELETE' then 1 end as dels, + case when code < 300 then 1 end as aoks, + case when code > 399 and code < 500 then 1 end as oops, + case when code > 499 and code < 999 then 1 end as fups, + case when bucket = 'clients' and method = 'GET' then 1 end as r_clients, + case when bucket = 'crypto' and method = 'GET' then 1 end as r_crypto, + case when bucket = 'forms' and method = 'GET' then 1 end as r_forms, + case when bucket = 'history' and method = 'GET' then 1 end as r_history, + case when bucket = 'keys' and method = 'GET' then 1 end as r_keys, + case when bucket = 'meta' and method = 'GET' then 1 end as r_meta, + case when bucket = 'bookmarks' and method = 'GET' then 1 end as r_bookmarks, + case when bucket = 'prefs' and method = 'GET' then 1 end as r_prefs, + case when bucket = 'tabs' and method = 'GET' then 1 end as r_tabs, + case when bucket = 'passwords' and method = 'GET' then 1 end as r_passwords, + case when bucket = 'addons' and method = 'GET' then 1 end as r_addons, + case when bucket = 'clients' and method = 'POST' then 1 end as w_clients, + case when bucket = 'crypto' and method = 'POST' then 1 end as w_crypto, + case when bucket = 'forms' and method = 'POST' then 1 end as w_forms, + case when bucket = 'history' and method = 'POST' then 1 end as w_history, + case when bucket = 'keys' and method = 'POST' then 1 end as w_keys, + case when bucket = 'meta' and method = 'POST' then 1 end as w_meta, + case when bucket = 'bookmarks' and method = 'POST' then 1 end as w_bookmarks, + case when bucket = 'prefs' and method = 'POST' then 1 end as w_prefs, + case when bucket = 'tabs' and method = 'POST' then 1 end as w_tabs, + case when bucket = 'passwords' and method = 'POST' then 1 end as w_passwords, + case when bucket = 'addons' and method = 'POST' then 1 end as w_addons + from sync +''' + +transformed = sqlContext.sql(sql_transform) +
transformed.registerTempTable("tx") + +sql_device_activity = ''' + select + uid, + dev, + max(ua_os) as ua_os, + max(ua_browser) as ua_browser, + max(ua_version) as ua_version, + min(t) as min_t, + max(t) as max_t, + sum(posts) as posts, + sum(gets) as gets, + sum(puts) as puts, + sum(dels) as dels, + sum(aoks) as aoks, + sum(oops) as oops, + sum(fups) as fups, + sum(r_clients) as r_clients, + sum(r_crypto) as r_crypto, + sum(r_forms) as r_forms, + sum(r_history) as r_history, + sum(r_keys) as r_keys, + sum(r_meta) as r_meta, + sum(r_bookmarks) as r_bookmarks, + sum(r_prefs) as r_prefs, + sum(r_tabs) as r_tabs, + sum(r_passwords) as r_passwords, + sum(r_addons) as r_addons, + sum(w_clients) as w_clients, + sum(w_crypto) as w_crypto, + sum(w_forms) as w_forms, + sum(w_history) as w_history, + sum(w_keys) as w_keys, + sum(w_meta) as w_meta, + sum(w_bookmarks) as w_bookmarks, + sum(w_prefs) as w_prefs, + sum(w_tabs) as w_tabs, + sum(w_passwords) as w_passwords, + sum(w_addons) as w_addons + from tx group by uid, dev +''' +rolled_up = sqlContext.sql(sql_device_activity) +
# Store device activity rollups +sync_log_device_activity_s3base = "{}/sync_log_device_activity/v1".format(dest_s3_prefix) +sync_log_device_activity_s3path = "{}/day={}".format(sync_log_device_activity_s3base, target_day) + +# TODO: Do we need to repartition? +rolled_up.repartition(5).write.parquet(sync_log_device_activity_s3path, mode="overwrite") +
def compute_device_counts(device_activity, target_day): + device_activity.registerTempTable("device_activity") + df = "%Y%m%d" + last_week_date = dt.strptime(target_day, df) - timedelta(7) + last_week = dt.strftime(last_week_date, df) + sql_device_counts = """ + select + uid, + count(distinct dev) as devs + from + (select + uid, + dev + from device_activity + where uid in + (select distinct(uid) from device_activity where day = '{}') + and day > '{}' + and day <= '{}') + group by uid + """.format(target_day, last_week, target_day) + + return sqlContext.sql(sql_device_counts) +
# Compute and store device counts + +# Re-read device activity data from S3 so we can look at historic info +device_activity = sqlContext.read.parquet(sync_log_device_activity_s3base) + +device_counts = compute_device_counts(device_activity, target_day) + +sync_log_device_counts_s3path = "{}/sync_log_device_counts/v1/day={}".format(dest_s3_prefix, target_day) +device_counts.repartition(1).write.parquet(sync_log_device_counts_s3path, mode="overwrite") +
from datetime import * +import dateutil.parser +from pyspark.sql.types import * + +from moztelemetry import get_pings_properties +from moztelemetry.dataset import Dataset + + +class ColumnConfig: + def __init__(self, name, path, cleaning_func, struct_type): + self.name = name + self.path = path + self.cleaning_func = cleaning_func + self.struct_type = struct_type + +class DataFrameConfig: + def __init__(self, col_configs): + self.columns = [ColumnConfig(*col) for col in col_configs] + + def toStructType(self): + return StructType(map( + lambda col: StructField(col.name, col.struct_type, True), + self.columns)) + + def get_names(self): + return map(lambda col: col.name, self.columns) + + def get_paths(self): + return map(lambda col: col.path, self.columns) + + + +def pings_to_df(sqlContext, pings, data_frame_config): + """Performs simple data pipelining on raw pings + + Arguments: + data_frame_config: a list of tuples of the form: + (name, path, cleaning_func, column_type) + """ + def build_cell(ping, column_config): + """Takes a json ping and a column config and returns a cleaned cell""" + raw_value = ping[column_config.path] + func = column_config.cleaning_func + if func is not None: + return func(raw_value) + else: + return raw_value + + def ping_to_row(ping): + return [build_cell(ping, col) for col in data_frame_config.columns] + + filtered_pings = get_pings_properties(pings, data_frame_config.get_paths()) + + return sqlContext.createDataFrame( + filtered_pings.map(ping_to_row), + schema = data_frame_config.toStructType()) + +def __main__(sc, sqlContext, submission_date): + if submission_date is None: + submission_date = (date.today() - timedelta(1)).strftime("%Y%m%d") + get_doctype_pings = lambda docType: Dataset.from_source("telemetry") \ + .where(docType=docType) \ + .where(submissionDate=submission_date) \ + .where(appName="Firefox") \ + .records(sc) + + return pings_to_df( + sqlContext, + get_doctype_pings("testpilottest"), + DataFrameConfig([ + ("method", "payload/payload/method", None, StringType()), + ("id", "payload/payload/id", None, StringType()), + ("type", "payload/payload/type", None, StringType()), + ("object", "payload/payload/object", None, StringType()), + ("category", "payload/payload/category", None, StringType()), + ("variant", "payload/payload/variant", None, StringType()), + ("details", "payload/payload/details", None, StringType()), + ("sentiment", "payload/payload/sentiment", None, IntegerType()), + ("reason", "payload/payload/reason", None, StringType()), + ("adBlocker", "payload/payload/adBlocker", None, BooleanType()), + ("addons", "payload/payload/addons", None, ArrayType(StringType())), + ("channel", "payload/payload/channel", None, StringType()), + ("hostname", "payload/payload/hostname", None, StringType()), + ("language", "payload/payload/language", None, StringType()), + ("openTabs", "payload/payload/openTabs", None, IntegerType()), + ("openWindows", "payload/payload/openWindows", None, IntegerType()), + ("platform", "payload/payload/platform", None, StringType()), + ("protocol", "payload/payload/protocol", None, StringType()), + ("telemetryId", "payload/payload/telemetryId", None, StringType()), + ("timerContentLoaded", "payload/payload/timerContentLoaded", None, LongType()), + ("timerFirstInteraction", "payload/payload/timerFirstInteraction", None, LongType()), + ("timerFirstPaint", "payload/payload/timerFirstPaint", None, LongType()), + ("timerWindowLoad", "payload/payload/timerWindowLoad", None, LongType()), + ("inner_timestamp", "payload/payload/timestamp", None, LongType()), + ("fx_version", "payload/payload/fx_version", None, StringType()), + ("creation_date", "creationDate", dateutil.parser.parse, TimestampType()), + ("test", "payload/test", None, StringType()), + ("variants", "payload/variants", None, StringType()), + ("timestamp", "payload/timestamp", None, LongType()), + ("version", "payload/version", None, StringType()) + ])).filter("test = 'pulse@mozilla.com'") +
submission_date = (date.today() - timedelta(1)).strftime("%Y%m%d") +
tpt = __main__(sc, sqlContext, submission_date) +
tpt.repartition(1).write.parquet('s3://telemetry-parquet/testpilot/txp_pulse/v1/submission_date={}'.format(submission_date)) +
from datetime import * +import dateutil.parser +from pyspark.sql.types import * + +from moztelemetry import get_pings_properties +from moztelemetry.dataset import Dataset + + +class ColumnConfig: + def __init__(self, name, path, cleaning_func, struct_type): + self.name = name + self.path = path + self.cleaning_func = cleaning_func + self.struct_type = struct_type + +class DataFrameConfig: + def __init__(self, col_configs): + self.columns = [ColumnConfig(*col) for col in col_configs] + + def toStructType(self): + return StructType(map( + lambda col: StructField(col.name, col.struct_type, True), + self.columns)) + + def get_names(self): + return map(lambda col: col.name, self.columns) + + def get_paths(self): + return map(lambda col: col.path, self.columns) + + + +def pings_to_df(sqlContext, pings, data_frame_config): + """Performs simple data pipelining on raw pings + + Arguments: + data_frame_config: a list of tuples of the form: + (name, path, cleaning_func, column_type) + """ + def build_cell(ping, column_config): + """Takes a json ping and a column config and returns a cleaned cell""" + raw_value = ping[column_config.path] + func = column_config.cleaning_func + if func is not None: + return func(raw_value) + else: + return raw_value + + def ping_to_row(ping): + return [build_cell(ping, col) for col in data_frame_config.columns] + + filtered_pings = get_pings_properties(pings, data_frame_config.get_paths()) + + return sqlContext.createDataFrame( + filtered_pings.map(ping_to_row), + schema = data_frame_config.toStructType()) + +def __main__(sc, sqlContext, submission_date): + if submission_date is None: + submission_date = (date.today() - timedelta(1)).strftime("%Y%m%d") + get_doctype_pings = lambda docType: Dataset.from_source("telemetry") \ + .where(docType=docType) \ + .where(submissionDate=submission_date) \ + .where(appName="Firefox") \ + .records(sc) + + return pings_to_df( + sqlContext, + get_doctype_pings("testpilottest"), + DataFrameConfig([ + ("method", "payload/payload/method", None, StringType()), + ("id", "payload/payload/id", None, StringType()), + ("type", "payload/payload/type", None, StringType()), + ("object", "payload/payload/object", None, StringType()), + ("category", "payload/payload/category", None, StringType()), + ("variant", "payload/payload/variant", None, StringType()), + ("details", "payload/payload/details", None, StringType()), + ("sentiment", "payload/payload/sentiment", None, IntegerType()), + ("reason", "payload/payload/reason", None, StringType()), + ("adBlocker", "payload/payload/adBlocker", None, BooleanType()), + ("addons", "payload/payload/addons", None, ArrayType(StringType())), + ("channel", "payload/payload/channel", None, StringType()), + ("hostname", "payload/payload/hostname", None, StringType()), + ("language", "payload/payload/language", None, StringType()), + ("openTabs", "payload/payload/openTabs", None, IntegerType()), + ("openWindows", "payload/payload/openWindows", None, IntegerType()), + ("platform", "payload/payload/platform", None, StringType()), + ("protocol", "payload/payload/protocol", None, StringType()), + ("telemetryId", "payload/payload/telemetryId", None, StringType()), + ("timerContentLoaded", "payload/payload/timerContentLoaded", None, LongType()), + ("timerFirstInteraction", "payload/payload/timerFirstInteraction", None, LongType()), + ("timerFirstPaint", "payload/payload/timerFirstPaint", None, LongType()), + ("timerWindowLoad", "payload/payload/timerWindowLoad", None, LongType()), + ("inner_timestamp", "payload/payload/timestamp", None, LongType()), + ("fx_version", "payload/payload/fx_version", None, StringType()), + ("creation_date", "creationDate", dateutil.parser.parse, TimestampType()), + ("test", "payload/test", None, StringType()), + ("variants", "payload/variants", None, StringType()), + ("timestamp", "payload/timestamp", None, LongType()), + ("version", "payload/version", None, StringType()) + ])).filter("test = 'pulse@mozilla.com'") +
submission_date = (date.today() - timedelta(1)).strftime("%Y%m%d") +
tpt = __main__(sc, sqlContext, submission_date) +
tpt.repartition(1).write.parquet('s3://telemetry-parquet/testpilot/txp_pulse/v1/submission_date={}'.format(submission_date)) +
from datetime import * +import dateutil.parser +from pyspark.sql.types import * +import boto3 + +from moztelemetry import get_pings_properties +from moztelemetry.dataset import Dataset + + +class ColumnConfig: + def __init__(self, name, path, cleaning_func, struct_type): + self.name = name + self.path = path + self.cleaning_func = cleaning_func + self.struct_type = struct_type + +class DataFrameConfig: + def __init__(self, col_configs): + self.columns = [ColumnConfig(*col) for col in col_configs] + + def toStructType(self): + return StructType(map( + lambda col: StructField(col.name, col.struct_type, True), + self.columns)) + + def get_names(self): + return map(lambda col: col.name, self.columns) + + def get_paths(self): + return map(lambda col: col.path, self.columns) + + + +def pings_to_df(sqlContext, pings, data_frame_config): + """Performs simple data pipelining on raw pings + + Arguments: + data_frame_config: a list of tuples of the form: + (name, path, cleaning_func, column_type) + """ + def build_cell(ping, column_config): + """Takes a json ping and a column config and returns a cleaned cell""" + raw_value = ping[column_config.path] + func = column_config.cleaning_func + if func is not None: + return func(raw_value) + else: + return raw_value + + def ping_to_row(ping): + return [build_cell(ping, col) for col in data_frame_config.columns] + + filtered_pings = get_pings_properties(pings, data_frame_config.get_paths()) + + return sqlContext.createDataFrame( + filtered_pings.map(ping_to_row), + schema = data_frame_config.toStructType()) + +def save_df(df, name, date_partition, partitions=1): + if date_partition is not None: + partition_str = "/submission={day}".format(day=date_partition) + else: + partition_str="" + + + path_fmt = "s3n://telemetry-parquet/harter/cliqz_{name}/v1{partition_str}" + path = path_fmt.format(name=name, partition_str=partition_str) + df.coalesce(partitions).write.mode("overwrite").parquet(path) + +def __main__(sc, sqlContext, submission_date): + if submission_date is None: + submission_date = (date.today() - timedelta(1)).strftime("%Y%m%d") + get_doctype_pings = lambda docType: Dataset.from_source("telemetry") \ + .where(docType=docType) \ + .where(submissionDate=submission_date) \ + .where(appName="Firefox") \ + .records(sc) + + old_st = pings_to_df( + sqlContext, + get_doctype_pings("testpilottest"), + DataFrameConfig([ + ("client_id", "clientId", None, StringType()), + ("event", "payload/payload/testpilotPingData/event", None, StringType()), + ("snooze_time", "payload/payload/testpilotPingData/snooze_time", None, LongType()), + ("snooze_time_type", "payload/payload/testpilotPingData/snooze_time_type", None, StringType()), + ("creation_date", "creationDate", dateutil.parser.parse, TimestampType()), + ("test", "payload/test", None, StringType()), + ("variants", "payload/variants", None, StringType()), + ("timestamp", "payload/timestamp", None, LongType()), + ("version", "payload/version", None, StringType()) + ])).filter("event IS NOT NULL") \ + .filter("test = 'snoozetabs@mozilla.com'") + + new_st = pings_to_df( + sqlContext, + get_doctype_pings("testpilottest"), + DataFrameConfig([ + ("client_id", "clientId", None, StringType()), + ("event", "payload/payload/event", None, StringType()), + ("snooze_time", "payload/payload/snooze_time", None, LongType()), + ("snooze_time_type", "payload/payload/snooze_time_type", None, StringType()), + ("creation_date", "creationDate", dateutil.parser.parse, TimestampType()), + ("test", "payload/test", None, StringType()), + ("variants", "payload/variants", None, StringType()), + ("timestamp", "payload/timestamp", None, LongType()), + ("version", "payload/version", None, StringType()) + ])).filter("event IS NOT NULL") \ + .filter("test = 'snoozetabs@mozilla.com'") + return old_st.union(new_st) +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
tpt = __main__(sc, sqlContext, submission_date) +
tpt.repartition(1).write.parquet('s3://telemetry-parquet/testpilot/txp_snoozetabs/v2/submission_date={}'.format(submission_date)) +
from datetime import * +import dateutil.parser +from pyspark.sql.types import * +import boto3 + +from moztelemetry import get_pings_properties +from moztelemetry.dataset import Dataset + + +class ColumnConfig: + def __init__(self, name, path, cleaning_func, struct_type): + self.name = name + self.path = path + self.cleaning_func = cleaning_func + self.struct_type = struct_type + +class DataFrameConfig: + def __init__(self, col_configs): + self.columns = [ColumnConfig(*col) for col in col_configs] + + def toStructType(self): + return StructType(map( + lambda col: StructField(col.name, col.struct_type, True), + self.columns)) + + def get_names(self): + return map(lambda col: col.name, self.columns) + + def get_paths(self): + return map(lambda col: col.path, self.columns) + + + +def pings_to_df(sqlContext, pings, data_frame_config): + """Performs simple data pipelining on raw pings + + Arguments: + data_frame_config: a list of tuples of the form: + (name, path, cleaning_func, column_type) + """ + def build_cell(ping, column_config): + """Takes a json ping and a column config and returns a cleaned cell""" + raw_value = ping[column_config.path] + func = column_config.cleaning_func + if func is not None: + return func(raw_value) + else: + return raw_value + + def ping_to_row(ping): + return [build_cell(ping, col) for col in data_frame_config.columns] + + filtered_pings = get_pings_properties(pings, data_frame_config.get_paths()) + + return sqlContext.createDataFrame( + filtered_pings.map(ping_to_row), + schema = data_frame_config.toStructType()) + +def save_df(df, name, date_partition, partitions=1): + if date_partition is not None: + partition_str = "/submission={day}".format(day=date_partition) + else: + partition_str="" + + + path_fmt = "s3n://telemetry-parquet/harter/cliqz_{name}/v1{partition_str}" + path = path_fmt.format(name=name, partition_str=partition_str) + df.coalesce(partitions).write.mode("overwrite").parquet(path) + +def __main__(sc, sqlContext, submission_date): + if submission_date is None: + submission_date = (date.today() - timedelta(1)).strftime("%Y%m%d") + get_doctype_pings = lambda docType: Dataset.from_source("telemetry") \ + .where(docType=docType) \ + .where(submissionDate=submission_date) \ + .where(appName="Firefox") \ + .records(sc) + + old_st = pings_to_df( + sqlContext, + get_doctype_pings("testpilottest"), + DataFrameConfig([ + ("client_id", "clientId", None, StringType()), + ("event", "payload/payload/testpilotPingData/event", None, StringType()), + ("snooze_time", "payload/payload/testpilotPingData/snooze_time", None, LongType()), + ("snooze_time_type", "payload/payload/testpilotPingData/snooze_time_type", None, StringType()), + ("creation_date", "creationDate", dateutil.parser.parse, TimestampType()), + ("test", "payload/test", None, StringType()), + ("variants", "payload/variants", None, StringType()), + ("timestamp", "payload/timestamp", None, LongType()), + ("version", "payload/version", None, StringType()) + ])).filter("event IS NOT NULL") \ + .filter("test = 'snoozetabs@mozilla.com'") + + new_st = pings_to_df( + sqlContext, + get_doctype_pings("testpilottest"), + DataFrameConfig([ + ("client_id", "clientId", None, StringType()), + ("event", "payload/payload/event", None, StringType()), + ("snooze_time", "payload/payload/snooze_time", None, LongType()), + ("snooze_time_type", "payload/payload/snooze_time_type", None, StringType()), + ("creation_date", "creationDate", dateutil.parser.parse, TimestampType()), + ("test", "payload/test", None, StringType()), + ("variants", "payload/variants", None, StringType()), + ("timestamp", "payload/timestamp", None, LongType()), + ("version", "payload/version", None, StringType()) + ])).filter("event IS NOT NULL") \ + .filter("test = 'snoozetabs@mozilla.com'") + return old_st.union(new_st) +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
tpt = __main__(sc, sqlContext, submission_date) +
tpt.repartition(1).write.parquet('s3://telemetry-parquet/testpilot/txp_snoozetabs/v2/submission_date={}'.format(submission_date)) +
NOTE: In the TL,DR, optimize for clarity and comprehensiveness. The goal is to convey the post with the least amount of friction, especially since ipython/beakers require much more scrolling than blog posts. Make the reader get a correct understanding of the post’s takeaway, and the points supporting that takeaway without having to strain through paragraphs and tons of prose. Bullet points are great here, but are up to you. Try to avoid academic paper style abstracts.
+NOTE: optimize in this section for context setting, as specifically as you can. For instance, this post is generally a set of standards for work in the repo. The specific motivation is to have least friction to current workflow while being able to painlessly aggregate it later.
+The knowledge repo was created to consolidate research work that is currently scattered in emails, blogposts, and presentations, so that people didn’t redo their work.
+import pandas as pd +import numpy as np +import matplotlib + +from matplotlib import pyplot as plt +from moztelemetry.dataset import Dataset +from moztelemetry import get_pings_properties, get_one_ping_per_client +
Unable to parse whitelist (/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
The goal of this example is to determine if Firefox has a similar startup time distribution on all Operating Systems. Let’s start by fetching 10% of Telemetry submissions for a given submission date…
+Dataset.from_source("telemetry").schema +
[u'submissionDate', + u'sourceName', + u'sourceVersion', + u'docType', + u'appName', + u'appUpdateChannel', + u'appVersion', + u'appBuildId'] +
pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(submissionDate="20161101") \ + .where(appUpdateChannel="nightly") \ + .records(sc, sample=0.1) +
… and extract only the attributes we need from the Telemetry submissions:
+subset = get_pings_properties(pings, ["clientId", + "environment/system/os/name", + "payload/simpleMeasurements/firstPaint"]) +
To prevent pseudoreplication, let’s consider only a single submission for each client. As this step requires a distributed shuffle, it should always be run only after extracting the attributes of interest with get_pings_properties.
+subset = get_one_ping_per_client(subset) +
Let’s group the startup timings by OS:
+grouped = subset.map(lambda p: (p["environment/system/os/name"], p["payload/simpleMeasurements/firstPaint"])).groupByKey().collectAsMap() +
And finally plot the data:
+frame = pd.DataFrame({x: np.log10(pd.Series(list(y))) for x, y in grouped.items()}) +plt.figure(figsize=(17, 7)) +frame.boxplot(return_type="axes") +plt.ylabel("log10(firstPaint)") +plt.xlabel("Operating System") +plt.show() +
NOTE: in graphs, optimize for being able to stand alone. Put enough labeling in your graph to be understood on its own. When aggregating and putting things in presentations, you won’t have to recreate and add code to each plot to make it understandable without the entire post around it. Will it be understandable without several paragraphs?
+Put all the stuff here that is not necessary for supporting the points above. Good place for documentation without distraction.
+NOTE: In the TL,DR, optimize for clarity and comprehensiveness. The goal is to convey the post with the least amount of friction, especially since ipython/beakers require much more scrolling than blog posts. Make the reader get a correct understanding of the post’s takeaway, and the points supporting that takeaway without having to strain through paragraphs and tons of prose. Bullet points are great here, but are up to you. Try to avoid academic paper style abstracts.
+NOTE: optimize in this section for context setting, as specifically as you can. For instance, this post is generally a set of standards for work in the repo. The specific motivation is to have least friction to current workflow while being able to painlessly aggregate it later.
+The knowledge repo was created to consolidate research work that is currently scattered in emails, blogposts, and presentations, so that people didn’t redo their work.
+import pandas as pd +import numpy as np +import matplotlib + +from matplotlib import pyplot as plt +from moztelemetry.dataset import Dataset +from moztelemetry import get_pings_properties, get_one_ping_per_client +
Unable to parse whitelist (/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
The goal of this example is to determine if Firefox has a similar startup time distribution on all Operating Systems. Let’s start by fetching 10% of Telemetry submissions for a given submission date…
+Dataset.from_source("telemetry").schema +
[u'submissionDate', + u'sourceName', + u'sourceVersion', + u'docType', + u'appName', + u'appUpdateChannel', + u'appVersion', + u'appBuildId'] +
pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(submissionDate="20161101") \ + .where(appUpdateChannel="nightly") \ + .records(sc, sample=0.1) +
… and extract only the attributes we need from the Telemetry submissions:
+subset = get_pings_properties(pings, ["clientId", + "environment/system/os/name", + "payload/simpleMeasurements/firstPaint"]) +
To prevent pseudoreplication, let’s consider only a single submission for each client. As this step requires a distributed shuffle, it should always be run only after extracting the attributes of interest with get_pings_properties.
+subset = get_one_ping_per_client(subset) +
Let’s group the startup timings by OS:
+grouped = subset.map(lambda p: (p["environment/system/os/name"], p["payload/simpleMeasurements/firstPaint"])).groupByKey().collectAsMap() +
And finally plot the data:
+frame = pd.DataFrame({x: np.log10(pd.Series(list(y))) for x, y in grouped.items()}) +plt.figure(figsize=(17, 7)) +frame.boxplot(return_type="axes") +plt.ylabel("log10(firstPaint)") +plt.xlabel("Operating System") +plt.show() +
NOTE: in graphs, optimize for being able to stand alone. Put enough labeling in your graph to be understood on its own. When aggregating and putting things in presentations, you won’t have to recreate and add code to each plot to make it understandable without the entire post around it. Will it be understandable without several paragraphs?
+Put all the stuff here that is not necessary for supporting the points above. Good place for documentation without distraction.
+Can we get rid of addonHistograms?
+import pandas as pd +import numpy as np +import matplotlib + +from matplotlib import pyplot as plt +from moztelemetry.dataset import Dataset +from moztelemetry import get_pings_properties, get_one_ping_per_client +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
Let’s just look at a non-representative 10% of main pings gathered on a recent Tuesday.
+pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(submissionDate="20170328") \ + .records(sc, sample=0.1) +
subset = get_pings_properties(pings, ["payload/addonHistograms"]) +
full_count = subset.count() +full_count +
37815981 +
filtered = subset.filter(lambda p: p["payload/addonHistograms"] is not None) +filtered_count = filtered.count() +filtered_count +
25794 +
1.0 * filtered_count / full_count +
0.0006820925787962502 +
addons = filtered.flatMap(lambda p: p['payload/addonHistograms'].keys()).map(lambda key: (key, 1)) +
addons.countByKey() +
defaultdict(int,
+ {u'Firebug': 92,
+ u'shumway@research.mozilla.org': 15,
+ u'uriloader@pdf.js': 4})
+Wow, so most of those addonHistograms sections are empty.
+…And those that aren’t are from defunct data collection sources. Looks like we can remove this without too many complaint. Excellent.
+Can we get rid of addonHistograms?
+import pandas as pd +import numpy as np +import matplotlib + +from matplotlib import pyplot as plt +from moztelemetry.dataset import Dataset +from moztelemetry import get_pings_properties, get_one_ping_per_client +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
Let’s just look at a non-representative 10% of main pings gathered on a recent Tuesday.
+pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(submissionDate="20170328") \ + .records(sc, sample=0.1) +
subset = get_pings_properties(pings, ["payload/addonHistograms"]) +
full_count = subset.count() +full_count +
37815981 +
filtered = subset.filter(lambda p: p["payload/addonHistograms"] is not None) +filtered_count = filtered.count() +filtered_count +
25794 +
1.0 * filtered_count / full_count +
0.0006820925787962502 +
addons = filtered.flatMap(lambda p: p['payload/addonHistograms'].keys()).map(lambda key: (key, 1)) +
addons.countByKey() +
defaultdict(int,
+ {u'Firebug': 92,
+ u'shumway@research.mozilla.org': 15,
+ u'uriloader@pdf.js': 4})
+Wow, so most of those addonHistograms sections are empty.
+…And those that aren’t are from defunct data collection sources. Looks like we can remove this without too many complaint. Excellent.
+Generate data for the Firefox Application Update Out Of Date dashboard
+import datetime as dt +import re +import urllib2 +import ujson as json +import boto3 +from boto3.s3.transfer import S3Transfer + +%pylab inline +
sc.defaultParallelism +
Get the time when this job started.
+start_time = dt.datetime.now() +print "Start: " + str(start_time.strftime("%Y-%m-%d %H:%M:%S")) +
Common settings.
+no_upload = None +today_str = None + +# Uncomment out the following two lines and adjust |today_str| as necessary to run manually without uploading the json. +no_upload = True +# today_str = "20161226" + +channel_to_process = "release" +min_version = 42 +up_to_date_releases = 2 +weeks_of_subsession_data = 12 +min_update_ping_count = 4 +min_subsession_hours = 2 +min_subsession_seconds = min_subsession_hours * 60 * 60 +
Get the settings based on dates.
+if today_str is None: + today_str = start_time.strftime("%Y%m%d") + +assert (today_str is not None), "The date environment parameter is missing." +today = dt.datetime.strptime(today_str, "%Y%m%d").date() + +# MON = 0, SAT = 5, SUN = 6 -> SUN = 0, MON = 1, SAT = 6 +day_index = (today.weekday() + 1) % 7 +# Filename used to save the report's JSON +report_filename = (today - datetime.timedelta(day_index)).strftime("%Y%m%d") +# Maximum report date which is the previous Saturday +max_report_date = today - datetime.timedelta(7 + day_index - 6) +# Suffix of the longitudinal datasource name to use +longitudinal_suffix = max_report_date.strftime("%Y%m%d") +# String used in the SQL queries to limit records to the maximum report date. +# Since the queries use less than this is the day after the previous Saturday. +max_report_date_sql = (max_report_date + dt.timedelta(days=1)).strftime("%Y-%m-%d") +# The Sunday prior to the last Saturday +min_report_date = max_report_date - dt.timedelta(days=6) +# String used in the SQL queries to limit records to the minimum report date +# Since the queries use greater than this is six days prior to the previous Saturday. +min_report_date_sql = min_report_date.strftime("%Y-%m-%d") +# Date used to limit records to the number of weeks specified by +# weeks_of_subsession_data prior to the maximum report date +min_subsession_date = max_report_date - dt.timedelta(weeks=weeks_of_subsession_data) +# Date used to compute the latest version from firefox_history_major_releases.json +latest_ver_date_str = (max_report_date - dt.timedelta(days=7)).strftime("%Y-%m-%d") + +print "max_report_date : " + max_report_date.strftime("%Y%m%d") +print "max_report_date_sql : " + max_report_date_sql +print "min_report_date : " + min_report_date.strftime("%Y%m%d") +print "min_report_date_sql : " + min_report_date_sql +print "min_subsession_date : " + min_subsession_date.strftime("%Y%m%d") +print "report_filename : " + report_filename +print "latest_ver_date_str : " + latest_ver_date_str +
Get the latest Firefox version available based on the date.
+def latest_version_on_date(date, major_releases): + latest_date = u"1900-01-01" + latest_ver = 0 + for version, release_date in major_releases.iteritems(): + version_int = int(version.split(".")[0]) + if release_date <= date and release_date >= latest_date and version_int >= latest_ver: + latest_date = release_date + latest_ver = version_int + + return latest_ver + +major_releases_json = urllib2.urlopen("https://product-details.mozilla.org/1.0/firefox_history_major_releases.json").read() +major_releases = json.loads(major_releases_json) +latest_version = latest_version_on_date(latest_ver_date_str, major_releases) +earliest_up_to_date_version = str(latest_version - up_to_date_releases) + +print "Latest Version: " + str(latest_version) +
Create a dictionary to store the general settings that will be written to a JSON file.
+report_details_dict = {"latestVersion": latest_version, + "upToDateReleases": up_to_date_releases, + "minReportDate": min_report_date.strftime("%Y-%m-%d"), + "maxReportDate": max_report_date.strftime("%Y-%m-%d"), + "weeksOfSubsessionData": weeks_of_subsession_data, + "minSubsessionDate": min_subsession_date.strftime("%Y-%m-%d"), + "minSubsessionHours": min_subsession_hours, + "minSubsessionSeconds": min_subsession_seconds, + "minUpdatePingCount": min_update_ping_count} +report_details_dict +
Create the common SQL FROM clause.
+# Note: using the parquet is as fast as using 'FROM longitudinal_vYYYMMDD' +# and it allows the query to go further back in time. + +#longitudinal_from_sql = ("FROM longitudinal_v{} ").format(longitudinal_suffix) +longitudinal_from_sql = ("FROM parquet.`s3://telemetry-parquet/longitudinal/v{}` ").format(longitudinal_suffix) +longitudinal_from_sql +
Create the common build.version SQL WHERE clause.
+build_version_where_sql = "(build.version[0] RLIKE '^[0-9]{2,3}\.0[\.0-9]*$' OR build.version[0] = '50.1.0')" +build_version_where_sql +
Create the remaining common SQL WHERE clause.
+common_where_sql = ("" + "build.application_name[0] = 'Firefox' AND " + "DATEDIFF(SUBSTR(subsession_start_date[0], 0, 10), '{}') >= 0 AND " + "DATEDIFF(SUBSTR(subsession_start_date[0], 0, 10), '{}') < 0 AND " + "settings.update.channel[0] = '{}'" +"").format(min_report_date_sql, + max_report_date_sql, + channel_to_process) +common_where_sql +
Create the SQL for the summary query.
+summary_sql = ("" +"SELECT " + "COUNT(CASE WHEN build.version[0] >= '{}.' AND build.version[0] < '{}.' THEN 1 END) AS versionUpToDate, " + "COUNT(CASE WHEN build.version[0] < '{}.' AND build.version[0] >= '{}.' THEN 1 END) AS versionOutOfDate, " + "COUNT(CASE WHEN build.version[0] < '{}.' THEN 1 END) AS versionTooLow, " + "COUNT(CASE WHEN build.version[0] > '{}.' THEN 1 END) AS versionTooHigh, " + "COUNT(CASE WHEN NOT build.version[0] > '0' THEN 1 END) AS versionMissing " +"{} " +"WHERE " + "{} AND " + "{}" +"").format(str(latest_version - up_to_date_releases), + str(latest_version + 1), + str(latest_version - up_to_date_releases), + str(min_version), + str(min_version), + str(latest_version + 1), + longitudinal_from_sql, + common_where_sql, + build_version_where_sql) +summary_sql +
Run the summary SQL query.
+summaryDF = sqlContext.sql(summary_sql) +
Create a dictionary to store the results from the summary query that will be written to a JSON file.
+summary_dict = summaryDF.first().asDict() +summary_dict +
Create the SQL for the out of date details query.
+# Only query for the columns and the records that are used to optimize +# for speed. Adding update_state_code_partial_stage and +# update_state_code_complete_stage increased the time it takes this +# notebook to run by 50 seconds when using 4 clusters. + +# Creating a temporary table of the data after the filters have been +# applied and joining it with the original datasource to include +# other columns doesn't appear to speed up the process but it doesn't +# appear to slow it down either so all columns of interest are in this +# query. + +out_of_date_details_sql = ("" +"SELECT " + "client_id, " + "build.version, " + "session_length, " + "subsession_start_date, " + "subsession_length, " + "update_check_code_notify, " + "update_check_extended_error_notify, " + "update_check_no_update_notify, " + "update_not_pref_update_enabled_notify, " + "update_not_pref_update_auto_notify, " + "update_ping_count_notify, " + "update_unable_to_apply_notify, " + "update_download_code_partial, " + "update_download_code_complete, " + "update_state_code_partial_stage, " + "update_state_code_complete_stage, " + "update_state_code_unknown_stage, " + "update_state_code_partial_startup, " + "update_state_code_complete_startup, " + "update_state_code_unknown_startup, " + "update_status_error_code_complete_startup, " + "update_status_error_code_partial_startup, " + "update_status_error_code_unknown_startup, " + "update_status_error_code_complete_stage, " + "update_status_error_code_partial_stage, " + "update_status_error_code_unknown_stage " +"{}" +"WHERE " + "{} AND " + "{} AND " + "build.version[0] < '{}.' AND " + "build.version[0] >= '{}.'" +"").format(longitudinal_from_sql, + common_where_sql, + build_version_where_sql, + str(latest_version - up_to_date_releases), + str(min_version)) +out_of_date_details_sql +
Run the out of date details SQL query.
+out_of_date_details_df = sqlContext.sql(out_of_date_details_sql) +
Create the RDD used to further restrict which clients are out of date +to focus on clients that are of concern and potentially of concern.
+out_of_date_details_rdd = out_of_date_details_df.rdd.cache() +
Create an RDD of out of date telemetry pings that have and don’t have +a previous telemetry ping with a version that is up to date along +with a dictionary of the count of True and False.
+def has_out_of_date_max_version_mapper(d): + ping = d + index = 0 + while (index < len(ping.version)): + if ((ping.version[index] == "50.1.0" or + p.match(ping.version[index])) and + ping.version[index] > earliest_up_to_date_version): + return False, ping + index += 1 + + return True, ping + +# RegEx for a valid release versions except for 50.1.0 which is handled separately. +p = re.compile('^[0-9]{2,3}\\.0[\\.0-9]*$') + +has_out_of_date_max_version_rdd = out_of_date_details_rdd.map(has_out_of_date_max_version_mapper).cache() +has_out_of_date_max_version_dict = has_out_of_date_max_version_rdd.countByKey() +has_out_of_date_max_version_dict +
Create an RDD of the telemetry pings that have a previous telemetry ping +with a version that is up to date.
+has_out_of_date_max_version_true_rdd = has_out_of_date_max_version_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that have and have not +sent an update telemtry ping for any version of Firefox along with a +dictionary of the count of True and False.
+def has_update_ping_mapper(d): + ping = d + if (ping.update_ping_count_notify is not None and + (ping.update_check_code_notify is not None or + ping.update_check_no_update_notify is not None)): + return True, ping + + return False, ping + +has_update_ping_rdd = has_out_of_date_max_version_true_rdd.map(has_update_ping_mapper).cache() +has_update_ping_dict = has_update_ping_rdd.countByKey() +has_update_ping_dict +
Create an RDD of the telemetry pings that have an update telemtry +ping for any version of Firefox.
+has_update_ping_true_rdd = has_update_ping_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that have and have not +ran this version of Firefox for more than the amount of seconds as +specified by min_subsession_seconds along with a dictionary of the +count of True and False.
+def has_min_subsession_length_mapper(d): + ping = d + seconds = 0 + index = 0 + current_version = ping.version[0] + while (seconds < min_subsession_seconds and + index < len(ping.subsession_start_date) and + index < len(ping.version) and + ping.version[index] == current_version): + try: + date = dt.datetime.strptime(ping.subsession_start_date[index][:10], + "%Y-%m-%d").date() + if date < min_subsession_date: + return False, ping + + seconds += ping.subsession_length[index] + index += 1 + except: # catch *all* exceptions + index += 1 + + if seconds >= min_subsession_seconds: + return True, ping + + return False, ping + +has_min_subsession_length_rdd = has_update_ping_true_rdd.map(has_min_subsession_length_mapper).cache() +has_min_subsession_length_dict = has_min_subsession_length_rdd.countByKey() +has_min_subsession_length_dict +
Create an RDD of the telemetry pings that have ran this version of +Firefox for more than the amount of seconds as specified by +min_subsession_seconds.
+has_min_subsession_length_true_rdd = has_min_subsession_length_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that have and have not +sent the minimum number of update pings as specified by +min_update_ping_count for this version of Firefox along with a +dictionary of the count of True and False.
+def has_min_update_ping_count_mapper(d): + ping = d + index = 0 + update_ping_count_total = 0 + current_version = ping.version[0] + while (update_ping_count_total < min_update_ping_count and + index < len(ping.update_ping_count_notify) and + index < len(ping.version) and + ping.version[index] == current_version): + + pingCount = ping.update_ping_count_notify[index] + # Is this an update ping or just a placeholder for the telemetry ping? + if pingCount > 0: + try: + date = dt.datetime.strptime(ping.subsession_start_date[index][:10], + "%Y-%m-%d").date() + if date < min_subsession_date: + return False, ping + + except: # catch *all* exceptions + index += 1 + continue + + # Is there also a valid update check code or no update telemetry ping? + if (ping.update_check_code_notify is not None and + len(ping.update_check_code_notify) > index): + for code_value in ping.update_check_code_notify[index]: + if code_value > 0: + update_ping_count_total += pingCount + index += 1 + continue + + if (ping.update_check_no_update_notify is not None and + len(ping.update_check_no_update_notify) > index and + ping.update_check_no_update_notify[index] > 0): + update_ping_count_total += pingCount + + index += 1 + + if update_ping_count_total < min_update_ping_count: + return False, ping + + return True, ping + +has_min_update_ping_count_rdd = has_min_subsession_length_true_rdd.map(has_min_update_ping_count_mapper).cache() +has_min_update_ping_count_dict = has_min_update_ping_count_rdd.countByKey() +has_min_update_ping_count_dict +
Create an RDD of the telemetry pings that have sent the minimum +number of update pings as specified by min_update_ping_count.
+has_min_update_ping_count_true_rdd = has_min_update_ping_count_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that are supported and +are not supported based on whether they have not received or have +received the unsupported update xml for the last update check along +with a dictionary of the count of True and False.
+def is_supported_mapper(d): + ping = d + index = 0 + update_ping_count_total = 0 + current_version = ping.version[0] + while (update_ping_count_total < min_update_ping_count and + index < len(ping.update_ping_count_notify) and + index < len(ping.version) and + ping.version[index] == current_version): + pingCount = ping.update_ping_count_notify[index] + # Is this an update ping or just a placeholder for the telemetry ping? + if pingCount > 0: + # Is there also a valid update check code or no update telemetry ping? + if (ping.update_check_code_notify is not None and + len(ping.update_check_code_notify) > index and + ping.update_check_code_notify[index][28] > 0): + return False, ping + + index += 1 + + return True, ping + +is_supported_rdd = has_min_update_ping_count_true_rdd.map(is_supported_mapper).cache() +is_supported_dict = is_supported_rdd.countByKey() +is_supported_dict +
Create an RDD of the telemetry pings that are supported based on +whether they have not received or have received the unsupported +update xml for the last update check.
+is_supported_true_rdd = is_supported_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that have and don’t have +the ability to apply an update along with a dictionary of the count +of True and False.
+def is_able_to_apply_mapper(d): + ping = d + index = 0 + current_version = ping.version[0] + while (index < len(ping.update_ping_count_notify) and + index < len(ping.version) and + ping.version[index] == current_version): + if ping.update_ping_count_notify[index] > 0: + # Only check the last value for update_unable_to_apply_notify + # to determine if the client is unable to apply. + if (ping.update_unable_to_apply_notify is not None and + ping.update_unable_to_apply_notify[index] > 0): + return False, ping + + return True, ping + + index += 1 + + raise ValueError("Missing update unable to apply value!") + +is_able_to_apply_rdd = is_supported_true_rdd.map(is_able_to_apply_mapper).cache() +is_able_to_apply_dict = is_able_to_apply_rdd.countByKey() +is_able_to_apply_dict +
Create an RDD of the telemetry pings that have the ability to apply +an update.
+is_able_to_apply_true_rdd = is_able_to_apply_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that have and don’t have +the application.update.enabled preference set to True / False along +with a dictionary of the count of True and False.
+def has_update_enabled_mapper(d): + ping = d + index = 0 + current_version = ping.version[0] + while (index < len(ping.update_ping_count_notify) and + index < len(ping.version) and + ping.version[index] == current_version): + if ping.update_ping_count_notify[index] > 0: + # If there is an update ping and update_not_pref_update_enabled_notify + # has a value greater than 0 then the preference is false. If there is + # a value of 0 or update_not_pref_update_enabled_notify is None then + # the preference is true. + if (ping.update_not_pref_update_enabled_notify is not None and + ping.update_not_pref_update_enabled_notify[index] > 0): + return False, ping + + return True, ping + + index += 1 + + raise ValueError("Missing update enabled value!") + +has_update_enabled_rdd = is_able_to_apply_true_rdd.map(has_update_enabled_mapper).cache() +has_update_enabled_dict = has_update_enabled_rdd.countByKey() +has_update_enabled_dict +
Create a reference to the dictionary which will be written to the +JSON that populates the web page data. This way the reference in the +web page never changes. A reference is all that is needed since the +dictionary is not modified.
+of_concern_dict = has_update_enabled_dict +
Create an RDD of the telemetry pings that have the +application.update.enabled preference set to True.
+This RDD is created from the last “out of date, potentially of concern” +RDD and it is named of_concern_true_rdd to simplify the addition of new code +without having to modify consumers of the RDD.
+of_concern_true_rdd = has_update_enabled_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date, of concern telemetry ping client +versions along with a dictionary of the count of each version.
+def by_version_mapper(d): + ping = d + return ping.version[0], ping + +of_concern_by_version_rdd = of_concern_true_rdd.map(by_version_mapper) +of_concern_by_version_dict = of_concern_by_version_rdd.countByKey() +of_concern_by_version_dict +
Create an RDD of out of date, of concern telemetry ping update check +codes along with a dictionary of the count of each update check code.
+def check_code_notify_mapper(d): + ping = d + index = 0 + current_version = ping.version[0] + while (index < len(ping.update_ping_count_notify) and + index < len(ping.version) and + ping.version[index] == current_version): + if (ping.update_ping_count_notify[index] > 0 and + ping.update_check_code_notify is not None): + code_index = 0 + for code_value in ping.update_check_code_notify[index]: + if code_value > 0: + return code_index, ping + code_index += 1 + + if (ping.update_check_no_update_notify is not None and + ping.update_check_no_update_notify[index] > 0): + return 0, ping + + index += 1 + + return -1, ping + +check_code_notify_of_concern_rdd = of_concern_true_rdd.map(check_code_notify_mapper) +check_code_notify_of_concern_dict = check_code_notify_of_concern_rdd.countByKey() +check_code_notify_of_concern_dict +
Create an RDD of out of date, of concern telemetry pings that had a +general failure for the update check. The general failure codes are: + CHK_GENERAL_ERROR_PROMPT: 22 + CHK_GENERAL_ERROR_SILENT: 23
+check_code_notify_general_error_of_concern_rdd = \ + check_code_notify_of_concern_rdd.filter(lambda p: p[0] == 22 or p[0] == 23).values().cache() +
Create an RDD of out of date, of concern telemetry ping update check +extended error values for the clients that had a general failure for +the update check along with a dictionary of the count of the error +values.
+def check_ex_error_notify_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if (ping.update_ping_count_notify[index] > 0 and + ping.update_check_extended_error_notify is not None): + for key_name in ping.update_check_extended_error_notify: + if ping.update_check_extended_error_notify[key_name][index] > 0: + if version == current_version: + key_name = key_name[17:] + if len(key_name) == 4: + key_name = key_name[1:] + return int(key_name), ping + return -1, ping + + return -2, ping + +check_ex_error_notify_of_concern_rdd = check_code_notify_general_error_of_concern_rdd.map(check_ex_error_notify_mapper) +check_ex_error_notify_of_concern_dict = check_ex_error_notify_of_concern_rdd.countByKey() +check_ex_error_notify_of_concern_dict +
Create an RDD of out of date, of concern telemetry ping update +download codes along with a dictionary of the count of the codes.
+def download_code_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if ping.update_download_code_partial is not None: + code_index = 0 + for code_value in ping.update_download_code_partial[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_download_code_complete is not None: + code_index = 0 + for code_value in ping.update_download_code_complete[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + return -2, ping + +download_code_of_concern_rdd = of_concern_true_rdd.map(download_code_mapper) +download_code_of_concern_dict = download_code_of_concern_rdd.countByKey() +download_code_of_concern_dict +
Create an RDD of out of date, of concern telemetry ping staged update +state codes along with a dictionary of the count of the codes.
+def state_code_stage_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if ping.update_state_code_partial_stage is not None: + code_index = 0 + for code_value in ping.update_state_code_partial_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_state_code_complete_stage is not None: + code_index = 0 + for code_value in ping.update_state_code_complete_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_state_code_unknown_stage is not None: + code_index = 0 + for code_value in ping.update_state_code_unknown_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + return -2, ping + +state_code_stage_of_concern_rdd = of_concern_true_rdd.map(state_code_stage_mapper).cache() +state_code_stage_of_concern_dict = state_code_stage_of_concern_rdd.countByKey() +state_code_stage_of_concern_dict +
Create an RDD of out of date, of concern telemetry pings that failed +to stage an update. +* STATE_FAILED: 12
+state_code_stage_failed_of_concern_rdd = \ + state_code_stage_of_concern_rdd.filter(lambda p: p[0] == 12).values().cache() +
Create an RDD of out of date, of concern telemetry ping staged update +state failure codes along with a dictionary of the count of the codes.
+def state_failure_code_stage_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if ping.update_status_error_code_partial_stage is not None: + code_index = 0 + for code_value in ping.update_status_error_code_partial_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_status_error_code_complete_stage is not None: + code_index = 0 + for code_value in ping.update_status_error_code_complete_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_status_error_code_unknown_stage is not None: + code_index = 0 + for code_value in ping.update_status_error_code_unknown_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + return -2, ping + +state_failure_code_stage_of_concern_rdd = state_code_stage_failed_of_concern_rdd.map(state_failure_code_stage_mapper) +state_failure_code_stage_of_concern_dict = state_failure_code_stage_of_concern_rdd.countByKey() +state_failure_code_stage_of_concern_dict +
Create an RDD of out of date, of concern telemetry ping startup +update state codes along with a dictionary of the count of the codes.
+def state_code_startup_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if ping.update_state_code_partial_startup is not None: + code_index = 0 + for code_value in ping.update_state_code_partial_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_state_code_complete_startup is not None: + code_index = 0 + for code_value in ping.update_state_code_complete_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_state_code_unknown_startup is not None: + code_index = 0 + for code_value in ping.update_state_code_unknown_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + return -2, ping + +state_code_startup_of_concern_rdd = of_concern_true_rdd.map(state_code_startup_mapper).cache() +state_code_startup_of_concern_dict = state_code_startup_of_concern_rdd.countByKey() +state_code_startup_of_concern_dict +
Create an RDD of the telemetry pings that have ping startup update state code equal to 12.
+state_code_startup_failed_of_concern_rdd = \ + state_code_startup_of_concern_rdd.filter(lambda p: p[0] == 12).values().cache() +
Create an RDD of out of date, of concern telemetry ping startup +update state failure codes along with a dictionary of the count of the +codes.
+def state_failure_code_startup_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if ping.update_status_error_code_partial_startup is not None: + code_index = 0 + for code_value in ping.update_status_error_code_partial_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_status_error_code_complete_startup is not None: + code_index = 0 + for code_value in ping.update_status_error_code_complete_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_status_error_code_unknown_startup is not None: + code_index = 0 + for code_value in ping.update_status_error_code_unknown_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + return -2, ping + +state_failure_code_startup_of_concern_rdd = state_code_startup_failed_of_concern_rdd.map(state_failure_code_startup_mapper) +state_failure_code_startup_of_concern_dict = state_failure_code_startup_of_concern_rdd.countByKey() +state_failure_code_startup_of_concern_dict +
Create an RDD of out of date, of concern telemetry pings that have +and have not received only no updates available during the update +check for their current version of Firefox along with a dictionary +of the count of values.
+def has_only_no_update_found_mapper(d): + ping = d + if ping.update_check_no_update_notify is None: + return False, ping + + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if current_version != version: + return True, ping + + if ping.update_ping_count_notify[index] > 0: + # If there is an update ping and update_check_no_update_notify + # has a value equal to 0 then the update check returned a + # value other than no update found. This could be improved by + # checking the check value for error conditions and ignoring + # those codes and ignoring the check below for those cases. + if (ping.update_check_no_update_notify[index] == 0): + return False, ping + + return True, ping + +has_only_no_update_found_rdd = of_concern_true_rdd.map(has_only_no_update_found_mapper).cache() +has_only_no_update_found_dict = has_only_no_update_found_rdd.countByKey() +has_only_no_update_found_dict +
Create an RDD of the telemetry pings that have not received only no updates +available during the update check for their current version of Firefox.
+has_only_no_update_found_false_rdd = has_only_no_update_found_rdd.filter(lambda p: p[0] == False).values().cache() +
Create an RDD of out of date, of concern telemetry pings that have and +don’t have any update download pings for their current version of +Firefox along with a dictionary of the count of the values.
+def has_no_download_code_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if current_version != version: + return True, ping + + if ping.update_download_code_partial is not None: + for code_value in ping.update_download_code_partial[index]: + if code_value > 0: + return False, ping + + if ping.update_download_code_complete is not None: + for code_value in ping.update_download_code_complete[index]: + if code_value > 0: + return False, ping + + return True, ping + +has_no_download_code_rdd = has_only_no_update_found_false_rdd.map(has_no_download_code_mapper).cache() +has_no_download_code_dict = has_no_download_code_rdd.countByKey() +has_no_download_code_dict +
Create an RDD of the telemetry pings that don’t have any update +download pings for their current version of Firefox.
+has_no_download_code_false_rdd = has_no_download_code_rdd.filter(lambda p: p[0] == False).values().cache() +
Create an RDD of out of date, of concern telemetry pings that have and +don’t have an update failure state for their current version of +Firefox along with a dictionary of the count of the values.
+def has_update_apply_failure_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if current_version != version: + return False, ping + + if (ping.update_state_code_partial_startup is not None and + ping.update_state_code_partial_startup[index][12] > 0): + return True, ping + + if (ping.update_state_code_complete_startup is not None and + ping.update_state_code_complete_startup[index][12] > 0): + return True, ping + + return False, ping + +has_update_apply_failure_rdd = has_no_download_code_false_rdd.map(has_update_apply_failure_mapper) +has_update_apply_failure_dict = has_update_apply_failure_rdd.countByKey() +has_update_apply_failure_dict +
Create a reference to the dictionary which will be written to the +JSON that populates the web page data. This way the reference in the +web page never changes. A reference is all that is needed since the +dictionary is not modified.
+of_concern_categorized_dict = has_update_apply_failure_dict +
Create the JSON that will be written to a file for the report.
+results_dict = {"reportDetails": report_details_dict, + "summary": summary_dict, + "hasOutOfDateMaxVersion": has_out_of_date_max_version_dict, + "hasUpdatePing": has_update_ping_dict, + "hasMinSubsessionLength": has_min_subsession_length_dict, + "hasMinUpdatePingCount": has_min_update_ping_count_dict, + "isSupported": is_supported_dict, + "isAbleToApply": is_able_to_apply_dict, + "hasUpdateEnabled": has_update_enabled_dict, + "ofConcern": of_concern_dict, + "hasOnlyNoUpdateFound": has_only_no_update_found_dict, + "hasNoDownloadCode": has_no_download_code_dict, + "hasUpdateApplyFailure": has_update_apply_failure_dict, + "ofConcernCategorized": of_concern_categorized_dict, + "ofConcernByVersion": of_concern_by_version_dict, + "checkCodeNotifyOfConcern": check_code_notify_of_concern_dict, + "checkExErrorNotifyOfConcern": check_ex_error_notify_of_concern_dict, + "downloadCodeOfConcern": download_code_of_concern_dict, + "stateCodeStageOfConcern": state_code_stage_of_concern_dict, + "stateFailureCodeStageOfConcern": state_failure_code_stage_of_concern_dict, + "stateCodeStartupOfConcern": state_code_startup_of_concern_dict, + "stateFailureCodeStartupOfConcern": state_failure_code_startup_of_concern_dict} +results_json = json.dumps(results_dict, ensure_ascii=False) +results_json +
Save the output to be uploaded automatically once the job completes. +The file will be stored at: +* https://analysis-output.telemetry.mozilla.org/app-update/data/out-of-date/FILENAME
+filename = "./output/" + report_filename + ".json" +if no_upload is None: + with open(filename, 'w') as f: + f.write(results_json) + + bucket = "telemetry-public-analysis-2" + path = "app-update/data/out-of-date/" + timestamped_s3_key = path + report_filename + ".json" + client = boto3.client('s3', 'us-west-2') + transfer = S3Transfer(client) + transfer.upload_file(filename, bucket, timestamped_s3_key, extra_args={'ContentType':'application/json'}) + +print "Filename: " + filename +
Get the time when this job ended.
+end_time = dt.datetime.now() +print "End: " + str(end_time.strftime("%Y-%m-%d %H:%M:%S")) +
Get the elapsed time it took to run this job.
+elapsed_time = end_time - start_time +print "Elapsed Seconds: " + str(int(elapsed_time.total_seconds())) +
Generate data for the Firefox Application Update Out Of Date dashboard
+import datetime as dt +import re +import urllib2 +import ujson as json +import boto3 +from boto3.s3.transfer import S3Transfer + +%pylab inline +
sc.defaultParallelism +
Get the time when this job started.
+start_time = dt.datetime.now() +print "Start: " + str(start_time.strftime("%Y-%m-%d %H:%M:%S")) +
Common settings.
+no_upload = None +today_str = None + +# Uncomment out the following two lines and adjust |today_str| as necessary to run manually without uploading the json. +no_upload = True +# today_str = "20161226" + +channel_to_process = "release" +min_version = 42 +up_to_date_releases = 2 +weeks_of_subsession_data = 12 +min_update_ping_count = 4 +min_subsession_hours = 2 +min_subsession_seconds = min_subsession_hours * 60 * 60 +
Get the settings based on dates.
+if today_str is None: + today_str = start_time.strftime("%Y%m%d") + +assert (today_str is not None), "The date environment parameter is missing." +today = dt.datetime.strptime(today_str, "%Y%m%d").date() + +# MON = 0, SAT = 5, SUN = 6 -> SUN = 0, MON = 1, SAT = 6 +day_index = (today.weekday() + 1) % 7 +# Filename used to save the report's JSON +report_filename = (today - datetime.timedelta(day_index)).strftime("%Y%m%d") +# Maximum report date which is the previous Saturday +max_report_date = today - datetime.timedelta(7 + day_index - 6) +# Suffix of the longitudinal datasource name to use +longitudinal_suffix = max_report_date.strftime("%Y%m%d") +# String used in the SQL queries to limit records to the maximum report date. +# Since the queries use less than this is the day after the previous Saturday. +max_report_date_sql = (max_report_date + dt.timedelta(days=1)).strftime("%Y-%m-%d") +# The Sunday prior to the last Saturday +min_report_date = max_report_date - dt.timedelta(days=6) +# String used in the SQL queries to limit records to the minimum report date +# Since the queries use greater than this is six days prior to the previous Saturday. +min_report_date_sql = min_report_date.strftime("%Y-%m-%d") +# Date used to limit records to the number of weeks specified by +# weeks_of_subsession_data prior to the maximum report date +min_subsession_date = max_report_date - dt.timedelta(weeks=weeks_of_subsession_data) +# Date used to compute the latest version from firefox_history_major_releases.json +latest_ver_date_str = (max_report_date - dt.timedelta(days=7)).strftime("%Y-%m-%d") + +print "max_report_date : " + max_report_date.strftime("%Y%m%d") +print "max_report_date_sql : " + max_report_date_sql +print "min_report_date : " + min_report_date.strftime("%Y%m%d") +print "min_report_date_sql : " + min_report_date_sql +print "min_subsession_date : " + min_subsession_date.strftime("%Y%m%d") +print "report_filename : " + report_filename +print "latest_ver_date_str : " + latest_ver_date_str +
Get the latest Firefox version available based on the date.
+def latest_version_on_date(date, major_releases): + latest_date = u"1900-01-01" + latest_ver = 0 + for version, release_date in major_releases.iteritems(): + version_int = int(version.split(".")[0]) + if release_date <= date and release_date >= latest_date and version_int >= latest_ver: + latest_date = release_date + latest_ver = version_int + + return latest_ver + +major_releases_json = urllib2.urlopen("https://product-details.mozilla.org/1.0/firefox_history_major_releases.json").read() +major_releases = json.loads(major_releases_json) +latest_version = latest_version_on_date(latest_ver_date_str, major_releases) +earliest_up_to_date_version = str(latest_version - up_to_date_releases) + +print "Latest Version: " + str(latest_version) +
Create a dictionary to store the general settings that will be written to a JSON file.
+report_details_dict = {"latestVersion": latest_version, + "upToDateReleases": up_to_date_releases, + "minReportDate": min_report_date.strftime("%Y-%m-%d"), + "maxReportDate": max_report_date.strftime("%Y-%m-%d"), + "weeksOfSubsessionData": weeks_of_subsession_data, + "minSubsessionDate": min_subsession_date.strftime("%Y-%m-%d"), + "minSubsessionHours": min_subsession_hours, + "minSubsessionSeconds": min_subsession_seconds, + "minUpdatePingCount": min_update_ping_count} +report_details_dict +
Create the common SQL FROM clause.
+# Note: using the parquet is as fast as using 'FROM longitudinal_vYYYMMDD' +# and it allows the query to go further back in time. + +#longitudinal_from_sql = ("FROM longitudinal_v{} ").format(longitudinal_suffix) +longitudinal_from_sql = ("FROM parquet.`s3://telemetry-parquet/longitudinal/v{}` ").format(longitudinal_suffix) +longitudinal_from_sql +
Create the common build.version SQL WHERE clause.
+build_version_where_sql = "(build.version[0] RLIKE '^[0-9]{2,3}\.0[\.0-9]*$' OR build.version[0] = '50.1.0')" +build_version_where_sql +
Create the remaining common SQL WHERE clause.
+common_where_sql = ("" + "build.application_name[0] = 'Firefox' AND " + "DATEDIFF(SUBSTR(subsession_start_date[0], 0, 10), '{}') >= 0 AND " + "DATEDIFF(SUBSTR(subsession_start_date[0], 0, 10), '{}') < 0 AND " + "settings.update.channel[0] = '{}'" +"").format(min_report_date_sql, + max_report_date_sql, + channel_to_process) +common_where_sql +
Create the SQL for the summary query.
+summary_sql = ("" +"SELECT " + "COUNT(CASE WHEN build.version[0] >= '{}.' AND build.version[0] < '{}.' THEN 1 END) AS versionUpToDate, " + "COUNT(CASE WHEN build.version[0] < '{}.' AND build.version[0] >= '{}.' THEN 1 END) AS versionOutOfDate, " + "COUNT(CASE WHEN build.version[0] < '{}.' THEN 1 END) AS versionTooLow, " + "COUNT(CASE WHEN build.version[0] > '{}.' THEN 1 END) AS versionTooHigh, " + "COUNT(CASE WHEN NOT build.version[0] > '0' THEN 1 END) AS versionMissing " +"{} " +"WHERE " + "{} AND " + "{}" +"").format(str(latest_version - up_to_date_releases), + str(latest_version + 1), + str(latest_version - up_to_date_releases), + str(min_version), + str(min_version), + str(latest_version + 1), + longitudinal_from_sql, + common_where_sql, + build_version_where_sql) +summary_sql +
Run the summary SQL query.
+summaryDF = sqlContext.sql(summary_sql) +
Create a dictionary to store the results from the summary query that will be written to a JSON file.
+summary_dict = summaryDF.first().asDict() +summary_dict +
Create the SQL for the out of date details query.
+# Only query for the columns and the records that are used to optimize +# for speed. Adding update_state_code_partial_stage and +# update_state_code_complete_stage increased the time it takes this +# notebook to run by 50 seconds when using 4 clusters. + +# Creating a temporary table of the data after the filters have been +# applied and joining it with the original datasource to include +# other columns doesn't appear to speed up the process but it doesn't +# appear to slow it down either so all columns of interest are in this +# query. + +out_of_date_details_sql = ("" +"SELECT " + "client_id, " + "build.version, " + "session_length, " + "subsession_start_date, " + "subsession_length, " + "update_check_code_notify, " + "update_check_extended_error_notify, " + "update_check_no_update_notify, " + "update_not_pref_update_enabled_notify, " + "update_not_pref_update_auto_notify, " + "update_ping_count_notify, " + "update_unable_to_apply_notify, " + "update_download_code_partial, " + "update_download_code_complete, " + "update_state_code_partial_stage, " + "update_state_code_complete_stage, " + "update_state_code_unknown_stage, " + "update_state_code_partial_startup, " + "update_state_code_complete_startup, " + "update_state_code_unknown_startup, " + "update_status_error_code_complete_startup, " + "update_status_error_code_partial_startup, " + "update_status_error_code_unknown_startup, " + "update_status_error_code_complete_stage, " + "update_status_error_code_partial_stage, " + "update_status_error_code_unknown_stage " +"{}" +"WHERE " + "{} AND " + "{} AND " + "build.version[0] < '{}.' AND " + "build.version[0] >= '{}.'" +"").format(longitudinal_from_sql, + common_where_sql, + build_version_where_sql, + str(latest_version - up_to_date_releases), + str(min_version)) +out_of_date_details_sql +
Run the out of date details SQL query.
+out_of_date_details_df = sqlContext.sql(out_of_date_details_sql) +
Create the RDD used to further restrict which clients are out of date +to focus on clients that are of concern and potentially of concern.
+out_of_date_details_rdd = out_of_date_details_df.rdd.cache() +
Create an RDD of out of date telemetry pings that have and don’t have +a previous telemetry ping with a version that is up to date along +with a dictionary of the count of True and False.
+def has_out_of_date_max_version_mapper(d): + ping = d + index = 0 + while (index < len(ping.version)): + if ((ping.version[index] == "50.1.0" or + p.match(ping.version[index])) and + ping.version[index] > earliest_up_to_date_version): + return False, ping + index += 1 + + return True, ping + +# RegEx for a valid release versions except for 50.1.0 which is handled separately. +p = re.compile('^[0-9]{2,3}\\.0[\\.0-9]*$') + +has_out_of_date_max_version_rdd = out_of_date_details_rdd.map(has_out_of_date_max_version_mapper).cache() +has_out_of_date_max_version_dict = has_out_of_date_max_version_rdd.countByKey() +has_out_of_date_max_version_dict +
Create an RDD of the telemetry pings that have a previous telemetry ping +with a version that is up to date.
+has_out_of_date_max_version_true_rdd = has_out_of_date_max_version_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that have and have not +sent an update telemtry ping for any version of Firefox along with a +dictionary of the count of True and False.
+def has_update_ping_mapper(d): + ping = d + if (ping.update_ping_count_notify is not None and + (ping.update_check_code_notify is not None or + ping.update_check_no_update_notify is not None)): + return True, ping + + return False, ping + +has_update_ping_rdd = has_out_of_date_max_version_true_rdd.map(has_update_ping_mapper).cache() +has_update_ping_dict = has_update_ping_rdd.countByKey() +has_update_ping_dict +
Create an RDD of the telemetry pings that have an update telemtry +ping for any version of Firefox.
+has_update_ping_true_rdd = has_update_ping_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that have and have not +ran this version of Firefox for more than the amount of seconds as +specified by min_subsession_seconds along with a dictionary of the +count of True and False.
+def has_min_subsession_length_mapper(d): + ping = d + seconds = 0 + index = 0 + current_version = ping.version[0] + while (seconds < min_subsession_seconds and + index < len(ping.subsession_start_date) and + index < len(ping.version) and + ping.version[index] == current_version): + try: + date = dt.datetime.strptime(ping.subsession_start_date[index][:10], + "%Y-%m-%d").date() + if date < min_subsession_date: + return False, ping + + seconds += ping.subsession_length[index] + index += 1 + except: # catch *all* exceptions + index += 1 + + if seconds >= min_subsession_seconds: + return True, ping + + return False, ping + +has_min_subsession_length_rdd = has_update_ping_true_rdd.map(has_min_subsession_length_mapper).cache() +has_min_subsession_length_dict = has_min_subsession_length_rdd.countByKey() +has_min_subsession_length_dict +
Create an RDD of the telemetry pings that have ran this version of +Firefox for more than the amount of seconds as specified by +min_subsession_seconds.
+has_min_subsession_length_true_rdd = has_min_subsession_length_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that have and have not +sent the minimum number of update pings as specified by +min_update_ping_count for this version of Firefox along with a +dictionary of the count of True and False.
+def has_min_update_ping_count_mapper(d): + ping = d + index = 0 + update_ping_count_total = 0 + current_version = ping.version[0] + while (update_ping_count_total < min_update_ping_count and + index < len(ping.update_ping_count_notify) and + index < len(ping.version) and + ping.version[index] == current_version): + + pingCount = ping.update_ping_count_notify[index] + # Is this an update ping or just a placeholder for the telemetry ping? + if pingCount > 0: + try: + date = dt.datetime.strptime(ping.subsession_start_date[index][:10], + "%Y-%m-%d").date() + if date < min_subsession_date: + return False, ping + + except: # catch *all* exceptions + index += 1 + continue + + # Is there also a valid update check code or no update telemetry ping? + if (ping.update_check_code_notify is not None and + len(ping.update_check_code_notify) > index): + for code_value in ping.update_check_code_notify[index]: + if code_value > 0: + update_ping_count_total += pingCount + index += 1 + continue + + if (ping.update_check_no_update_notify is not None and + len(ping.update_check_no_update_notify) > index and + ping.update_check_no_update_notify[index] > 0): + update_ping_count_total += pingCount + + index += 1 + + if update_ping_count_total < min_update_ping_count: + return False, ping + + return True, ping + +has_min_update_ping_count_rdd = has_min_subsession_length_true_rdd.map(has_min_update_ping_count_mapper).cache() +has_min_update_ping_count_dict = has_min_update_ping_count_rdd.countByKey() +has_min_update_ping_count_dict +
Create an RDD of the telemetry pings that have sent the minimum +number of update pings as specified by min_update_ping_count.
+has_min_update_ping_count_true_rdd = has_min_update_ping_count_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that are supported and +are not supported based on whether they have not received or have +received the unsupported update xml for the last update check along +with a dictionary of the count of True and False.
+def is_supported_mapper(d): + ping = d + index = 0 + update_ping_count_total = 0 + current_version = ping.version[0] + while (update_ping_count_total < min_update_ping_count and + index < len(ping.update_ping_count_notify) and + index < len(ping.version) and + ping.version[index] == current_version): + pingCount = ping.update_ping_count_notify[index] + # Is this an update ping or just a placeholder for the telemetry ping? + if pingCount > 0: + # Is there also a valid update check code or no update telemetry ping? + if (ping.update_check_code_notify is not None and + len(ping.update_check_code_notify) > index and + ping.update_check_code_notify[index][28] > 0): + return False, ping + + index += 1 + + return True, ping + +is_supported_rdd = has_min_update_ping_count_true_rdd.map(is_supported_mapper).cache() +is_supported_dict = is_supported_rdd.countByKey() +is_supported_dict +
Create an RDD of the telemetry pings that are supported based on +whether they have not received or have received the unsupported +update xml for the last update check.
+is_supported_true_rdd = is_supported_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that have and don’t have +the ability to apply an update along with a dictionary of the count +of True and False.
+def is_able_to_apply_mapper(d): + ping = d + index = 0 + current_version = ping.version[0] + while (index < len(ping.update_ping_count_notify) and + index < len(ping.version) and + ping.version[index] == current_version): + if ping.update_ping_count_notify[index] > 0: + # Only check the last value for update_unable_to_apply_notify + # to determine if the client is unable to apply. + if (ping.update_unable_to_apply_notify is not None and + ping.update_unable_to_apply_notify[index] > 0): + return False, ping + + return True, ping + + index += 1 + + raise ValueError("Missing update unable to apply value!") + +is_able_to_apply_rdd = is_supported_true_rdd.map(is_able_to_apply_mapper).cache() +is_able_to_apply_dict = is_able_to_apply_rdd.countByKey() +is_able_to_apply_dict +
Create an RDD of the telemetry pings that have the ability to apply +an update.
+is_able_to_apply_true_rdd = is_able_to_apply_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date telemetry pings that have and don’t have +the application.update.enabled preference set to True / False along +with a dictionary of the count of True and False.
+def has_update_enabled_mapper(d): + ping = d + index = 0 + current_version = ping.version[0] + while (index < len(ping.update_ping_count_notify) and + index < len(ping.version) and + ping.version[index] == current_version): + if ping.update_ping_count_notify[index] > 0: + # If there is an update ping and update_not_pref_update_enabled_notify + # has a value greater than 0 then the preference is false. If there is + # a value of 0 or update_not_pref_update_enabled_notify is None then + # the preference is true. + if (ping.update_not_pref_update_enabled_notify is not None and + ping.update_not_pref_update_enabled_notify[index] > 0): + return False, ping + + return True, ping + + index += 1 + + raise ValueError("Missing update enabled value!") + +has_update_enabled_rdd = is_able_to_apply_true_rdd.map(has_update_enabled_mapper).cache() +has_update_enabled_dict = has_update_enabled_rdd.countByKey() +has_update_enabled_dict +
Create a reference to the dictionary which will be written to the +JSON that populates the web page data. This way the reference in the +web page never changes. A reference is all that is needed since the +dictionary is not modified.
+of_concern_dict = has_update_enabled_dict +
Create an RDD of the telemetry pings that have the +application.update.enabled preference set to True.
+This RDD is created from the last “out of date, potentially of concern” +RDD and it is named of_concern_true_rdd to simplify the addition of new code +without having to modify consumers of the RDD.
+of_concern_true_rdd = has_update_enabled_rdd.filter(lambda p: p[0] == True).values().cache() +
Create an RDD of out of date, of concern telemetry ping client +versions along with a dictionary of the count of each version.
+def by_version_mapper(d): + ping = d + return ping.version[0], ping + +of_concern_by_version_rdd = of_concern_true_rdd.map(by_version_mapper) +of_concern_by_version_dict = of_concern_by_version_rdd.countByKey() +of_concern_by_version_dict +
Create an RDD of out of date, of concern telemetry ping update check +codes along with a dictionary of the count of each update check code.
+def check_code_notify_mapper(d): + ping = d + index = 0 + current_version = ping.version[0] + while (index < len(ping.update_ping_count_notify) and + index < len(ping.version) and + ping.version[index] == current_version): + if (ping.update_ping_count_notify[index] > 0 and + ping.update_check_code_notify is not None): + code_index = 0 + for code_value in ping.update_check_code_notify[index]: + if code_value > 0: + return code_index, ping + code_index += 1 + + if (ping.update_check_no_update_notify is not None and + ping.update_check_no_update_notify[index] > 0): + return 0, ping + + index += 1 + + return -1, ping + +check_code_notify_of_concern_rdd = of_concern_true_rdd.map(check_code_notify_mapper) +check_code_notify_of_concern_dict = check_code_notify_of_concern_rdd.countByKey() +check_code_notify_of_concern_dict +
Create an RDD of out of date, of concern telemetry pings that had a +general failure for the update check. The general failure codes are: + CHK_GENERAL_ERROR_PROMPT: 22 + CHK_GENERAL_ERROR_SILENT: 23
+check_code_notify_general_error_of_concern_rdd = \ + check_code_notify_of_concern_rdd.filter(lambda p: p[0] == 22 or p[0] == 23).values().cache() +
Create an RDD of out of date, of concern telemetry ping update check +extended error values for the clients that had a general failure for +the update check along with a dictionary of the count of the error +values.
+def check_ex_error_notify_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if (ping.update_ping_count_notify[index] > 0 and + ping.update_check_extended_error_notify is not None): + for key_name in ping.update_check_extended_error_notify: + if ping.update_check_extended_error_notify[key_name][index] > 0: + if version == current_version: + key_name = key_name[17:] + if len(key_name) == 4: + key_name = key_name[1:] + return int(key_name), ping + return -1, ping + + return -2, ping + +check_ex_error_notify_of_concern_rdd = check_code_notify_general_error_of_concern_rdd.map(check_ex_error_notify_mapper) +check_ex_error_notify_of_concern_dict = check_ex_error_notify_of_concern_rdd.countByKey() +check_ex_error_notify_of_concern_dict +
Create an RDD of out of date, of concern telemetry ping update +download codes along with a dictionary of the count of the codes.
+def download_code_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if ping.update_download_code_partial is not None: + code_index = 0 + for code_value in ping.update_download_code_partial[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_download_code_complete is not None: + code_index = 0 + for code_value in ping.update_download_code_complete[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + return -2, ping + +download_code_of_concern_rdd = of_concern_true_rdd.map(download_code_mapper) +download_code_of_concern_dict = download_code_of_concern_rdd.countByKey() +download_code_of_concern_dict +
Create an RDD of out of date, of concern telemetry ping staged update +state codes along with a dictionary of the count of the codes.
+def state_code_stage_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if ping.update_state_code_partial_stage is not None: + code_index = 0 + for code_value in ping.update_state_code_partial_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_state_code_complete_stage is not None: + code_index = 0 + for code_value in ping.update_state_code_complete_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_state_code_unknown_stage is not None: + code_index = 0 + for code_value in ping.update_state_code_unknown_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + return -2, ping + +state_code_stage_of_concern_rdd = of_concern_true_rdd.map(state_code_stage_mapper).cache() +state_code_stage_of_concern_dict = state_code_stage_of_concern_rdd.countByKey() +state_code_stage_of_concern_dict +
Create an RDD of out of date, of concern telemetry pings that failed +to stage an update. +* STATE_FAILED: 12
+state_code_stage_failed_of_concern_rdd = \ + state_code_stage_of_concern_rdd.filter(lambda p: p[0] == 12).values().cache() +
Create an RDD of out of date, of concern telemetry ping staged update +state failure codes along with a dictionary of the count of the codes.
+def state_failure_code_stage_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if ping.update_status_error_code_partial_stage is not None: + code_index = 0 + for code_value in ping.update_status_error_code_partial_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_status_error_code_complete_stage is not None: + code_index = 0 + for code_value in ping.update_status_error_code_complete_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_status_error_code_unknown_stage is not None: + code_index = 0 + for code_value in ping.update_status_error_code_unknown_stage[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + return -2, ping + +state_failure_code_stage_of_concern_rdd = state_code_stage_failed_of_concern_rdd.map(state_failure_code_stage_mapper) +state_failure_code_stage_of_concern_dict = state_failure_code_stage_of_concern_rdd.countByKey() +state_failure_code_stage_of_concern_dict +
Create an RDD of out of date, of concern telemetry ping startup +update state codes along with a dictionary of the count of the codes.
+def state_code_startup_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if ping.update_state_code_partial_startup is not None: + code_index = 0 + for code_value in ping.update_state_code_partial_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_state_code_complete_startup is not None: + code_index = 0 + for code_value in ping.update_state_code_complete_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_state_code_unknown_startup is not None: + code_index = 0 + for code_value in ping.update_state_code_unknown_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + return -2, ping + +state_code_startup_of_concern_rdd = of_concern_true_rdd.map(state_code_startup_mapper).cache() +state_code_startup_of_concern_dict = state_code_startup_of_concern_rdd.countByKey() +state_code_startup_of_concern_dict +
Create an RDD of the telemetry pings that have ping startup update state code equal to 12.
+state_code_startup_failed_of_concern_rdd = \ + state_code_startup_of_concern_rdd.filter(lambda p: p[0] == 12).values().cache() +
Create an RDD of out of date, of concern telemetry ping startup +update state failure codes along with a dictionary of the count of the +codes.
+def state_failure_code_startup_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if ping.update_status_error_code_partial_startup is not None: + code_index = 0 + for code_value in ping.update_status_error_code_partial_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_status_error_code_complete_startup is not None: + code_index = 0 + for code_value in ping.update_status_error_code_complete_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + if ping.update_status_error_code_unknown_startup is not None: + code_index = 0 + for code_value in ping.update_status_error_code_unknown_startup[index]: + if code_value > 0: + if version == current_version: + return code_index, ping + return -1, ping + code_index += 1 + + return -2, ping + +state_failure_code_startup_of_concern_rdd = state_code_startup_failed_of_concern_rdd.map(state_failure_code_startup_mapper) +state_failure_code_startup_of_concern_dict = state_failure_code_startup_of_concern_rdd.countByKey() +state_failure_code_startup_of_concern_dict +
Create an RDD of out of date, of concern telemetry pings that have +and have not received only no updates available during the update +check for their current version of Firefox along with a dictionary +of the count of values.
+def has_only_no_update_found_mapper(d): + ping = d + if ping.update_check_no_update_notify is None: + return False, ping + + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if current_version != version: + return True, ping + + if ping.update_ping_count_notify[index] > 0: + # If there is an update ping and update_check_no_update_notify + # has a value equal to 0 then the update check returned a + # value other than no update found. This could be improved by + # checking the check value for error conditions and ignoring + # those codes and ignoring the check below for those cases. + if (ping.update_check_no_update_notify[index] == 0): + return False, ping + + return True, ping + +has_only_no_update_found_rdd = of_concern_true_rdd.map(has_only_no_update_found_mapper).cache() +has_only_no_update_found_dict = has_only_no_update_found_rdd.countByKey() +has_only_no_update_found_dict +
Create an RDD of the telemetry pings that have not received only no updates +available during the update check for their current version of Firefox.
+has_only_no_update_found_false_rdd = has_only_no_update_found_rdd.filter(lambda p: p[0] == False).values().cache() +
Create an RDD of out of date, of concern telemetry pings that have and +don’t have any update download pings for their current version of +Firefox along with a dictionary of the count of the values.
+def has_no_download_code_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if current_version != version: + return True, ping + + if ping.update_download_code_partial is not None: + for code_value in ping.update_download_code_partial[index]: + if code_value > 0: + return False, ping + + if ping.update_download_code_complete is not None: + for code_value in ping.update_download_code_complete[index]: + if code_value > 0: + return False, ping + + return True, ping + +has_no_download_code_rdd = has_only_no_update_found_false_rdd.map(has_no_download_code_mapper).cache() +has_no_download_code_dict = has_no_download_code_rdd.countByKey() +has_no_download_code_dict +
Create an RDD of the telemetry pings that don’t have any update +download pings for their current version of Firefox.
+has_no_download_code_false_rdd = has_no_download_code_rdd.filter(lambda p: p[0] == False).values().cache() +
Create an RDD of out of date, of concern telemetry pings that have and +don’t have an update failure state for their current version of +Firefox along with a dictionary of the count of the values.
+def has_update_apply_failure_mapper(d): + ping = d + current_version = ping.version[0] + for index, version in enumerate(ping.version): + if current_version != version: + return False, ping + + if (ping.update_state_code_partial_startup is not None and + ping.update_state_code_partial_startup[index][12] > 0): + return True, ping + + if (ping.update_state_code_complete_startup is not None and + ping.update_state_code_complete_startup[index][12] > 0): + return True, ping + + return False, ping + +has_update_apply_failure_rdd = has_no_download_code_false_rdd.map(has_update_apply_failure_mapper) +has_update_apply_failure_dict = has_update_apply_failure_rdd.countByKey() +has_update_apply_failure_dict +
Create a reference to the dictionary which will be written to the +JSON that populates the web page data. This way the reference in the +web page never changes. A reference is all that is needed since the +dictionary is not modified.
+of_concern_categorized_dict = has_update_apply_failure_dict +
Create the JSON that will be written to a file for the report.
+results_dict = {"reportDetails": report_details_dict, + "summary": summary_dict, + "hasOutOfDateMaxVersion": has_out_of_date_max_version_dict, + "hasUpdatePing": has_update_ping_dict, + "hasMinSubsessionLength": has_min_subsession_length_dict, + "hasMinUpdatePingCount": has_min_update_ping_count_dict, + "isSupported": is_supported_dict, + "isAbleToApply": is_able_to_apply_dict, + "hasUpdateEnabled": has_update_enabled_dict, + "ofConcern": of_concern_dict, + "hasOnlyNoUpdateFound": has_only_no_update_found_dict, + "hasNoDownloadCode": has_no_download_code_dict, + "hasUpdateApplyFailure": has_update_apply_failure_dict, + "ofConcernCategorized": of_concern_categorized_dict, + "ofConcernByVersion": of_concern_by_version_dict, + "checkCodeNotifyOfConcern": check_code_notify_of_concern_dict, + "checkExErrorNotifyOfConcern": check_ex_error_notify_of_concern_dict, + "downloadCodeOfConcern": download_code_of_concern_dict, + "stateCodeStageOfConcern": state_code_stage_of_concern_dict, + "stateFailureCodeStageOfConcern": state_failure_code_stage_of_concern_dict, + "stateCodeStartupOfConcern": state_code_startup_of_concern_dict, + "stateFailureCodeStartupOfConcern": state_failure_code_startup_of_concern_dict} +results_json = json.dumps(results_dict, ensure_ascii=False) +results_json +
Save the output to be uploaded automatically once the job completes. +The file will be stored at: +* https://analysis-output.telemetry.mozilla.org/app-update/data/out-of-date/FILENAME
+filename = "./output/" + report_filename + ".json" +if no_upload is None: + with open(filename, 'w') as f: + f.write(results_json) + + bucket = "telemetry-public-analysis-2" + path = "app-update/data/out-of-date/" + timestamped_s3_key = path + report_filename + ".json" + client = boto3.client('s3', 'us-west-2') + transfer = S3Transfer(client) + transfer.upload_file(filename, bucket, timestamped_s3_key, extra_args={'ContentType':'application/json'}) + +print "Filename: " + filename +
Get the time when this job ended.
+end_time = dt.datetime.now() +print "End: " + str(end_time.strftime("%Y-%m-%d %H:%M:%S")) +
Get the elapsed time it took to run this job.
+elapsed_time = end_time - start_time +print "Elapsed Seconds: " + str(int(elapsed_time.total_seconds())) +
I ran into some Spark weirdness when working on some ETL.
+Specifically, when repartitioning a parquet file with coalesce(), the parallelism for the entire job (including upstream tasks) was constrained by the number of coalesce partitions.
+Instead, I expected the upstream jobs to use all available cores.
+We should be limited by the number of file partitions only when its time to actually write the file.
It’s probably easier if I demonstrate. +Below I’ll create a small example dataframe containing 10 rows. +I’ll map a slow function over the example dataframe in a few different ways. +I’d expect these calculations to take a fixed amount of time, since they’re happening in parallel. +However, for one example, execution time will increase linearly with the number of rows.
+import time +from pyspark.sql.types import LongType + +path = "~/tmp.parquet" +
sc.defaultParallelism +
32 +
def slow_func(ping): + """Identity function that takes 1s to return""" + time.sleep(1) + return(ping) +
def timer(func): + """Times the execution of a function""" + start_time = time.time() + func() + return time.time() - start_time +
# Example usage: +timer(lambda: slow_func(10)) +
1.001082181930542 +
def create_frame(rdd): + return sqlContext.createDataFrame(rdd, schema=LongType()) +
First, let’s look at a simple RDD. Everything seems to work as expected here. Execution time levels off to ~3.7 as the dataset increases:
+map(lambda x: timer(lambda: sc.parallelize(range(x)).map(slow_func).take(x)), range(10)) +
[0.07758498191833496, + 118.664391040802, + 2.453991174697876, + 2.390385866165161, + 2.3567309379577637, + 2.3262758255004883, + 2.3200111389160156, + 3.3115720748901367, + 3.3115429878234863, + 3.274951934814453] +
Let’s create a Spark DataFrame and write the contents to parquet without any modification. Again, things seem to be behaving here. Execution time is fairly flat.
+map(lambda x: timer(lambda: create_frame(sc.parallelize(range(x)))\ + .coalesce(1).write.mode("overwrite").parquet(path)), + range(10)) +
[5.700469017028809, + 1.5091090202331543, + 1.4622771739959717, + 1.448883056640625, + 1.4437789916992188, + 1.4351229667663574, + 1.4368910789489746, + 1.4349958896636963, + 1.4199819564819336, + 1.4395389556884766] +
Now, let’s map the slow function over the DataFrame before saving. This should increase execution time by one second for every dataset. However, it looks like execution time is increasing by one second for each row.
+map(lambda x: timer(lambda: create_frame(sc.parallelize(range(x))\ + .map(slow_func))\ + .coalesce(1).write.mode("overwrite").parquet(path)), + range(10)) +
[1.42529296875, + 2.436065912246704, + 3.3423829078674316, + 4.332568883895874, + 5.268526077270508, + 6.280202865600586, + 7.169728994369507, + 8.18229603767395, + 9.098582029342651, + 10.119444131851196] +
Using repartition instead of coalesce fixes the issue.
map(lambda x: timer(lambda: create_frame(sc.parallelize(range(x))\ + .map(slow_func))\ + .repartition(1).write.mode("overwrite").parquet(path)), + range(10)) +
[0.8304200172424316, + 1.276075839996338, + 1.2515549659729004, + 1.2429919242858887, + 1.2587580680847168, + 1.2490499019622803, + 1.6439399719238281, + 1.229665994644165, + 1.2340660095214844, + 1.2454640865325928] +
sc.cancelAllJobs() +
I ran into some Spark weirdness when working on some ETL.
+Specifically, when repartitioning a parquet file with coalesce(), the parallelism for the entire job (including upstream tasks) was constrained by the number of coalesce partitions.
+Instead, I expected the upstream jobs to use all available cores.
+We should be limited by the number of file partitions only when its time to actually write the file.
It’s probably easier if I demonstrate. +Below I’ll create a small example dataframe containing 10 rows. +I’ll map a slow function over the example dataframe in a few different ways. +I’d expect these calculations to take a fixed amount of time, since they’re happening in parallel. +However, for one example, execution time will increase linearly with the number of rows.
+import time +from pyspark.sql.types import LongType + +path = "~/tmp.parquet" +
sc.defaultParallelism +
32 +
def slow_func(ping): + """Identity function that takes 1s to return""" + time.sleep(1) + return(ping) +
def timer(func): + """Times the execution of a function""" + start_time = time.time() + func() + return time.time() - start_time +
# Example usage: +timer(lambda: slow_func(10)) +
1.001082181930542 +
def create_frame(rdd): + return sqlContext.createDataFrame(rdd, schema=LongType()) +
First, let’s look at a simple RDD. Everything seems to work as expected here. Execution time levels off to ~3.7 as the dataset increases:
+map(lambda x: timer(lambda: sc.parallelize(range(x)).map(slow_func).take(x)), range(10)) +
[0.07758498191833496, + 118.664391040802, + 2.453991174697876, + 2.390385866165161, + 2.3567309379577637, + 2.3262758255004883, + 2.3200111389160156, + 3.3115720748901367, + 3.3115429878234863, + 3.274951934814453] +
Let’s create a Spark DataFrame and write the contents to parquet without any modification. Again, things seem to be behaving here. Execution time is fairly flat.
+map(lambda x: timer(lambda: create_frame(sc.parallelize(range(x)))\ + .coalesce(1).write.mode("overwrite").parquet(path)), + range(10)) +
[5.700469017028809, + 1.5091090202331543, + 1.4622771739959717, + 1.448883056640625, + 1.4437789916992188, + 1.4351229667663574, + 1.4368910789489746, + 1.4349958896636963, + 1.4199819564819336, + 1.4395389556884766] +
Now, let’s map the slow function over the DataFrame before saving. This should increase execution time by one second for every dataset. However, it looks like execution time is increasing by one second for each row.
+map(lambda x: timer(lambda: create_frame(sc.parallelize(range(x))\ + .map(slow_func))\ + .coalesce(1).write.mode("overwrite").parquet(path)), + range(10)) +
[1.42529296875, + 2.436065912246704, + 3.3423829078674316, + 4.332568883895874, + 5.268526077270508, + 6.280202865600586, + 7.169728994369507, + 8.18229603767395, + 9.098582029342651, + 10.119444131851196] +
Using repartition instead of coalesce fixes the issue.
map(lambda x: timer(lambda: create_frame(sc.parallelize(range(x))\ + .map(slow_func))\ + .repartition(1).write.mode("overwrite").parquet(path)), + range(10)) +
[0.8304200172424316, + 1.276075839996338, + 1.2515549659729004, + 1.2429919242858887, + 1.2587580680847168, + 1.2490499019622803, + 1.6439399719238281, + 1.229665994644165, + 1.2340660095214844, + 1.2454640865325928] +
sc.cancelAllJobs() +
This is follow-up analysis to the Main Ping Submission and Recording Delays by Channel analysis previously performed.
+Specifically investigating what typical values of “recording delay” and “submission delay” might be.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Looking at Jan 10, 2017 to parallel the previous analysis.
+pings = Dataset.from_source("telemetry") \ + .where(docType='crash') \ + .where(submissionDate="20170110") \ + .records(sc, sample=0.5) +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+creationDate - The time the Telemetry code in Firefox created the ping, according to the client’s clock, expressed as an ISO string. meta/creationTimestamp is the same time, but expressed in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
payload/crashDate - Sadly the only time info associated with the crash event itself is at day resolution. I expect cliffs to show at multiples of 24 hours on the CDFs.
subset = get_pings_properties(pings, ["application/channel", + "creationDate", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "payload/crashDate"]) +
p = subset.take(1)[0] +
p
+{'application/channel': u'release',
+ 'creationDate': u'2017-01-10T10:20:56.247Z',
+ 'meta/Date': u'Tue, 10 Jan 2017 10:20:59 GMT',
+ 'meta/Timestamp': 1484043660992423424L,
+ 'meta/creationTimestamp': 1.484043656247e+18,
+ 'payload/crashDate': u'2017-01-09'}
+Quick normalization: ditch any ping that doesn’t have a subsessionLength, creationTimestamp, or Timestamp:
+prev_count = subset.count() +subset = subset.filter(lambda p:\ + p["payload/crashDate"] is not None\ + and p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +filtered_count = subset.count() +print "Filtered {} of {} pings ({:.2f}%)".format(prev_count - filtered_count, prev_count, (prev_count - filtered_count) / prev_count) +
Filtered 0 of 1191175 pings (0.00%) +
We’ll be plotting Cumulative Distribution Functions today.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +CHANNELS = ['release', 'beta', 'aurora', 'nightly'] +
def setup_plot(title, max_x): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0, 1.0) + plt.xlim(0.0, max_x) + + plt.grid(True) + +def plot_cdf(data): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys) +
def calculate_delays(p): + + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + reporting_delay = (created.date() - datetime.strptime(p["payload/crashDate"], "%Y-%m-%d").date()).total_seconds() + submission_delay = (received - created - clock_skew).total_seconds() + return (reporting_delay, submission_delay) +
delays_by_chan = subset.map(lambda p: (p["application/channel"], calculate_delays(p))) +
Recording Delay is the time from when the data “happens” to the time we record it in a ping.
+Due to only having day-resolution time information about the crash, this will be approximate and might look weird.
+setup_plot("Recording Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: d[1][0] / HOUR_IN_S if d[1][0] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="lower right") +
<matplotlib.legend.Legend at 0x7f8dcb4c8810> +
Since we don’t know when in the day a crash happened, we can’t use the precise time of day the ping was created to tell us how long it’s been. Thus we get this stair-step pattern as each ping is some quantum of days.
+Still, it’s enough to show us that Nightly is a clear winner with over 95% of its crashes recorded within a day. Release and beta still manage over 70% within a day and over 80% within two. However, it takes at least four days to reach 90%.
+Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+Here we run into a problem with clock skew. Clients’ clocks aren’t guaranteed to align with our server’s clock, so we cannot necessarily compare the two. Luckily, with bug 1144778 we introduced an HTTP Date header which tells us what time the client’s clock thinks it is when it is sending the data. Coupled with the Timestamp field recorded which is what time the server’s clock thinks it is when it receives the data, we can subtract the more egregious examples of clock skew and get values that are closer to reality.
setup_plot("Submission Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: d[1][1] / HOUR_IN_S if d[1][1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="lower right") +
<matplotlib.legend.Legend at 0x7f8dbe677a10> +
Submission delays are really low across the board meaning there is very little time between the crash ping being created and it being received by our servers.
+This echoes the code where the creation of the crash ping happens on the next restart of the browser, and is then sent almost immediately.
+Nightly is an interesting subject, though, in that it starts out as the slowest performer before becoming the channel with the most submitted crashes after 24 hours.
+And, summing the delays together and graphing them we get…
+setup_plot("Combined Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: (d[1][0] + d[1][1]) / HOUR_IN_S if (d[1][0] + d[1][1]) < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="lower right") +
<matplotlib.legend.Legend at 0x7f8dcafd1310> +
Any measure that depends on a count of all crashes will be waiting a long time. Nightly’s pretty quick about getting you to 95% within 24 hours, but every other channel requires more (possibly lots more in the case of release) than four days to get us information about 95% of their crashes.
+There is active work to improve these speeds. I look forward to its affect on these delays.
+This is follow-up analysis to the Main Ping Submission and Recording Delays by Channel analysis previously performed.
+Specifically investigating what typical values of “recording delay” and “submission delay” might be.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Looking at Jan 10, 2017 to parallel the previous analysis.
+pings = Dataset.from_source("telemetry") \ + .where(docType='crash') \ + .where(submissionDate="20170110") \ + .records(sc, sample=0.5) +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+creationDate - The time the Telemetry code in Firefox created the ping, according to the client’s clock, expressed as an ISO string. meta/creationTimestamp is the same time, but expressed in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
payload/crashDate - Sadly the only time info associated with the crash event itself is at day resolution. I expect cliffs to show at multiples of 24 hours on the CDFs.
subset = get_pings_properties(pings, ["application/channel", + "creationDate", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "payload/crashDate"]) +
p = subset.take(1)[0] +
p
+{'application/channel': u'release',
+ 'creationDate': u'2017-01-10T10:20:56.247Z',
+ 'meta/Date': u'Tue, 10 Jan 2017 10:20:59 GMT',
+ 'meta/Timestamp': 1484043660992423424L,
+ 'meta/creationTimestamp': 1.484043656247e+18,
+ 'payload/crashDate': u'2017-01-09'}
+Quick normalization: ditch any ping that doesn’t have a subsessionLength, creationTimestamp, or Timestamp:
+prev_count = subset.count() +subset = subset.filter(lambda p:\ + p["payload/crashDate"] is not None\ + and p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +filtered_count = subset.count() +print "Filtered {} of {} pings ({:.2f}%)".format(prev_count - filtered_count, prev_count, (prev_count - filtered_count) / prev_count) +
Filtered 0 of 1191175 pings (0.00%) +
We’ll be plotting Cumulative Distribution Functions today.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +CHANNELS = ['release', 'beta', 'aurora', 'nightly'] +
def setup_plot(title, max_x): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0, 1.0) + plt.xlim(0.0, max_x) + + plt.grid(True) + +def plot_cdf(data): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys) +
def calculate_delays(p): + + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + reporting_delay = (created.date() - datetime.strptime(p["payload/crashDate"], "%Y-%m-%d").date()).total_seconds() + submission_delay = (received - created - clock_skew).total_seconds() + return (reporting_delay, submission_delay) +
delays_by_chan = subset.map(lambda p: (p["application/channel"], calculate_delays(p))) +
Recording Delay is the time from when the data “happens” to the time we record it in a ping.
+Due to only having day-resolution time information about the crash, this will be approximate and might look weird.
+setup_plot("Recording Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: d[1][0] / HOUR_IN_S if d[1][0] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="lower right") +
<matplotlib.legend.Legend at 0x7f8dcb4c8810> +
Since we don’t know when in the day a crash happened, we can’t use the precise time of day the ping was created to tell us how long it’s been. Thus we get this stair-step pattern as each ping is some quantum of days.
+Still, it’s enough to show us that Nightly is a clear winner with over 95% of its crashes recorded within a day. Release and beta still manage over 70% within a day and over 80% within two. However, it takes at least four days to reach 90%.
+Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+Here we run into a problem with clock skew. Clients’ clocks aren’t guaranteed to align with our server’s clock, so we cannot necessarily compare the two. Luckily, with bug 1144778 we introduced an HTTP Date header which tells us what time the client’s clock thinks it is when it is sending the data. Coupled with the Timestamp field recorded which is what time the server’s clock thinks it is when it receives the data, we can subtract the more egregious examples of clock skew and get values that are closer to reality.
setup_plot("Submission Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: d[1][1] / HOUR_IN_S if d[1][1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="lower right") +
<matplotlib.legend.Legend at 0x7f8dbe677a10> +
Submission delays are really low across the board meaning there is very little time between the crash ping being created and it being received by our servers.
+This echoes the code where the creation of the crash ping happens on the next restart of the browser, and is then sent almost immediately.
+Nightly is an interesting subject, though, in that it starts out as the slowest performer before becoming the channel with the most submitted crashes after 24 hours.
+And, summing the delays together and graphing them we get…
+setup_plot("Combined Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: (d[1][0] + d[1][1]) / HOUR_IN_S if (d[1][0] + d[1][1]) < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="lower right") +
<matplotlib.legend.Legend at 0x7f8dcafd1310> +
Any measure that depends on a count of all crashes will be waiting a long time. Nightly’s pretty quick about getting you to 95% within 24 hours, but every other channel requires more (possibly lots more in the case of release) than four days to get us information about 95% of their crashes.
+There is active work to improve these speeds. I look forward to its affect on these delays.
+This is follow-up analysis to the Crash Ping Submission and Recording Delays by Channel analysis previously performed.
+Specifically, this one investigates the difference between typical values of “recording delay” and “submission delay” before and after pingSender started sending pings.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
We’ll be looking at two cohorts: Feb 1-14 and Feb 16 - Mar 1. pingSender was released Feb 15. We will be limiting to crashes submitted by builds built during the cohort range that were submitted during the cohort range.
pre_pings = Dataset.from_source("telemetry") \ + .where(docType='crash') \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: x >= "20170201" and x < "20170214") \ + .where(appBuildId=lambda x: x >= "20170201" and x < "20170214") \ + .records(sc, sample=1) + +post_pings = Dataset.from_source("telemetry") \ + .where(docType='crash') \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: x >= "20170216" and x < "20170301") \ + .where(appBuildId=lambda x: x >= "20170216" and x < "20170301") \ + .records(sc, sample=1) +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+creationDate - The time the Telemetry code in Firefox created the ping, according to the client’s clock, expressed as an ISO string. meta/creationTimestamp is the same time, but expressed in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
payload/crashDate - Sadly the only time info associated with the crash event itself is at day resolution. I expect cliffs to show at multiples of 24 hours on the CDFs.
pre_subset = get_pings_properties(pre_pings, ["application/channel", + "id", + "payload/processType", + "creationDate", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "payload/crashDate"]) + +post_subset = get_pings_properties(post_pings, ["application/channel", + "id", + "payload/processType", + "creationDate", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "payload/crashDate"]) +
The rest of the analysis is cleaner if we combine the two cohorts here.
+def add_pre(p): + p['pre'] = 'pre' + return p + +def add_post(p): + p['pre'] = 'post' + return p + +combined = pre_subset.map(add_pre).union(post_subset.map(add_post)) +
Quick normalization: ditch any ping that doesn’t have a subsessionLength, creationTimestamp, or Timestamp:
+prev_count = combined.count() +combined = combined.filter(lambda p:\ + p["payload/crashDate"] is not None\ + and p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +filtered_count = combined.count() +print "Filtered {} of {} pings ({:.2f}%)".format(prev_count - filtered_count, prev_count, 100.0 * (prev_count - filtered_count) / prev_count) +
Filtered 0 of 186321 pings (0.00%) +
pingSender only submits “crash” pings for main-process crashes, so let’s limit ourselves to those.
prev_count = combined.count() +combined = combined.filter(lambda p: p["payload/processType"] == "main") +filtered_count = combined.count() +print "Filtered {} of {} pings ({:.2f}%)".format(prev_count - filtered_count, prev_count, 100.0 * (prev_count - filtered_count) / prev_count) +
Filtered 138409 of 186321 pings (74.29%) +
We sometimes receive crash pings more than once (identical document ids). This is usually below 1%, but there was a known issue that can complicate measurement.
+So we’ll dedupe here.
+combined_deduped = combined\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < a["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) +
combined_count = combined.count() +combined_deduped_count = combined_deduped.count() +print "Filtered {} of {} crash pings ({:.2f}%)".format(combined_count - combined_deduped_count, combined_count, 100.0 * (combined_count - combined_deduped_count) / combined_count) +
Filtered 9684 of 47912 crash pings (20.21%) +
p = combined_deduped.take(1)[0] +
p
+<omitted just in case> +
We’ll be plotting Cumulative Distribution Functions today.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +PRES = ['pre', 'post'] +
def setup_plot(title, max_x): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0, 1.0) + plt.xlim(0.0, max_x) + + plt.grid(True) + +def plot_cdf(data): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys) +
def calculate_delays(p): + + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + reporting_delay = (created.date() - datetime.strptime(p["payload/crashDate"], "%Y-%m-%d").date()).total_seconds() + submission_delay = (received - created - clock_skew).total_seconds() + return (reporting_delay, submission_delay) +
delays_by_chan = combined_deduped.map(lambda p: (p["pre"], calculate_delays(p))) +
Recording Delay is the time from when the data “happens” to the time we record it in a ping.
+Due to only having day-resolution time information about the crash, this will be approximate and might look weird.
+setup_plot("Recording Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == pre)\ + .map(lambda d: d[1][0] / HOUR_IN_S if d[1][0] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(PRES, loc="lower right") +
<matplotlib.legend.Legend at 0x7f37f4a49450> +
As expected, more main-process “crash” pings are recorded more quickly with pingSender.
The only reason this isn’t 100% at 0 days is probably due to “crash” pings failing to be received when sent by pingSender. (it tries at most once to send a ping). We will have better information on pingSender’s success rate when it sends some identifying headers.
+Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+Here we run into a problem with clock skew. Clients’ clocks aren’t guaranteed to align with our server’s clock, so we cannot necessarily compare the two. Luckily, with bug 1144778 we introduced an HTTP Date header which tells us what time the client’s clock thinks it is when it is sending the data. Coupled with the Timestamp field recorded which is what time the server’s clock thinks it is when it receives the data, we can subtract the more egregious examples of clock skew and get values that are closer to reality.
setup_plot("Submission Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == pre)\ + .map(lambda d: d[1][1] / HOUR_IN_S if d[1][1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(PRES, loc="lower right") +
<matplotlib.legend.Legend at 0x7f37f4642550> +
I did not expect any large difference in submission delay as, regardless of whether pingSender is doing it or CrashManager is doing it, we attempt to send main-process “crash” pings immediately upon creation.
Likely the only reason this isn’t 100% at 0 is because of failing the initial transmission.
+And, summing the delays together and graphing them we get…
+setup_plot("Combined Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == pre)\ + .map(lambda d: (d[1][0] + d[1][1]) / HOUR_IN_S if (d[1][0] + d[1][1]) < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(PRES, loc="lower right") +
<matplotlib.legend.Legend at 0x7f37f449d2d0> +
The use of pingSender results in an improvement in main-process “crash” ping client delay.
This is follow-up analysis to the Crash Ping Submission and Recording Delays by Channel analysis previously performed.
+Specifically, this one investigates the difference between typical values of “recording delay” and “submission delay” before and after pingSender started sending pings.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
We’ll be looking at two cohorts: Feb 1-14 and Feb 16 - Mar 1. pingSender was released Feb 15. We will be limiting to crashes submitted by builds built during the cohort range that were submitted during the cohort range.
pre_pings = Dataset.from_source("telemetry") \ + .where(docType='crash') \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: x >= "20170201" and x < "20170214") \ + .where(appBuildId=lambda x: x >= "20170201" and x < "20170214") \ + .records(sc, sample=1) + +post_pings = Dataset.from_source("telemetry") \ + .where(docType='crash') \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: x >= "20170216" and x < "20170301") \ + .where(appBuildId=lambda x: x >= "20170216" and x < "20170301") \ + .records(sc, sample=1) +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+creationDate - The time the Telemetry code in Firefox created the ping, according to the client’s clock, expressed as an ISO string. meta/creationTimestamp is the same time, but expressed in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
payload/crashDate - Sadly the only time info associated with the crash event itself is at day resolution. I expect cliffs to show at multiples of 24 hours on the CDFs.
pre_subset = get_pings_properties(pre_pings, ["application/channel", + "id", + "payload/processType", + "creationDate", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "payload/crashDate"]) + +post_subset = get_pings_properties(post_pings, ["application/channel", + "id", + "payload/processType", + "creationDate", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "payload/crashDate"]) +
The rest of the analysis is cleaner if we combine the two cohorts here.
+def add_pre(p): + p['pre'] = 'pre' + return p + +def add_post(p): + p['pre'] = 'post' + return p + +combined = pre_subset.map(add_pre).union(post_subset.map(add_post)) +
Quick normalization: ditch any ping that doesn’t have a subsessionLength, creationTimestamp, or Timestamp:
+prev_count = combined.count() +combined = combined.filter(lambda p:\ + p["payload/crashDate"] is not None\ + and p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +filtered_count = combined.count() +print "Filtered {} of {} pings ({:.2f}%)".format(prev_count - filtered_count, prev_count, 100.0 * (prev_count - filtered_count) / prev_count) +
Filtered 0 of 186321 pings (0.00%) +
pingSender only submits “crash” pings for main-process crashes, so let’s limit ourselves to those.
prev_count = combined.count() +combined = combined.filter(lambda p: p["payload/processType"] == "main") +filtered_count = combined.count() +print "Filtered {} of {} pings ({:.2f}%)".format(prev_count - filtered_count, prev_count, 100.0 * (prev_count - filtered_count) / prev_count) +
Filtered 138409 of 186321 pings (74.29%) +
We sometimes receive crash pings more than once (identical document ids). This is usually below 1%, but there was a known issue that can complicate measurement.
+So we’ll dedupe here.
+combined_deduped = combined\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < a["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) +
combined_count = combined.count() +combined_deduped_count = combined_deduped.count() +print "Filtered {} of {} crash pings ({:.2f}%)".format(combined_count - combined_deduped_count, combined_count, 100.0 * (combined_count - combined_deduped_count) / combined_count) +
Filtered 9684 of 47912 crash pings (20.21%) +
p = combined_deduped.take(1)[0] +
p
+<omitted just in case> +
We’ll be plotting Cumulative Distribution Functions today.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +PRES = ['pre', 'post'] +
def setup_plot(title, max_x): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0, 1.0) + plt.xlim(0.0, max_x) + + plt.grid(True) + +def plot_cdf(data): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys) +
def calculate_delays(p): + + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + reporting_delay = (created.date() - datetime.strptime(p["payload/crashDate"], "%Y-%m-%d").date()).total_seconds() + submission_delay = (received - created - clock_skew).total_seconds() + return (reporting_delay, submission_delay) +
delays_by_chan = combined_deduped.map(lambda p: (p["pre"], calculate_delays(p))) +
Recording Delay is the time from when the data “happens” to the time we record it in a ping.
+Due to only having day-resolution time information about the crash, this will be approximate and might look weird.
+setup_plot("Recording Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == pre)\ + .map(lambda d: d[1][0] / HOUR_IN_S if d[1][0] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(PRES, loc="lower right") +
<matplotlib.legend.Legend at 0x7f37f4a49450> +
As expected, more main-process “crash” pings are recorded more quickly with pingSender.
The only reason this isn’t 100% at 0 days is probably due to “crash” pings failing to be received when sent by pingSender. (it tries at most once to send a ping). We will have better information on pingSender’s success rate when it sends some identifying headers.
+Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+Here we run into a problem with clock skew. Clients’ clocks aren’t guaranteed to align with our server’s clock, so we cannot necessarily compare the two. Luckily, with bug 1144778 we introduced an HTTP Date header which tells us what time the client’s clock thinks it is when it is sending the data. Coupled with the Timestamp field recorded which is what time the server’s clock thinks it is when it receives the data, we can subtract the more egregious examples of clock skew and get values that are closer to reality.
setup_plot("Submission Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == pre)\ + .map(lambda d: d[1][1] / HOUR_IN_S if d[1][1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(PRES, loc="lower right") +
<matplotlib.legend.Legend at 0x7f37f4642550> +
I did not expect any large difference in submission delay as, regardless of whether pingSender is doing it or CrashManager is doing it, we attempt to send main-process “crash” pings immediately upon creation.
Likely the only reason this isn’t 100% at 0 is because of failing the initial transmission.
+And, summing the delays together and graphing them we get…
+setup_plot("Combined Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == pre)\ + .map(lambda d: (d[1][0] + d[1][1]) / HOUR_IN_S if (d[1][0] + d[1][1]) < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(PRES, loc="lower right") +
<matplotlib.legend.Legend at 0x7f37f449d2d0> +
The use of pingSender results in an improvement in main-process “crash” ping client delay.
import pandas as pd +import numpy as np +import matplotlib + +from matplotlib import pyplot as plt +from moztelemetry.dataset import Dataset +from moztelemetry import get_pings_properties, get_one_ping_per_client +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
pings = Dataset.from_source("telemetry")\ + .where(docType='crash')\ + .where(appName='Firefox')\ + .where(appUpdateChannel=lambda x: x == 'nightly' or x == 'aurora')\ + .where(appBuildId=lambda x: x > '20170210' and x < '20170408')\ + .records(sc, sample=1) +
subset = get_pings_properties(pings, ["id", "application/channel", "application/buildId"]) +
To get the proportions of each builds’ crash pings that were duplicated, get the full count and the deduplicated count per-build.
+build_counts = subset.map(lambda s: ((s["application/buildId"][:8], s["application/channel"]), 1)).countByKey() +
deduped_counts = subset\ + .map(lambda s: (s["id"], s))\ + .reduceByKey(lambda a, b: a)\ + .map(lambda pair: pair[1])\ + .map(lambda s: ((s["application/buildId"][:8], s["application/channel"]), 1)).countByKey() +
from datetime import datetime +
sorted_counts = sorted(build_counts.iteritems()) +
sorted_deduped = sorted(deduped_counts.iteritems()) +
plt.figure(figsize=(16, 10)) +plt.plot([datetime.strptime(k[0], '%Y%m%d') for k,v in sorted_deduped if k[1] == 'nightly'], [100.0 * (build_counts[k] - v) / build_counts[k] for k,v in sorted_deduped if k[1] == 'nightly']) +plt.plot([datetime.strptime(k[0], '%Y%m%d') for k,v in sorted_deduped if k[1] == 'aurora'], [100.0 * (build_counts[k] - v) / build_counts[k] for k,v in sorted_deduped if k[1] == 'aurora']) +plt.ylabel("% of submitted crash pings that are duplicate") +plt.xlabel("Build date") +plt.show() +
Looks like something happened on March 30 on Nightly and April 5 on Aurora to drastically reduce the proportion of duplicate crash pings we’ve been seeing.
+import pandas as pd +import numpy as np +import matplotlib + +from matplotlib import pyplot as plt +from moztelemetry.dataset import Dataset +from moztelemetry import get_pings_properties, get_one_ping_per_client +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
pings = Dataset.from_source("telemetry")\ + .where(docType='crash')\ + .where(appName='Firefox')\ + .where(appUpdateChannel=lambda x: x == 'nightly' or x == 'aurora')\ + .where(appBuildId=lambda x: x > '20170210' and x < '20170408')\ + .records(sc, sample=1) +
subset = get_pings_properties(pings, ["id", "application/channel", "application/buildId"]) +
To get the proportions of each builds’ crash pings that were duplicated, get the full count and the deduplicated count per-build.
+build_counts = subset.map(lambda s: ((s["application/buildId"][:8], s["application/channel"]), 1)).countByKey() +
deduped_counts = subset\ + .map(lambda s: (s["id"], s))\ + .reduceByKey(lambda a, b: a)\ + .map(lambda pair: pair[1])\ + .map(lambda s: ((s["application/buildId"][:8], s["application/channel"]), 1)).countByKey() +
from datetime import datetime +
sorted_counts = sorted(build_counts.iteritems()) +
sorted_deduped = sorted(deduped_counts.iteritems()) +
plt.figure(figsize=(16, 10)) +plt.plot([datetime.strptime(k[0], '%Y%m%d') for k,v in sorted_deduped if k[1] == 'nightly'], [100.0 * (build_counts[k] - v) / build_counts[k] for k,v in sorted_deduped if k[1] == 'nightly']) +plt.plot([datetime.strptime(k[0], '%Y%m%d') for k,v in sorted_deduped if k[1] == 'aurora'], [100.0 * (build_counts[k] - v) / build_counts[k] for k,v in sorted_deduped if k[1] == 'aurora']) +plt.ylabel("% of submitted crash pings that are duplicate") +plt.xlabel("Build date") +plt.show() +
Looks like something happened on March 30 on Nightly and April 5 on Aurora to drastically reduce the proportion of duplicate crash pings we’ve been seeing.
+import numpy as np +import matplotlib.pyplot as plt +import pandas as pd +from datetime import datetime, timedelta +from moztelemetry import get_pings_properties +from moztelemetry.dataset import Dataset +from scipy.stats import linregress + +%matplotlib inline +
start_date = (datetime.today() + timedelta(days=-22)) +start_date_str = start_date.strftime("%Y%m%d") +end_date = (datetime.today() + timedelta(days=-6)) +end_date_str = end_date.strftime("%Y%m%d") + +pings = (Dataset.from_source("telemetry") + .where(docType='main') + .where(appBuildId=lambda b: (b.startswith(start_date_str) or b > start_date_str) + and (b.startswith(end_date_str) or b < end_date_str)) + .where(appUpdateChannel="nightly") + .records(sc, sample=1.0)) + +subset = get_pings_properties(pings, [ + 'environment/system/os/name', + 'application/buildId', + 'payload/info/subsessionLength', + 'payload/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS', + 'payload/processes/content/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS/values', + 'payload/childPayloads', + 'payload/threadHangStats', + ]) +
fetching 96379.93220MB in 95251 files... +
This analysis is oriented toward understanding the relationship between BHR data (which can be viewed here), and input hang data (the “Input Lag measures here). If for most BHR hangs, we have a corresponding “Input Lag” hang, and vice versa, then that means the stacks we visualize in the BHR dashboard are of high value for bettering our score on the “Input Lag” metric.
+Our first step: let’s just gather numbers for both of our metrics - one for the parent process, and one for content.
+def hang_has_user_interaction(hang): + if 'annotations' not in hang: + return False + if len(hang['annotations']) == 0: + return False + return any('UserInteracting' in a and a['UserInteracting'] == 'true' for a in hang['annotations']) + +def flatten_hangs(thread_hang): + if 'name' not in thread_hang: + return [] + + hangs = thread_hang['hangs'] + + return [ + { + 'thread_name': thread_hang['name'], + 'hang': x + } + for x in hangs + if hang_has_user_interaction(x) + ] + +def flatten_all_hangs(ping): + result = [] + + if ping['payload/childPayloads'] is not None: + for payload in ping['payload/childPayloads']: + if 'threadHangStats' not in payload: + continue + + for thread_hang in payload['threadHangStats']: + result = result + flatten_hangs(thread_hang) + + if ping['payload/threadHangStats'] is not None: + for thread_hang in ping['payload/threadHangStats']: + result = result + flatten_hangs(thread_hang) + + return result + +def count_bhr_hangs(thread_name, hangs): + count = 0 + for hang in hangs: + if hang['thread_name'] == thread_name: + hist_data = hang['hang']['histogram']['values'] + key_ints = map(int, hist_data.keys()) + hist = pd.Series(hist_data.values(), index=key_ints) + count += hist[hist.index > 2048].sum() + return count + +def count_parent_input_delays(ping): + if ping['payload/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS'] is None: + return 0 + data = ping['payload/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS'] + zipped = zip(data.values, map(int, data.keys())) + vals = sorted(zipped, key=lambda x: x[1]) + + return pd.Series([v for v,k in vals], index=[k for v,k in vals]).truncate(before=2048).sum() + +def count_content_input_delays(ping): + if ping['payload/processes/content/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS/values'] is None: + return 0 + data = ping['payload/processes/content/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS/values'] + zipped = zip(data.values(), map(int, data.keys())) + vals = sorted(zipped, key=lambda x: x[1]) + + return pd.Series([v for v,k in vals], index=[k for v,k in vals]).truncate(before=2048).sum() + +def get_counts(ping): + hangs = flatten_all_hangs(ping) + subsession_length = ping['payload/info/subsessionLength'] + return (ping['application/buildId'], { + 'subsession_length': subsession_length, + 'parent_bhr': count_bhr_hangs('Gecko', hangs), + 'content_bhr': count_bhr_hangs('Gecko_Child', hangs), + 'parent_input': count_parent_input_delays(ping), + 'content_input': count_content_input_delays(ping), + }) + +def merge_counts(a, b): + return {k: a[k] + b[k] for k in a.iterkeys()} + +def ping_is_valid(ping): + if not isinstance(ping["application/buildId"], basestring): + return False + if type(ping["payload/info/subsessionLength"]) != int: + return False + + return ping["environment/system/os/name"] == "Windows_NT" + +cached = subset.filter(ping_is_valid).map(get_counts).cache() +counts_result = cached.reduceByKey(merge_counts).collect() +
sorted_result = sorted(counts_result, key=lambda x: x[1]) +
plot_data = np.array([ + [float(x['parent_bhr']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['content_bhr']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['parent_input']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['content_input']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result] +], np.float32) +
Let’s take a look at the parent numbers over time to get an intuition for their relationship:
+plt.title("Parent Hang Stats") +plt.xlabel("Build date") +plt.ylabel("Hangs per kuh") + +bhr_index = 0 +input_index = 2 + +plt.xticks(range(0, len(sorted_result))) +max_y = max(np.amax(plot_data[bhr_index]), np.amax(plot_data[input_index])) +plt.yticks(np.arange(0., max_y, max_y / 20.)) + +plt.grid(True) + +plt.plot(range(0, len(sorted_result)), plot_data[bhr_index]) +plt.plot(range(0, len(sorted_result)), plot_data[input_index]) +plt.legend(["bhr", "input"], loc="upper right") +
<matplotlib.legend.Legend at 0x7fa35f3e4890> +
Looks plausibly correlated to me - let’s try a scatter plot:
+plt.title("Parent Hang Stats") +plt.ylabel("BHR hangs per kuh") +plt.xlabel("Input hangs per kuh") + +bhr_index = 0 +input_index = 2 + +max_val = max(np.amax(plot_data[bhr_index]), np.amax(plot_data[input_index])) +ticks = np.arange(0., max_val, max_val / 10.) +plt.yticks(ticks) +plt.xticks(ticks) + + +plt.grid(True) + +plt.scatter(plot_data[input_index], plot_data[bhr_index]) + +slope, intercept, rvalue, pvalue, stderr = linregress(plot_data[input_index], plot_data[bhr_index]) + +max_x = np.amax(plot_data[input_index]) +plt.plot([0, max_x], [intercept, intercept + max_x * slope], '--') +rvalue # print the correlation coefficient +
0.71141966513446731 +
Correlation coefficient of ~0.711, so, moderately correlated. Let’s try the content process:
+plt.title("Content Hang Stats") +plt.xlabel("Build date") +plt.ylabel("Hangs per kuh") + +bhr_index = 1 +input_index = 3 + +plt.xticks(range(0, len(sorted_result))) +max_y = max(np.amax(plot_data[bhr_index]), np.amax(plot_data[input_index])) +plt.yticks(np.arange(0., max_y, max_y / 20.)) + +plt.grid(True) + +plt.plot(range(0, len(sorted_result)), plot_data[bhr_index]) +plt.plot(range(0, len(sorted_result)), plot_data[input_index]) +plt.legend(["bhr", "input"], loc="upper right") +
<matplotlib.legend.Legend at 0x7fa35d5a0e50> +
plt.title("Content Hang Stats") +plt.ylabel("BHR hangs per kuh") +plt.xlabel("Input hangs per kuh") + +bhr_index = 1 +input_index = 3 + +max_val = max(np.amax(plot_data[bhr_index]), np.amax(plot_data[input_index])) +ticks = np.arange(0., max_val, max_val / 10.) +plt.yticks(ticks) +plt.xticks(ticks) + +plt.grid(True) + +plt.scatter(plot_data[input_index], plot_data[bhr_index]) + +slope, intercept, rvalue, pvalue, stderr = linregress(plot_data[input_index], plot_data[bhr_index]) + +max_x = np.amax(plot_data[input_index]) +plt.plot([0, max_x], [intercept, intercept + max_x * slope], '--') +rvalue +
0.92448071222652162 +
~0.924. Much more strongly correlated. So it’s plausible that BHR hangs are a strong cause of content hangs. They could still be a significant cause of parent hangs, but it seems weaker.
+Each data point in the above scatter plots is the sum of hang stats for a given build date. The correlation across build dates for the content process is high. How about across individual pings?
+collected = cached.collect() +plt.title("Content Hang Stats (per ping)") +plt.ylabel("BHR hangs") +plt.xlabel("Input hangs") + +bhr_index = 1 +input_index = 3 + +content_filtered = [x for k,x in collected if x['content_bhr'] < 100 and x['content_input'] < 100] + +per_ping_data = plot_data = np.array([ + [x['parent_bhr'] for x in content_filtered], + [x['content_bhr'] for x in content_filtered], + [x['parent_input'] for x in content_filtered], + [x['content_input'] for x in content_filtered] +], np.int32) + +ticks = np.arange(0, 100, 10) +plt.yticks(ticks) +plt.xticks(ticks) + +plt.grid(True) + +plt.scatter(per_ping_data[input_index], per_ping_data[bhr_index]) + +slope, intercept, rvalue, pvalue, stderr = linregress(per_ping_data[input_index], per_ping_data[bhr_index]) + +max_x = 10. +plt.plot([0, max_x], [intercept, intercept + max_x * slope], '--') +rvalue +
0.41093048863293707 +
Interesting - the data split out per ping is significantly less correlated than the aggregate data by build date. This might suggest that individual BHR hangs don’t seem to cause Input Lag events, which is unfortunate for us. That would imply however that there must be some third cause for the high correlation in the aggregate data.
+Let’s see if we can observe any strong correlation between BHR data between processes. This should give us a feel for whether there might be any forces external to FF that are influencing the numbers:
+plot_data = np.array([ + [float(x['parent_bhr']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['content_bhr']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['parent_input']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['content_input']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result] +], np.float32) +
plt.title("BHR Stats") +plt.xlabel("Build date") +plt.ylabel("Hangs per kuh") + +parent_index = 0 +content_index = 1 + +plt.xticks(range(0, len(sorted_result))) +max_y = max(np.amax(plot_data[parent_index]), np.amax(plot_data[content_index])) +plt.yticks(np.arange(0., max_y, max_y / 20.)) + +plt.grid(True) + +plt.plot(range(0, len(sorted_result)), plot_data[parent_index]) +plt.plot(range(0, len(sorted_result)), plot_data[content_index]) +plt.legend(["parent", "content"], loc="upper right") +
<matplotlib.legend.Legend at 0x7fa34abfabd0> +
plt.title("BHR Stats") +plt.ylabel("Parent Input hangs per kuh") +plt.xlabel("Content Input hangs per kuh") + +parent_index = 0 +content_index = 1 + +max_val = max(np.amax(plot_data[parent_index]), np.amax(plot_data[content_index])) +ticks = np.arange(0., max_val, max_val / 10.) +plt.yticks(ticks) +plt.xticks(ticks) + + +plt.grid(True) + +plt.scatter(plot_data[parent_index], plot_data[content_index]) + +slope, intercept, rvalue, pvalue, stderr = linregress(plot_data[parent_index], plot_data[content_index]) + +max_x = np.amax(plot_data[parent_index]) +plt.plot([0, max_x], [intercept, intercept + max_x * slope], '--') +rvalue +
0.61208744161139772 +
Significantly lower than the correlation between BHR and input lag in the content process.
+Let’s look at input lag across processes:
+plt.title("Input Hang Stats") +plt.xlabel("Build date") +plt.ylabel("Hangs per kuh") + +parent_index = 2 +content_index = 3 + +plt.xticks(range(0, len(sorted_result))) +max_y = max(np.amax(plot_data[parent_index]), np.amax(plot_data[content_index])) +plt.yticks(np.arange(0., max_y, max_y / 20.)) + +plt.grid(True) + +plt.plot(range(0, len(sorted_result)), plot_data[parent_index]) +plt.plot(range(0, len(sorted_result)), plot_data[content_index]) +plt.legend(["parent", "content"], loc="upper right") +
<matplotlib.legend.Legend at 0x7fa34b21d090> +
plt.title("Interprocess Hang Stats (Input)") +plt.ylabel("Parent Input hangs per kuh") +plt.xlabel("Content Input hangs per kuh") + +parent_index = 2 +content_index = 3 + +max_val = max(np.amax(plot_data[parent_index]), np.amax(plot_data[content_index])) +ticks = np.arange(0., max_val, max_val / 10.) +plt.yticks(ticks) +plt.xticks(ticks) + + +plt.grid(True) + +plt.scatter(plot_data[parent_index], plot_data[content_index]) + +slope, intercept, rvalue, pvalue, stderr = linregress(plot_data[parent_index], plot_data[content_index]) + +max_x = np.amax(plot_data[parent_index]) +plt.plot([0, max_x], [intercept, intercept + max_x * slope], '--') +rvalue +
0.99701789074391722 +
Extremely high correlation. There’s some discussion of this in Bug 1383924. Essentially, the parent process gets all of the content process’s events, so it makes sense that there’s a good deal of overlap. However, this could help explain why the content process BHR and input lag are so tightly correlated while the parent process’s aren’t.
+In any case, we have some interesting data here, but the biggest unanswered question I have at the end of this is why is the aggregate correlation between BHR and input lag in the content process so high, while the correlation in individual pings is so low?
+import numpy as np +import matplotlib.pyplot as plt +import pandas as pd +from datetime import datetime, timedelta +from moztelemetry import get_pings_properties +from moztelemetry.dataset import Dataset +from scipy.stats import linregress + +%matplotlib inline +
start_date = (datetime.today() + timedelta(days=-22)) +start_date_str = start_date.strftime("%Y%m%d") +end_date = (datetime.today() + timedelta(days=-6)) +end_date_str = end_date.strftime("%Y%m%d") + +pings = (Dataset.from_source("telemetry") + .where(docType='main') + .where(appBuildId=lambda b: (b.startswith(start_date_str) or b > start_date_str) + and (b.startswith(end_date_str) or b < end_date_str)) + .where(appUpdateChannel="nightly") + .records(sc, sample=1.0)) + +subset = get_pings_properties(pings, [ + 'environment/system/os/name', + 'application/buildId', + 'payload/info/subsessionLength', + 'payload/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS', + 'payload/processes/content/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS/values', + 'payload/childPayloads', + 'payload/threadHangStats', + ]) +
fetching 96379.93220MB in 95251 files... +
This analysis is oriented toward understanding the relationship between BHR data (which can be viewed here), and input hang data (the “Input Lag measures here). If for most BHR hangs, we have a corresponding “Input Lag” hang, and vice versa, then that means the stacks we visualize in the BHR dashboard are of high value for bettering our score on the “Input Lag” metric.
+Our first step: let’s just gather numbers for both of our metrics - one for the parent process, and one for content.
+def hang_has_user_interaction(hang): + if 'annotations' not in hang: + return False + if len(hang['annotations']) == 0: + return False + return any('UserInteracting' in a and a['UserInteracting'] == 'true' for a in hang['annotations']) + +def flatten_hangs(thread_hang): + if 'name' not in thread_hang: + return [] + + hangs = thread_hang['hangs'] + + return [ + { + 'thread_name': thread_hang['name'], + 'hang': x + } + for x in hangs + if hang_has_user_interaction(x) + ] + +def flatten_all_hangs(ping): + result = [] + + if ping['payload/childPayloads'] is not None: + for payload in ping['payload/childPayloads']: + if 'threadHangStats' not in payload: + continue + + for thread_hang in payload['threadHangStats']: + result = result + flatten_hangs(thread_hang) + + if ping['payload/threadHangStats'] is not None: + for thread_hang in ping['payload/threadHangStats']: + result = result + flatten_hangs(thread_hang) + + return result + +def count_bhr_hangs(thread_name, hangs): + count = 0 + for hang in hangs: + if hang['thread_name'] == thread_name: + hist_data = hang['hang']['histogram']['values'] + key_ints = map(int, hist_data.keys()) + hist = pd.Series(hist_data.values(), index=key_ints) + count += hist[hist.index > 2048].sum() + return count + +def count_parent_input_delays(ping): + if ping['payload/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS'] is None: + return 0 + data = ping['payload/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS'] + zipped = zip(data.values, map(int, data.keys())) + vals = sorted(zipped, key=lambda x: x[1]) + + return pd.Series([v for v,k in vals], index=[k for v,k in vals]).truncate(before=2048).sum() + +def count_content_input_delays(ping): + if ping['payload/processes/content/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS/values'] is None: + return 0 + data = ping['payload/processes/content/histograms/INPUT_EVENT_RESPONSE_COALESCED_MS/values'] + zipped = zip(data.values(), map(int, data.keys())) + vals = sorted(zipped, key=lambda x: x[1]) + + return pd.Series([v for v,k in vals], index=[k for v,k in vals]).truncate(before=2048).sum() + +def get_counts(ping): + hangs = flatten_all_hangs(ping) + subsession_length = ping['payload/info/subsessionLength'] + return (ping['application/buildId'], { + 'subsession_length': subsession_length, + 'parent_bhr': count_bhr_hangs('Gecko', hangs), + 'content_bhr': count_bhr_hangs('Gecko_Child', hangs), + 'parent_input': count_parent_input_delays(ping), + 'content_input': count_content_input_delays(ping), + }) + +def merge_counts(a, b): + return {k: a[k] + b[k] for k in a.iterkeys()} + +def ping_is_valid(ping): + if not isinstance(ping["application/buildId"], basestring): + return False + if type(ping["payload/info/subsessionLength"]) != int: + return False + + return ping["environment/system/os/name"] == "Windows_NT" + +cached = subset.filter(ping_is_valid).map(get_counts).cache() +counts_result = cached.reduceByKey(merge_counts).collect() +
sorted_result = sorted(counts_result, key=lambda x: x[1]) +
plot_data = np.array([ + [float(x['parent_bhr']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['content_bhr']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['parent_input']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['content_input']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result] +], np.float32) +
Let’s take a look at the parent numbers over time to get an intuition for their relationship:
+plt.title("Parent Hang Stats") +plt.xlabel("Build date") +plt.ylabel("Hangs per kuh") + +bhr_index = 0 +input_index = 2 + +plt.xticks(range(0, len(sorted_result))) +max_y = max(np.amax(plot_data[bhr_index]), np.amax(plot_data[input_index])) +plt.yticks(np.arange(0., max_y, max_y / 20.)) + +plt.grid(True) + +plt.plot(range(0, len(sorted_result)), plot_data[bhr_index]) +plt.plot(range(0, len(sorted_result)), plot_data[input_index]) +plt.legend(["bhr", "input"], loc="upper right") +
<matplotlib.legend.Legend at 0x7fa35f3e4890> +
Looks plausibly correlated to me - let’s try a scatter plot:
+plt.title("Parent Hang Stats") +plt.ylabel("BHR hangs per kuh") +plt.xlabel("Input hangs per kuh") + +bhr_index = 0 +input_index = 2 + +max_val = max(np.amax(plot_data[bhr_index]), np.amax(plot_data[input_index])) +ticks = np.arange(0., max_val, max_val / 10.) +plt.yticks(ticks) +plt.xticks(ticks) + + +plt.grid(True) + +plt.scatter(plot_data[input_index], plot_data[bhr_index]) + +slope, intercept, rvalue, pvalue, stderr = linregress(plot_data[input_index], plot_data[bhr_index]) + +max_x = np.amax(plot_data[input_index]) +plt.plot([0, max_x], [intercept, intercept + max_x * slope], '--') +rvalue # print the correlation coefficient +
0.71141966513446731 +
Correlation coefficient of ~0.711, so, moderately correlated. Let’s try the content process:
+plt.title("Content Hang Stats") +plt.xlabel("Build date") +plt.ylabel("Hangs per kuh") + +bhr_index = 1 +input_index = 3 + +plt.xticks(range(0, len(sorted_result))) +max_y = max(np.amax(plot_data[bhr_index]), np.amax(plot_data[input_index])) +plt.yticks(np.arange(0., max_y, max_y / 20.)) + +plt.grid(True) + +plt.plot(range(0, len(sorted_result)), plot_data[bhr_index]) +plt.plot(range(0, len(sorted_result)), plot_data[input_index]) +plt.legend(["bhr", "input"], loc="upper right") +
<matplotlib.legend.Legend at 0x7fa35d5a0e50> +
plt.title("Content Hang Stats") +plt.ylabel("BHR hangs per kuh") +plt.xlabel("Input hangs per kuh") + +bhr_index = 1 +input_index = 3 + +max_val = max(np.amax(plot_data[bhr_index]), np.amax(plot_data[input_index])) +ticks = np.arange(0., max_val, max_val / 10.) +plt.yticks(ticks) +plt.xticks(ticks) + +plt.grid(True) + +plt.scatter(plot_data[input_index], plot_data[bhr_index]) + +slope, intercept, rvalue, pvalue, stderr = linregress(plot_data[input_index], plot_data[bhr_index]) + +max_x = np.amax(plot_data[input_index]) +plt.plot([0, max_x], [intercept, intercept + max_x * slope], '--') +rvalue +
0.92448071222652162 +
~0.924. Much more strongly correlated. So it’s plausible that BHR hangs are a strong cause of content hangs. They could still be a significant cause of parent hangs, but it seems weaker.
+Each data point in the above scatter plots is the sum of hang stats for a given build date. The correlation across build dates for the content process is high. How about across individual pings?
+collected = cached.collect() +plt.title("Content Hang Stats (per ping)") +plt.ylabel("BHR hangs") +plt.xlabel("Input hangs") + +bhr_index = 1 +input_index = 3 + +content_filtered = [x for k,x in collected if x['content_bhr'] < 100 and x['content_input'] < 100] + +per_ping_data = plot_data = np.array([ + [x['parent_bhr'] for x in content_filtered], + [x['content_bhr'] for x in content_filtered], + [x['parent_input'] for x in content_filtered], + [x['content_input'] for x in content_filtered] +], np.int32) + +ticks = np.arange(0, 100, 10) +plt.yticks(ticks) +plt.xticks(ticks) + +plt.grid(True) + +plt.scatter(per_ping_data[input_index], per_ping_data[bhr_index]) + +slope, intercept, rvalue, pvalue, stderr = linregress(per_ping_data[input_index], per_ping_data[bhr_index]) + +max_x = 10. +plt.plot([0, max_x], [intercept, intercept + max_x * slope], '--') +rvalue +
0.41093048863293707 +
Interesting - the data split out per ping is significantly less correlated than the aggregate data by build date. This might suggest that individual BHR hangs don’t seem to cause Input Lag events, which is unfortunate for us. That would imply however that there must be some third cause for the high correlation in the aggregate data.
+Let’s see if we can observe any strong correlation between BHR data between processes. This should give us a feel for whether there might be any forces external to FF that are influencing the numbers:
+plot_data = np.array([ + [float(x['parent_bhr']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['content_bhr']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['parent_input']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result], + [float(x['content_input']) / x['subsession_length'] * 3600. * 1000. for k,x in sorted_result] +], np.float32) +
plt.title("BHR Stats") +plt.xlabel("Build date") +plt.ylabel("Hangs per kuh") + +parent_index = 0 +content_index = 1 + +plt.xticks(range(0, len(sorted_result))) +max_y = max(np.amax(plot_data[parent_index]), np.amax(plot_data[content_index])) +plt.yticks(np.arange(0., max_y, max_y / 20.)) + +plt.grid(True) + +plt.plot(range(0, len(sorted_result)), plot_data[parent_index]) +plt.plot(range(0, len(sorted_result)), plot_data[content_index]) +plt.legend(["parent", "content"], loc="upper right") +
<matplotlib.legend.Legend at 0x7fa34abfabd0> +
plt.title("BHR Stats") +plt.ylabel("Parent Input hangs per kuh") +plt.xlabel("Content Input hangs per kuh") + +parent_index = 0 +content_index = 1 + +max_val = max(np.amax(plot_data[parent_index]), np.amax(plot_data[content_index])) +ticks = np.arange(0., max_val, max_val / 10.) +plt.yticks(ticks) +plt.xticks(ticks) + + +plt.grid(True) + +plt.scatter(plot_data[parent_index], plot_data[content_index]) + +slope, intercept, rvalue, pvalue, stderr = linregress(plot_data[parent_index], plot_data[content_index]) + +max_x = np.amax(plot_data[parent_index]) +plt.plot([0, max_x], [intercept, intercept + max_x * slope], '--') +rvalue +
0.61208744161139772 +
Significantly lower than the correlation between BHR and input lag in the content process.
+Let’s look at input lag across processes:
+plt.title("Input Hang Stats") +plt.xlabel("Build date") +plt.ylabel("Hangs per kuh") + +parent_index = 2 +content_index = 3 + +plt.xticks(range(0, len(sorted_result))) +max_y = max(np.amax(plot_data[parent_index]), np.amax(plot_data[content_index])) +plt.yticks(np.arange(0., max_y, max_y / 20.)) + +plt.grid(True) + +plt.plot(range(0, len(sorted_result)), plot_data[parent_index]) +plt.plot(range(0, len(sorted_result)), plot_data[content_index]) +plt.legend(["parent", "content"], loc="upper right") +
<matplotlib.legend.Legend at 0x7fa34b21d090> +
plt.title("Interprocess Hang Stats (Input)") +plt.ylabel("Parent Input hangs per kuh") +plt.xlabel("Content Input hangs per kuh") + +parent_index = 2 +content_index = 3 + +max_val = max(np.amax(plot_data[parent_index]), np.amax(plot_data[content_index])) +ticks = np.arange(0., max_val, max_val / 10.) +plt.yticks(ticks) +plt.xticks(ticks) + + +plt.grid(True) + +plt.scatter(plot_data[parent_index], plot_data[content_index]) + +slope, intercept, rvalue, pvalue, stderr = linregress(plot_data[parent_index], plot_data[content_index]) + +max_x = np.amax(plot_data[parent_index]) +plt.plot([0, max_x], [intercept, intercept + max_x * slope], '--') +rvalue +
0.99701789074391722 +
Extremely high correlation. There’s some discussion of this in Bug 1383924. Essentially, the parent process gets all of the content process’s events, so it makes sense that there’s a good deal of overlap. However, this could help explain why the content process BHR and input lag are so tightly correlated while the parent process’s aren’t.
+In any case, we have some interesting data here, but the biggest unanswered question I have at the end of this is why is the aggregate correlation between BHR and input lag in the content process so high, while the correlation in individual pings is so low?
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import pandas as pd + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from collections import Counter +import operator + +get_ipython().magic(u'matplotlib inline') +
pings = Dataset.from_source("telemetry") \ + .where(docType='health', appUpdateChannel="nightly") \ + .records(sc, sample=1) + +cachedData = get_pings_properties(pings, ["creationDate", "payload/pingDiscardedForSize", "payload/sendFailure", + "clientId", "meta/submissionDate", "payload/os", "payload/reason", "application/version"]).cache() +
fetching 522.25817MB in 132777 files... +
cachedDataNightly57 = cachedData.filter(lambda ping: ping["meta/submissionDate"] > '20170801').cache() +
def aggregateFailures(first, second): + if first is None: + return second + if second is None: + return first + + res = first + for k, v in second.items(): + if isinstance(v, int): + if k in res: + res[k] += v + else: + res[k] = v; + return res +
# return array of pairs [(failureName, {failureStatistic: count, ....}), ...] +# e.g. [(discardedForSize, {"main": 3, "crash": 5}), (sendFailure, {"timeout" : 34})] +def getFailuresStatPerFailureName(pings, failureNames): + def reduceFailure(failureName): + return pings.map(lambda p: p[failureName]).reduce(aggregateFailures) + + return [(name, reduceFailure(name)) for name in failureNames] +
failuresNames = ["payload/pingDiscardedForSize", "payload/sendFailure"] +
result = cachedData.map(lambda p: p['payload/pingDiscardedForSize']).filter(lambda p: p != None).collect() +
failuresStat = getFailuresStatPerFailureName(cachedData, failuresNames) +for fs in failuresStat: + plt.title(fs[0]) + plt.bar(range(len(fs[1])), fs[1].values(), align='center') + plt.xticks(range(len(fs[1])), fs[1].keys(), rotation=90) + plt.show() + print fs +
('payload/pingDiscardedForSize', {u'<unknown>': 4523, u'main': 1810, u'crash': 35, u'bhr': 49})
+
('payload/sendFailure', {u'eUnreachable': 2725838, u'abort': 13191, u'eChannelOpen': 4789989, u'timeout': 771011})
+Unknown currently represent all oversized pending pings. (https://bugzilla.mozilla.org/show_bug.cgi?id=1384903)
+failuresStat = getFailuresStatPerFailureName(cachedDataNightly57, failuresNames) +for fs in failuresStat: + plt.title(fs[0]) + plt.bar(range(len(fs[1])), fs[1].values(), align='center') + plt.xticks(range(len(fs[1])), fs[1].keys(), rotation=90) + plt.show() + print fs +
('payload/pingDiscardedForSize', {u'<unknown>': 4523, u'main': 1657, u'crash': 29, u'bhr': 49})
+
('payload/sendFailure', {u'eUnreachable': 2565221, u'abort': 12709, u'eChannelOpen': 4510507, u'timeout': 736047})
+import matplotlib.dates as mdates +def plotlistofTuples(listOfTuples, title="", inColor='blue'): + keys = [t[0] for t in listOfTuples] + values = [t[1] for t in listOfTuples] + + plt.figure(1) + fig = plt.gcf() + fig.set_size_inches(15, 7) + + plt.title(title) + plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y')) + plt.bar(range(len(listOfTuples)), values, align='center', color=inColor) + plt.xticks(range(len(listOfTuples)), keys, rotation=90) +
EXPECTED_SENDFAILURES_COUNT = 60 +def failureCountReasonPerPing(pings, failureName): + pingsWithSendFailure = pings.filter(lambda ping: ping[failureName] != None) + return pingsWithSendFailure.map(lambda ping: (sum(ping[failureName].values()), ping["payload/reason"])).collect() + +def describefailureDistribution(sendFailureCountDistr, failureName): + failuresCount = [k for k, v in sendFailureCountDistr] + pingsPerDaySeries = pd.Series(failuresCount) + print pingsPerDaySeries.describe([.25, .5, .75, .95]) + + plt.title(failureName + " per ping distribution.") + plt.yscale('log') + plt.ylabel('log(' + failureName + ' count)') + plt.xlabel('ping') + plt.plot(sorted(failuresCount)) + plt.show() + +def decribeReasonDistribution(sendFailureCountDistr, failureName): + unexpectedPingsCount = [(k, v) for k, v in sendFailureCountDistr if k > EXPECTED_SENDFAILURES_COUNT] + print "Pings reported more than " + str(EXPECTED_SENDFAILURES_COUNT) + " " + str(len(unexpectedPingsCount)) + + if len(unexpectedPingsCount) != 0: + reasonStat = Counter([v for k, v in unexpectedPingsCount]) + plotlistofTuples(reasonStat.items(), title="Reason distribution for pings reported more than " + str(EXPECTED_SENDFAILURES_COUNT) + " " + failureName) + plt.xlabel('reason') + plt.ylabel('count') + plt.show() + + +def describe(pings, failure): + print "\n COMPUTATION FOR " + failure + "\n" + countAndReason = failureCountReasonPerPing(pings, failure) + decribeReasonDistribution(countAndReason, failure) + describefailureDistribution(countAndReason, failure) +
for f in failuresNames: + describe(cachedData, f) +
COMPUTATION FOR payload/pingDiscardedForSize + +Pings reported more than 60 0 +count 6146.000000 +mean 1.044094 +std 0.696504 +min 1.000000 +25% 1.000000 +50% 1.000000 +75% 1.000000 +95% 1.000000 +max 48.000000 +dtype: float64 +
COMPUTATION FOR payload/sendFailure + +Pings reported more than 60 6831 +
count 3.177612e+06 +mean 2.612034e+00 +std 1.191218e+01 +min 1.000000e+00 +25% 1.000000e+00 +50% 1.000000e+00 +75% 2.000000e+00 +95% 9.000000e+00 +max 6.237000e+03 +dtype: float64 +
Check that: + required fields are non-empty. + payload/reason contains only expected values (“immediate”, “delayed”, “shutdown”). + payload/sendFailure and payload/discardedForSize are non empty together. + count paramter in payload/sendFailure and payload/discardedForSize has type int. + sendFailureType contains only expected values (“eOK”, “eRequest”, “eUnreachable”, “eChannelOpen”, “eRedirect”, “abort”, “timeout”). + payload/discardedForSize contains only 10 records. +* check the distribution of sendFailures (sum) per ping. We expected to have this number not more than 60.
+def validate(ping): + OK = "" + MUST_NOT_BE_EMPTY = "must not be empty" + + # validate os + clientId = ping["clientId"] + + if clientId == None: + return ("clientId " + MUST_NOT_BE_EMPTY, ping) + + os = ping["payload/os"] + if os == None: + return ("OS " + MUST_NOT_BE_EMPTY, ping) + + name, version = os.items() + if name == None: + return ("OS name " + MUST_NOT_BE_EMPTY, ping) + if version == None: + return ("OS version " + MUST_NOT_BE_EMPTY, ping) + + # validate reason + reason = ping["payload/reason"] + if reason == None: + return ("Reason " + MUST_NOT_BE_EMPTY, ping) + + if not reason in ["immediate", "delayed", "shutdown"]: + return ("Reason must be equal to immediate, delayed or shutdown", ping) + + # doesn't contain failures + sendFailure = ping["payload/sendFailure"] + pingDiscardedForSize = ping["payload/pingDiscardedForSize"] + if sendFailure == None and pingDiscardedForSize == None: + return ("Ping must countain at least one of the failures", ping) + + + # validate sendFailure + supportedFailureTypes = ["eOK", "eRequest", "eUnreachable", "eChannelOpen", "eRedirect", "abort", "timeout"] + if sendFailure != None and len(sendFailure) > len(supportedFailureTypes): + return ("send Failure accept only 8 send failures", ping) + + if sendFailure != None: + for key in sendFailure.keys(): + if not key in supportedFailureTypes: + return (key + " type is not supported", ping) + for count in sendFailure.values(): + if not isinstance(count, int): + return ("Count must be int type", ping) + if sum(sendFailure.values()) > 60: + return ("sendFailure count must not be more than 60", ping) + + + # validate pingDiscardedForSize + if pingDiscardedForSize != None: + if len(pingDiscardedForSize) > 10: + return ("pingDicardedForSize accept only top ten pings types", ping) + for count in pingDiscardedForSize.values(): + if not isinstance(count, int): + return ("Count must be int type", ping) + + return (OK, ping) + +# retrieve all needed fields +validatedData = cachedData.map(validate) +errorsPerProblem = validatedData.countByKey() +errorsPerProblem +
defaultdict(int,
+ {'': 3176610, 'sendFailure count must not be more than 60': 6831})
+def printOSReason(data): + return "os: " + str(data[0]) + " reason: " + str(data[1]) + +def osAndReasonForErros(error): + result = validatedData.filter(lambda pair: pair[0] == error).map(lambda pair: (pair[1]["payload/os"], pair[1]["payload/reason"])).collect() + return result[:min(10, len(result))] + +print "Show only 10 info lines per problem \n" +for err in errorsPerProblem.keys(): + if err != '': + print err + print "\n".join(map(printOSReason, osAndReasonForErros(err))) + print "\n" +
Show only 10 info lines per problem
+
+sendFailure count must not be more than 60
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+This includes showing diagrams and printing stats
+from datetime import datetime + +def pingsCountPerDay(pings): + return pings.map(lambda ping: ping["meta/submissionDate"]).countByValue() + +resultDictionary = pingsCountPerDay(cachedData) +
import matplotlib.dates as mdates +def plotlistofTuples(listOfTuples, title="", inColor='blue'): + keys = [t[0] for t in listOfTuples] + values = [t[1] for t in listOfTuples] + + plt.figure(1) + fig = plt.gcf() + fig.set_size_inches(15, 7) + + plt.title(title) + plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y')) + plt.bar(range(len(listOfTuples)), values, align='center', color=inColor) + plt.xticks(range(len(listOfTuples)), keys, rotation=90) +
plotlistofTuples(sorted(resultDictionary.items(), key=lambda tup: tup[0]), "Pings count per day") +plt.xlabel('meta/submissionDate') +plt.ylabel('pings count') +plt.show() +
pingsPerDaySeries = pd.Series(resultDictionary.values()) +pingsPerDaySeries.describe([.25, .5, .75, .95]) +
count 34.000000 +mean 93630.617647 +std 49146.705206 +min 6475.000000 +25% 58565.750000 +50% 77670.500000 +75% 137599.500000 +95% 172715.950000 +max 176918.000000 +dtype: float64 +
def getClients(pings): + clients = pings.map(lambda ping: ping["clientId"]).distinct() + return clients.collect() +
print str(len(getClients(cachedData))) + " clients are reporting health ping." +
148428 clients are reporting health ping. +
from collections import Counter + +def getAvgPerDate(iterable): + aggregare = Counter(iterable) + result = sum(aggregare.values()) * 1.0 / len(aggregare) + return result + +def pingsPerClientPerDay(pings, date): + return pings.map(lambda ping: (ping["clientId"], ping[date])).groupByKey() + +def avgPingsPerClientPerDay(pings, date): + idDateRDD = pingsPerClientPerDay(pings, date) + return idDateRDD.map(lambda pair: getAvgPerDate(pair[1])).collect() +
def plotAvgPingPerDateDistr(pings, date): + PINGS_COUNT_PER_DAY = 24 + resultDistributionList = avgPingsPerClientPerDay(pings, date) + values = [v for v in resultDistributionList if v > PINGS_COUNT_PER_DAY] + print date + " - clients sending too many \"health\" pings per day - " + str(len(values)) + if len(values) > 0: + plt.title("Average pings per day per client") + plt.ylabel('log(average ping count)') + plt.xlabel("clientId (anonymized)") + plt.yscale('log') + plt.plot(sorted(values)) + plt.show() +
plotAvgPingPerDateDistr(cachedData, "meta/submissionDate") +plotAvgPingPerDateDistr(cachedData, "creationDate") +
meta/submissionDate - clients sending too many "health" pings per day - 569 +
creationDate - clients sending too many "health" pings per day - 0 +
Turns out, clients submit health pings properly (less that 24/day) but we get them on server with some delay
+DAU57, DAU56 from re:dash https://sql.telemetry.mozilla.org/queries/15337/source#table
+DAU57 = [('20170802', 9497), ('20170803', 20923), ('20170804', 25515), ('20170805', 24669), ('20170806', 25604), \ + ('20170807', 32762), ('20170808', 36011), ('20170809', 37101), ('20170810', 38934), ('20170811', 38128), \ + ('20170812', 33403), ('20170813', 33301), ('20170814', 38519), ('20170815', 39053), ('20170816', 41439), \ + ('20170817', 41982), ('20170818', 41557), ('20170819', 35654), ('20170820', 35820), ('20170821', 43760), \ + ('20170822', 45038)] +DAU56 = [('20170727', 34419), ('20170728', 33142), ('20170729', 27740), ('20170730', 28194), ('20170731', 34340), ('20170801', 37308)] +
def getClientsPerDay(pings, version, dateFrom, dateTo): + clientsPerDay = pings.filter(lambda ping: ping["application/version"] == version) \ + .map(lambda ping: (ping["meta/submissionDate"], ping["clientId"])) \ + .groupByKey() \ + .map(lambda pair: (pair[0], len(set(pair[1])))).collect() + + return filter(lambda pair: pair[0] >= dateFrom and pair[0] <= dateTo, clientsPerDay) + +filtered57 = getClientsPerDay(cachedData, '57.0a1', '20170801', '20170822') +filtered56 = getClientsPerDay(cachedData, '56.0a1', '20170727', '20170801') +
plotlistofTuples(sorted(DAU56 + DAU57), inColor='red') +plotlistofTuples(sorted(filtered56 + filtered57), "Nightly 56&57. 56 from 20170727 to 20170801. 57 from 20170802 to 20170822") +plt.legend(['DAU', 'Health ping clients/day']) +plt.xlabel('meta/submissionDate') +plt.ylabel('clients count') +
<matplotlib.text.Text at 0x7fe88652ccd0> +
Conclusion: Almost half of the DAU submits health ping. It is seemsed to be because of sendFailure types: eChannelOpen and eUnreachable.
+eChannelOpen - This error happen when we failed to open channel, maybe it is better to avoid closing the channel and reuse existed channels instead.
+eUnreachable - Probably internet connection problems.
++
import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import pandas as pd + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from collections import Counter +import operator + +get_ipython().magic(u'matplotlib inline') +
pings = Dataset.from_source("telemetry") \ + .where(docType='health', appUpdateChannel="nightly") \ + .records(sc, sample=1) + +cachedData = get_pings_properties(pings, ["creationDate", "payload/pingDiscardedForSize", "payload/sendFailure", + "clientId", "meta/submissionDate", "payload/os", "payload/reason", "application/version"]).cache() +
fetching 522.25817MB in 132777 files... +
cachedDataNightly57 = cachedData.filter(lambda ping: ping["meta/submissionDate"] > '20170801').cache() +
def aggregateFailures(first, second): + if first is None: + return second + if second is None: + return first + + res = first + for k, v in second.items(): + if isinstance(v, int): + if k in res: + res[k] += v + else: + res[k] = v; + return res +
# return array of pairs [(failureName, {failureStatistic: count, ....}), ...] +# e.g. [(discardedForSize, {"main": 3, "crash": 5}), (sendFailure, {"timeout" : 34})] +def getFailuresStatPerFailureName(pings, failureNames): + def reduceFailure(failureName): + return pings.map(lambda p: p[failureName]).reduce(aggregateFailures) + + return [(name, reduceFailure(name)) for name in failureNames] +
failuresNames = ["payload/pingDiscardedForSize", "payload/sendFailure"] +
result = cachedData.map(lambda p: p['payload/pingDiscardedForSize']).filter(lambda p: p != None).collect() +
failuresStat = getFailuresStatPerFailureName(cachedData, failuresNames) +for fs in failuresStat: + plt.title(fs[0]) + plt.bar(range(len(fs[1])), fs[1].values(), align='center') + plt.xticks(range(len(fs[1])), fs[1].keys(), rotation=90) + plt.show() + print fs +
('payload/pingDiscardedForSize', {u'<unknown>': 4523, u'main': 1810, u'crash': 35, u'bhr': 49})
+
('payload/sendFailure', {u'eUnreachable': 2725838, u'abort': 13191, u'eChannelOpen': 4789989, u'timeout': 771011})
+Unknown currently represent all oversized pending pings. (https://bugzilla.mozilla.org/show_bug.cgi?id=1384903)
+failuresStat = getFailuresStatPerFailureName(cachedDataNightly57, failuresNames) +for fs in failuresStat: + plt.title(fs[0]) + plt.bar(range(len(fs[1])), fs[1].values(), align='center') + plt.xticks(range(len(fs[1])), fs[1].keys(), rotation=90) + plt.show() + print fs +
('payload/pingDiscardedForSize', {u'<unknown>': 4523, u'main': 1657, u'crash': 29, u'bhr': 49})
+
('payload/sendFailure', {u'eUnreachable': 2565221, u'abort': 12709, u'eChannelOpen': 4510507, u'timeout': 736047})
+import matplotlib.dates as mdates +def plotlistofTuples(listOfTuples, title="", inColor='blue'): + keys = [t[0] for t in listOfTuples] + values = [t[1] for t in listOfTuples] + + plt.figure(1) + fig = plt.gcf() + fig.set_size_inches(15, 7) + + plt.title(title) + plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y')) + plt.bar(range(len(listOfTuples)), values, align='center', color=inColor) + plt.xticks(range(len(listOfTuples)), keys, rotation=90) +
EXPECTED_SENDFAILURES_COUNT = 60 +def failureCountReasonPerPing(pings, failureName): + pingsWithSendFailure = pings.filter(lambda ping: ping[failureName] != None) + return pingsWithSendFailure.map(lambda ping: (sum(ping[failureName].values()), ping["payload/reason"])).collect() + +def describefailureDistribution(sendFailureCountDistr, failureName): + failuresCount = [k for k, v in sendFailureCountDistr] + pingsPerDaySeries = pd.Series(failuresCount) + print pingsPerDaySeries.describe([.25, .5, .75, .95]) + + plt.title(failureName + " per ping distribution.") + plt.yscale('log') + plt.ylabel('log(' + failureName + ' count)') + plt.xlabel('ping') + plt.plot(sorted(failuresCount)) + plt.show() + +def decribeReasonDistribution(sendFailureCountDistr, failureName): + unexpectedPingsCount = [(k, v) for k, v in sendFailureCountDistr if k > EXPECTED_SENDFAILURES_COUNT] + print "Pings reported more than " + str(EXPECTED_SENDFAILURES_COUNT) + " " + str(len(unexpectedPingsCount)) + + if len(unexpectedPingsCount) != 0: + reasonStat = Counter([v for k, v in unexpectedPingsCount]) + plotlistofTuples(reasonStat.items(), title="Reason distribution for pings reported more than " + str(EXPECTED_SENDFAILURES_COUNT) + " " + failureName) + plt.xlabel('reason') + plt.ylabel('count') + plt.show() + + +def describe(pings, failure): + print "\n COMPUTATION FOR " + failure + "\n" + countAndReason = failureCountReasonPerPing(pings, failure) + decribeReasonDistribution(countAndReason, failure) + describefailureDistribution(countAndReason, failure) +
for f in failuresNames: + describe(cachedData, f) +
COMPUTATION FOR payload/pingDiscardedForSize + +Pings reported more than 60 0 +count 6146.000000 +mean 1.044094 +std 0.696504 +min 1.000000 +25% 1.000000 +50% 1.000000 +75% 1.000000 +95% 1.000000 +max 48.000000 +dtype: float64 +
COMPUTATION FOR payload/sendFailure + +Pings reported more than 60 6831 +
count 3.177612e+06 +mean 2.612034e+00 +std 1.191218e+01 +min 1.000000e+00 +25% 1.000000e+00 +50% 1.000000e+00 +75% 2.000000e+00 +95% 9.000000e+00 +max 6.237000e+03 +dtype: float64 +
Check that: + required fields are non-empty. + payload/reason contains only expected values (“immediate”, “delayed”, “shutdown”). + payload/sendFailure and payload/discardedForSize are non empty together. + count paramter in payload/sendFailure and payload/discardedForSize has type int. + sendFailureType contains only expected values (“eOK”, “eRequest”, “eUnreachable”, “eChannelOpen”, “eRedirect”, “abort”, “timeout”). + payload/discardedForSize contains only 10 records. +* check the distribution of sendFailures (sum) per ping. We expected to have this number not more than 60.
+def validate(ping): + OK = "" + MUST_NOT_BE_EMPTY = "must not be empty" + + # validate os + clientId = ping["clientId"] + + if clientId == None: + return ("clientId " + MUST_NOT_BE_EMPTY, ping) + + os = ping["payload/os"] + if os == None: + return ("OS " + MUST_NOT_BE_EMPTY, ping) + + name, version = os.items() + if name == None: + return ("OS name " + MUST_NOT_BE_EMPTY, ping) + if version == None: + return ("OS version " + MUST_NOT_BE_EMPTY, ping) + + # validate reason + reason = ping["payload/reason"] + if reason == None: + return ("Reason " + MUST_NOT_BE_EMPTY, ping) + + if not reason in ["immediate", "delayed", "shutdown"]: + return ("Reason must be equal to immediate, delayed or shutdown", ping) + + # doesn't contain failures + sendFailure = ping["payload/sendFailure"] + pingDiscardedForSize = ping["payload/pingDiscardedForSize"] + if sendFailure == None and pingDiscardedForSize == None: + return ("Ping must countain at least one of the failures", ping) + + + # validate sendFailure + supportedFailureTypes = ["eOK", "eRequest", "eUnreachable", "eChannelOpen", "eRedirect", "abort", "timeout"] + if sendFailure != None and len(sendFailure) > len(supportedFailureTypes): + return ("send Failure accept only 8 send failures", ping) + + if sendFailure != None: + for key in sendFailure.keys(): + if not key in supportedFailureTypes: + return (key + " type is not supported", ping) + for count in sendFailure.values(): + if not isinstance(count, int): + return ("Count must be int type", ping) + if sum(sendFailure.values()) > 60: + return ("sendFailure count must not be more than 60", ping) + + + # validate pingDiscardedForSize + if pingDiscardedForSize != None: + if len(pingDiscardedForSize) > 10: + return ("pingDicardedForSize accept only top ten pings types", ping) + for count in pingDiscardedForSize.values(): + if not isinstance(count, int): + return ("Count must be int type", ping) + + return (OK, ping) + +# retrieve all needed fields +validatedData = cachedData.map(validate) +errorsPerProblem = validatedData.countByKey() +errorsPerProblem +
defaultdict(int,
+ {'': 3176610, 'sendFailure count must not be more than 60': 6831})
+def printOSReason(data): + return "os: " + str(data[0]) + " reason: " + str(data[1]) + +def osAndReasonForErros(error): + result = validatedData.filter(lambda pair: pair[0] == error).map(lambda pair: (pair[1]["payload/os"], pair[1]["payload/reason"])).collect() + return result[:min(10, len(result))] + +print "Show only 10 info lines per problem \n" +for err in errorsPerProblem.keys(): + if err != '': + print err + print "\n".join(map(printOSReason, osAndReasonForErros(err))) + print "\n" +
Show only 10 info lines per problem
+
+sendFailure count must not be more than 60
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+This includes showing diagrams and printing stats
+from datetime import datetime + +def pingsCountPerDay(pings): + return pings.map(lambda ping: ping["meta/submissionDate"]).countByValue() + +resultDictionary = pingsCountPerDay(cachedData) +
import matplotlib.dates as mdates +def plotlistofTuples(listOfTuples, title="", inColor='blue'): + keys = [t[0] for t in listOfTuples] + values = [t[1] for t in listOfTuples] + + plt.figure(1) + fig = plt.gcf() + fig.set_size_inches(15, 7) + + plt.title(title) + plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y')) + plt.bar(range(len(listOfTuples)), values, align='center', color=inColor) + plt.xticks(range(len(listOfTuples)), keys, rotation=90) +
plotlistofTuples(sorted(resultDictionary.items(), key=lambda tup: tup[0]), "Pings count per day") +plt.xlabel('meta/submissionDate') +plt.ylabel('pings count') +plt.show() +
pingsPerDaySeries = pd.Series(resultDictionary.values()) +pingsPerDaySeries.describe([.25, .5, .75, .95]) +
count 34.000000 +mean 93630.617647 +std 49146.705206 +min 6475.000000 +25% 58565.750000 +50% 77670.500000 +75% 137599.500000 +95% 172715.950000 +max 176918.000000 +dtype: float64 +
def getClients(pings): + clients = pings.map(lambda ping: ping["clientId"]).distinct() + return clients.collect() +
print str(len(getClients(cachedData))) + " clients are reporting health ping." +
148428 clients are reporting health ping. +
from collections import Counter + +def getAvgPerDate(iterable): + aggregare = Counter(iterable) + result = sum(aggregare.values()) * 1.0 / len(aggregare) + return result + +def pingsPerClientPerDay(pings, date): + return pings.map(lambda ping: (ping["clientId"], ping[date])).groupByKey() + +def avgPingsPerClientPerDay(pings, date): + idDateRDD = pingsPerClientPerDay(pings, date) + return idDateRDD.map(lambda pair: getAvgPerDate(pair[1])).collect() +
def plotAvgPingPerDateDistr(pings, date): + PINGS_COUNT_PER_DAY = 24 + resultDistributionList = avgPingsPerClientPerDay(pings, date) + values = [v for v in resultDistributionList if v > PINGS_COUNT_PER_DAY] + print date + " - clients sending too many \"health\" pings per day - " + str(len(values)) + if len(values) > 0: + plt.title("Average pings per day per client") + plt.ylabel('log(average ping count)') + plt.xlabel("clientId (anonymized)") + plt.yscale('log') + plt.plot(sorted(values)) + plt.show() +
plotAvgPingPerDateDistr(cachedData, "meta/submissionDate") +plotAvgPingPerDateDistr(cachedData, "creationDate") +
meta/submissionDate - clients sending too many "health" pings per day - 569 +
creationDate - clients sending too many "health" pings per day - 0 +
Turns out, clients submit health pings properly (less that 24/day) but we get them on server with some delay
+DAU57, DAU56 from re:dash https://sql.telemetry.mozilla.org/queries/15337/source#table
+DAU57 = [('20170802', 9497), ('20170803', 20923), ('20170804', 25515), ('20170805', 24669), ('20170806', 25604), \ + ('20170807', 32762), ('20170808', 36011), ('20170809', 37101), ('20170810', 38934), ('20170811', 38128), \ + ('20170812', 33403), ('20170813', 33301), ('20170814', 38519), ('20170815', 39053), ('20170816', 41439), \ + ('20170817', 41982), ('20170818', 41557), ('20170819', 35654), ('20170820', 35820), ('20170821', 43760), \ + ('20170822', 45038)] +DAU56 = [('20170727', 34419), ('20170728', 33142), ('20170729', 27740), ('20170730', 28194), ('20170731', 34340), ('20170801', 37308)] +
def getClientsPerDay(pings, version, dateFrom, dateTo): + clientsPerDay = pings.filter(lambda ping: ping["application/version"] == version) \ + .map(lambda ping: (ping["meta/submissionDate"], ping["clientId"])) \ + .groupByKey() \ + .map(lambda pair: (pair[0], len(set(pair[1])))).collect() + + return filter(lambda pair: pair[0] >= dateFrom and pair[0] <= dateTo, clientsPerDay) + +filtered57 = getClientsPerDay(cachedData, '57.0a1', '20170801', '20170822') +filtered56 = getClientsPerDay(cachedData, '56.0a1', '20170727', '20170801') +
plotlistofTuples(sorted(DAU56 + DAU57), inColor='red') +plotlistofTuples(sorted(filtered56 + filtered57), "Nightly 56&57. 56 from 20170727 to 20170801. 57 from 20170802 to 20170822") +plt.legend(['DAU', 'Health ping clients/day']) +plt.xlabel('meta/submissionDate') +plt.ylabel('clients count') +
<matplotlib.text.Text at 0x7fe88652ccd0> +
Conclusion: Almost half of the DAU submits health ping. It is seemsed to be because of sendFailure types: eChannelOpen and eUnreachable.
+eChannelOpen - This error happen when we failed to open channel, maybe it is better to avoid closing the channel and reuse existed channels instead.
+eUnreachable - Probably internet connection problems.
++
import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import pandas as pd + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from collections import Counter +import operator + +get_ipython().magic(u'matplotlib inline') +
pings = Dataset.from_source("telemetry") \ + .where(docType='health', appUpdateChannel="beta") \ + .records(sc, sample=1) + +cachedData = get_pings_properties(pings, ["creationDate", "payload/pingDiscardedForSize", "payload/sendFailure", + "clientId", "meta/submissionDate", "payload/os", "payload/reason", "application/version"]).cache() +
fetching 3855.58046MB in 38405 files... +
def aggregateFailures(first, second): + if first is None: + return second + if second is None: + return first + + res = first + for k, v in second.items(): + if isinstance(v, int): + if k in res: + res[k] += v + else: + res[k] = v; + return res +
# return array of pairs [(failureName, {failureStatistic: count, ....}), ...] +# e.g. [(discardedForSize, {"main": 3, "crash": 5}), (sendFailure, {"timeout" : 34})] +def getFailuresStatPerFailureName(pings, failureNames): + def reduceFailure(failureName): + return pings.map(lambda p: p[failureName]).reduce(aggregateFailures) + + return [(name, reduceFailure(name)) for name in failureNames] +
failuresNames = ["payload/pingDiscardedForSize", "payload/sendFailure"] +
failuresStat = getFailuresStatPerFailureName(cachedData, failuresNames) +for fs in failuresStat: + plt.title(fs[0]) + plt.bar(range(len(fs[1])), fs[1].values(), align='center') + plt.xticks(range(len(fs[1])), fs[1].keys(), rotation=90) + plt.show() + print fs +
('payload/pingDiscardedForSize', {u'main': 2013, u'crash': 222})
+
('payload/sendFailure', {u'eUnreachable': 46180480, u'abort': 47104, u'eChannelOpen': 45322465, u'timeout': 12014520, u'eChanndlOpen': 1})
+Unknown currently represent all oversized pending pings. (https://bugzilla.mozilla.org/show_bug.cgi?id=1384903)
+import matplotlib.dates as mdates +def plotlistofTuples(listOfTuples, title="", inColor='blue'): + keys = [t[0] for t in listOfTuples] + values = [t[1] for t in listOfTuples] + + plt.figure(1) + fig = plt.gcf() + fig.set_size_inches(15, 7) + + plt.title(title) + plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y')) + plt.bar(range(len(listOfTuples)), values, align='center', color=inColor) + plt.xticks(range(len(listOfTuples)), keys, rotation=90) +
EXPECTED_SENDFAILURES_COUNT = 60 +def failureCountReasonPerPing(pings, failureName): + pingsWithSendFailure = pings.filter(lambda ping: ping[failureName] != None) + return pingsWithSendFailure.map(lambda ping: (sum(ping[failureName].values()), ping["payload/reason"])).collect() + +def describefailureDistribution(sendFailureCountDistr, failureName): + failuresCount = [k for k, v in sendFailureCountDistr] + pingsPerDaySeries = pd.Series(failuresCount) + print pingsPerDaySeries.describe([.25, .5, .75, .95]) + + plt.title(failureName + " per ping distribution.") + plt.yscale('log') + plt.ylabel('log(' + failureName + ' count)') + plt.xlabel('ping') + plt.plot(sorted(failuresCount)) + plt.show() + +def decribeReasonDistribution(sendFailureCountDistr, failureName): + unexpectedPingsCount = [(k, v) for k, v in sendFailureCountDistr if k > EXPECTED_SENDFAILURES_COUNT] + print "Pings reported more than " + str(EXPECTED_SENDFAILURES_COUNT) + " " + str(len(unexpectedPingsCount)) + + if len(unexpectedPingsCount) != 0: + reasonStat = Counter([v for k, v in unexpectedPingsCount]) + plotlistofTuples(reasonStat.items(), title="Reason distribution for pings reported more than " + str(EXPECTED_SENDFAILURES_COUNT) + " " + failureName) + plt.xlabel('reason') + plt.ylabel('count') + plt.show() + + +def describe(pings, failure): + print "\n COMPUTATION FOR " + failure + "\n" + countAndReason = failureCountReasonPerPing(pings, failure) + decribeReasonDistribution(countAndReason, failure) + describefailureDistribution(countAndReason, failure) +
for f in failuresNames: + describe(cachedData, f) +
COMPUTATION FOR payload/pingDiscardedForSize + +Pings reported more than 60 0 +count 1364.000000 +mean 1.638563 +std 1.745934 +min 1.000000 +25% 1.000000 +50% 1.000000 +75% 1.000000 +95% 5.000000 +max 21.000000 +dtype: float64 +
COMPUTATION FOR payload/sendFailure + +Pings reported more than 60 49712 +
count 3.076923e+07 +mean 3.365849e+00 +std 7.436627e+00 +min 1.000000e+00 +25% 1.000000e+00 +50% 1.000000e+00 +75% 2.000000e+00 +95% 1.000000e+01 +max 1.133700e+04 +dtype: float64 +
Check that: + required fields are non-empty. + payload/reason contains only expected values (“immediate”, “delayed”, “shutdown”). + payload/sendFailure and payload/discardedForSize are non empty together. + count paramter in payload/sendFailure and payload/discardedForSize has type int. + sendFailureType contains only expected values (“eOK”, “eRequest”, “eUnreachable”, “eChannelOpen”, “eRedirect”, “abort”, “timeout”). + payload/discardedForSize contains only 10 records. +* check the distribution of sendFailures (sum) per ping. We expected to have this number not more than 60.
+def validate(ping): + OK = "" + MUST_NOT_BE_EMPTY = "must not be empty" + + # validate os + clientId = ping["clientId"] + + if clientId == None: + return ("clientId " + MUST_NOT_BE_EMPTY, ping) + + os = ping["payload/os"] + if os == None: + return ("OS " + MUST_NOT_BE_EMPTY, ping) + + name, version = os.items() + if name == None: + return ("OS name " + MUST_NOT_BE_EMPTY, ping) + if version == None: + return ("OS version " + MUST_NOT_BE_EMPTY, ping) + + # validate reason + reason = ping["payload/reason"] + if reason == None: + return ("Reason " + MUST_NOT_BE_EMPTY, ping) + + if not reason in ["immediate", "delayed", "shutdown"]: + return ("Reason must be equal to immediate, delayed or shutdown", ping) + + # doesn't contain failures + sendFailure = ping["payload/sendFailure"] + pingDiscardedForSize = ping["payload/pingDiscardedForSize"] + if sendFailure == None and pingDiscardedForSize == None: + return ("Ping must countain at least one of the failures", ping) + + + # validate sendFailure + supportedFailureTypes = ["eOK", "eRequest", "eUnreachable", "eChannelOpen", "eRedirect", "abort", "timeout"] + if sendFailure != None and len(sendFailure) > len(supportedFailureTypes): + return ("send Failure accept only 8 send failures", ping) + + if sendFailure != None: + for key in sendFailure.keys(): + if not key in supportedFailureTypes: + return (key + " type is not supported", ping) + for count in sendFailure.values(): + if not isinstance(count, int): + return ("Count must be int type", ping) + if sum(sendFailure.values()) > 60: + return ("sendFailure count must not be more than 60", ping) + + + # validate pingDiscardedForSize + if pingDiscardedForSize != None: + if len(pingDiscardedForSize) > 10: + return ("pingDicardedForSize accept only top ten pings types", ping) + for count in pingDiscardedForSize.values(): + if not isinstance(count, int): + return ("Count must be int type", ping) + + return (OK, ping) + +# retrieve all needed fields +validatedData = cachedData.map(validate) +errorsPerProblem = validatedData.countByKey() +errorsPerProblem +
defaultdict(int,
+ {'': 30720671,
+ 'OS must not be empty': 9,
+ 'Ping must countain at least one of the failures': 13,
+ 'Reason must be equal to immediate, delayed or shutdown': 4,
+ 'clientId must not be empty': 4,
+ 'sendFailure count must not be more than 60': 49712})
+def printOSReason(data): + return "os: " + str(data[0]) + " reason: " + str(data[1]) + +def osAndReasonForErros(error): + result = validatedData.filter(lambda pair: pair[0] == error).map(lambda pair: (pair[1]["payload/os"], pair[1]["payload/reason"])).collect() + return result[:min(10, len(result))] + +print "Show only 10 info lines per problem \n" +for err in errorsPerProblem.keys(): + if err != '': + print err + print "\n".join(map(printOSReason, osAndReasonForErros(err))) + print "\n" +
Show only 10 info lines per problem
+
+Reason must be equal to immediate, delayed or shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+
+
+sendFailure count must not be more than 60
+os: {u'version': u'6.3', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'6.3', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+
+
+Ping must countain at least one of the failures
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+
+
+clientId must not be empty
+os: None reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+
+
+OS must not be empty
+os: None reason: immediate
+os: None reason: immediate
+os: None reason: shutdown
+os: None reason: shutdown
+os: None reason: immediate
+os: None reason: shutdown
+os: None reason: shutdown
+os: None reason: shutdown
+os: None reason: immediate
+This includes showing diagrams and printing stats
+from datetime import datetime + +def pingsCountPerDay(pings): + return pings.map(lambda ping: ping["meta/submissionDate"]).countByValue() + +resultDictionary = pingsCountPerDay(cachedData) +
import matplotlib.dates as mdates +def plotlistofTuples(listOfTuples, title="", inColor='blue'): + keys = [t[0] for t in listOfTuples] + values = [t[1] for t in listOfTuples] + + plt.figure(1) + fig = plt.gcf() + fig.set_size_inches(15, 7) + + plt.title(title) + plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y')) + plt.bar(range(len(listOfTuples)), values, align='center', color=inColor) + plt.xticks(range(len(listOfTuples)), keys, rotation=90) +
plotlistofTuples(sorted(resultDictionary.items(), key=lambda tup: tup[0]), "Pings count per day") +plt.xlabel('meta/submissionDate') +plt.ylabel('pings count') +plt.show() +
pingsPerDaySeries = pd.Series(resultDictionary.values()) +pingsPerDaySeries.describe([.25, .5, .75, .95]) +
count 3.000000e+01 +mean 1.025680e+06 +std 1.028015e+06 +min 2.000000e+00 +25% 5.970000e+02 +50% 6.205345e+05 +75% 1.962817e+06 +95% 2.665410e+06 +max 2.998241e+06 +dtype: float64 +
def getClients(pings): + clients = pings.map(lambda ping: ping["clientId"]).distinct() + return clients +
clientsCount = getClients(cachedData).count() +print "Clients number = " + str(clientsCount) +
Clients number = 2413716 +
from collections import Counter + +def getAvgPerDate(iterable): + aggregare = Counter(iterable) + result = sum(aggregare.values()) * 1.0 / len(aggregare) + return result + +def pingsPerClientPerDay(pings, date): + return pings.map(lambda ping: (ping["clientId"], ping[date])).groupByKey() + +def avgPingsPerClientPerDay(pings, date): + idDateRDD = pingsPerClientPerDay(pings, date) + return idDateRDD.map(lambda pair: getAvgPerDate(pair[1])).collect() +
def plotAvgPingPerDateDistr(pings, date): + PINGS_COUNT_PER_DAY = 24 + resultDistributionList = avgPingsPerClientPerDay(pings, date) + values = [v for v in resultDistributionList if v > PINGS_COUNT_PER_DAY] + print date + " : clients sending too many \"health\" pings per day - " + str(len(values)) + if len(values) > 0: + plt.title("Average pings per day per client") + plt.ylabel('log(average ping count)') + plt.xlabel('clientId (anonymized)') + plt.yscale('log') + plt.plot(sorted(values)) + plt.show() +
plotAvgPingPerDateDistr(cachedData, "meta/submissionDate") +plotAvgPingPerDateDistr(cachedData, "creationDate") +
meta/submissionDate : clients sending too many "health" pings per day - 8248 +
creationDate : clients sending too many "health" pings per day - 3 +
Turns out, clients submit health pings properly (less that 24/day) but we get them on server with some delay
+ratio = DAU beta / health ping clients count.
+DAU beta from here: https://sql.telemetry.mozilla.org/queries/15337/source#table +health ping clients count: precomputed
+ratio = [('20170810', 0.75), ('20170811', 0.35), \ + ('20170812', 0.4), ('20170813', 0.59), ('20170814', 0.43), ('20170815', 0.73), ('20170816', 0.78), \ + ('20170817', 0.52), ('20170818', 0.38), ('20170819', 0.46), ('20170820', 0.62), ('20170821', 0.43), \ + ('20170822', 0.42), ('20170823', 0.45)] +
plotlistofTuples(sorted(ratio), inColor='red') +plt.xlabel('meta/submissionDate') +plt.ylabel('ratio') +
<matplotlib.text.Text at 0x7f5dac59ac90> +
Conclusion: Almost half of the DAU submits health ping. It is seemsed to be because of sendFailure types: eChannelOpen and eUnreachable.
+eChannelOpen - This error happen when we failed to open channel, maybe it is better to avoid closing the channel and reuse existed channels instead.
+eUnreachable - Probably internet connection problems.
++
import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import pandas as pd + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from collections import Counter +import operator + +get_ipython().magic(u'matplotlib inline') +
pings = Dataset.from_source("telemetry") \ + .where(docType='health', appUpdateChannel="beta") \ + .records(sc, sample=1) + +cachedData = get_pings_properties(pings, ["creationDate", "payload/pingDiscardedForSize", "payload/sendFailure", + "clientId", "meta/submissionDate", "payload/os", "payload/reason", "application/version"]).cache() +
fetching 3855.58046MB in 38405 files... +
def aggregateFailures(first, second): + if first is None: + return second + if second is None: + return first + + res = first + for k, v in second.items(): + if isinstance(v, int): + if k in res: + res[k] += v + else: + res[k] = v; + return res +
# return array of pairs [(failureName, {failureStatistic: count, ....}), ...] +# e.g. [(discardedForSize, {"main": 3, "crash": 5}), (sendFailure, {"timeout" : 34})] +def getFailuresStatPerFailureName(pings, failureNames): + def reduceFailure(failureName): + return pings.map(lambda p: p[failureName]).reduce(aggregateFailures) + + return [(name, reduceFailure(name)) for name in failureNames] +
failuresNames = ["payload/pingDiscardedForSize", "payload/sendFailure"] +
failuresStat = getFailuresStatPerFailureName(cachedData, failuresNames) +for fs in failuresStat: + plt.title(fs[0]) + plt.bar(range(len(fs[1])), fs[1].values(), align='center') + plt.xticks(range(len(fs[1])), fs[1].keys(), rotation=90) + plt.show() + print fs +
('payload/pingDiscardedForSize', {u'main': 2013, u'crash': 222})
+
('payload/sendFailure', {u'eUnreachable': 46180480, u'abort': 47104, u'eChannelOpen': 45322465, u'timeout': 12014520, u'eChanndlOpen': 1})
+Unknown currently represent all oversized pending pings. (https://bugzilla.mozilla.org/show_bug.cgi?id=1384903)
+import matplotlib.dates as mdates +def plotlistofTuples(listOfTuples, title="", inColor='blue'): + keys = [t[0] for t in listOfTuples] + values = [t[1] for t in listOfTuples] + + plt.figure(1) + fig = plt.gcf() + fig.set_size_inches(15, 7) + + plt.title(title) + plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y')) + plt.bar(range(len(listOfTuples)), values, align='center', color=inColor) + plt.xticks(range(len(listOfTuples)), keys, rotation=90) +
EXPECTED_SENDFAILURES_COUNT = 60 +def failureCountReasonPerPing(pings, failureName): + pingsWithSendFailure = pings.filter(lambda ping: ping[failureName] != None) + return pingsWithSendFailure.map(lambda ping: (sum(ping[failureName].values()), ping["payload/reason"])).collect() + +def describefailureDistribution(sendFailureCountDistr, failureName): + failuresCount = [k for k, v in sendFailureCountDistr] + pingsPerDaySeries = pd.Series(failuresCount) + print pingsPerDaySeries.describe([.25, .5, .75, .95]) + + plt.title(failureName + " per ping distribution.") + plt.yscale('log') + plt.ylabel('log(' + failureName + ' count)') + plt.xlabel('ping') + plt.plot(sorted(failuresCount)) + plt.show() + +def decribeReasonDistribution(sendFailureCountDistr, failureName): + unexpectedPingsCount = [(k, v) for k, v in sendFailureCountDistr if k > EXPECTED_SENDFAILURES_COUNT] + print "Pings reported more than " + str(EXPECTED_SENDFAILURES_COUNT) + " " + str(len(unexpectedPingsCount)) + + if len(unexpectedPingsCount) != 0: + reasonStat = Counter([v for k, v in unexpectedPingsCount]) + plotlistofTuples(reasonStat.items(), title="Reason distribution for pings reported more than " + str(EXPECTED_SENDFAILURES_COUNT) + " " + failureName) + plt.xlabel('reason') + plt.ylabel('count') + plt.show() + + +def describe(pings, failure): + print "\n COMPUTATION FOR " + failure + "\n" + countAndReason = failureCountReasonPerPing(pings, failure) + decribeReasonDistribution(countAndReason, failure) + describefailureDistribution(countAndReason, failure) +
for f in failuresNames: + describe(cachedData, f) +
COMPUTATION FOR payload/pingDiscardedForSize + +Pings reported more than 60 0 +count 1364.000000 +mean 1.638563 +std 1.745934 +min 1.000000 +25% 1.000000 +50% 1.000000 +75% 1.000000 +95% 5.000000 +max 21.000000 +dtype: float64 +
COMPUTATION FOR payload/sendFailure + +Pings reported more than 60 49712 +
count 3.076923e+07 +mean 3.365849e+00 +std 7.436627e+00 +min 1.000000e+00 +25% 1.000000e+00 +50% 1.000000e+00 +75% 2.000000e+00 +95% 1.000000e+01 +max 1.133700e+04 +dtype: float64 +
Check that: + required fields are non-empty. + payload/reason contains only expected values (“immediate”, “delayed”, “shutdown”). + payload/sendFailure and payload/discardedForSize are non empty together. + count paramter in payload/sendFailure and payload/discardedForSize has type int. + sendFailureType contains only expected values (“eOK”, “eRequest”, “eUnreachable”, “eChannelOpen”, “eRedirect”, “abort”, “timeout”). + payload/discardedForSize contains only 10 records. +* check the distribution of sendFailures (sum) per ping. We expected to have this number not more than 60.
+def validate(ping): + OK = "" + MUST_NOT_BE_EMPTY = "must not be empty" + + # validate os + clientId = ping["clientId"] + + if clientId == None: + return ("clientId " + MUST_NOT_BE_EMPTY, ping) + + os = ping["payload/os"] + if os == None: + return ("OS " + MUST_NOT_BE_EMPTY, ping) + + name, version = os.items() + if name == None: + return ("OS name " + MUST_NOT_BE_EMPTY, ping) + if version == None: + return ("OS version " + MUST_NOT_BE_EMPTY, ping) + + # validate reason + reason = ping["payload/reason"] + if reason == None: + return ("Reason " + MUST_NOT_BE_EMPTY, ping) + + if not reason in ["immediate", "delayed", "shutdown"]: + return ("Reason must be equal to immediate, delayed or shutdown", ping) + + # doesn't contain failures + sendFailure = ping["payload/sendFailure"] + pingDiscardedForSize = ping["payload/pingDiscardedForSize"] + if sendFailure == None and pingDiscardedForSize == None: + return ("Ping must countain at least one of the failures", ping) + + + # validate sendFailure + supportedFailureTypes = ["eOK", "eRequest", "eUnreachable", "eChannelOpen", "eRedirect", "abort", "timeout"] + if sendFailure != None and len(sendFailure) > len(supportedFailureTypes): + return ("send Failure accept only 8 send failures", ping) + + if sendFailure != None: + for key in sendFailure.keys(): + if not key in supportedFailureTypes: + return (key + " type is not supported", ping) + for count in sendFailure.values(): + if not isinstance(count, int): + return ("Count must be int type", ping) + if sum(sendFailure.values()) > 60: + return ("sendFailure count must not be more than 60", ping) + + + # validate pingDiscardedForSize + if pingDiscardedForSize != None: + if len(pingDiscardedForSize) > 10: + return ("pingDicardedForSize accept only top ten pings types", ping) + for count in pingDiscardedForSize.values(): + if not isinstance(count, int): + return ("Count must be int type", ping) + + return (OK, ping) + +# retrieve all needed fields +validatedData = cachedData.map(validate) +errorsPerProblem = validatedData.countByKey() +errorsPerProblem +
defaultdict(int,
+ {'': 30720671,
+ 'OS must not be empty': 9,
+ 'Ping must countain at least one of the failures': 13,
+ 'Reason must be equal to immediate, delayed or shutdown': 4,
+ 'clientId must not be empty': 4,
+ 'sendFailure count must not be more than 60': 49712})
+def printOSReason(data): + return "os: " + str(data[0]) + " reason: " + str(data[1]) + +def osAndReasonForErros(error): + result = validatedData.filter(lambda pair: pair[0] == error).map(lambda pair: (pair[1]["payload/os"], pair[1]["payload/reason"])).collect() + return result[:min(10, len(result))] + +print "Show only 10 info lines per problem \n" +for err in errorsPerProblem.keys(): + if err != '': + print err + print "\n".join(map(printOSReason, osAndReasonForErros(err))) + print "\n" +
Show only 10 info lines per problem
+
+Reason must be equal to immediate, delayed or shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+
+
+sendFailure count must not be more than 60
+os: {u'version': u'6.3', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'6.3', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+
+
+Ping must countain at least one of the failures
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+
+
+clientId must not be empty
+os: None reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+
+
+OS must not be empty
+os: None reason: immediate
+os: None reason: immediate
+os: None reason: shutdown
+os: None reason: shutdown
+os: None reason: immediate
+os: None reason: shutdown
+os: None reason: shutdown
+os: None reason: shutdown
+os: None reason: immediate
+This includes showing diagrams and printing stats
+from datetime import datetime + +def pingsCountPerDay(pings): + return pings.map(lambda ping: ping["meta/submissionDate"]).countByValue() + +resultDictionary = pingsCountPerDay(cachedData) +
import matplotlib.dates as mdates +def plotlistofTuples(listOfTuples, title="", inColor='blue'): + keys = [t[0] for t in listOfTuples] + values = [t[1] for t in listOfTuples] + + plt.figure(1) + fig = plt.gcf() + fig.set_size_inches(15, 7) + + plt.title(title) + plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y')) + plt.bar(range(len(listOfTuples)), values, align='center', color=inColor) + plt.xticks(range(len(listOfTuples)), keys, rotation=90) +
plotlistofTuples(sorted(resultDictionary.items(), key=lambda tup: tup[0]), "Pings count per day") +plt.xlabel('meta/submissionDate') +plt.ylabel('pings count') +plt.show() +
pingsPerDaySeries = pd.Series(resultDictionary.values()) +pingsPerDaySeries.describe([.25, .5, .75, .95]) +
count 3.000000e+01 +mean 1.025680e+06 +std 1.028015e+06 +min 2.000000e+00 +25% 5.970000e+02 +50% 6.205345e+05 +75% 1.962817e+06 +95% 2.665410e+06 +max 2.998241e+06 +dtype: float64 +
def getClients(pings): + clients = pings.map(lambda ping: ping["clientId"]).distinct() + return clients +
clientsCount = getClients(cachedData).count() +print "Clients number = " + str(clientsCount) +
Clients number = 2413716 +
from collections import Counter + +def getAvgPerDate(iterable): + aggregare = Counter(iterable) + result = sum(aggregare.values()) * 1.0 / len(aggregare) + return result + +def pingsPerClientPerDay(pings, date): + return pings.map(lambda ping: (ping["clientId"], ping[date])).groupByKey() + +def avgPingsPerClientPerDay(pings, date): + idDateRDD = pingsPerClientPerDay(pings, date) + return idDateRDD.map(lambda pair: getAvgPerDate(pair[1])).collect() +
def plotAvgPingPerDateDistr(pings, date): + PINGS_COUNT_PER_DAY = 24 + resultDistributionList = avgPingsPerClientPerDay(pings, date) + values = [v for v in resultDistributionList if v > PINGS_COUNT_PER_DAY] + print date + " : clients sending too many \"health\" pings per day - " + str(len(values)) + if len(values) > 0: + plt.title("Average pings per day per client") + plt.ylabel('log(average ping count)') + plt.xlabel('clientId (anonymized)') + plt.yscale('log') + plt.plot(sorted(values)) + plt.show() +
plotAvgPingPerDateDistr(cachedData, "meta/submissionDate") +plotAvgPingPerDateDistr(cachedData, "creationDate") +
meta/submissionDate : clients sending too many "health" pings per day - 8248 +
creationDate : clients sending too many "health" pings per day - 3 +
Turns out, clients submit health pings properly (less that 24/day) but we get them on server with some delay
+ratio = DAU beta / health ping clients count.
+DAU beta from here: https://sql.telemetry.mozilla.org/queries/15337/source#table +health ping clients count: precomputed
+ratio = [('20170810', 0.75), ('20170811', 0.35), \ + ('20170812', 0.4), ('20170813', 0.59), ('20170814', 0.43), ('20170815', 0.73), ('20170816', 0.78), \ + ('20170817', 0.52), ('20170818', 0.38), ('20170819', 0.46), ('20170820', 0.62), ('20170821', 0.43), \ + ('20170822', 0.42), ('20170823', 0.45)] +
plotlistofTuples(sorted(ratio), inColor='red') +plt.xlabel('meta/submissionDate') +plt.ylabel('ratio') +
<matplotlib.text.Text at 0x7f5dac59ac90> +
Conclusion: Almost half of the DAU submits health ping. It is seemsed to be because of sendFailure types: eChannelOpen and eUnreachable.
+eChannelOpen - This error happen when we failed to open channel, maybe it is better to avoid closing the channel and reuse existed channels instead.
+eUnreachable - Probably internet connection problems.
++
import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import pandas as pd + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from collections import Counter +import operator + +get_ipython().magic(u'matplotlib inline') +
pings = Dataset.from_source("telemetry") \ + .where(docType='health', appUpdateChannel="beta") \ + .records(sc, sample=1) + +cachedData = get_pings_properties(pings, ["creationDate", "payload/pingDiscardedForSize", "payload/sendFailure", + "clientId", "meta/submissionDate", "payload/os", "payload/reason", "application/version"]).cache() +
fetching 3855.58046MB in 38405 files... +
def aggregateFailures(first, second): + if first is None: + return second + if second is None: + return first + + res = first + for k, v in second.items(): + if isinstance(v, int): + if k in res: + res[k] += v + else: + res[k] = v; + return res +
# return array of pairs [(failureName, {failureStatistic: count, ....}), ...] +# e.g. [(discardedForSize, {"main": 3, "crash": 5}), (sendFailure, {"timeout" : 34})] +def getFailuresStatPerFailureName(pings, failureNames): + def reduceFailure(failureName): + return pings.map(lambda p: p[failureName]).reduce(aggregateFailures) + + return [(name, reduceFailure(name)) for name in failureNames] +
failuresNames = ["payload/pingDiscardedForSize", "payload/sendFailure"] +
failuresStat = getFailuresStatPerFailureName(cachedData, failuresNames) +for fs in failuresStat: + plt.title(fs[0]) + plt.bar(range(len(fs[1])), fs[1].values(), align='center') + plt.xticks(range(len(fs[1])), fs[1].keys(), rotation=90) + plt.show() + print fs +
('payload/pingDiscardedForSize', {u'main': 2013, u'crash': 222})
+
('payload/sendFailure', {u'eUnreachable': 46180480, u'abort': 47104, u'eChannelOpen': 45322465, u'timeout': 12014520, u'eChanndlOpen': 1})
+Unknown currently represent all oversized pending pings. (https://bugzilla.mozilla.org/show_bug.cgi?id=1384903)
+import matplotlib.dates as mdates +def plotlistofTuples(listOfTuples, title="", inColor='blue'): + keys = [t[0] for t in listOfTuples] + values = [t[1] for t in listOfTuples] + + plt.figure(1) + fig = plt.gcf() + fig.set_size_inches(15, 7) + + plt.title(title) + plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y')) + plt.bar(range(len(listOfTuples)), values, align='center', color=inColor) + plt.xticks(range(len(listOfTuples)), keys, rotation=90) +
EXPECTED_SENDFAILURES_COUNT = 60 +def failureCountReasonPerPing(pings, failureName): + pingsWithSendFailure = pings.filter(lambda ping: ping[failureName] != None) + return pingsWithSendFailure.map(lambda ping: (sum(ping[failureName].values()), ping["payload/reason"])).collect() + +def describefailureDistribution(sendFailureCountDistr, failureName): + failuresCount = [k for k, v in sendFailureCountDistr] + pingsPerDaySeries = pd.Series(failuresCount) + print pingsPerDaySeries.describe([.25, .5, .75, .95]) + + plt.title(failureName + " per ping distribution.") + plt.yscale('log') + plt.ylabel('log(' + failureName + ' count)') + plt.xlabel('ping') + plt.plot(sorted(failuresCount)) + plt.show() + +def decribeReasonDistribution(sendFailureCountDistr, failureName): + unexpectedPingsCount = [(k, v) for k, v in sendFailureCountDistr if k > EXPECTED_SENDFAILURES_COUNT] + print "Pings reported more than " + str(EXPECTED_SENDFAILURES_COUNT) + " " + str(len(unexpectedPingsCount)) + + if len(unexpectedPingsCount) != 0: + reasonStat = Counter([v for k, v in unexpectedPingsCount]) + plotlistofTuples(reasonStat.items(), title="Reason distribution for pings reported more than " + str(EXPECTED_SENDFAILURES_COUNT) + " " + failureName) + plt.xlabel('reason') + plt.ylabel('count') + plt.show() + + +def describe(pings, failure): + print "\n COMPUTATION FOR " + failure + "\n" + countAndReason = failureCountReasonPerPing(pings, failure) + decribeReasonDistribution(countAndReason, failure) + describefailureDistribution(countAndReason, failure) +
for f in failuresNames: + describe(cachedData, f) +
COMPUTATION FOR payload/pingDiscardedForSize + +Pings reported more than 60 0 +count 1364.000000 +mean 1.638563 +std 1.745934 +min 1.000000 +25% 1.000000 +50% 1.000000 +75% 1.000000 +95% 5.000000 +max 21.000000 +dtype: float64 +
COMPUTATION FOR payload/sendFailure + +Pings reported more than 60 49712 +
count 3.076923e+07 +mean 3.365849e+00 +std 7.436627e+00 +min 1.000000e+00 +25% 1.000000e+00 +50% 1.000000e+00 +75% 2.000000e+00 +95% 1.000000e+01 +max 1.133700e+04 +dtype: float64 +
Check that: + required fields are non-empty. + payload/reason contains only expected values (“immediate”, “delayed”, “shutdown”). + payload/sendFailure and payload/discardedForSize are non empty together. + count paramter in payload/sendFailure and payload/discardedForSize has type int. + sendFailureType contains only expected values (“eOK”, “eRequest”, “eUnreachable”, “eChannelOpen”, “eRedirect”, “abort”, “timeout”). + payload/discardedForSize contains only 10 records. +* check the distribution of sendFailures (sum) per ping. We expected to have this number not more than 60.
+def validate(ping): + OK = "" + MUST_NOT_BE_EMPTY = "must not be empty" + + # validate os + clientId = ping["clientId"] + + if clientId == None: + return ("clientId " + MUST_NOT_BE_EMPTY, ping) + + os = ping["payload/os"] + if os == None: + return ("OS " + MUST_NOT_BE_EMPTY, ping) + + name, version = os.items() + if name == None: + return ("OS name " + MUST_NOT_BE_EMPTY, ping) + if version == None: + return ("OS version " + MUST_NOT_BE_EMPTY, ping) + + # validate reason + reason = ping["payload/reason"] + if reason == None: + return ("Reason " + MUST_NOT_BE_EMPTY, ping) + + if not reason in ["immediate", "delayed", "shutdown"]: + return ("Reason must be equal to immediate, delayed or shutdown", ping) + + # doesn't contain failures + sendFailure = ping["payload/sendFailure"] + pingDiscardedForSize = ping["payload/pingDiscardedForSize"] + if sendFailure == None and pingDiscardedForSize == None: + return ("Ping must countain at least one of the failures", ping) + + + # validate sendFailure + supportedFailureTypes = ["eOK", "eRequest", "eUnreachable", "eChannelOpen", "eRedirect", "abort", "timeout"] + if sendFailure != None and len(sendFailure) > len(supportedFailureTypes): + return ("send Failure accept only 8 send failures", ping) + + if sendFailure != None: + for key in sendFailure.keys(): + if not key in supportedFailureTypes: + return (key + " type is not supported", ping) + for count in sendFailure.values(): + if not isinstance(count, int): + return ("Count must be int type", ping) + if sum(sendFailure.values()) > 60: + return ("sendFailure count must not be more than 60", ping) + + + # validate pingDiscardedForSize + if pingDiscardedForSize != None: + if len(pingDiscardedForSize) > 10: + return ("pingDicardedForSize accept only top ten pings types", ping) + for count in pingDiscardedForSize.values(): + if not isinstance(count, int): + return ("Count must be int type", ping) + + return (OK, ping) + +# retrieve all needed fields +validatedData = cachedData.map(validate) +errorsPerProblem = validatedData.countByKey() +errorsPerProblem +
defaultdict(int,
+ {'': 30720671,
+ 'OS must not be empty': 9,
+ 'Ping must countain at least one of the failures': 13,
+ 'Reason must be equal to immediate, delayed or shutdown': 4,
+ 'clientId must not be empty': 4,
+ 'sendFailure count must not be more than 60': 49712})
+def printOSReason(data): + return "os: " + str(data[0]) + " reason: " + str(data[1]) + +def osAndReasonForErros(error): + result = validatedData.filter(lambda pair: pair[0] == error).map(lambda pair: (pair[1]["payload/os"], pair[1]["payload/reason"])).collect() + return result[:min(10, len(result))] + +print "Show only 10 info lines per problem \n" +for err in errorsPerProblem.keys(): + if err != '': + print err + print "\n".join(map(printOSReason, osAndReasonForErros(err))) + print "\n" +
Show only 10 info lines per problem
+
+Reason must be equal to immediate, delayed or shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: ilmediate
+
+
+sendFailure count must not be more than 60
+os: {u'version': u'6.3', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'10.0', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+os: {u'version': u'6.3', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: delayed
+
+
+Ping must countain at least one of the failures
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: shutdown
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+
+
+clientId must not be empty
+os: None reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+os: {u'version': u'6.1', u'name': u'WINNT'} reason: immediate
+
+
+OS must not be empty
+os: None reason: immediate
+os: None reason: immediate
+os: None reason: shutdown
+os: None reason: shutdown
+os: None reason: immediate
+os: None reason: shutdown
+os: None reason: shutdown
+os: None reason: shutdown
+os: None reason: immediate
+This includes showing diagrams and printing stats
+from datetime import datetime + +def pingsCountPerDay(pings): + return pings.map(lambda ping: ping["meta/submissionDate"]).countByValue() + +resultDictionary = pingsCountPerDay(cachedData) +
import matplotlib.dates as mdates +def plotlistofTuples(listOfTuples, title="", inColor='blue'): + keys = [t[0] for t in listOfTuples] + values = [t[1] for t in listOfTuples] + + plt.figure(1) + fig = plt.gcf() + fig.set_size_inches(15, 7) + + plt.title(title) + plt.gca().xaxis.set_major_formatter(mdates.DateFormatter('%m/%d/%Y')) + plt.bar(range(len(listOfTuples)), values, align='center', color=inColor) + plt.xticks(range(len(listOfTuples)), keys, rotation=90) +
plotlistofTuples(sorted(resultDictionary.items(), key=lambda tup: tup[0]), "Pings count per day") +plt.xlabel('meta/submissionDate') +plt.ylabel('pings count') +plt.show() +
pingsPerDaySeries = pd.Series(resultDictionary.values()) +pingsPerDaySeries.describe([.25, .5, .75, .95]) +
count 3.000000e+01 +mean 1.025680e+06 +std 1.028015e+06 +min 2.000000e+00 +25% 5.970000e+02 +50% 6.205345e+05 +75% 1.962817e+06 +95% 2.665410e+06 +max 2.998241e+06 +dtype: float64 +
def getClients(pings): + clients = pings.map(lambda ping: ping["clientId"]).distinct() + return clients +
clientsCount = getClients(cachedData).count() +print "Clients number = " + str(clientsCount) +
Clients number = 2413716 +
from collections import Counter + +def getAvgPerDate(iterable): + aggregare = Counter(iterable) + result = sum(aggregare.values()) * 1.0 / len(aggregare) + return result + +def pingsPerClientPerDay(pings, date): + return pings.map(lambda ping: (ping["clientId"], ping[date])).groupByKey() + +def avgPingsPerClientPerDay(pings, date): + idDateRDD = pingsPerClientPerDay(pings, date) + return idDateRDD.map(lambda pair: getAvgPerDate(pair[1])).collect() +
def plotAvgPingPerDateDistr(pings, date): + PINGS_COUNT_PER_DAY = 24 + resultDistributionList = avgPingsPerClientPerDay(pings, date) + values = [v for v in resultDistributionList if v > PINGS_COUNT_PER_DAY] + print date + " : clients sending too many \"health\" pings per day - " + str(len(values)) + if len(values) > 0: + plt.title("Average pings per day per client") + plt.ylabel('log(average ping count)') + plt.xlabel('clientId (anonymized)') + plt.yscale('log') + plt.plot(sorted(values)) + plt.show() +
plotAvgPingPerDateDistr(cachedData, "meta/submissionDate") +plotAvgPingPerDateDistr(cachedData, "creationDate") +
meta/submissionDate : clients sending too many "health" pings per day - 8248 +
creationDate : clients sending too many "health" pings per day - 3 +
Turns out, clients submit health pings properly (less that 24/day) but we get them on server with some delay
+ratio = DAU beta / health ping clients count.
+DAU beta from here: https://sql.telemetry.mozilla.org/queries/15337/source#table +health ping clients count: precomputed
+ratio = [('20170810', 0.75), ('20170811', 0.35), \ + ('20170812', 0.4), ('20170813', 0.59), ('20170814', 0.43), ('20170815', 0.73), ('20170816', 0.78), \ + ('20170817', 0.52), ('20170818', 0.38), ('20170819', 0.46), ('20170820', 0.62), ('20170821', 0.43), \ + ('20170822', 0.42), ('20170823', 0.45)] +
plotlistofTuples(sorted(ratio), inColor='red') +plt.xlabel('meta/submissionDate') +plt.ylabel('ratio') +
<matplotlib.text.Text at 0x7f5dac59ac90> +
Conclusion: Almost half of the DAU submits health ping. It is seemsed to be because of sendFailure types: eChannelOpen and eUnreachable.
+eChannelOpen - This error happen when we failed to open channel, maybe it is better to avoid closing the channel and reuse existed channels instead.
+eUnreachable - Probably internet connection problems.
++
This analysis is an adaptation of the one performed on crash pings to validate the effectiveness of the pingsender to reduce data latency.
+Specifically, this one investigates the difference between typical values of “recording delay” and “submission delay” before and after pingSender started sending “shutdown” pings.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
We’ll be looking at two cohorts: April 2-8 and April 30 - May 6. The pingsender started sending shudown pings on the 14th of April, but due to some crashes we disabled it shortly after. We enabled sending the shutdown ping using the pingsender, again, on the 28th of April.
We will examing two cohorts: the first with shutdown pings sent without the pingsender, the second with shutdown pings sent with the pingsender.
pre_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170402" <= x < "20170408") \ + .where(appBuildId=lambda x: "20170402" <= x < "20170408") \ + .records(sc, sample=1) + +post_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170430" <= x < "20170506") \ + .where(appBuildId=lambda x: "20170430" <= x < "20170506") \ + .records(sc, sample=1) +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+meta/creationTimestamp The time the Telemetry code in Firefox created the ping, according to the client’s clock, in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
pre_subset = get_pings_properties(pre_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) + +post_subset = get_pings_properties(post_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) +
The shutdown ping is a particular kind of main ping with the reason field set to shutdown, as it’s saved during shutdown.
pre_subset = pre_subset.filter(lambda p: p.get("payload/info/reason") == "shutdown") +post_subset = post_subset.filter(lambda p: p.get("payload/info/reason") == "shutdown") +
The rest of the analysis is cleaner if we combine the two cohorts here.
+def add_pre(p): + p['pre'] = 'pre' + return p + +def add_post(p): + p['pre'] = 'post' + return p + +combined = pre_subset.map(add_pre).union(post_subset.map(add_post)) +
Quick normalization: ditch any ping that doesn’t have a creationTimestamp or Timestamp:
+prev_count = combined.count() +combined = combined.filter(lambda p:\ + p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +filtered_count = combined.count() +print "Filtered {} of {} pings ({:.2f}%)"\ + .format(prev_count - filtered_count, prev_count, 100.0 * (prev_count - filtered_count) / prev_count) +
Filtered 0 of 570853 pings (0.00%) +
We sometimes receive main pings more than once (identical document ids). This is usually low, but let’s check if this is still true after using the pingsender.
+So we’ll dedupe here.
+def dedupe(pings): + return pings\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) + +combined_deduped = dedupe(combined) +
combined_count = combined.count() +combined_deduped_count = combined_deduped.count() +print "Filtered {} of {} shutdown pings ({:.2f}%)"\ + .format(combined_count - combined_deduped_count, combined_count, + 100.0 * (combined_count - combined_deduped_count) / combined_count) +
Filtered 6136 of 570853 shutdown pings (1.07%) +
We’ll be plotting Cumulative Distribution Functions today.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +PRES = ['pre', 'post'] +
def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys) +
def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
delays_by_chan = combined_deduped.map(lambda p: (p["pre"], calculate_submission_delay(p))) +
Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+setup_plot("'shutdown' ping submission delay CDF", MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == pre)\ + .map(lambda d: d[1] / HOUR_IN_S if d[1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(["No pingsender", "With pingsender"], loc="lower right") +
<matplotlib.legend.Legend at 0x7fd034c91ad0> +
The use of pingsender results in an improvement in the submission delay of the shutdown “main” ping. We receive almost 85% of the mentioned pings as soon as they are generated, instead of just ~30% within the first hour.
We don’t receive 100% of the pings sooner for builds having the pingsender enabled because the pingsender can fail submitting the ping (e.g. the system or Firefox uses a proxy, poor connection, …) and, when this happen, no retrasmission is attempted; the ping will be sent on the next restart by Firefox.
Let’s start by separating the pings coming from the pingsender from the ones coming from the normal Firefox flow since the pingsender started sending the shutdown pings.
post_pingsender_only = post_subset.filter(lambda p: p.get("meta/X-PingSender-Version") is not None) +post_no_pingsender = post_subset.filter(lambda p: p.get("meta/X-PingSender-Version") is None) +
num_from_pingsender = post_pingsender_only.count() +num_no_pingsender = post_no_pingsender.count() +total_post = post_subset.count() +num_sent_by_both =\ + post_pingsender_only.map(lambda p: p["id"]).intersection(post_no_pingsender.map(lambda p: p["id"])).count() +
We want to understand how many pings were sent by the pingsender, correctly received from the server, and sent again next time Firefox starts.
+def pct(a, b): + return 100 * float(a) / b + +print("Duplicate pings percentage: {:.2f}%".format(pct(num_sent_by_both, total_post))) +
Duplicate pings percentage: 1.38% +
Do we get many more duplicates after landing the shutdown pingsender?
count_deduped_pre = dedupe(pre_subset).count() +count_pre = pre_subset.count() +count_deduped_post = dedupe(post_subset).count() +count_post = post_subset.count() + +print("Duplicates with shutdown pingsender:\nBefore:\t{:.2f}%\nAfter:\t{:.2f}%\n"\ + .format(pct(count_pre - count_deduped_pre, count_pre), + pct(count_post - count_deduped_post, count_post))) +
Duplicates with shutdown pingsender: +Before: 0.50% +After: 1.61% +
It looks like 1% of the pings sent by the pingsender are also being sent by Firefox next time it restarts. This is potentially due to pingsender:
It’s important to note that the percentages of duplicate pings from the previous cells are not the same. The first, 1.38%, is the percentage of duplicates that were sent at least once by pingsender whereas the last, 1.61%, includes all duplicates regardless of whether pingsender was involved.
+Start off by getting the pings that were sent by both the pingsender and the normal Firefox flow. This is basically mimicking an intersectByKey, which is not available on pySpark.
pingsender_dupes = post_pingsender_only\ + .map(lambda p: (p["id"], calculate_submission_delay(p)))\ + .cogroup(post_no_pingsender\ + .map(lambda p: (p["id"], calculate_submission_delay(p))))\ + .filter(lambda p: p[1][0] and p[1][1])\ + .map(lambda p: (p[0], (list(p[1][0]), list(p[1][1])))) +
The pingsender_dupes RDD should now contain only the data for the pings sent by both systems. Each entry is in the form:
{ping-id: ([ delays for duplicates from the pingsender ], [delays for duplicates by FF])}
We assume that the pingsender only sends a ping once and that Firefox might attempt to send more than once, hence might have more than one ping delay in its list. Let’s see if these claims hold true.
pingsender_dupes.first() +
(u'77426bfe-4b24-4c81-b4ae-a8de2ee56736', ([-0.457], [3846.543])) +
# Number of duplicates, for each duped ping, from the pingsender. +print pingsender_dupes.map(lambda p: len(p[1][0])).countByValue() +# Number of duplicates, for each duped ping, from Firefox. +print pingsender_dupes.map(lambda p: len(p[1][1])).countByValue() +
defaultdict(<type 'int'>, {1: 4067, 2: 2})
+defaultdict(<type 'int'>, {1: 4053, 2: 15, 3: 1})
+It seems that the pingsender can, sometimes, send the ping more than once. That’s unexpected, but it has a relatively low occurrence (just twice over 4069 duplicated pings). The same issue can be seen with Firefox, with the occurrence being a little higher.
Finally, compute the average delay between the duplicates from the pingsender and Firefox.
delay_between_duplicates =\ + pingsender_dupes.map(lambda t: np.fabs(np.max(t[1][1]) - np.min(t[1][0]))) +
setup_plot("'shutdown' duplicates submission delay CDF", MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +plot_cdf(delay_between_duplicates\ + .map(lambda d: d / HOUR_IN_S if d < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) +plt.axvline(x=4, ymin=0.0, ymax = 1.0, linewidth=1, linestyle='dashed', color='r') +plt.legend(["Duplicates delay", "4 hour filter limit"], loc="lower right") +
<matplotlib.legend.Legend at 0x7fd02e6807d0> +
About ~65% of the duplicates can be caught by the deduplicator because they will arrive within a 4 hour window.
+collected_delays = delay_between_duplicates.collect() +
plt.title("The distribution of 'shutdown' ping delays for duplicate submissions") +plt.xlabel("Delay (seconds)") +plt.ylabel("Frequency") + +# Use 50 bins for values up to the clip value, and accumulate the +# rest in the last bucket (instead of having a super-long tail). +plt.hist(np.clip(collected_delays, 0, 48.0 * HOUR_IN_S), + alpha=0.5, bins=50, label="Delays") +# Plot some convenience marker for 4, 12 and 24 hours. +for m in [4.0, 12.0, 24.0]: + plt.axvline(x=m * HOUR_IN_S, ymin=0.0, ymax = 1.0, linewidth=1, linestyle='dashed', color='r', + label="{} hours".format(m)) +plt.legend() +
<matplotlib.legend.Legend at 0x7fd02e1a8390> +
The shutdownDuration is defined as the time it takes to complete the Firefox shutdown process, in milliseconds. Extract the data from the two series: before the shutdown pingsender was enabled and after. Plot the data as two distinct distributions on the same plot.
pre_shutdown_durations = pre_subset.map(lambda p: p.get("payload/simpleMeasurements/shutdownDuration", None))\ + .filter(lambda p: p is not None)\ + .collect() +post_shutdown_durations = post_subset.map(lambda p: p.get("payload/simpleMeasurements/shutdownDuration", None))\ + .filter(lambda p: p is not None)\ + .collect() +
plt.title("'shutdown' pingsender effect on the shutdown duration") +plt.xlabel("shutdownDuration (milliseconds)") +plt.ylabel("Number of pings") + +# Use 50 bins for values up to the clip value, and accumulate the +# rest in the last bucket (instead of having a super-long tail). +CLIP_VALUE = 10000 # 10s +plt.hist([np.clip(pre_shutdown_durations, 0, CLIP_VALUE), np.clip(post_shutdown_durations, 0, CLIP_VALUE)], + alpha=0.5, bins=50, label=["No pingsender", "With pingsender"]) +plt.legend() +
<matplotlib.legend.Legend at 0x7fd02f2ce250> +
It seems that the distribution of shutdown durations for builds with the pingsender enabled has a different shape compared to the distribution of shutdown durations for builds with no pingsender. The former seems to be a bit shifted toward higher values of the duration times. The same trend can be spotted on TMO.
+Let’s dig more into this by looking at some statistics about the durations.
+def stats(data, label): + print("\n{}\n".format(label)) + print("Min:\t{}".format(np.min(data))) + print("Max:\t{}".format(np.max(data))) + print("Average:\t{}".format(np.mean(data))) + print("50, 90 and 99 percentiles:\t{}\n".format(np.percentile(data, [0.5, 0.9, 0.99]))) + +stats(pre_shutdown_durations, "No pingsender (ms)") +stats(post_shutdown_durations, "With pingsender (ms)") +
No pingsender (ms) + +Min: 25 +Max: 146671322 +Average: 8797.70181773 +50, 90 and 99 percentiles: [ 351. 374. 378.] + + +With pingsender (ms) + +Min: 19 +Max: 94115063 +Average: 7670.99524021 +50, 90 and 99 percentiles: [ 352. 377. 382.] +
It seems that builds that are sending shutdown pings at shutdown are taking up to about 4ms more to close.
+
This analysis is an adaptation of the one performed on crash pings to validate the effectiveness of the pingsender to reduce data latency.
+Specifically, this one investigates the difference between typical values of “recording delay” and “submission delay” before and after pingSender started sending “shutdown” pings.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
We’ll be looking at two cohorts: April 2-8 and April 30 - May 6. The pingsender started sending shudown pings on the 14th of April, but due to some crashes we disabled it shortly after. We enabled sending the shutdown ping using the pingsender, again, on the 28th of April.
We will examing two cohorts: the first with shutdown pings sent without the pingsender, the second with shutdown pings sent with the pingsender.
pre_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170402" <= x < "20170408") \ + .where(appBuildId=lambda x: "20170402" <= x < "20170408") \ + .records(sc, sample=1) + +post_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170430" <= x < "20170506") \ + .where(appBuildId=lambda x: "20170430" <= x < "20170506") \ + .records(sc, sample=1) +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+meta/creationTimestamp The time the Telemetry code in Firefox created the ping, according to the client’s clock, in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
pre_subset = get_pings_properties(pre_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) + +post_subset = get_pings_properties(post_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) +
The shutdown ping is a particular kind of main ping with the reason field set to shutdown, as it’s saved during shutdown.
pre_subset = pre_subset.filter(lambda p: p.get("payload/info/reason") == "shutdown") +post_subset = post_subset.filter(lambda p: p.get("payload/info/reason") == "shutdown") +
The rest of the analysis is cleaner if we combine the two cohorts here.
+def add_pre(p): + p['pre'] = 'pre' + return p + +def add_post(p): + p['pre'] = 'post' + return p + +combined = pre_subset.map(add_pre).union(post_subset.map(add_post)) +
Quick normalization: ditch any ping that doesn’t have a creationTimestamp or Timestamp:
+prev_count = combined.count() +combined = combined.filter(lambda p:\ + p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +filtered_count = combined.count() +print "Filtered {} of {} pings ({:.2f}%)"\ + .format(prev_count - filtered_count, prev_count, 100.0 * (prev_count - filtered_count) / prev_count) +
Filtered 0 of 570853 pings (0.00%) +
We sometimes receive main pings more than once (identical document ids). This is usually low, but let’s check if this is still true after using the pingsender.
+So we’ll dedupe here.
+def dedupe(pings): + return pings\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) + +combined_deduped = dedupe(combined) +
combined_count = combined.count() +combined_deduped_count = combined_deduped.count() +print "Filtered {} of {} shutdown pings ({:.2f}%)"\ + .format(combined_count - combined_deduped_count, combined_count, + 100.0 * (combined_count - combined_deduped_count) / combined_count) +
Filtered 6136 of 570853 shutdown pings (1.07%) +
We’ll be plotting Cumulative Distribution Functions today.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +PRES = ['pre', 'post'] +
def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys) +
def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
delays_by_chan = combined_deduped.map(lambda p: (p["pre"], calculate_submission_delay(p))) +
Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+setup_plot("'shutdown' ping submission delay CDF", MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == pre)\ + .map(lambda d: d[1] / HOUR_IN_S if d[1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(["No pingsender", "With pingsender"], loc="lower right") +
<matplotlib.legend.Legend at 0x7fd034c91ad0> +
The use of pingsender results in an improvement in the submission delay of the shutdown “main” ping. We receive almost 85% of the mentioned pings as soon as they are generated, instead of just ~30% within the first hour.
We don’t receive 100% of the pings sooner for builds having the pingsender enabled because the pingsender can fail submitting the ping (e.g. the system or Firefox uses a proxy, poor connection, …) and, when this happen, no retrasmission is attempted; the ping will be sent on the next restart by Firefox.
Let’s start by separating the pings coming from the pingsender from the ones coming from the normal Firefox flow since the pingsender started sending the shutdown pings.
post_pingsender_only = post_subset.filter(lambda p: p.get("meta/X-PingSender-Version") is not None) +post_no_pingsender = post_subset.filter(lambda p: p.get("meta/X-PingSender-Version") is None) +
num_from_pingsender = post_pingsender_only.count() +num_no_pingsender = post_no_pingsender.count() +total_post = post_subset.count() +num_sent_by_both =\ + post_pingsender_only.map(lambda p: p["id"]).intersection(post_no_pingsender.map(lambda p: p["id"])).count() +
We want to understand how many pings were sent by the pingsender, correctly received from the server, and sent again next time Firefox starts.
+def pct(a, b): + return 100 * float(a) / b + +print("Duplicate pings percentage: {:.2f}%".format(pct(num_sent_by_both, total_post))) +
Duplicate pings percentage: 1.38% +
Do we get many more duplicates after landing the shutdown pingsender?
count_deduped_pre = dedupe(pre_subset).count() +count_pre = pre_subset.count() +count_deduped_post = dedupe(post_subset).count() +count_post = post_subset.count() + +print("Duplicates with shutdown pingsender:\nBefore:\t{:.2f}%\nAfter:\t{:.2f}%\n"\ + .format(pct(count_pre - count_deduped_pre, count_pre), + pct(count_post - count_deduped_post, count_post))) +
Duplicates with shutdown pingsender: +Before: 0.50% +After: 1.61% +
It looks like 1% of the pings sent by the pingsender are also being sent by Firefox next time it restarts. This is potentially due to pingsender:
It’s important to note that the percentages of duplicate pings from the previous cells are not the same. The first, 1.38%, is the percentage of duplicates that were sent at least once by pingsender whereas the last, 1.61%, includes all duplicates regardless of whether pingsender was involved.
+Start off by getting the pings that were sent by both the pingsender and the normal Firefox flow. This is basically mimicking an intersectByKey, which is not available on pySpark.
pingsender_dupes = post_pingsender_only\ + .map(lambda p: (p["id"], calculate_submission_delay(p)))\ + .cogroup(post_no_pingsender\ + .map(lambda p: (p["id"], calculate_submission_delay(p))))\ + .filter(lambda p: p[1][0] and p[1][1])\ + .map(lambda p: (p[0], (list(p[1][0]), list(p[1][1])))) +
The pingsender_dupes RDD should now contain only the data for the pings sent by both systems. Each entry is in the form:
{ping-id: ([ delays for duplicates from the pingsender ], [delays for duplicates by FF])}
We assume that the pingsender only sends a ping once and that Firefox might attempt to send more than once, hence might have more than one ping delay in its list. Let’s see if these claims hold true.
pingsender_dupes.first() +
(u'77426bfe-4b24-4c81-b4ae-a8de2ee56736', ([-0.457], [3846.543])) +
# Number of duplicates, for each duped ping, from the pingsender. +print pingsender_dupes.map(lambda p: len(p[1][0])).countByValue() +# Number of duplicates, for each duped ping, from Firefox. +print pingsender_dupes.map(lambda p: len(p[1][1])).countByValue() +
defaultdict(<type 'int'>, {1: 4067, 2: 2})
+defaultdict(<type 'int'>, {1: 4053, 2: 15, 3: 1})
+It seems that the pingsender can, sometimes, send the ping more than once. That’s unexpected, but it has a relatively low occurrence (just twice over 4069 duplicated pings). The same issue can be seen with Firefox, with the occurrence being a little higher.
Finally, compute the average delay between the duplicates from the pingsender and Firefox.
delay_between_duplicates =\ + pingsender_dupes.map(lambda t: np.fabs(np.max(t[1][1]) - np.min(t[1][0]))) +
setup_plot("'shutdown' duplicates submission delay CDF", MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +plot_cdf(delay_between_duplicates\ + .map(lambda d: d / HOUR_IN_S if d < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) +plt.axvline(x=4, ymin=0.0, ymax = 1.0, linewidth=1, linestyle='dashed', color='r') +plt.legend(["Duplicates delay", "4 hour filter limit"], loc="lower right") +
<matplotlib.legend.Legend at 0x7fd02e6807d0> +
About ~65% of the duplicates can be caught by the deduplicator because they will arrive within a 4 hour window.
+collected_delays = delay_between_duplicates.collect() +
plt.title("The distribution of 'shutdown' ping delays for duplicate submissions") +plt.xlabel("Delay (seconds)") +plt.ylabel("Frequency") + +# Use 50 bins for values up to the clip value, and accumulate the +# rest in the last bucket (instead of having a super-long tail). +plt.hist(np.clip(collected_delays, 0, 48.0 * HOUR_IN_S), + alpha=0.5, bins=50, label="Delays") +# Plot some convenience marker for 4, 12 and 24 hours. +for m in [4.0, 12.0, 24.0]: + plt.axvline(x=m * HOUR_IN_S, ymin=0.0, ymax = 1.0, linewidth=1, linestyle='dashed', color='r', + label="{} hours".format(m)) +plt.legend() +
<matplotlib.legend.Legend at 0x7fd02e1a8390> +
The shutdownDuration is defined as the time it takes to complete the Firefox shutdown process, in milliseconds. Extract the data from the two series: before the shutdown pingsender was enabled and after. Plot the data as two distinct distributions on the same plot.
pre_shutdown_durations = pre_subset.map(lambda p: p.get("payload/simpleMeasurements/shutdownDuration", None))\ + .filter(lambda p: p is not None)\ + .collect() +post_shutdown_durations = post_subset.map(lambda p: p.get("payload/simpleMeasurements/shutdownDuration", None))\ + .filter(lambda p: p is not None)\ + .collect() +
plt.title("'shutdown' pingsender effect on the shutdown duration") +plt.xlabel("shutdownDuration (milliseconds)") +plt.ylabel("Number of pings") + +# Use 50 bins for values up to the clip value, and accumulate the +# rest in the last bucket (instead of having a super-long tail). +CLIP_VALUE = 10000 # 10s +plt.hist([np.clip(pre_shutdown_durations, 0, CLIP_VALUE), np.clip(post_shutdown_durations, 0, CLIP_VALUE)], + alpha=0.5, bins=50, label=["No pingsender", "With pingsender"]) +plt.legend() +
<matplotlib.legend.Legend at 0x7fd02f2ce250> +
It seems that the distribution of shutdown durations for builds with the pingsender enabled has a different shape compared to the distribution of shutdown durations for builds with no pingsender. The former seems to be a bit shifted toward higher values of the duration times. The same trend can be spotted on TMO.
+Let’s dig more into this by looking at some statistics about the durations.
+def stats(data, label): + print("\n{}\n".format(label)) + print("Min:\t{}".format(np.min(data))) + print("Max:\t{}".format(np.max(data))) + print("Average:\t{}".format(np.mean(data))) + print("50, 90 and 99 percentiles:\t{}\n".format(np.percentile(data, [0.5, 0.9, 0.99]))) + +stats(pre_shutdown_durations, "No pingsender (ms)") +stats(post_shutdown_durations, "With pingsender (ms)") +
No pingsender (ms) + +Min: 25 +Max: 146671322 +Average: 8797.70181773 +50, 90 and 99 percentiles: [ 351. 374. 378.] + + +With pingsender (ms) + +Min: 19 +Max: 94115063 +Average: 7670.99524021 +50, 90 and 99 percentiles: [ 352. 377. 382.] +
It seems that builds that are sending shutdown pings at shutdown are taking up to about 4ms more to close.
+
This analysis is an update of the one performed in the Nightly channel to validate the effectiveness of the pingsender to reduce data latency.
+Specifically, this one investigates the difference between typical values of “recording delay” and “submission delay” between the previous Beta build and the latest one. The latter includes the pingSender that started sending “shutdown” pings.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Unable to parse whitelist: /mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json. +Assuming all histograms are acceptable. +
We’ll be looking at two cohorts: May 31 - June 7 (Beta 54) and June 14 - 20 (Beta 55). The pingsender started sending shudown pings in Beta 55.
We will examing two cohorts: the first with shutdown pings sent without the pingsender, the second with shutdown pings sent with the pingsender.
pre_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170531" <= x < "20170607") \ + .where(appBuildId=lambda x: "20170420" <= x < "20170611") \ + .records(sc, sample=1) + +post_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170614" <= x < "20170620") \ + .where(appBuildId=lambda x: "20170612" <= x < "20170622") \ + .records(sc, sample=1) +
fetching 1147169.41811MB in 39626 files... +fetching 20894.21218MB in 8408 files... +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+meta/creationTimestamp The time the Telemetry code in Firefox created the ping, according to the client’s clock, in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
pre_subset = get_pings_properties(pre_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) + +post_subset = get_pings_properties(post_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) +
The shutdown ping is a particular kind of main ping with the reason field set to shutdown, as it’s saved during shutdown.
pre_subset = pre_subset.filter(lambda p: p.get("payload/info/reason") == "shutdown") +post_subset = post_subset.filter(lambda p: p.get("payload/info/reason") == "shutdown") +
The rest of the analysis is cleaner if we combine the two cohorts here.
+def add_pre(p): + p['pre'] = 'pre' + return p + +def add_post(p): + p['pre'] = 'post' + return p + +combined = pre_subset.map(add_pre).union(post_subset.map(add_post)) +
Quick normalization: ditch any ping that doesn’t have a creationTimestamp or Timestamp:
+prev_count = combined.count() +combined = combined.filter(lambda p:\ + p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +filtered_count = combined.count() +print "Filtered {} of {} pings ({:.2f}%)"\ + .format(prev_count - filtered_count, prev_count, 100.0 * (prev_count - filtered_count) / prev_count) +
Filtered 0 of 34682752 pings (0.00%) +
We sometimes receive main pings more than once (identical document ids). This is usually low, but let’s check if this is still true after using the pingsender.
+So we’ll dedupe here.
+def dedupe(pings): + return pings\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) + +combined_deduped = dedupe(combined) +
combined_count = combined.count() +combined_deduped_count = combined_deduped.count() +print "Filtered {} of {} shutdown pings ({:.2f}%)"\ + .format(combined_count - combined_deduped_count, combined_count, + 100.0 * (combined_count - combined_deduped_count) / combined_count) +
Filtered 481634 of 34682752 shutdown pings (1.39%) +
This is slightly higher than our Nightly analysis, which reported 1.07%, but still in the expected range of about 1%.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +PRES = ['pre', 'post'] +
def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys) +
def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
delays_by_chan = combined_deduped.map(lambda p: (p["pre"], calculate_submission_delay(p))) +
Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+setup_plot("'shutdown' ping submission delay CDF", MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == pre)\ + .map(lambda d: d[1] / HOUR_IN_S if d[1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(["No pingsender", "With pingsender"], loc="lower right") +
<matplotlib.legend.Legend at 0x7fda065b6f10> +
The use of pingsender results in an improvement in the submission delay of the shutdown “main” ping:
shutdown pings within 8 hours, compared to ~95% received after over 90 hours without it. We don’t receive 100% of the pings sooner for builds having the pingsender enabled because the pingsender can fail submitting the ping (e.g. the system or Firefox uses a proxy, poor connection, …) and, when this happen, no retrasmission is attempted; the ping will be sent on the next restart by Firefox.
Let’s start by separating the pings coming from the pingsender from the ones coming from the normal Firefox flow since the pingsender started sending the shutdown pings.
post_pingsender_only = post_subset.filter(lambda p: p.get("meta/X-PingSender-Version") is not None) +post_no_pingsender = post_subset.filter(lambda p: p.get("meta/X-PingSender-Version") is None) +
num_from_pingsender = post_pingsender_only.count() +num_no_pingsender = post_no_pingsender.count() +total_post = post_subset.count() +num_sent_by_both =\ + post_pingsender_only.map(lambda p: p["id"]).intersection(post_no_pingsender.map(lambda p: p["id"])).count() +
We want to understand how many pings were sent by the pingsender, correctly received from the server, and sent again next time Firefox starts.
+def pct(a, b): + return 100 * float(a) / b + +print("Duplicate pings percentage: {:.2f}%".format(pct(num_sent_by_both, total_post))) +
Duplicate pings percentage: 0.41% +
Do we get many more duplicates after landing the shutdown pingsender?
count_deduped_pre = dedupe(pre_subset).count() +count_pre = pre_subset.count() +count_deduped_post = dedupe(post_subset).count() +count_post = post_subset.count() + +print("Duplicates with shutdown pingsender:\nBefore:\t{:.2f}%\nAfter:\t{:.2f}%\n"\ + .format(pct(count_pre - count_deduped_pre, count_pre), + pct(count_post - count_deduped_post, count_post))) +
Duplicates with shutdown pingsender: +Before: 1.40% +After: 0.85% +
It looks like 1% of the pings sent by the pingsender are also being sent by Firefox next time it restarts. This is potentially due to pingsender:
It’s important to note that the percentages of duplicate pings from the previous cells are not the same. The first, 0.41%, is the percentage of duplicates that were sent at least once by pingsender whereas the last, 0.85%, includes all duplicates regardless of whether pingsender was involved. This looks way better than the numbers in the Nightly channel possibly due to disabling the pingsender when the OS is shutting down, which happened in bug 1372202 (after the Nightly analysis was performed).
+Start off by getting the pings that were sent by both the pingsender and the normal Firefox flow. This is basically mimicking an intersectByKey, which is not available on pySpark.
pingsender_dupes = post_pingsender_only\ + .map(lambda p: (p["id"], calculate_submission_delay(p)))\ + .cogroup(post_no_pingsender\ + .map(lambda p: (p["id"], calculate_submission_delay(p))))\ + .filter(lambda p: p[1][0] and p[1][1])\ + .map(lambda p: (p[0], (list(p[1][0]), list(p[1][1])))) +
The pingsender_dupes RDD should now contain only the data for the pings sent by both systems. Each entry is in the form:
{ping-id: ([ delays for duplicates from the pingsender ], [delays for duplicates by FF])}
We assume that the pingsender only sends a ping once and that Firefox might attempt to send more than once, hence might have more than one ping delay in its list. Let’s see if these claims hold true.
# Number of duplicates, for each duped ping, from the pingsender. +print pingsender_dupes.map(lambda p: len(p[1][0])).countByValue() +# Number of duplicates, for each duped ping, from Firefox. +print pingsender_dupes.map(lambda p: len(p[1][1])).countByValue() +
defaultdict(<type 'int'>, {1: 2295})
+defaultdict(<type 'int'>, {1: 2243, 2: 38, 3: 10, 4: 3, 6: 1})
+Unlike the data from the Nightly channel, we see no proof that the pingsender can send the same ping ping more than once. On Nightly it had a very low occurrence, while on Beta it isn’t happening at all in the current data. We still can see some duplicates coming from Firefox, with the occurrence being a little higher.
Finally, compute the average delay between the duplicates from the pingsender and Firefox.
delay_between_duplicates =\ + pingsender_dupes.map(lambda t: np.fabs(np.max(t[1][1]) - np.min(t[1][0]))) +
setup_plot("'shutdown' duplicates submission delay CDF", MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +plot_cdf(delay_between_duplicates\ + .map(lambda d: d / HOUR_IN_S if d < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) +plt.axvline(x=4, ymin=0.0, ymax = 1.0, linewidth=1, linestyle='dashed', color='r') +plt.legend(["Duplicates delay", "4 hour filter limit"], loc="lower right") +
<matplotlib.legend.Legend at 0x7fda04742ad0> +
About ~55% of the duplicates can be caught by the pipeline ping deduplicator because they will arrive within a 4 hour window.
+collected_delays = delay_between_duplicates.collect() +
plt.title("The distribution of 'shutdown' ping delays for duplicate submissions") +plt.xlabel("Delay (seconds)") +plt.ylabel("Frequency") + +# Use 50 bins for values up to the clip value, and accumulate the +# rest in the last bucket (instead of having a super-long tail). +plt.hist(np.clip(collected_delays, 0, 48.0 * HOUR_IN_S), + alpha=0.5, bins=50, label="Delays") +# Plot some convenience marker for 4, 12 and 24 hours. +for m in [4.0, 12.0, 24.0]: + plt.axvline(x=m * HOUR_IN_S, ymin=0.0, ymax = 1.0, linewidth=1, linestyle='dashed', color='r', + label="{} hours".format(m)) +plt.legend() +
<matplotlib.legend.Legend at 0x7fda07e1f3d0> +
The shutdownDuration is defined as the time it takes to complete the Firefox shutdown process, in milliseconds. Extract the data from the two series: before the shutdown pingsender was enabled and after. Plot the data as two distinct distributions on the same plot.
pre_shutdown_durations = pre_subset.map(lambda p: p.get("payload/simpleMeasurements/shutdownDuration", None))\ + .filter(lambda p: p is not None)\ + .collect() +post_shutdown_durations = post_subset.map(lambda p: p.get("payload/simpleMeasurements/shutdownDuration", None))\ + .filter(lambda p: p is not None)\ + .collect() +
plt.title("'shutdown' pingsender effect on the shutdown duration") +plt.xlabel("shutdownDuration (milliseconds)") +plt.ylabel("Number of pings") + +# Use 50 bins for values up to the clip value, and accumulate the +# rest in the last bucket (instead of having a super-long tail). +CLIP_VALUE = 10000 # 10s +plt.hist([np.clip(pre_shutdown_durations, 0, CLIP_VALUE), np.clip(post_shutdown_durations, 0, CLIP_VALUE)], + alpha=0.5, bins=50, label=["No pingsender", "With pingsender"]) +plt.gca().set_yscale("log", nonposy='clip') +plt.legend() +
<matplotlib.legend.Legend at 0x7fda047687d0> +
It seems that the distribution of shutdown durations for builds with the pingsender enabled has a different shape compared to the distribution of shutdown durations for builds with no pingsender. The former seems to be a bit shifted toward higher values of the duration times. The same trend can be spotted on TMO.
+Let’s dig more into this by looking at some statistics about the durations.
+def stats(data, label): + print("\n{}\n".format(label)) + print("Min:\t{}".format(np.min(data))) + print("Max:\t{}".format(np.max(data))) + print("Average:\t{}".format(np.mean(data))) + print("50, 90 and 99 percentiles:\t{}\n".format(np.percentile(data, [0.5, 0.9, 0.99]))) + +stats(pre_shutdown_durations, "No pingsender (ms)") +stats(post_shutdown_durations, "With pingsender (ms)") +
No pingsender (ms) + +Min: 4 +Max: 3706649472 +Average: 10254.4014656 +50, 90 and 99 percentiles: [ 373. 399. 403.] + + +With pingsender (ms) + +Min: 12 +Max: 127666866 +Average: 5622.89186625 +50, 90 and 99 percentiles: [ 375. 406. 411.] +
It seems that builds that are sending shutdown pings at shutdown are taking up to about 8ms more to close.
+
This analysis is an update of the one performed in the Nightly channel to validate the effectiveness of the pingsender to reduce data latency.
+Specifically, this one investigates the difference between typical values of “recording delay” and “submission delay” between the previous Beta build and the latest one. The latter includes the pingSender that started sending “shutdown” pings.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Unable to parse whitelist: /mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json. +Assuming all histograms are acceptable. +
We’ll be looking at two cohorts: May 31 - June 7 (Beta 54) and June 14 - 20 (Beta 55). The pingsender started sending shudown pings in Beta 55.
We will examing two cohorts: the first with shutdown pings sent without the pingsender, the second with shutdown pings sent with the pingsender.
pre_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170531" <= x < "20170607") \ + .where(appBuildId=lambda x: "20170420" <= x < "20170611") \ + .records(sc, sample=1) + +post_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170614" <= x < "20170620") \ + .where(appBuildId=lambda x: "20170612" <= x < "20170622") \ + .records(sc, sample=1) +
fetching 1147169.41811MB in 39626 files... +fetching 20894.21218MB in 8408 files... +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+meta/creationTimestamp The time the Telemetry code in Firefox created the ping, according to the client’s clock, in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
pre_subset = get_pings_properties(pre_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) + +post_subset = get_pings_properties(post_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) +
The shutdown ping is a particular kind of main ping with the reason field set to shutdown, as it’s saved during shutdown.
pre_subset = pre_subset.filter(lambda p: p.get("payload/info/reason") == "shutdown") +post_subset = post_subset.filter(lambda p: p.get("payload/info/reason") == "shutdown") +
The rest of the analysis is cleaner if we combine the two cohorts here.
+def add_pre(p): + p['pre'] = 'pre' + return p + +def add_post(p): + p['pre'] = 'post' + return p + +combined = pre_subset.map(add_pre).union(post_subset.map(add_post)) +
Quick normalization: ditch any ping that doesn’t have a creationTimestamp or Timestamp:
+prev_count = combined.count() +combined = combined.filter(lambda p:\ + p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +filtered_count = combined.count() +print "Filtered {} of {} pings ({:.2f}%)"\ + .format(prev_count - filtered_count, prev_count, 100.0 * (prev_count - filtered_count) / prev_count) +
Filtered 0 of 34682752 pings (0.00%) +
We sometimes receive main pings more than once (identical document ids). This is usually low, but let’s check if this is still true after using the pingsender.
+So we’ll dedupe here.
+def dedupe(pings): + return pings\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) + +combined_deduped = dedupe(combined) +
combined_count = combined.count() +combined_deduped_count = combined_deduped.count() +print "Filtered {} of {} shutdown pings ({:.2f}%)"\ + .format(combined_count - combined_deduped_count, combined_count, + 100.0 * (combined_count - combined_deduped_count) / combined_count) +
Filtered 481634 of 34682752 shutdown pings (1.39%) +
This is slightly higher than our Nightly analysis, which reported 1.07%, but still in the expected range of about 1%.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +PRES = ['pre', 'post'] +
def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys) +
def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
delays_by_chan = combined_deduped.map(lambda p: (p["pre"], calculate_submission_delay(p))) +
Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+setup_plot("'shutdown' ping submission delay CDF", MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == pre)\ + .map(lambda d: d[1] / HOUR_IN_S if d[1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(["No pingsender", "With pingsender"], loc="lower right") +
<matplotlib.legend.Legend at 0x7fda065b6f10> +
The use of pingsender results in an improvement in the submission delay of the shutdown “main” ping:
shutdown pings within 8 hours, compared to ~95% received after over 90 hours without it. We don’t receive 100% of the pings sooner for builds having the pingsender enabled because the pingsender can fail submitting the ping (e.g. the system or Firefox uses a proxy, poor connection, …) and, when this happen, no retrasmission is attempted; the ping will be sent on the next restart by Firefox.
Let’s start by separating the pings coming from the pingsender from the ones coming from the normal Firefox flow since the pingsender started sending the shutdown pings.
post_pingsender_only = post_subset.filter(lambda p: p.get("meta/X-PingSender-Version") is not None) +post_no_pingsender = post_subset.filter(lambda p: p.get("meta/X-PingSender-Version") is None) +
num_from_pingsender = post_pingsender_only.count() +num_no_pingsender = post_no_pingsender.count() +total_post = post_subset.count() +num_sent_by_both =\ + post_pingsender_only.map(lambda p: p["id"]).intersection(post_no_pingsender.map(lambda p: p["id"])).count() +
We want to understand how many pings were sent by the pingsender, correctly received from the server, and sent again next time Firefox starts.
+def pct(a, b): + return 100 * float(a) / b + +print("Duplicate pings percentage: {:.2f}%".format(pct(num_sent_by_both, total_post))) +
Duplicate pings percentage: 0.41% +
Do we get many more duplicates after landing the shutdown pingsender?
count_deduped_pre = dedupe(pre_subset).count() +count_pre = pre_subset.count() +count_deduped_post = dedupe(post_subset).count() +count_post = post_subset.count() + +print("Duplicates with shutdown pingsender:\nBefore:\t{:.2f}%\nAfter:\t{:.2f}%\n"\ + .format(pct(count_pre - count_deduped_pre, count_pre), + pct(count_post - count_deduped_post, count_post))) +
Duplicates with shutdown pingsender: +Before: 1.40% +After: 0.85% +
It looks like 1% of the pings sent by the pingsender are also being sent by Firefox next time it restarts. This is potentially due to pingsender:
It’s important to note that the percentages of duplicate pings from the previous cells are not the same. The first, 0.41%, is the percentage of duplicates that were sent at least once by pingsender whereas the last, 0.85%, includes all duplicates regardless of whether pingsender was involved. This looks way better than the numbers in the Nightly channel possibly due to disabling the pingsender when the OS is shutting down, which happened in bug 1372202 (after the Nightly analysis was performed).
+Start off by getting the pings that were sent by both the pingsender and the normal Firefox flow. This is basically mimicking an intersectByKey, which is not available on pySpark.
pingsender_dupes = post_pingsender_only\ + .map(lambda p: (p["id"], calculate_submission_delay(p)))\ + .cogroup(post_no_pingsender\ + .map(lambda p: (p["id"], calculate_submission_delay(p))))\ + .filter(lambda p: p[1][0] and p[1][1])\ + .map(lambda p: (p[0], (list(p[1][0]), list(p[1][1])))) +
The pingsender_dupes RDD should now contain only the data for the pings sent by both systems. Each entry is in the form:
{ping-id: ([ delays for duplicates from the pingsender ], [delays for duplicates by FF])}
We assume that the pingsender only sends a ping once and that Firefox might attempt to send more than once, hence might have more than one ping delay in its list. Let’s see if these claims hold true.
# Number of duplicates, for each duped ping, from the pingsender. +print pingsender_dupes.map(lambda p: len(p[1][0])).countByValue() +# Number of duplicates, for each duped ping, from Firefox. +print pingsender_dupes.map(lambda p: len(p[1][1])).countByValue() +
defaultdict(<type 'int'>, {1: 2295})
+defaultdict(<type 'int'>, {1: 2243, 2: 38, 3: 10, 4: 3, 6: 1})
+Unlike the data from the Nightly channel, we see no proof that the pingsender can send the same ping ping more than once. On Nightly it had a very low occurrence, while on Beta it isn’t happening at all in the current data. We still can see some duplicates coming from Firefox, with the occurrence being a little higher.
Finally, compute the average delay between the duplicates from the pingsender and Firefox.
delay_between_duplicates =\ + pingsender_dupes.map(lambda t: np.fabs(np.max(t[1][1]) - np.min(t[1][0]))) +
setup_plot("'shutdown' duplicates submission delay CDF", MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +plot_cdf(delay_between_duplicates\ + .map(lambda d: d / HOUR_IN_S if d < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) +plt.axvline(x=4, ymin=0.0, ymax = 1.0, linewidth=1, linestyle='dashed', color='r') +plt.legend(["Duplicates delay", "4 hour filter limit"], loc="lower right") +
<matplotlib.legend.Legend at 0x7fda04742ad0> +
About ~55% of the duplicates can be caught by the pipeline ping deduplicator because they will arrive within a 4 hour window.
+collected_delays = delay_between_duplicates.collect() +
plt.title("The distribution of 'shutdown' ping delays for duplicate submissions") +plt.xlabel("Delay (seconds)") +plt.ylabel("Frequency") + +# Use 50 bins for values up to the clip value, and accumulate the +# rest in the last bucket (instead of having a super-long tail). +plt.hist(np.clip(collected_delays, 0, 48.0 * HOUR_IN_S), + alpha=0.5, bins=50, label="Delays") +# Plot some convenience marker for 4, 12 and 24 hours. +for m in [4.0, 12.0, 24.0]: + plt.axvline(x=m * HOUR_IN_S, ymin=0.0, ymax = 1.0, linewidth=1, linestyle='dashed', color='r', + label="{} hours".format(m)) +plt.legend() +
<matplotlib.legend.Legend at 0x7fda07e1f3d0> +
The shutdownDuration is defined as the time it takes to complete the Firefox shutdown process, in milliseconds. Extract the data from the two series: before the shutdown pingsender was enabled and after. Plot the data as two distinct distributions on the same plot.
pre_shutdown_durations = pre_subset.map(lambda p: p.get("payload/simpleMeasurements/shutdownDuration", None))\ + .filter(lambda p: p is not None)\ + .collect() +post_shutdown_durations = post_subset.map(lambda p: p.get("payload/simpleMeasurements/shutdownDuration", None))\ + .filter(lambda p: p is not None)\ + .collect() +
plt.title("'shutdown' pingsender effect on the shutdown duration") +plt.xlabel("shutdownDuration (milliseconds)") +plt.ylabel("Number of pings") + +# Use 50 bins for values up to the clip value, and accumulate the +# rest in the last bucket (instead of having a super-long tail). +CLIP_VALUE = 10000 # 10s +plt.hist([np.clip(pre_shutdown_durations, 0, CLIP_VALUE), np.clip(post_shutdown_durations, 0, CLIP_VALUE)], + alpha=0.5, bins=50, label=["No pingsender", "With pingsender"]) +plt.gca().set_yscale("log", nonposy='clip') +plt.legend() +
<matplotlib.legend.Legend at 0x7fda047687d0> +
It seems that the distribution of shutdown durations for builds with the pingsender enabled has a different shape compared to the distribution of shutdown durations for builds with no pingsender. The former seems to be a bit shifted toward higher values of the duration times. The same trend can be spotted on TMO.
+Let’s dig more into this by looking at some statistics about the durations.
+def stats(data, label): + print("\n{}\n".format(label)) + print("Min:\t{}".format(np.min(data))) + print("Max:\t{}".format(np.max(data))) + print("Average:\t{}".format(np.mean(data))) + print("50, 90 and 99 percentiles:\t{}\n".format(np.percentile(data, [0.5, 0.9, 0.99]))) + +stats(pre_shutdown_durations, "No pingsender (ms)") +stats(post_shutdown_durations, "With pingsender (ms)") +
No pingsender (ms) + +Min: 4 +Max: 3706649472 +Average: 10254.4014656 +50, 90 and 99 percentiles: [ 373. 399. 403.] + + +With pingsender (ms) + +Min: 12 +Max: 127666866 +Average: 5622.89186625 +50, 90 and 99 percentiles: [ 375. 406. 411.] +
It seems that builds that are sending shutdown pings at shutdown are taking up to about 8ms more to close.
+
In this notebook we extend the shutdown main-ping submission delay analysis done here, for Firefox Beta, to all the main-ping types regardless of their reason (i.e. shutdown, daily, …).
Specifically, this one investigates the difference between typical values “submission delay” between the previous Beta build and the latest one. The latter includes the pingSender that started sending “shutdown” pings.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
We’ll be looking at two cohorts: May 31 - June 7 (Beta 54, no pingsender) and June 14 - 20 (Beta 55, with the pingsender sending shutdown pings). The pingsender started sending shudown pings in Beta 55.
pre_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170531" <= x < "20170607") \ + .where(appBuildId=lambda x: "20170420" <= x < "20170611") \ + .records(sc, sample=1) + +post_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170614" <= x < "20170620") \ + .where(appBuildId=lambda x: "20170612" <= x < "20170622") \ + .records(sc, sample=1) +
fetching 1147169.41811MB in 39626 files... +fetching 20894.21218MB in 8408 files... +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+meta/creationTimestamp The time the Telemetry code in Firefox created the ping, according to the client’s clock, in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
pre_subset = get_pings_properties(pre_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) + +post_subset = get_pings_properties(post_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) +
The rest of the analysis is cleaner if we combine the two cohorts here.
+def add_pre(p): + p['pre'] = 'pre' + return p + +def add_post(p): + p['pre'] = 'post' + return p + +combined = pre_subset.map(add_pre).union(post_subset.map(add_post)) +
Quick normalization: ditch any ping that doesn’t have a creationTimestamp or Timestamp:
+prev_count = combined.count() +combined = combined.filter(lambda p:\ + p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +filtered_count = combined.count() +print "Filtered {} of {} pings ({:.2f}%)"\ + .format(prev_count - filtered_count, prev_count, 100.0 * (prev_count - filtered_count) / prev_count) +
Filtered 0 of 42046459 pings (0.00%) +
We sometimes receive main pings more than once (identical document ids). This is usually low, but let’s check if this is still true after using the pingsender.
+So we’ll dedupe here.
+def dedupe(pings): + return pings\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) + +combined_deduped = dedupe(combined) +
combined_count = combined.count() +combined_deduped_count = combined_deduped.count() +print "Filtered {} of {} main pings ({:.2f}%)"\ + .format(combined_count - combined_deduped_count, combined_count, + 100.0 * (combined_count - combined_deduped_count) / combined_count) +
Filtered 538004 of 42046459 main pings (1.28%) +
The previous 1.28% is the duplicate rate over all the main pings.
MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +PRES = ['pre', 'post'] +MAIN_PING_REASONS = [ + 'aborted-session', 'environment-change', 'shutdown', 'daily', 'environment-change' +] +
def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data, **kwargs): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys, **kwargs) +
def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
delays_by_chan = combined_deduped.map(lambda p: ((p["pre"], p["payload/info/reason"]), calculate_submission_delay(p))) +
Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+The following block of code plots the CDF of the submission delay for each reason of the main-ping (even though the pingsender is only used for the shutdown reason).
for reason in MAIN_PING_REASONS: + setup_plot("'main-ping' ({}) ping submission delay CDF".format(reason), + MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + + for pre in PRES: + # Build an useful label. + using_pingsender = pre != 'pre' + label = "'{}'{}".format(reason, ", with pingsender" if using_pingsender else "") + + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0][0] == pre and d[0][1] == reason)\ + .map(lambda d: d[1] / HOUR_IN_S if d[1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect(), label=label, linestyle="solid" if using_pingsender else "dashed") + + plt.legend(loc="lower right") + plt.show() +
Interestingly enough, it looks like enabling the pingsender on the shutdown main-ping allowed the latency to decrease for other ping types too. One possible reason for this is that the Telemetry sending queue has fewer pings to deal with and can deal with other types more efficiently.
Let’s plot the overall latency, below.
+setup_plot("'main-ping' (all reasons) ping submission delay CDF", + MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0][0] == pre)\ + .map(lambda d: d[1] / HOUR_IN_S if d[1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(["No pingsender", "With pingsender"], loc="lower right") +
<matplotlib.legend.Legend at 0x7fc0b4161610> +
The use of pingsender results in an improvement in the submission delay of the main-ping:
main-ping within 8 hours, compared to ~95% received after over 90 hours without it. We don’t receive 100% of the pings sooner for builds having the pingsender enabled because the pingsender can fail submitting the ping (e.g. the system or Firefox uses a proxy, poor connection, …) and, when this happen, no retrasmission is attempted; the ping will be sent on the next restart by Firefox.
+
In this notebook we extend the shutdown main-ping submission delay analysis done here, for Firefox Beta, to all the main-ping types regardless of their reason (i.e. shutdown, daily, …).
Specifically, this one investigates the difference between typical values “submission delay” between the previous Beta build and the latest one. The latter includes the pingSender that started sending “shutdown” pings.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
We’ll be looking at two cohorts: May 31 - June 7 (Beta 54, no pingsender) and June 14 - 20 (Beta 55, with the pingsender sending shutdown pings). The pingsender started sending shudown pings in Beta 55.
pre_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170531" <= x < "20170607") \ + .where(appBuildId=lambda x: "20170420" <= x < "20170611") \ + .records(sc, sample=1) + +post_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170614" <= x < "20170620") \ + .where(appBuildId=lambda x: "20170612" <= x < "20170622") \ + .records(sc, sample=1) +
fetching 1147169.41811MB in 39626 files... +fetching 20894.21218MB in 8408 files... +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+meta/creationTimestamp The time the Telemetry code in Firefox created the ping, according to the client’s clock, in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
pre_subset = get_pings_properties(pre_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) + +post_subset = get_pings_properties(post_pings, ["application/channel", + "id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "payload/info/reason", + "payload/simpleMeasurements/shutdownDuration"]) +
The rest of the analysis is cleaner if we combine the two cohorts here.
+def add_pre(p): + p['pre'] = 'pre' + return p + +def add_post(p): + p['pre'] = 'post' + return p + +combined = pre_subset.map(add_pre).union(post_subset.map(add_post)) +
Quick normalization: ditch any ping that doesn’t have a creationTimestamp or Timestamp:
+prev_count = combined.count() +combined = combined.filter(lambda p:\ + p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +filtered_count = combined.count() +print "Filtered {} of {} pings ({:.2f}%)"\ + .format(prev_count - filtered_count, prev_count, 100.0 * (prev_count - filtered_count) / prev_count) +
Filtered 0 of 42046459 pings (0.00%) +
We sometimes receive main pings more than once (identical document ids). This is usually low, but let’s check if this is still true after using the pingsender.
+So we’ll dedupe here.
+def dedupe(pings): + return pings\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) + +combined_deduped = dedupe(combined) +
combined_count = combined.count() +combined_deduped_count = combined_deduped.count() +print "Filtered {} of {} main pings ({:.2f}%)"\ + .format(combined_count - combined_deduped_count, combined_count, + 100.0 * (combined_count - combined_deduped_count) / combined_count) +
Filtered 538004 of 42046459 main pings (1.28%) +
The previous 1.28% is the duplicate rate over all the main pings.
MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +PRES = ['pre', 'post'] +MAIN_PING_REASONS = [ + 'aborted-session', 'environment-change', 'shutdown', 'daily', 'environment-change' +] +
def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data, **kwargs): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys, **kwargs) +
def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
delays_by_chan = combined_deduped.map(lambda p: ((p["pre"], p["payload/info/reason"]), calculate_submission_delay(p))) +
Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+The following block of code plots the CDF of the submission delay for each reason of the main-ping (even though the pingsender is only used for the shutdown reason).
for reason in MAIN_PING_REASONS: + setup_plot("'main-ping' ({}) ping submission delay CDF".format(reason), + MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + + for pre in PRES: + # Build an useful label. + using_pingsender = pre != 'pre' + label = "'{}'{}".format(reason, ", with pingsender" if using_pingsender else "") + + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0][0] == pre and d[0][1] == reason)\ + .map(lambda d: d[1] / HOUR_IN_S if d[1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect(), label=label, linestyle="solid" if using_pingsender else "dashed") + + plt.legend(loc="lower right") + plt.show() +
Interestingly enough, it looks like enabling the pingsender on the shutdown main-ping allowed the latency to decrease for other ping types too. One possible reason for this is that the Telemetry sending queue has fewer pings to deal with and can deal with other types more efficiently.
Let’s plot the overall latency, below.
+setup_plot("'main-ping' (all reasons) ping submission delay CDF", + MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +for pre in PRES: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0][0] == pre)\ + .map(lambda d: d[1] / HOUR_IN_S if d[1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(["No pingsender", "With pingsender"], loc="lower right") +
<matplotlib.legend.Legend at 0x7fc0b4161610> +
The use of pingsender results in an improvement in the submission delay of the main-ping:
main-ping within 8 hours, compared to ~95% received after over 90 hours without it. We don’t receive 100% of the pings sooner for builds having the pingsender enabled because the pingsender can fail submitting the ping (e.g. the system or Firefox uses a proxy, poor connection, …) and, when this happen, no retrasmission is attempted; the ping will be sent on the next restart by Firefox.
+
new-profile submissions on BetaThis analysis validates the new-profile pings submitted by Beta builds for one week since it hit that channel. We are going to verify that:
new-profile ping is received within a reasonable time after the profile creation;import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from datetime import datetime, timedelta + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
We’ll be looking at the pings that have been coming in since June 14th to June 20th 2017 (Beta 55).
+# Note that the 'new-profile' ping needs to use underscores in the Dataset API due to bug. +pings = Dataset.from_source("telemetry") \ + .where(docType='new_profile') \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170614" <= x < "20170620") \ + .where(appBuildId=lambda x: "20170612" <= x < "20170622") \ + .records(sc, sample=1.0) +
fetching 12.91282MB in 1408 files... +
ping_count = pings.count() +
The new-profile ping can be sent either during the browsing session, 30 minutes after the browser starts, or at shutdown (docs). Let’s see how many pings we get in each case.
raw_subset = get_pings_properties(pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/reason"]) +
Discard and count any ping that’s missing creationTimestamp or Timestamp.
+def pct(a, b): + return 100.0 * a / b + +subset = raw_subset.filter(lambda p: p["meta/creationTimestamp"] is not None and p["meta/Timestamp"] is not None) +print("'new-profile' pings with missing timestamps:\t{:.2f}%".format(pct(ping_count - subset.count(), ping_count))) +
'new-profile' pings with missing timestamps: 0.00% +
reason_counts = subset.map(lambda p: p.get("payload/reason")).countByValue() + +for reason, count in reason_counts.iteritems(): + print("'new-profile' pings with reason '{}':\t{:.2f}%".format(reason, pct(count, ping_count))) +
'new-profile' pings with reason 'startup': 21.87% +'new-profile' pings with reason 'shutdown': 78.13% +
This means that, among all the new-profile pings, the majority was sent at shutdown. This could mean different things:
def dedupe(pings, duping_key): + return pings\ + .map(lambda p: (p[duping_key], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) + +deduped_docid = dedupe(subset, "id") +deduped_docid_count = deduped_docid.count() +total_duplicates = ping_count - deduped_docid_count +print("Duplicate pings percentage (by document id): {:.2f}%".format(pct(total_duplicates, ping_count))) +
Duplicate pings percentage (by document id): 0.43% +
The 0.43% of ping duplicates is nice, compared to ~1% we usually get from the main and crash pings. However, nowdays we’re running de-duplication by document id at the pipeline ingestion, so this might be a bit higher. To check that, we have a telemetry_duplicates_parquet table and this handy query that says 4 duplicates were filtered on the pipeline. This means that our 0.43% is basically the real duplicate rate for the new-profile ping on Beta.
Did we send different pings for the same client id? We shouldn’t, as we send at most one ‘new-profile’ ping per client.
+deduped_clientid = dedupe(deduped_docid, "clientId") +total_duplicates_clientid = deduped_docid_count - deduped_clientid.count() +print("Duplicate pings percentage (by client id): {:.2f}%".format(pct(total_duplicates_clientid, deduped_docid_count))) +
Duplicate pings percentage (by client id): 0.81% +
That’s disappointing: it looks like we’re receiving multiple new-profile pings for some clients. Let’s dig into this by analysing the set of pings deduped by document id. To have a clearer picture of the problem, let’s make sure to aggregate the duplicates ordered by the time they were created on the client.
# Builds an RDD with (<client id>, [<ordered reason>, <ordered reason>, ...]) +clients_with_dupes = deduped_docid.map(lambda p: (p["clientId"], [(p["payload/reason"], p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) + +# Check how often each case occurs. Hide the counts. +[k for k, v in\ + sorted(clients_with_dupes.map(lambda p: tuple(p[1])).countByValue().items(), key=lambda k: k[1], reverse=True)] +
[(u'shutdown', u'shutdown'), + (u'shutdown', u'startup'), + (u'startup', u'startup'), + (u'startup', u'shutdown'), + (u'shutdown', u'shutdown', u'shutdown'), + (u'shutdown', u'startup', u'startup'), + (u'shutdown', u'startup', u'shutdown'), + (u'shutdown', + u'startup', + u'shutdown', + u'shutdown', + u'shutdown', + u'startup', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'startup'), + (u'startup', u'shutdown', u'shutdown', u'shutdown', u'shutdown', u'shutdown'), + (u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'startup', + u'shutdown', + u'shutdown'), + (u'shutdown', u'shutdown', u'shutdown', u'shutdown', u'shutdown')] +
collected_clientid_reasons = clients_with_dupes.collect() +
num_shutdown_dupes = sum([len(filter(lambda e: e == 'shutdown', t[1])) for t in collected_clientid_reasons]) +print("Duplicate 'shutdown' pings percentage (by client id): {:.2f}%"\ + .format(pct(num_shutdown_dupes, ping_count))) +
Duplicate 'shutdown' pings percentage (by client id): 1.10% +
The multiple pings we’re receiving for the same client id are mostly new-profile pings with reason shutdown. This is not too surprising, as most of the new-profile pings are being sent at shutdown (78%).
But does the pingsender have anything to do with this? Let’s attack the problem like this:
+misbehaving_clients = list(set([client_id for client_id, data in collected_clientid_reasons])) +
count_reason_pingsender = subset\ + .filter(lambda p: p.get("clientId") in misbehaving_clients)\ + .map(lambda p: (p.get("payload/reason"), p.get("meta/X-PingSender-Version")))\ + .countByValue() + +for reason, count in count_reason_pingsender.items(): + print("{}:\t{}".format(reason, pct(count, sum(count_reason_pingsender.values())))) +
(u'startup', None): 23.8532110092 +(u'shutdown', u'1.0'): 56.2691131498 +(u'shutdown', None): 19.877675841 +
Looks like some of these pings are missing the pingsender header from the request. While this is expected for new-profile pings with reason startup, as they are being sent while Firefox is still active, there might be reasons why this also happens at shutdown:
This leads to other questions:
+newprofile_shutdown_from_bad_clients =\ + subset.filter(lambda p: p.get("payload/reason") == 'shutdown')\ + .filter(lambda p: p.get("clientId") in misbehaving_clients) + +newprofile_shutdown_from_good_clients =\ + subset.filter(lambda p: p.get("payload/reason") == 'shutdown')\ + .filter(lambda p: p.get("clientId") not in misbehaving_clients) +
dict_dump = newprofile_shutdown_from_bad_clients\ + .map(lambda p: p.get("meta/X-PingSender-Version")).countByValue() +# Just print the percentages. +print("Pingsender header breakdown for misbehaving clients:") +den = sum(dict_dump.values()) +for k, v in dict_dump.items(): + print("{}:\t{}".format(k, pct(v, den))) +
Pingsender header breakdown for misbehaving clients: +1.0: 73.8955823293 +None: 26.1044176707 +
This is telling us that most of the shutdown new-profile pings are coming from the pingsender, about 73% (the 1.0 header represents the pingsender).
dict_dump = newprofile_shutdown_from_good_clients\ + .map(lambda p: p.get("meta/X-PingSender-Version")).countByValue() +# Just print the percentages. +print("Pingsender header breakdown for well behaving clients:") +den = sum(dict_dump.values()) +for k, v in dict_dump.items(): + print("{}:\t{}".format(k, pct(v, den))) +
Pingsender header breakdown for well behaving clients: +1.0: 69.3411573321 +None: 30.6588426679 +
This is somehow true with well-behaved clients, as 69% of the same pings are coming with the pingsender. The pingsender doesn’t seem to be the issue here: if we generate a ping at shutdown and try to send it with the pingsender, and fail, then it’s normal for Firefox to pick it back up and send it. As long as we don’t generate a new, different, new-profile ping for the same client.
def datetime_from_daysepoch(days_from_epoch): + return datetime(1970, 1, 1, 0, 0) + timedelta(days=days_from_epoch) + +def datetime_from_nanosepoch(nanos_from_epoch): + return datetime.fromtimestamp(nanos_from_epoch / 1000.0 / 1000.0 / 1000.0) + +def get_times(p): + profile_creation = datetime_from_daysepoch(p["environment/profile/creationDate"])\ + if p["environment/profile/creationDate"] else None + ping_creation = datetime_from_nanosepoch(p["meta/creationTimestamp"]) + ping_recv = datetime_from_nanosepoch(p["meta/Timestamp"]) + + return (p["id"], profile_creation, ping_creation, ping_recv) + +ping_times = deduped_clientid.map(get_times) +
ping_creation_delay_days = ping_times.filter(lambda p: p[1] is not None)\ + .map(lambda p: abs((p[1].date() - p[2].date()).days)).collect() +
plt.title("The distribution of the days between the profile creationDate and the 'new-profile' ping creation date") +plt.xlabel("Difference in days") +plt.ylabel("Frequency") + +CLIP_DAY = 30 +plt.xticks(range(0, CLIP_DAY + 1, 1)) +plt.hist(np.clip(ping_creation_delay_days, 0, CLIP_DAY), + alpha=0.5, bins=50, label="Delays") +
(array([ 1.94340000e+04, 3.80000000e+02, 0.00000000e+00, + 1.13000000e+02, 0.00000000e+00, 5.60000000e+01, + 5.10000000e+01, 0.00000000e+00, 2.30000000e+01, + 0.00000000e+00, 2.10000000e+01, 1.80000000e+01, + 0.00000000e+00, 2.00000000e+01, 0.00000000e+00, + 1.50000000e+01, 8.00000000e+00, 0.00000000e+00, + 1.50000000e+01, 0.00000000e+00, 1.40000000e+01, + 2.10000000e+01, 0.00000000e+00, 1.60000000e+01, + 0.00000000e+00, 1.10000000e+01, 1.00000000e+01, + 0.00000000e+00, 1.70000000e+01, 0.00000000e+00, + 2.40000000e+01, 9.00000000e+00, 0.00000000e+00, + 3.00000000e+00, 0.00000000e+00, 9.00000000e+00, + 5.00000000e+00, 0.00000000e+00, 5.00000000e+00, + 0.00000000e+00, 5.00000000e+00, 5.00000000e+00, + 0.00000000e+00, 4.00000000e+00, 0.00000000e+00, + 8.00000000e+00, 6.00000000e+00, 0.00000000e+00, + 4.00000000e+00, 1.73800000e+03]), + array([ 0. , 0.6, 1.2, 1.8, 2.4, 3. , 3.6, 4.2, 4.8, + 5.4, 6. , 6.6, 7.2, 7.8, 8.4, 9. , 9.6, 10.2, + 10.8, 11.4, 12. , 12.6, 13.2, 13.8, 14.4, 15. , 15.6, + 16.2, 16.8, 17.4, 18. , 18.6, 19.2, 19.8, 20.4, 21. , + 21.6, 22.2, 22.8, 23.4, 24. , 24.6, 25.2, 25.8, 26.4, + 27. , 27.6, 28.2, 28.8, 29.4, 30. ]), + <a list of 50 Patch objects>) +
np.percentile(np.array(ping_creation_delay_days), [50, 70, 80, 95, 99]) +
array([ 0. , 0. , 0. , 274. , 1836.6]) +
The plot shows that most of the creation dates for new-profile pings match exactly with the date reported in the environment, creationDate. That’s good, as this ping should be created very close to the profile creation. The percentile computation confirms that’s true for 80% of the new-profile pings.
new-profile and main pingsmain_pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170614" <= x < "20170620") \ + .where(appBuildId=lambda x: "20170612" <= x < "20170622") \ + .records(sc, sample=1.0) +
fetching 20894.21218MB in 8408 files... +
main_subset = get_pings_properties(main_pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/info/reason", + "payload/info/sessionLength", + "payload/info/subsessionLength", + "payload/info/profileSubsessionCounter", + "payload/info/previousSessionId"]) +
Dedupe by document id and restrict the main ping data to the pings from the misbehaving and well behaving clients.
well_behaving_clients =\ + set(subset.filter(lambda p: p.get("clientId") not in misbehaving_clients).map(lambda p: p.get("clientId")).collect()) + +all_clients = misbehaving_clients + list(well_behaving_clients) +
main_deduped = dedupe(main_subset.filter(lambda p: p.get("clientId") in all_clients), "id") +main_deduped_count = main_deduped.count() +
Try to pair each new-profile ping with reason shutdown to the very first main ping with reason shutdown received from that client, to make sure that the former were sent at the right time.
first_main = main_deduped.filter(lambda p:\ + p.get("payload/info/previousSessionId") == None and\ + p.get("payload/info/reason") == "shutdown") +
newping_shutdown = deduped_docid.filter(lambda p: p.get("payload/reason") == "shutdown") +
newprofile_plus_main = first_main.union(newping_shutdown) +sorted_per_client = newprofile_plus_main.map(lambda p: (p["clientId"], [(p, p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) +num_analysed_clients = sorted_per_client.count() +
HALF_HOUR_IN_S = 30 * 60 + +def is_newprofile(p): + # The 'main' ping has the reason field in 'payload/info/reason' + return "payload/reason" in p and p.get("payload/reason") in ["startup", "shutdown"] + +def validate_newprofile_shutdown(client_data): + ordered_pings = client_data[1] + + newprofile_mask = [is_newprofile(p) for p in ordered_pings] + + # Do we have at least a 'new-profile' ping? + num_newprofile_pings = sum(newprofile_mask) + if num_newprofile_pings < 1: + return ("No shutdown 'new-profile' ping found", 1) + + # Do we have multiple 'new-profile' pings? + if num_newprofile_pings > 1: + return ("Duplicate 'new-profile' ping.", 1) + + if not newprofile_mask[0]: + return ("The 'new-profile' ping is not the first ping", 1) + + # If there's a new-profile ping with reason 'shutdown', look for the closest next + # 'main' ping with reason shutdown. + for i, p in enumerate(ordered_pings): + # Skip until we find the 'new-profile' ping. + if not is_newprofile(p): + continue + + # We found the 'new-profile' ping. Do we have any other ping + # after this? + next_index = i + 1 + if next_index >= len(ordered_pings): + return ("No more pings after the 'new-profile'", 1) + + # Did we schedule the 'new-profile' ping at the right moment? + next_ping = ordered_pings[next_index] + if next_ping.get("payload/info/sessionLength") <= HALF_HOUR_IN_S: + return ("The 'new-profile' ping was correctly scheduled", 1) + + return ("The 'new-profile' ping was scheduled at the wrong time", 1) + + return ("Unknown condition", 1) + +scheduling_error_counts = sorted_per_client.map(validate_newprofile_shutdown).countByKey() +
for error, count in scheduling_error_counts.items(): + print("{}:\t{}".format(error, pct(count, num_analysed_clients))) +
No shutdown 'new-profile' ping found: 0.037503750375 +The 'new-profile' ping was correctly scheduled: 98.6198619862 +The 'new-profile' ping was scheduled at the wrong time: 0.630063006301 +Duplicate 'new-profile' ping.: 0.652565256526 +The 'new-profile' ping is not the first ping: 0.0600060006001 +
Most of the new-profile pings sent at shutdown, 98.61%, were correctly generated because the session lasted less than 30 minutes. Only 0.63% were scheduled at the wrong time. The rest of the clients either sent the new-profile at startup or we’re still waiting for their main ping with reason shutdown.
new-profile/startup pings only from sessions > 30 minutes?newping_startup = deduped_docid.filter(lambda p: p.get("payload/reason") == "startup") +newprofile_start_main = first_main.union(newping_startup) +sorted_per_client = newprofile_start_main.map(lambda p: (p["clientId"], [(p, p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) +num_analysed_clients = sorted_per_client.count() +
def validate_newprofile_startup(client_data): + ordered_pings = client_data[1] + + newprofile_mask = [is_newprofile(p) for p in ordered_pings] + + # Do we have at least a 'new-profile' ping? + num_newprofile_pings = sum(newprofile_mask) + if num_newprofile_pings < 1: + return ("No startup 'new-profile' ping found", 1) + + # Do we have multiple 'new-profile' pings? + if num_newprofile_pings > 1: + return ("Duplicate 'new-profile' ping", 1) + + if not newprofile_mask[0]: + return ("The 'new-profile' ping it's not the first ping", 1) + + # If there's a new-profile ping with reason 'startup', look for the closest next + # 'main' ping with reason shutdown. + for i, p in enumerate(ordered_pings): + # Skip until we find the 'new-profile' ping. + if not is_newprofile(p): + continue + + # We found the 'new-profile' ping. Do we have any other ping + # after this? + next_index = i + 1 + if next_index >= len(ordered_pings): + return ("No more pings after the 'new-profile'", 1) + + # Did we schedule the 'new-profile' ping at the right moment? + next_ping = ordered_pings[next_index] + if next_ping.get("payload/info/sessionLength") > HALF_HOUR_IN_S: + return ("The 'new-profile' ping was correctly scheduled", 1) + + return ("The 'new-profile' ping was scheduled at the wrong time", 1) + + return ("Unknown condition", 1) + +startup_newprofile_errors = sorted_per_client.map(validate_newprofile_startup).countByKey() +for error, count in startup_newprofile_errors.items(): + print("{}:\t{}".format(error, pct(count, num_analysed_clients))) +
The 'new-profile' ping it's not the first ping: 1.41059855128 +The 'new-profile' ping was correctly scheduled: 95.50133435 +Duplicate 'new-profile' ping: 0.609988562714 +The 'new-profile' ping was scheduled at the wrong time: 0.228745711018 +No startup 'new-profile' ping found: 2.24933282501 +
The results look good and in line with the previous case of the new-profile ping being sent at shutdown. The number of times the new-profile ping isn’t the first generated ping is slightly higher (0.06% vs 1.41%), but this can be explained by the fact that nothing prevents Firefox from sending new pings after Telemetry starts up (60s into the Firefox startup some addon is installed), while the new-profile ping is strictly scheduled 30 minutes after the startup.
crash ping from bad-behaved clients?If that happened close to when we generated a new-profile ping, it could hint at some correlation between crashes and the duplicates per client id.
crash_pings = Dataset.from_source("telemetry") \ + .where(docType='crash') \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170614" <= x < "20170620") \ + .where(appBuildId=lambda x: "20170612" <= x < "20170622") \ + .records(sc, sample=1.0) +
fetching 126.79929MB in 1764 files... +
Restrict the crashes to a set of useful fields, just for the misbehaving clients, and dedupe them by document id.
+crash_subset = get_pings_properties(crash_pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/crashDate", + "payload/crashTime", + "payload/processType", + "payload/sessionId"]) +crashes_misbehaving_clients = dedupe(crash_subset.filter(lambda p:\ + p.get("clientId") in misbehaving_clients and\ + p.get("payload/processType") == 'main'), "id") +newprofile_bad_clients = subset.filter(lambda p: p.get("clientId") in misbehaving_clients) +
Let’s also check how many clients are reporting crashes compared to the number of misbehaving ones.
+from operator import add +clients_with_crashes =\ + crashes_misbehaving_clients.map(lambda p: (p.get('clientId'), 1)).reduceByKey(add).map(lambda p: p[0]).collect() +
print("Percentages of bad clients with crash pings:\t{}".format(pct(len(clients_with_crashes), len(misbehaving_clients)))) +
Percentages of bad clients with crash pings: 13.8888888889 +
def get_ping_type(p): + return "crash" if "payload/crashDate" in p else "new-profile" + +newprofile_and_crashes = crashes_misbehaving_clients.union(newprofile_bad_clients) + +# Builds an RDD with (<client id>, [<ordered reason>, <ordered reason>, ...]) +joint_ordered_pings = newprofile_and_crashes\ + .map(lambda p: (p["clientId"], [(get_ping_type(p), p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) + +# Just show the pings, the most occurring first. Hide the counts. +[k for k, v in\ + sorted(joint_ordered_pings.map(lambda p: tuple(p[1])).countByValue().items(), key=lambda k: k[1], reverse=True)] +
[('new-profile', 'new-profile'),
+ ('new-profile', 'new-profile', 'new-profile'),
+ ('new-profile', 'crash', 'new-profile'),
+ ('new-profile', 'new-profile', 'crash'),
+ ('crash', 'new-profile', 'new-profile'),
+ ('new-profile', 'crash', 'crash', 'crash', 'crash', 'crash', 'new-profile'),
+ ('new-profile', 'crash', 'crash', 'crash', 'crash', 'new-profile'),
+ ('new-profile',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'new-profile',
+ 'crash',
+ 'crash',
+ 'crash'),
+ ('crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'new-profile',
+ 'crash',
+ 'new-profile'),
+ ('new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile'),
+ ('new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile'),
+ ('new-profile',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'new-profile',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash'),
+ ('new-profile', 'new-profile', 'new-profile', 'crash'),
+ ('new-profile', 'new-profile', 'new-profile', 'new-profile', 'new-profile'),
+ ('crash', 'new-profile', 'new-profile', 'crash'),
+ ('crash', 'new-profile', 'crash', 'new-profile', 'crash'),
+ ('new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile')]
+The first groups of reported ping sequences, don’t contain any crash ping and account for most of the new-profile duplicates pattern. The other sequences interleave new-profile and main process crash pings, suggesting that crashes might play a role in per-client duplicates. However, we only have crashes for 13% of the clients that do not behave correctly: this probably means that there is a weak correlation between crashes and getting multiple new-profile pings, but this is not the main problem. There’s some potential bug lurking around in the client code.
+
new-profile submissions on BetaThis analysis validates the new-profile pings submitted by Beta builds for one week since it hit that channel. We are going to verify that:
new-profile ping is received within a reasonable time after the profile creation;import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from datetime import datetime, timedelta + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
We’ll be looking at the pings that have been coming in since June 14th to June 20th 2017 (Beta 55).
+# Note that the 'new-profile' ping needs to use underscores in the Dataset API due to bug. +pings = Dataset.from_source("telemetry") \ + .where(docType='new_profile') \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170614" <= x < "20170620") \ + .where(appBuildId=lambda x: "20170612" <= x < "20170622") \ + .records(sc, sample=1.0) +
fetching 12.91282MB in 1408 files... +
ping_count = pings.count() +
The new-profile ping can be sent either during the browsing session, 30 minutes after the browser starts, or at shutdown (docs). Let’s see how many pings we get in each case.
raw_subset = get_pings_properties(pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/reason"]) +
Discard and count any ping that’s missing creationTimestamp or Timestamp.
+def pct(a, b): + return 100.0 * a / b + +subset = raw_subset.filter(lambda p: p["meta/creationTimestamp"] is not None and p["meta/Timestamp"] is not None) +print("'new-profile' pings with missing timestamps:\t{:.2f}%".format(pct(ping_count - subset.count(), ping_count))) +
'new-profile' pings with missing timestamps: 0.00% +
reason_counts = subset.map(lambda p: p.get("payload/reason")).countByValue() + +for reason, count in reason_counts.iteritems(): + print("'new-profile' pings with reason '{}':\t{:.2f}%".format(reason, pct(count, ping_count))) +
'new-profile' pings with reason 'startup': 21.87% +'new-profile' pings with reason 'shutdown': 78.13% +
This means that, among all the new-profile pings, the majority was sent at shutdown. This could mean different things:
def dedupe(pings, duping_key): + return pings\ + .map(lambda p: (p[duping_key], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) + +deduped_docid = dedupe(subset, "id") +deduped_docid_count = deduped_docid.count() +total_duplicates = ping_count - deduped_docid_count +print("Duplicate pings percentage (by document id): {:.2f}%".format(pct(total_duplicates, ping_count))) +
Duplicate pings percentage (by document id): 0.43% +
The 0.43% of ping duplicates is nice, compared to ~1% we usually get from the main and crash pings. However, nowdays we’re running de-duplication by document id at the pipeline ingestion, so this might be a bit higher. To check that, we have a telemetry_duplicates_parquet table and this handy query that says 4 duplicates were filtered on the pipeline. This means that our 0.43% is basically the real duplicate rate for the new-profile ping on Beta.
Did we send different pings for the same client id? We shouldn’t, as we send at most one ‘new-profile’ ping per client.
+deduped_clientid = dedupe(deduped_docid, "clientId") +total_duplicates_clientid = deduped_docid_count - deduped_clientid.count() +print("Duplicate pings percentage (by client id): {:.2f}%".format(pct(total_duplicates_clientid, deduped_docid_count))) +
Duplicate pings percentage (by client id): 0.81% +
That’s disappointing: it looks like we’re receiving multiple new-profile pings for some clients. Let’s dig into this by analysing the set of pings deduped by document id. To have a clearer picture of the problem, let’s make sure to aggregate the duplicates ordered by the time they were created on the client.
# Builds an RDD with (<client id>, [<ordered reason>, <ordered reason>, ...]) +clients_with_dupes = deduped_docid.map(lambda p: (p["clientId"], [(p["payload/reason"], p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) + +# Check how often each case occurs. Hide the counts. +[k for k, v in\ + sorted(clients_with_dupes.map(lambda p: tuple(p[1])).countByValue().items(), key=lambda k: k[1], reverse=True)] +
[(u'shutdown', u'shutdown'), + (u'shutdown', u'startup'), + (u'startup', u'startup'), + (u'startup', u'shutdown'), + (u'shutdown', u'shutdown', u'shutdown'), + (u'shutdown', u'startup', u'startup'), + (u'shutdown', u'startup', u'shutdown'), + (u'shutdown', + u'startup', + u'shutdown', + u'shutdown', + u'shutdown', + u'startup', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'startup'), + (u'startup', u'shutdown', u'shutdown', u'shutdown', u'shutdown', u'shutdown'), + (u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'startup', + u'shutdown', + u'shutdown'), + (u'shutdown', u'shutdown', u'shutdown', u'shutdown', u'shutdown')] +
collected_clientid_reasons = clients_with_dupes.collect() +
num_shutdown_dupes = sum([len(filter(lambda e: e == 'shutdown', t[1])) for t in collected_clientid_reasons]) +print("Duplicate 'shutdown' pings percentage (by client id): {:.2f}%"\ + .format(pct(num_shutdown_dupes, ping_count))) +
Duplicate 'shutdown' pings percentage (by client id): 1.10% +
The multiple pings we’re receiving for the same client id are mostly new-profile pings with reason shutdown. This is not too surprising, as most of the new-profile pings are being sent at shutdown (78%).
But does the pingsender have anything to do with this? Let’s attack the problem like this:
+misbehaving_clients = list(set([client_id for client_id, data in collected_clientid_reasons])) +
count_reason_pingsender = subset\ + .filter(lambda p: p.get("clientId") in misbehaving_clients)\ + .map(lambda p: (p.get("payload/reason"), p.get("meta/X-PingSender-Version")))\ + .countByValue() + +for reason, count in count_reason_pingsender.items(): + print("{}:\t{}".format(reason, pct(count, sum(count_reason_pingsender.values())))) +
(u'startup', None): 23.8532110092 +(u'shutdown', u'1.0'): 56.2691131498 +(u'shutdown', None): 19.877675841 +
Looks like some of these pings are missing the pingsender header from the request. While this is expected for new-profile pings with reason startup, as they are being sent while Firefox is still active, there might be reasons why this also happens at shutdown:
This leads to other questions:
+newprofile_shutdown_from_bad_clients =\ + subset.filter(lambda p: p.get("payload/reason") == 'shutdown')\ + .filter(lambda p: p.get("clientId") in misbehaving_clients) + +newprofile_shutdown_from_good_clients =\ + subset.filter(lambda p: p.get("payload/reason") == 'shutdown')\ + .filter(lambda p: p.get("clientId") not in misbehaving_clients) +
dict_dump = newprofile_shutdown_from_bad_clients\ + .map(lambda p: p.get("meta/X-PingSender-Version")).countByValue() +# Just print the percentages. +print("Pingsender header breakdown for misbehaving clients:") +den = sum(dict_dump.values()) +for k, v in dict_dump.items(): + print("{}:\t{}".format(k, pct(v, den))) +
Pingsender header breakdown for misbehaving clients: +1.0: 73.8955823293 +None: 26.1044176707 +
This is telling us that most of the shutdown new-profile pings are coming from the pingsender, about 73% (the 1.0 header represents the pingsender).
dict_dump = newprofile_shutdown_from_good_clients\ + .map(lambda p: p.get("meta/X-PingSender-Version")).countByValue() +# Just print the percentages. +print("Pingsender header breakdown for well behaving clients:") +den = sum(dict_dump.values()) +for k, v in dict_dump.items(): + print("{}:\t{}".format(k, pct(v, den))) +
Pingsender header breakdown for well behaving clients: +1.0: 69.3411573321 +None: 30.6588426679 +
This is somehow true with well-behaved clients, as 69% of the same pings are coming with the pingsender. The pingsender doesn’t seem to be the issue here: if we generate a ping at shutdown and try to send it with the pingsender, and fail, then it’s normal for Firefox to pick it back up and send it. As long as we don’t generate a new, different, new-profile ping for the same client.
def datetime_from_daysepoch(days_from_epoch): + return datetime(1970, 1, 1, 0, 0) + timedelta(days=days_from_epoch) + +def datetime_from_nanosepoch(nanos_from_epoch): + return datetime.fromtimestamp(nanos_from_epoch / 1000.0 / 1000.0 / 1000.0) + +def get_times(p): + profile_creation = datetime_from_daysepoch(p["environment/profile/creationDate"])\ + if p["environment/profile/creationDate"] else None + ping_creation = datetime_from_nanosepoch(p["meta/creationTimestamp"]) + ping_recv = datetime_from_nanosepoch(p["meta/Timestamp"]) + + return (p["id"], profile_creation, ping_creation, ping_recv) + +ping_times = deduped_clientid.map(get_times) +
ping_creation_delay_days = ping_times.filter(lambda p: p[1] is not None)\ + .map(lambda p: abs((p[1].date() - p[2].date()).days)).collect() +
plt.title("The distribution of the days between the profile creationDate and the 'new-profile' ping creation date") +plt.xlabel("Difference in days") +plt.ylabel("Frequency") + +CLIP_DAY = 30 +plt.xticks(range(0, CLIP_DAY + 1, 1)) +plt.hist(np.clip(ping_creation_delay_days, 0, CLIP_DAY), + alpha=0.5, bins=50, label="Delays") +
(array([ 1.94340000e+04, 3.80000000e+02, 0.00000000e+00, + 1.13000000e+02, 0.00000000e+00, 5.60000000e+01, + 5.10000000e+01, 0.00000000e+00, 2.30000000e+01, + 0.00000000e+00, 2.10000000e+01, 1.80000000e+01, + 0.00000000e+00, 2.00000000e+01, 0.00000000e+00, + 1.50000000e+01, 8.00000000e+00, 0.00000000e+00, + 1.50000000e+01, 0.00000000e+00, 1.40000000e+01, + 2.10000000e+01, 0.00000000e+00, 1.60000000e+01, + 0.00000000e+00, 1.10000000e+01, 1.00000000e+01, + 0.00000000e+00, 1.70000000e+01, 0.00000000e+00, + 2.40000000e+01, 9.00000000e+00, 0.00000000e+00, + 3.00000000e+00, 0.00000000e+00, 9.00000000e+00, + 5.00000000e+00, 0.00000000e+00, 5.00000000e+00, + 0.00000000e+00, 5.00000000e+00, 5.00000000e+00, + 0.00000000e+00, 4.00000000e+00, 0.00000000e+00, + 8.00000000e+00, 6.00000000e+00, 0.00000000e+00, + 4.00000000e+00, 1.73800000e+03]), + array([ 0. , 0.6, 1.2, 1.8, 2.4, 3. , 3.6, 4.2, 4.8, + 5.4, 6. , 6.6, 7.2, 7.8, 8.4, 9. , 9.6, 10.2, + 10.8, 11.4, 12. , 12.6, 13.2, 13.8, 14.4, 15. , 15.6, + 16.2, 16.8, 17.4, 18. , 18.6, 19.2, 19.8, 20.4, 21. , + 21.6, 22.2, 22.8, 23.4, 24. , 24.6, 25.2, 25.8, 26.4, + 27. , 27.6, 28.2, 28.8, 29.4, 30. ]), + <a list of 50 Patch objects>) +
np.percentile(np.array(ping_creation_delay_days), [50, 70, 80, 95, 99]) +
array([ 0. , 0. , 0. , 274. , 1836.6]) +
The plot shows that most of the creation dates for new-profile pings match exactly with the date reported in the environment, creationDate. That’s good, as this ping should be created very close to the profile creation. The percentile computation confirms that’s true for 80% of the new-profile pings.
new-profile and main pingsmain_pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170614" <= x < "20170620") \ + .where(appBuildId=lambda x: "20170612" <= x < "20170622") \ + .records(sc, sample=1.0) +
fetching 20894.21218MB in 8408 files... +
main_subset = get_pings_properties(main_pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/info/reason", + "payload/info/sessionLength", + "payload/info/subsessionLength", + "payload/info/profileSubsessionCounter", + "payload/info/previousSessionId"]) +
Dedupe by document id and restrict the main ping data to the pings from the misbehaving and well behaving clients.
well_behaving_clients =\ + set(subset.filter(lambda p: p.get("clientId") not in misbehaving_clients).map(lambda p: p.get("clientId")).collect()) + +all_clients = misbehaving_clients + list(well_behaving_clients) +
main_deduped = dedupe(main_subset.filter(lambda p: p.get("clientId") in all_clients), "id") +main_deduped_count = main_deduped.count() +
Try to pair each new-profile ping with reason shutdown to the very first main ping with reason shutdown received from that client, to make sure that the former were sent at the right time.
first_main = main_deduped.filter(lambda p:\ + p.get("payload/info/previousSessionId") == None and\ + p.get("payload/info/reason") == "shutdown") +
newping_shutdown = deduped_docid.filter(lambda p: p.get("payload/reason") == "shutdown") +
newprofile_plus_main = first_main.union(newping_shutdown) +sorted_per_client = newprofile_plus_main.map(lambda p: (p["clientId"], [(p, p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) +num_analysed_clients = sorted_per_client.count() +
HALF_HOUR_IN_S = 30 * 60 + +def is_newprofile(p): + # The 'main' ping has the reason field in 'payload/info/reason' + return "payload/reason" in p and p.get("payload/reason") in ["startup", "shutdown"] + +def validate_newprofile_shutdown(client_data): + ordered_pings = client_data[1] + + newprofile_mask = [is_newprofile(p) for p in ordered_pings] + + # Do we have at least a 'new-profile' ping? + num_newprofile_pings = sum(newprofile_mask) + if num_newprofile_pings < 1: + return ("No shutdown 'new-profile' ping found", 1) + + # Do we have multiple 'new-profile' pings? + if num_newprofile_pings > 1: + return ("Duplicate 'new-profile' ping.", 1) + + if not newprofile_mask[0]: + return ("The 'new-profile' ping is not the first ping", 1) + + # If there's a new-profile ping with reason 'shutdown', look for the closest next + # 'main' ping with reason shutdown. + for i, p in enumerate(ordered_pings): + # Skip until we find the 'new-profile' ping. + if not is_newprofile(p): + continue + + # We found the 'new-profile' ping. Do we have any other ping + # after this? + next_index = i + 1 + if next_index >= len(ordered_pings): + return ("No more pings after the 'new-profile'", 1) + + # Did we schedule the 'new-profile' ping at the right moment? + next_ping = ordered_pings[next_index] + if next_ping.get("payload/info/sessionLength") <= HALF_HOUR_IN_S: + return ("The 'new-profile' ping was correctly scheduled", 1) + + return ("The 'new-profile' ping was scheduled at the wrong time", 1) + + return ("Unknown condition", 1) + +scheduling_error_counts = sorted_per_client.map(validate_newprofile_shutdown).countByKey() +
for error, count in scheduling_error_counts.items(): + print("{}:\t{}".format(error, pct(count, num_analysed_clients))) +
No shutdown 'new-profile' ping found: 0.037503750375 +The 'new-profile' ping was correctly scheduled: 98.6198619862 +The 'new-profile' ping was scheduled at the wrong time: 0.630063006301 +Duplicate 'new-profile' ping.: 0.652565256526 +The 'new-profile' ping is not the first ping: 0.0600060006001 +
Most of the new-profile pings sent at shutdown, 98.61%, were correctly generated because the session lasted less than 30 minutes. Only 0.63% were scheduled at the wrong time. The rest of the clients either sent the new-profile at startup or we’re still waiting for their main ping with reason shutdown.
new-profile/startup pings only from sessions > 30 minutes?newping_startup = deduped_docid.filter(lambda p: p.get("payload/reason") == "startup") +newprofile_start_main = first_main.union(newping_startup) +sorted_per_client = newprofile_start_main.map(lambda p: (p["clientId"], [(p, p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) +num_analysed_clients = sorted_per_client.count() +
def validate_newprofile_startup(client_data): + ordered_pings = client_data[1] + + newprofile_mask = [is_newprofile(p) for p in ordered_pings] + + # Do we have at least a 'new-profile' ping? + num_newprofile_pings = sum(newprofile_mask) + if num_newprofile_pings < 1: + return ("No startup 'new-profile' ping found", 1) + + # Do we have multiple 'new-profile' pings? + if num_newprofile_pings > 1: + return ("Duplicate 'new-profile' ping", 1) + + if not newprofile_mask[0]: + return ("The 'new-profile' ping it's not the first ping", 1) + + # If there's a new-profile ping with reason 'startup', look for the closest next + # 'main' ping with reason shutdown. + for i, p in enumerate(ordered_pings): + # Skip until we find the 'new-profile' ping. + if not is_newprofile(p): + continue + + # We found the 'new-profile' ping. Do we have any other ping + # after this? + next_index = i + 1 + if next_index >= len(ordered_pings): + return ("No more pings after the 'new-profile'", 1) + + # Did we schedule the 'new-profile' ping at the right moment? + next_ping = ordered_pings[next_index] + if next_ping.get("payload/info/sessionLength") > HALF_HOUR_IN_S: + return ("The 'new-profile' ping was correctly scheduled", 1) + + return ("The 'new-profile' ping was scheduled at the wrong time", 1) + + return ("Unknown condition", 1) + +startup_newprofile_errors = sorted_per_client.map(validate_newprofile_startup).countByKey() +for error, count in startup_newprofile_errors.items(): + print("{}:\t{}".format(error, pct(count, num_analysed_clients))) +
The 'new-profile' ping it's not the first ping: 1.41059855128 +The 'new-profile' ping was correctly scheduled: 95.50133435 +Duplicate 'new-profile' ping: 0.609988562714 +The 'new-profile' ping was scheduled at the wrong time: 0.228745711018 +No startup 'new-profile' ping found: 2.24933282501 +
The results look good and in line with the previous case of the new-profile ping being sent at shutdown. The number of times the new-profile ping isn’t the first generated ping is slightly higher (0.06% vs 1.41%), but this can be explained by the fact that nothing prevents Firefox from sending new pings after Telemetry starts up (60s into the Firefox startup some addon is installed), while the new-profile ping is strictly scheduled 30 minutes after the startup.
crash ping from bad-behaved clients?If that happened close to when we generated a new-profile ping, it could hint at some correlation between crashes and the duplicates per client id.
crash_pings = Dataset.from_source("telemetry") \ + .where(docType='crash') \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170614" <= x < "20170620") \ + .where(appBuildId=lambda x: "20170612" <= x < "20170622") \ + .records(sc, sample=1.0) +
fetching 126.79929MB in 1764 files... +
Restrict the crashes to a set of useful fields, just for the misbehaving clients, and dedupe them by document id.
+crash_subset = get_pings_properties(crash_pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/crashDate", + "payload/crashTime", + "payload/processType", + "payload/sessionId"]) +crashes_misbehaving_clients = dedupe(crash_subset.filter(lambda p:\ + p.get("clientId") in misbehaving_clients and\ + p.get("payload/processType") == 'main'), "id") +newprofile_bad_clients = subset.filter(lambda p: p.get("clientId") in misbehaving_clients) +
Let’s also check how many clients are reporting crashes compared to the number of misbehaving ones.
+from operator import add +clients_with_crashes =\ + crashes_misbehaving_clients.map(lambda p: (p.get('clientId'), 1)).reduceByKey(add).map(lambda p: p[0]).collect() +
print("Percentages of bad clients with crash pings:\t{}".format(pct(len(clients_with_crashes), len(misbehaving_clients)))) +
Percentages of bad clients with crash pings: 13.8888888889 +
def get_ping_type(p): + return "crash" if "payload/crashDate" in p else "new-profile" + +newprofile_and_crashes = crashes_misbehaving_clients.union(newprofile_bad_clients) + +# Builds an RDD with (<client id>, [<ordered reason>, <ordered reason>, ...]) +joint_ordered_pings = newprofile_and_crashes\ + .map(lambda p: (p["clientId"], [(get_ping_type(p), p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) + +# Just show the pings, the most occurring first. Hide the counts. +[k for k, v in\ + sorted(joint_ordered_pings.map(lambda p: tuple(p[1])).countByValue().items(), key=lambda k: k[1], reverse=True)] +
[('new-profile', 'new-profile'),
+ ('new-profile', 'new-profile', 'new-profile'),
+ ('new-profile', 'crash', 'new-profile'),
+ ('new-profile', 'new-profile', 'crash'),
+ ('crash', 'new-profile', 'new-profile'),
+ ('new-profile', 'crash', 'crash', 'crash', 'crash', 'crash', 'new-profile'),
+ ('new-profile', 'crash', 'crash', 'crash', 'crash', 'new-profile'),
+ ('new-profile',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'new-profile',
+ 'crash',
+ 'crash',
+ 'crash'),
+ ('crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'new-profile',
+ 'crash',
+ 'new-profile'),
+ ('new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile'),
+ ('new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile'),
+ ('new-profile',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'new-profile',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash'),
+ ('new-profile', 'new-profile', 'new-profile', 'crash'),
+ ('new-profile', 'new-profile', 'new-profile', 'new-profile', 'new-profile'),
+ ('crash', 'new-profile', 'new-profile', 'crash'),
+ ('crash', 'new-profile', 'crash', 'new-profile', 'crash'),
+ ('new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile')]
+The first groups of reported ping sequences, don’t contain any crash ping and account for most of the new-profile duplicates pattern. The other sequences interleave new-profile and main process crash pings, suggesting that crashes might play a role in per-client duplicates. However, we only have crashes for 13% of the clients that do not behave correctly: this probably means that there is a weak correlation between crashes and getting multiple new-profile pings, but this is not the main problem. There’s some potential bug lurking around in the client code.
+
This analysis validates the ‘new-profile’ pings submitted by Nightly builds for one week since the ‘new-profile’ ping was enabled in bug 1364068. We are going to verify that:
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from datetime import datetime, timedelta + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Unable to parse whitelist: /mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json. +Assuming all histograms are acceptable. +
We’ll be looking at the pings that have been coming in since the 31st May 2017 to the 7th June 2017.
+# Note that the 'new-profile' ping needs to use underscores in the Dataset API due to bug. +pings = Dataset.from_source("telemetry") \ + .where(docType='new_profile') \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170531" <= x < "20170607") \ + .where(appBuildId=lambda x: "20170531" <= x < "20170607") \ + .records(sc, sample=1.0) +
fetching 5.53331MB in 1438 files... +
ping_count = pings.count() +ping_count +
3840 +
The new-profile ping can be sent either during the browsing session, 30 minutes after the browser starts, or at shutdown (docs). Let’s see how many pings we get in each case.
raw_subset = get_pings_properties(pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/reason"]) +
Discard and count any ping that’s missing creationTimestamp or Timestamp.
+def pct(a, b): + return 100.0 * a / b + +subset = raw_subset.filter(lambda p: p["meta/creationTimestamp"] is not None and p["meta/Timestamp"] is not None) +print("'new-profile' pings with missing timestamps:\t{:.2f}%".format(pct(ping_count - subset.count(), ping_count))) +
'new-profile' pings with missing timestamps: 0.00% +
reason_counts = subset.map(lambda p: p.get("payload/reason")).countByValue() + +for reason, count in reason_counts.iteritems(): + print("'new-profile' pings with reason '{}':\t{:.2f}%".format(reason, pct(count, ping_count))) +
'new-profile' pings with reason 'startup': 24.32% +'new-profile' pings with reason 'shutdown': 75.68% +
This means that, among all the new-profile pings, the majority was sent at shutdown. This could mean different things:
def dedupe(pings, duping_key): + return pings\ + .map(lambda p: (p[duping_key], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) + +deduped_docid = dedupe(subset, "id") +deduped_docid_count = deduped_docid.count() +total_duplicates = ping_count - deduped_docid_count +print("Duplicate pings percentage (by document id): {:.2f}%".format(pct(total_duplicates, ping_count))) +
Duplicate pings percentage (by document id): 0.21% +
The 0.21% of ping duplicates is nice, compared to ~1% we usually get from the main and crash pings. However, nowdays we’re running de-duplication by document id at the pipeline ingestion, so this might be a bit higher. To check that, we have a telemetry_duplicates_parquet table and this handy query that says 0 duplicates were filtered on the pipeline. This means that our 0.21% is the real duplicate rate for the new-profile ping on Nightly.
Did we send different pings for the same client id? We shouldn’t, as we send at most one ‘new-profile’ ping per client.
+deduped_clientid = dedupe(deduped_docid, "clientId") +total_duplicates_clientid = deduped_docid_count - deduped_clientid.count() +print("Duplicate pings percentage (by client id): {:.2f}%".format(pct(total_duplicates_clientid, deduped_docid_count))) +
Duplicate pings percentage (by client id): 0.89% +
That’s disappointing: it looks like we’re receiving multiple new-profile pings for some clients. Let’s dig into this by analysing the set of pings deduped by document id. To have a clearer picture of the problem, let’s make sure to aggregate the duplicates ordered by the time they were created on the client.
# Builds an RDD with (<client id>, [<ordered reason>, <ordered reason>, ...]) +clients_with_dupes = deduped_docid.map(lambda p: (p["clientId"], [(p["payload/reason"], p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) + +# Check how often each case occurs. +sorted(clients_with_dupes.map(lambda p: tuple(p[1])).countByValue().items(), key=lambda k: k[1], reverse=True) +
[((u'shutdown', u'shutdown'), 7), + ((u'shutdown', u'startup'), 4), + ((u'startup', u'startup'), 3), + ((u'startup', u'shutdown'), 3), + ((u'shutdown', u'shutdown', u'shutdown'), 2), + ((u'shutdown', u'shutdown', u'startup'), 1), + ((u'shutdown', u'startup', u'shutdown'), 1), + ((u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown'), + 1), + ((u'shutdown', u'shutdown', u'shutdown', u'startup', u'shutdown'), 1)] +
collected_clientid_reasons = clients_with_dupes.collect() +
num_shutdown_dupes = sum([len(filter(lambda e: e == 'shutdown', t[1])) for t in collected_clientid_reasons]) +print("Duplicate 'shutdown' pings percentage (by client id): {:.2f}%"\ + .format(pct(num_shutdown_dupes, ping_count))) +
Duplicate 'shutdown' pings percentage (by client id): 1.07% +
The multiple pings we’re receiving for the same client id are mostly new-profile pings with reason shutdown. This is not too surprising, as most of the new-profile pings are being sent at shutdown (75%).
But does the pingsender have anything to do with this? Let’s attack the problem like this:
+count_reason_pingsender = subset\ + .filter(lambda p: p.get("clientId") in misbehaving_clients)\ + .map(lambda p: (p.get("payload/reason"), p.get("meta/X-PingSender-Version")))\ + .countByValue() + +for reason, count in count_reason_pingsender.items(): + print("{}:\t{}".format(reason, pct(count, sum(count_reason_pingsender.values())))) +
(u'startup', None): 27.5862068966 +(u'shutdown', u'1.0'): 55.1724137931 +(u'shutdown', None): 17.2413793103 +
Looks like some of these pings are missing the pingsender header from the request. While this is expected for new-profile pings with reason startup, as they are being sent while Firefox is still active, there might be reasons why this also happens at shutdown:
This leads to other questions:
+newprofile_shutdown_from_bad_clients =\ + subset.filter(lambda p: p.get("payload/reason") == 'shutdown')\ + .filter(lambda p: p.get("clientId") in misbehaving_clients) + +newprofile_shutdown_from_good_clients =\ + subset.filter(lambda p: p.get("payload/reason") == 'shutdown')\ + .filter(lambda p: p.get("clientId") not in misbehaving_clients) +
newprofile_shutdown_from_bad_clients\ + .map(lambda p: p.get("meta/X-PingSender-Version")).countByValue() +
defaultdict(int, {None: 10, u'1.0': 32})
+This is telling us that most of the shutdown new-profile pings are coming from the pingsender, about 76%.
newprofile_shutdown_from_good_clients\ + .map(lambda p: p.get("meta/X-PingSender-Version")).countByValue() +
defaultdict(int, {None: 528, u'1.0': 2336})
+This is still true with well-behaved clients, as 81% of the same pings are coming with the pingsender. The pingsender doesn’t seem to be the issue here: if we generate a ping at shutdown and try to send it with the pingsender, and fail, then it’s normal for Firefox to pick it back up and send it. As long as we don’t generate a new, different, new-profile ping for the same client.
def datetime_from_daysepoch(days_from_epoch): + return datetime(1970, 1, 1, 0, 0) + timedelta(days=days_from_epoch) + +def datetime_from_nanosepoch(nanos_from_epoch): + return datetime.fromtimestamp(nanos_from_epoch / 1000.0 / 1000.0 / 1000.0) + +def get_times(p): + profile_creation = datetime_from_daysepoch(p["environment/profile/creationDate"])\ + if p["environment/profile/creationDate"] else None + ping_creation = datetime_from_nanosepoch(p["meta/creationTimestamp"]) + ping_recv = datetime_from_nanosepoch(p["meta/Timestamp"]) + + return (p["id"], profile_creation, ping_creation, ping_recv) + +ping_times = deduped_clientid.map(get_times) +
ping_creation_delay_days = ping_times.filter(lambda p: p[1] is not None)\ + .map(lambda p: abs((p[1].date() - p[2].date()).days)).collect() +
plt.title("The distribution of the days between the profile creationDate and the 'new-profile' ping creation date") +plt.xlabel("Difference in days") +plt.ylabel("Frequency") + +CLIP_DAY = 30 +plt.xticks(range(0, CLIP_DAY + 1, 1)) +plt.hist(np.clip(ping_creation_delay_days, 0, CLIP_DAY), + alpha=0.5, bins=50, label="Delays") +
(array([ 2.84000000e+03, 4.00000000e+01, 0.00000000e+00, + 1.50000000e+01, 0.00000000e+00, 2.70000000e+01, + 2.20000000e+01, 0.00000000e+00, 2.70000000e+01, + 0.00000000e+00, 3.40000000e+01, 3.80000000e+01, + 0.00000000e+00, 3.20000000e+01, 0.00000000e+00, + 3.40000000e+01, 3.40000000e+01, 0.00000000e+00, + 4.00000000e+01, 0.00000000e+00, 2.90000000e+01, + 2.90000000e+01, 0.00000000e+00, 3.40000000e+01, + 0.00000000e+00, 2.00000000e+01, 2.50000000e+01, + 0.00000000e+00, 3.30000000e+01, 0.00000000e+00, + 1.40000000e+01, 2.00000000e+01, 0.00000000e+00, + 1.50000000e+01, 0.00000000e+00, 1.50000000e+01, + 8.00000000e+00, 0.00000000e+00, 8.00000000e+00, + 0.00000000e+00, 4.00000000e+00, 2.00000000e+00, + 0.00000000e+00, 1.00000000e+00, 0.00000000e+00, + 2.00000000e+00, 3.00000000e+00, 0.00000000e+00, + 1.00000000e+00, 3.51000000e+02]), + array([ 0. , 0.6, 1.2, 1.8, 2.4, 3. , 3.6, 4.2, 4.8, + 5.4, 6. , 6.6, 7.2, 7.8, 8.4, 9. , 9.6, 10.2, + 10.8, 11.4, 12. , 12.6, 13.2, 13.8, 14.4, 15. , 15.6, + 16.2, 16.8, 17.4, 18. , 18.6, 19.2, 19.8, 20.4, 21. , + 21.6, 22.2, 22.8, 23.4, 24. , 24.6, 25.2, 25.8, 26.4, + 27. , 27.6, 28.2, 28.8, 29.4, 30. ]), + <a list of 50 Patch objects>) +
np.percentile(np.array(ping_creation_delay_days), [50, 70, 80, 95, 99]) +
array([ 0. , 0. , 7. , 385.6, 1294.2]) +
The plot shows that most of the creation dates for new-profile pings match exactly with the date reported in the environment, creationDate. That’s good, as this ping should be created very close to the profile creation. The percentile computation confirms that’s true for 70% of the new-profile pings.
However, things get tricky. The new-profile ping was enabled on the 30th of May, 2017 and should only be sent my new profiles. A delay longer than 7 days suggests that either the profile creationDate is terribly wrong or that we’re sending the new-profile ping from existing profiles as well.
new-profile and main pingsmain_pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170531" <= x < "20170607") \ + .where(appBuildId=lambda x: "20170531" <= x < "20170607") \ + .records(sc, sample=1.0) +
fetching 23348.80037MB in 8027 files... +
main_subset = get_pings_properties(main_pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/info/reason", + "payload/info/sessionLength", + "payload/info/subsessionLength", + "payload/info/profileSubsessionCounter", + "payload/info/previousSessionId"]) +
Dedupe by document id and restrict the main ping data to the pings from the misbehaving and well behaving clients.
well_behaving_clients =\ + set(subset.filter(lambda p: p.get("clientId") not in misbehaving_clients).map(lambda p: p.get("clientId")).collect()) + +all_clients = misbehaving_clients + list(well_behaving_clients) +
main_deduped = dedupe(main_subset.filter(lambda p: p.get("clientId") in all_clients), "id") +main_deduped_count = main_deduped.count() +
Try to pair each new-profile ping with reason shutdown to the very first main ping with reason shutdown received from that client, to make sure that the former were sent at the right time.
first_main = main_deduped.filter(lambda p:\ + p.get("payload/info/previousSessionId") == None and\ + p.get("payload/info/reason") == "shutdown") +
newping_shutdown = deduped_docid.filter(lambda p: p.get("payload/reason") == "shutdown") +
newprofile_plus_main = first_main.union(newping_shutdown) +sorted_per_client = newprofile_plus_main.map(lambda p: (p["clientId"], [(p, p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) +num_analysed_clients = sorted_per_client.count() +
HALF_HOUR_IN_S = 30 * 60 + +def is_newprofile(p): + # The 'main' ping has the reason field in 'payload/info/reason' + return "payload/reason" in p and p.get("payload/reason") in ["startup", "shutdown"] + +def validate_newprofile_shutdown(client_data): + ordered_pings = client_data[1] + + newprofile_mask = [is_newprofile(p) for p in ordered_pings] + + # Do we have at least a 'new-profile' ping? + num_newprofile_pings = sum(newprofile_mask) + if num_newprofile_pings < 1: + return ("No shutdown 'new-profile' ping found", 1) + + # Do we have multiple 'new-profile' pings? + if num_newprofile_pings > 1: + return ("Duplicate 'new-profile' ping.", 1) + + if not newprofile_mask[0]: + return ("The 'new-profile' ping is not the first ping", 1) + + # If there's a new-profile ping with reason 'shutdown', look for the closest next + # 'main' ping with reason shutdown. + for i, p in enumerate(ordered_pings): + # Skip until we find the 'new-profile' ping. + if not is_newprofile(p): + continue + + # We found the 'new-profile' ping. Do we have any other ping + # after this? + next_index = i + 1 + if next_index >= len(ordered_pings): + return ("No more pings after the 'new-profile'", 1) + + # Did we schedule the 'new-profile' ping at the right moment? + next_ping = ordered_pings[next_index] + if next_ping.get("payload/info/sessionLength") <= HALF_HOUR_IN_S: + return ("The 'new-profile' ping was correctly scheduled", 1) + + return ("The 'new-profile' ping was scheduled at the wrong time", 1) + + return ("Unknown condition", 1) + +scheduling_error_counts = sorted_per_client.map(validate_newprofile_shutdown).countByKey() +
for error, count in scheduling_error_counts.items(): + print("{}:\t{}".format(error, pct(count, num_analysed_clients))) +
The 'new-profile' ping is not the first ping: 0.135685210312 +No shutdown 'new-profile' ping found: 0.203527815468 +The 'new-profile' ping was correctly scheduled: 97.8968792402 +The 'new-profile' ping was scheduled at the wrong time: 0.881953867028 +Duplicate 'new-profile' ping.: 0.881953867028 +
Most of the new-profile pings sent at shutdown, 97.89%, were correctly generated because the session lasted less than 30 minutes. Only 0.88% were scheduled at the wrong time. The rest of the clients either sent the new-profile at startup or we’re still waiting for their main ping with reason shutdown.
new-profile/startup pings only from sessions > 30 minutes?newping_startup = deduped_docid.filter(lambda p: p.get("payload/reason") == "startup") +newprofile_start_main = first_main.union(newping_startup) +sorted_per_client = newprofile_start_main.map(lambda p: (p["clientId"], [(p, p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) +num_analysed_clients = sorted_per_client.count() +
def validate_newprofile_startup(client_data): + ordered_pings = client_data[1] + + newprofile_mask = [is_newprofile(p) for p in ordered_pings] + + # Do we have at least a 'new-profile' ping? + num_newprofile_pings = sum(newprofile_mask) + if num_newprofile_pings < 1: + return ("No startup 'new-profile' ping found", 1) + + # Do we have multiple 'new-profile' pings? + if num_newprofile_pings > 1: + return ("Duplicate 'new-profile' ping", 1) + + if not newprofile_mask[0]: + return ("The 'new-profile' ping it's not the first ping", 1) + + # If there's a new-profile ping with reason 'startup', look for the closest next + # 'main' ping with reason shutdown. + for i, p in enumerate(ordered_pings): + # Skip until we find the 'new-profile' ping. + if not is_newprofile(p): + continue + + # We found the 'new-profile' ping. Do we have any other ping + # after this? + next_index = i + 1 + if next_index >= len(ordered_pings): + return ("No more pings after the 'new-profile'", 1) + + # Did we schedule the 'new-profile' ping at the right moment? + next_ping = ordered_pings[next_index] + if next_ping.get("payload/info/sessionLength") > HALF_HOUR_IN_S: + return ("The 'new-profile' ping was correctly scheduled", 1) + + return ("The 'new-profile' ping was scheduled at the wrong time", 1) + + return ("Unknown condition", 1) + +startup_newprofile_errors = sorted_per_client.map(validate_newprofile_startup).countByKey() +for error, count in startup_newprofile_errors.items(): + print("{}:\t{}".format(error, pct(count, num_analysed_clients))) +
The 'new-profile' ping it's not the first ping: 1.41342756184 +The 'new-profile' ping was correctly scheduled: 96.1130742049 +Duplicate 'new-profile' ping: 1.06007067138 +The 'new-profile' ping was scheduled at the wrong time: 0.353356890459 +No startup 'new-profile' ping found: 1.06007067138 +
The results look good and in line with the previous case of the new-profile ping being sent at shutdown. The number of times the new-profile ping isn’t the first generated ping is slightly higher (0.13% vs 1.41%), but this can be explained by the fact that nothing prevents Firefox from sending new pings after Telemetry starts up (60s into the Firefox startup some addon is installed), while the new-profile ping is strictly scheduled 30 minutes after the startup.
crash ping from bad-behaved clients?If that happened close to when we generated a new-profile ping, it could hint at some correlation between crashes and the duplicates per client id.
crash_pings = Dataset.from_source("telemetry") \ + .where(docType='crash') \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170531" <= x < "20170607") \ + .where(appBuildId=lambda x: "20170531" <= x < "20170607") \ + .records(sc, sample=1.0) +
fetching 356.90416MB in 3290 files... +
Restrict the crashes to a set of useful fields, just for the misbehaving clients, and dedupe them by document id.
+crash_subset = get_pings_properties(crash_pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/crashDate", + "payload/crashTime", + "payload/processType", + "payload/sessionId"]) +crashes_misbehaving_clients = dedupe(crash_subset.filter(lambda p:\ + p.get("clientId") in misbehaving_clients and\ + p.get("payload/processType") == 'main'), "id") +newprofile_bad_clients = subset.filter(lambda p: p.get("clientId") in misbehaving_clients) +
def get_ping_type(p): + return "crash" if "payload/crashDate" in p else "new-profile" + +newprofile_and_crashes = crashes_misbehaving_clients.union(newprofile_bad_clients) + +# Builds an RDD with (<client id>, [<ordered reason>, <ordered reason>, ...]) +joint_ordered_pings = newprofile_and_crashes\ + .map(lambda p: (p["clientId"], [(get_ping_type(p), p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) + +sorted(joint_ordered_pings.map(lambda p: tuple(p[1])).countByValue().items(), key=lambda k: k[1], reverse=True) +
[(('new-profile', 'new-profile'), 15),
+ (('new-profile', 'new-profile', 'new-profile'), 4),
+ (('new-profile',
+ 'crash',
+ 'crash',
+ 'new-profile',
+ 'crash',
+ 'new-profile',
+ 'new-profile',
+ 'crash',
+ 'new-profile',
+ 'crash',
+ 'new-profile'),
+ 1),
+ (('new-profile', 'crash', 'new-profile'), 1),
+ (('new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'new-profile'),
+ 1),
+ (('new-profile', 'new-profile', 'crash'), 1)]
+The first groups of reported ping sequences, don’t contain any crash ping and account for most of the new-profile duplicates pattern. The other sequences interleave new-profile and main process crash pings, suggesting that crashes might play a role in per-client duplicates. However, we only have crashes for 3 clients over 23 misbehaving ones, not enough to draw conclusions from.
+
This analysis validates the ‘new-profile’ pings submitted by Nightly builds for one week since the ‘new-profile’ ping was enabled in bug 1364068. We are going to verify that:
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from datetime import datetime, timedelta + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Unable to parse whitelist: /mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json. +Assuming all histograms are acceptable. +
We’ll be looking at the pings that have been coming in since the 31st May 2017 to the 7th June 2017.
+# Note that the 'new-profile' ping needs to use underscores in the Dataset API due to bug. +pings = Dataset.from_source("telemetry") \ + .where(docType='new_profile') \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170531" <= x < "20170607") \ + .where(appBuildId=lambda x: "20170531" <= x < "20170607") \ + .records(sc, sample=1.0) +
fetching 5.53331MB in 1438 files... +
ping_count = pings.count() +ping_count +
3840 +
The new-profile ping can be sent either during the browsing session, 30 minutes after the browser starts, or at shutdown (docs). Let’s see how many pings we get in each case.
raw_subset = get_pings_properties(pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/reason"]) +
Discard and count any ping that’s missing creationTimestamp or Timestamp.
+def pct(a, b): + return 100.0 * a / b + +subset = raw_subset.filter(lambda p: p["meta/creationTimestamp"] is not None and p["meta/Timestamp"] is not None) +print("'new-profile' pings with missing timestamps:\t{:.2f}%".format(pct(ping_count - subset.count(), ping_count))) +
'new-profile' pings with missing timestamps: 0.00% +
reason_counts = subset.map(lambda p: p.get("payload/reason")).countByValue() + +for reason, count in reason_counts.iteritems(): + print("'new-profile' pings with reason '{}':\t{:.2f}%".format(reason, pct(count, ping_count))) +
'new-profile' pings with reason 'startup': 24.32% +'new-profile' pings with reason 'shutdown': 75.68% +
This means that, among all the new-profile pings, the majority was sent at shutdown. This could mean different things:
def dedupe(pings, duping_key): + return pings\ + .map(lambda p: (p[duping_key], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) + +deduped_docid = dedupe(subset, "id") +deduped_docid_count = deduped_docid.count() +total_duplicates = ping_count - deduped_docid_count +print("Duplicate pings percentage (by document id): {:.2f}%".format(pct(total_duplicates, ping_count))) +
Duplicate pings percentage (by document id): 0.21% +
The 0.21% of ping duplicates is nice, compared to ~1% we usually get from the main and crash pings. However, nowdays we’re running de-duplication by document id at the pipeline ingestion, so this might be a bit higher. To check that, we have a telemetry_duplicates_parquet table and this handy query that says 0 duplicates were filtered on the pipeline. This means that our 0.21% is the real duplicate rate for the new-profile ping on Nightly.
Did we send different pings for the same client id? We shouldn’t, as we send at most one ‘new-profile’ ping per client.
+deduped_clientid = dedupe(deduped_docid, "clientId") +total_duplicates_clientid = deduped_docid_count - deduped_clientid.count() +print("Duplicate pings percentage (by client id): {:.2f}%".format(pct(total_duplicates_clientid, deduped_docid_count))) +
Duplicate pings percentage (by client id): 0.89% +
That’s disappointing: it looks like we’re receiving multiple new-profile pings for some clients. Let’s dig into this by analysing the set of pings deduped by document id. To have a clearer picture of the problem, let’s make sure to aggregate the duplicates ordered by the time they were created on the client.
# Builds an RDD with (<client id>, [<ordered reason>, <ordered reason>, ...]) +clients_with_dupes = deduped_docid.map(lambda p: (p["clientId"], [(p["payload/reason"], p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) + +# Check how often each case occurs. +sorted(clients_with_dupes.map(lambda p: tuple(p[1])).countByValue().items(), key=lambda k: k[1], reverse=True) +
[((u'shutdown', u'shutdown'), 7), + ((u'shutdown', u'startup'), 4), + ((u'startup', u'startup'), 3), + ((u'startup', u'shutdown'), 3), + ((u'shutdown', u'shutdown', u'shutdown'), 2), + ((u'shutdown', u'shutdown', u'startup'), 1), + ((u'shutdown', u'startup', u'shutdown'), 1), + ((u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown', + u'shutdown'), + 1), + ((u'shutdown', u'shutdown', u'shutdown', u'startup', u'shutdown'), 1)] +
collected_clientid_reasons = clients_with_dupes.collect() +
num_shutdown_dupes = sum([len(filter(lambda e: e == 'shutdown', t[1])) for t in collected_clientid_reasons]) +print("Duplicate 'shutdown' pings percentage (by client id): {:.2f}%"\ + .format(pct(num_shutdown_dupes, ping_count))) +
Duplicate 'shutdown' pings percentage (by client id): 1.07% +
The multiple pings we’re receiving for the same client id are mostly new-profile pings with reason shutdown. This is not too surprising, as most of the new-profile pings are being sent at shutdown (75%).
But does the pingsender have anything to do with this? Let’s attack the problem like this:
+count_reason_pingsender = subset\ + .filter(lambda p: p.get("clientId") in misbehaving_clients)\ + .map(lambda p: (p.get("payload/reason"), p.get("meta/X-PingSender-Version")))\ + .countByValue() + +for reason, count in count_reason_pingsender.items(): + print("{}:\t{}".format(reason, pct(count, sum(count_reason_pingsender.values())))) +
(u'startup', None): 27.5862068966 +(u'shutdown', u'1.0'): 55.1724137931 +(u'shutdown', None): 17.2413793103 +
Looks like some of these pings are missing the pingsender header from the request. While this is expected for new-profile pings with reason startup, as they are being sent while Firefox is still active, there might be reasons why this also happens at shutdown:
This leads to other questions:
+newprofile_shutdown_from_bad_clients =\ + subset.filter(lambda p: p.get("payload/reason") == 'shutdown')\ + .filter(lambda p: p.get("clientId") in misbehaving_clients) + +newprofile_shutdown_from_good_clients =\ + subset.filter(lambda p: p.get("payload/reason") == 'shutdown')\ + .filter(lambda p: p.get("clientId") not in misbehaving_clients) +
newprofile_shutdown_from_bad_clients\ + .map(lambda p: p.get("meta/X-PingSender-Version")).countByValue() +
defaultdict(int, {None: 10, u'1.0': 32})
+This is telling us that most of the shutdown new-profile pings are coming from the pingsender, about 76%.
newprofile_shutdown_from_good_clients\ + .map(lambda p: p.get("meta/X-PingSender-Version")).countByValue() +
defaultdict(int, {None: 528, u'1.0': 2336})
+This is still true with well-behaved clients, as 81% of the same pings are coming with the pingsender. The pingsender doesn’t seem to be the issue here: if we generate a ping at shutdown and try to send it with the pingsender, and fail, then it’s normal for Firefox to pick it back up and send it. As long as we don’t generate a new, different, new-profile ping for the same client.
def datetime_from_daysepoch(days_from_epoch): + return datetime(1970, 1, 1, 0, 0) + timedelta(days=days_from_epoch) + +def datetime_from_nanosepoch(nanos_from_epoch): + return datetime.fromtimestamp(nanos_from_epoch / 1000.0 / 1000.0 / 1000.0) + +def get_times(p): + profile_creation = datetime_from_daysepoch(p["environment/profile/creationDate"])\ + if p["environment/profile/creationDate"] else None + ping_creation = datetime_from_nanosepoch(p["meta/creationTimestamp"]) + ping_recv = datetime_from_nanosepoch(p["meta/Timestamp"]) + + return (p["id"], profile_creation, ping_creation, ping_recv) + +ping_times = deduped_clientid.map(get_times) +
ping_creation_delay_days = ping_times.filter(lambda p: p[1] is not None)\ + .map(lambda p: abs((p[1].date() - p[2].date()).days)).collect() +
plt.title("The distribution of the days between the profile creationDate and the 'new-profile' ping creation date") +plt.xlabel("Difference in days") +plt.ylabel("Frequency") + +CLIP_DAY = 30 +plt.xticks(range(0, CLIP_DAY + 1, 1)) +plt.hist(np.clip(ping_creation_delay_days, 0, CLIP_DAY), + alpha=0.5, bins=50, label="Delays") +
(array([ 2.84000000e+03, 4.00000000e+01, 0.00000000e+00, + 1.50000000e+01, 0.00000000e+00, 2.70000000e+01, + 2.20000000e+01, 0.00000000e+00, 2.70000000e+01, + 0.00000000e+00, 3.40000000e+01, 3.80000000e+01, + 0.00000000e+00, 3.20000000e+01, 0.00000000e+00, + 3.40000000e+01, 3.40000000e+01, 0.00000000e+00, + 4.00000000e+01, 0.00000000e+00, 2.90000000e+01, + 2.90000000e+01, 0.00000000e+00, 3.40000000e+01, + 0.00000000e+00, 2.00000000e+01, 2.50000000e+01, + 0.00000000e+00, 3.30000000e+01, 0.00000000e+00, + 1.40000000e+01, 2.00000000e+01, 0.00000000e+00, + 1.50000000e+01, 0.00000000e+00, 1.50000000e+01, + 8.00000000e+00, 0.00000000e+00, 8.00000000e+00, + 0.00000000e+00, 4.00000000e+00, 2.00000000e+00, + 0.00000000e+00, 1.00000000e+00, 0.00000000e+00, + 2.00000000e+00, 3.00000000e+00, 0.00000000e+00, + 1.00000000e+00, 3.51000000e+02]), + array([ 0. , 0.6, 1.2, 1.8, 2.4, 3. , 3.6, 4.2, 4.8, + 5.4, 6. , 6.6, 7.2, 7.8, 8.4, 9. , 9.6, 10.2, + 10.8, 11.4, 12. , 12.6, 13.2, 13.8, 14.4, 15. , 15.6, + 16.2, 16.8, 17.4, 18. , 18.6, 19.2, 19.8, 20.4, 21. , + 21.6, 22.2, 22.8, 23.4, 24. , 24.6, 25.2, 25.8, 26.4, + 27. , 27.6, 28.2, 28.8, 29.4, 30. ]), + <a list of 50 Patch objects>) +
np.percentile(np.array(ping_creation_delay_days), [50, 70, 80, 95, 99]) +
array([ 0. , 0. , 7. , 385.6, 1294.2]) +
The plot shows that most of the creation dates for new-profile pings match exactly with the date reported in the environment, creationDate. That’s good, as this ping should be created very close to the profile creation. The percentile computation confirms that’s true for 70% of the new-profile pings.
However, things get tricky. The new-profile ping was enabled on the 30th of May, 2017 and should only be sent my new profiles. A delay longer than 7 days suggests that either the profile creationDate is terribly wrong or that we’re sending the new-profile ping from existing profiles as well.
new-profile and main pingsmain_pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170531" <= x < "20170607") \ + .where(appBuildId=lambda x: "20170531" <= x < "20170607") \ + .records(sc, sample=1.0) +
fetching 23348.80037MB in 8027 files... +
main_subset = get_pings_properties(main_pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/info/reason", + "payload/info/sessionLength", + "payload/info/subsessionLength", + "payload/info/profileSubsessionCounter", + "payload/info/previousSessionId"]) +
Dedupe by document id and restrict the main ping data to the pings from the misbehaving and well behaving clients.
well_behaving_clients =\ + set(subset.filter(lambda p: p.get("clientId") not in misbehaving_clients).map(lambda p: p.get("clientId")).collect()) + +all_clients = misbehaving_clients + list(well_behaving_clients) +
main_deduped = dedupe(main_subset.filter(lambda p: p.get("clientId") in all_clients), "id") +main_deduped_count = main_deduped.count() +
Try to pair each new-profile ping with reason shutdown to the very first main ping with reason shutdown received from that client, to make sure that the former were sent at the right time.
first_main = main_deduped.filter(lambda p:\ + p.get("payload/info/previousSessionId") == None and\ + p.get("payload/info/reason") == "shutdown") +
newping_shutdown = deduped_docid.filter(lambda p: p.get("payload/reason") == "shutdown") +
newprofile_plus_main = first_main.union(newping_shutdown) +sorted_per_client = newprofile_plus_main.map(lambda p: (p["clientId"], [(p, p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) +num_analysed_clients = sorted_per_client.count() +
HALF_HOUR_IN_S = 30 * 60 + +def is_newprofile(p): + # The 'main' ping has the reason field in 'payload/info/reason' + return "payload/reason" in p and p.get("payload/reason") in ["startup", "shutdown"] + +def validate_newprofile_shutdown(client_data): + ordered_pings = client_data[1] + + newprofile_mask = [is_newprofile(p) for p in ordered_pings] + + # Do we have at least a 'new-profile' ping? + num_newprofile_pings = sum(newprofile_mask) + if num_newprofile_pings < 1: + return ("No shutdown 'new-profile' ping found", 1) + + # Do we have multiple 'new-profile' pings? + if num_newprofile_pings > 1: + return ("Duplicate 'new-profile' ping.", 1) + + if not newprofile_mask[0]: + return ("The 'new-profile' ping is not the first ping", 1) + + # If there's a new-profile ping with reason 'shutdown', look for the closest next + # 'main' ping with reason shutdown. + for i, p in enumerate(ordered_pings): + # Skip until we find the 'new-profile' ping. + if not is_newprofile(p): + continue + + # We found the 'new-profile' ping. Do we have any other ping + # after this? + next_index = i + 1 + if next_index >= len(ordered_pings): + return ("No more pings after the 'new-profile'", 1) + + # Did we schedule the 'new-profile' ping at the right moment? + next_ping = ordered_pings[next_index] + if next_ping.get("payload/info/sessionLength") <= HALF_HOUR_IN_S: + return ("The 'new-profile' ping was correctly scheduled", 1) + + return ("The 'new-profile' ping was scheduled at the wrong time", 1) + + return ("Unknown condition", 1) + +scheduling_error_counts = sorted_per_client.map(validate_newprofile_shutdown).countByKey() +
for error, count in scheduling_error_counts.items(): + print("{}:\t{}".format(error, pct(count, num_analysed_clients))) +
The 'new-profile' ping is not the first ping: 0.135685210312 +No shutdown 'new-profile' ping found: 0.203527815468 +The 'new-profile' ping was correctly scheduled: 97.8968792402 +The 'new-profile' ping was scheduled at the wrong time: 0.881953867028 +Duplicate 'new-profile' ping.: 0.881953867028 +
Most of the new-profile pings sent at shutdown, 97.89%, were correctly generated because the session lasted less than 30 minutes. Only 0.88% were scheduled at the wrong time. The rest of the clients either sent the new-profile at startup or we’re still waiting for their main ping with reason shutdown.
new-profile/startup pings only from sessions > 30 minutes?newping_startup = deduped_docid.filter(lambda p: p.get("payload/reason") == "startup") +newprofile_start_main = first_main.union(newping_startup) +sorted_per_client = newprofile_start_main.map(lambda p: (p["clientId"], [(p, p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .filter(lambda p: len(p[1]) > 1)\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) +num_analysed_clients = sorted_per_client.count() +
def validate_newprofile_startup(client_data): + ordered_pings = client_data[1] + + newprofile_mask = [is_newprofile(p) for p in ordered_pings] + + # Do we have at least a 'new-profile' ping? + num_newprofile_pings = sum(newprofile_mask) + if num_newprofile_pings < 1: + return ("No startup 'new-profile' ping found", 1) + + # Do we have multiple 'new-profile' pings? + if num_newprofile_pings > 1: + return ("Duplicate 'new-profile' ping", 1) + + if not newprofile_mask[0]: + return ("The 'new-profile' ping it's not the first ping", 1) + + # If there's a new-profile ping with reason 'startup', look for the closest next + # 'main' ping with reason shutdown. + for i, p in enumerate(ordered_pings): + # Skip until we find the 'new-profile' ping. + if not is_newprofile(p): + continue + + # We found the 'new-profile' ping. Do we have any other ping + # after this? + next_index = i + 1 + if next_index >= len(ordered_pings): + return ("No more pings after the 'new-profile'", 1) + + # Did we schedule the 'new-profile' ping at the right moment? + next_ping = ordered_pings[next_index] + if next_ping.get("payload/info/sessionLength") > HALF_HOUR_IN_S: + return ("The 'new-profile' ping was correctly scheduled", 1) + + return ("The 'new-profile' ping was scheduled at the wrong time", 1) + + return ("Unknown condition", 1) + +startup_newprofile_errors = sorted_per_client.map(validate_newprofile_startup).countByKey() +for error, count in startup_newprofile_errors.items(): + print("{}:\t{}".format(error, pct(count, num_analysed_clients))) +
The 'new-profile' ping it's not the first ping: 1.41342756184 +The 'new-profile' ping was correctly scheduled: 96.1130742049 +Duplicate 'new-profile' ping: 1.06007067138 +The 'new-profile' ping was scheduled at the wrong time: 0.353356890459 +No startup 'new-profile' ping found: 1.06007067138 +
The results look good and in line with the previous case of the new-profile ping being sent at shutdown. The number of times the new-profile ping isn’t the first generated ping is slightly higher (0.13% vs 1.41%), but this can be explained by the fact that nothing prevents Firefox from sending new pings after Telemetry starts up (60s into the Firefox startup some addon is installed), while the new-profile ping is strictly scheduled 30 minutes after the startup.
crash ping from bad-behaved clients?If that happened close to when we generated a new-profile ping, it could hint at some correlation between crashes and the duplicates per client id.
crash_pings = Dataset.from_source("telemetry") \ + .where(docType='crash') \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170531" <= x < "20170607") \ + .where(appBuildId=lambda x: "20170531" <= x < "20170607") \ + .records(sc, sample=1.0) +
fetching 356.90416MB in 3290 files... +
Restrict the crashes to a set of useful fields, just for the misbehaving clients, and dedupe them by document id.
+crash_subset = get_pings_properties(crash_pings, ["id", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "meta/X-PingSender-Version", + "clientId", + "environment/profile/creationDate", + "payload/crashDate", + "payload/crashTime", + "payload/processType", + "payload/sessionId"]) +crashes_misbehaving_clients = dedupe(crash_subset.filter(lambda p:\ + p.get("clientId") in misbehaving_clients and\ + p.get("payload/processType") == 'main'), "id") +newprofile_bad_clients = subset.filter(lambda p: p.get("clientId") in misbehaving_clients) +
def get_ping_type(p): + return "crash" if "payload/crashDate" in p else "new-profile" + +newprofile_and_crashes = crashes_misbehaving_clients.union(newprofile_bad_clients) + +# Builds an RDD with (<client id>, [<ordered reason>, <ordered reason>, ...]) +joint_ordered_pings = newprofile_and_crashes\ + .map(lambda p: (p["clientId"], [(get_ping_type(p), p["meta/creationTimestamp"])]))\ + .reduceByKey(lambda a, b: sorted(a + b, key=lambda k: k[1]))\ + .map(lambda p: (p[0], [r[0] for r in p[1]])) + +sorted(joint_ordered_pings.map(lambda p: tuple(p[1])).countByValue().items(), key=lambda k: k[1], reverse=True) +
[(('new-profile', 'new-profile'), 15),
+ (('new-profile', 'new-profile', 'new-profile'), 4),
+ (('new-profile',
+ 'crash',
+ 'crash',
+ 'new-profile',
+ 'crash',
+ 'new-profile',
+ 'new-profile',
+ 'crash',
+ 'new-profile',
+ 'crash',
+ 'new-profile'),
+ 1),
+ (('new-profile', 'crash', 'new-profile'), 1),
+ (('new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'new-profile',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'crash',
+ 'new-profile'),
+ 1),
+ (('new-profile', 'new-profile', 'crash'), 1)]
+The first groups of reported ping sequences, don’t contain any crash ping and account for most of the new-profile duplicates pattern. The other sequences interleave new-profile and main process crash pings, suggesting that crashes might play a role in per-client duplicates. However, we only have crashes for 3 clients over 23 misbehaving ones, not enough to draw conclusions from.
+
This is primarily a summary of Bug 1333960 for the public repo.
+I ran into some strangeness when trying to count users for major OS’s. +Specifically, my queries consistently showed more Linux users than Mac users +(example query). +However, if we take the exact same data and look at users per day we show the opposite trend: +more Mac than Linux users every day (query).
+It turns out the root of this problem is client_id churn.
+The queries showing more users on Linux than Darwin
+state that we’ve seen more Linux client_id‘s than we have Darwin client_id‘s over time.
+But, what if a large portion of those Linux client_id‘s haven’t been active for months?
Consider this graph showing the most recent ping for each Linux and Mac client_id.
+There are many more stale Linux client_id‘s.
+If it’s hard to see look at this graph for a clearer image based off of the same data.
In short, consider your time window when trying to count users with client_ids.
+client_id churn is a growing problem as you expand your window.
This is primarily a summary of Bug 1333960 for the public repo.
+I ran into some strangeness when trying to count users for major OS’s. +Specifically, my queries consistently showed more Linux users than Mac users +(example query). +However, if we take the exact same data and look at users per day we show the opposite trend: +more Mac than Linux users every day (query).
+It turns out the root of this problem is client_id churn.
+The queries showing more users on Linux than Darwin
+state that we’ve seen more Linux client_id‘s than we have Darwin client_id‘s over time.
+But, what if a large portion of those Linux client_id‘s haven’t been active for months?
Consider this graph showing the most recent ping for each Linux and Mac client_id.
+There are many more stale Linux client_id‘s.
+If it’s hard to see look at this graph for a clearer image based off of the same data.
In short, consider your time window when trying to count users with client_ids.
+client_id churn is a growing problem as you expand your window.
This is some analysis to clarify some stuff I wrote in a blog post.
+Specifically investigating what typical values of “recording delay” and “submission delay” might be.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Unable to parse whitelist (/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
Start with the bread-and-butter of Telemetry reporting: the “main” ping
+Looking at Jan 10, 2017 because it’s a recent Tuesday that isn’t too close to any holidays.
+pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(submissionDate="20170110") \ + .records(sc, sample=0.01) +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+creationDate - The time the Telemetry code in Firefox created the ping, according to the client’s clock, expressed as an ISO string. meta/creationTimestamp is the same time, but expressed in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
payload/info/subsessionLength - The length of time over which the ping was collecting data, according to the client’s clock, expressed as a number of seconds. Working backwards from creationDate this gives us a subsession start time which allows us to determine Reporting Delay.
subset = get_pings_properties(pings, ["application/channel", + "creationDate", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "payload/info/subsessionLength"]) +
p = subset.take(1)[0] +
p
+{'application/channel': u'release',
+ 'creationDate': u'2017-01-10T02:01:31.551Z',
+ 'meta/Date': u'Tue, 10 Jan 2017 02:01:31 GMT',
+ 'meta/Timestamp': 1484013691737682688L,
+ 'meta/creationTimestamp': 1.484013691551e+18,
+ 'payload/info/subsessionLength': 67}
+Quick normalization: ditch any ping that doesn’t have a subsessionLength, creationTimestamp, or Timestamp:
+subset = subset.filter(lambda p:\ + p["payload/info/subsessionLength"] is not None\ + and p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +
We’ll be plotting Cumulative Distribution Functions today.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +CHANNELS = ['release', 'beta', 'aurora', 'nightly'] +
def setup_plot(title, max_x): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 10.0, range(0, 11, 1))) + + plt.ylim(0.0, 1.0) + plt.xlim(0.0, max_x) + + plt.grid(True) + +def plot_cdf(data): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys) +
def calculate_delays(p): + reporting_delay = p["payload/info/subsessionLength"] + + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + submission_delay = (received - created - clock_skew).total_seconds() + return (reporting_delay, submission_delay) +
delays_by_chan = subset.map(lambda p: (p["application/channel"], calculate_delays(p))) +
Recording Delay is the time from when the data “happens” to the time we record it in a ping.
+The maximum value for this is the subsessionLength: the length of time from the beginning of the interval over which this ping is reporting to the end, where the ping is recorded.
setup_plot("Recording Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: d[1][0] / HOUR_IN_S if d[1][0] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="best") +
<matplotlib.legend.Legend at 0x7fa3940834d0> +
So it seems as though about 80% of recording delays on release are 2 hours or less, not much difference amongst the channels (though it is interesting that Aurora has the longer subsessions).
+Note the cliff at 24 hours. We have code that tries to ensure that our data is recorded at least every day, around local midnight. Nice to see that it appears to be working.
+Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+Here we run into a problem with clock skew. Clients’ clocks aren’t guaranteed to align with our server’s clock, so we cannot necessarily compare the two. Luckily, with bug 1144778 we introduced an HTTP Date header which tells us what time the client’s clock thinks it is when it is sending the data. Coupled with the Timestamp field recorded which is what time the server’s clock thinks it is when it receives the data, we can subtract the more egregious examples of clock skew and get values that are closer to reality.
setup_plot("Submission Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: d[1][1] / HOUR_IN_S if d[1][1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="lower right") +
<matplotlib.legend.Legend at 0x7fa38761a910> +
Here we see a much larger variation in delays across the channels. Nightly, as you could expect, tends to have the shortest delays. I suspect this is because its rapid update cycle tends to encourage users to restart more often. Beta is a bit of a surprise for me as having the longest delays.
+Maybe Beta users use their browsers less than other channels? But then I’d expect them to be bottom of the pile for engagement ratio, and they’re more middle of the road.
+Something to follow up on, maybe.
+Anyhoo, we get 80% of main pings from nightly within 6 hours of them being created by the client. Which is pretty awesome. From beta, we have to wait a little over 24 hours to get 80% of main pings.
+If we’re waiting for 90% the spread’s even greater with nightly getting 9 out of every 10 pings before 18 hours is up, and beta having to wait more than 96 hours.
+And, summing the delays together and graphing them we get…
+setup_plot("Combined Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: (d[1][0] + d[1][1]) / HOUR_IN_S if (d[1][0] + d[1][1]) < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="lower right") +
<matplotlib.legend.Legend at 0x7fa38743dad0> +
The 80% numbers for the combined delay again have Nightly being speediest at just over 10 hours. Beta is once again the laggiest at 27 hours.
+This is some analysis to clarify some stuff I wrote in a blog post.
+Specifically investigating what typical values of “recording delay” and “submission delay” might be.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
Unable to parse whitelist (/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
Start with the bread-and-butter of Telemetry reporting: the “main” ping
+Looking at Jan 10, 2017 because it’s a recent Tuesday that isn’t too close to any holidays.
+pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(submissionDate="20170110") \ + .records(sc, sample=0.01) +
To look at delays, we need to look at times. There are a lot of times, and they are recorded relative to different clocks.
+creationDate - The time the Telemetry code in Firefox created the ping, according to the client’s clock, expressed as an ISO string. meta/creationTimestamp is the same time, but expressed in nanoseconds since the epoch.
meta/Date - The time the Telemetry code in Firefox sent the ping to the server, according to the client’s clock, expressed as a Date string conforming to RFC 7231.
meta/Timestamp - The time the ping was received by the server, according to the server’s
+clock, expressed in nanoseconds since the epoch.
payload/info/subsessionLength - The length of time over which the ping was collecting data, according to the client’s clock, expressed as a number of seconds. Working backwards from creationDate this gives us a subsession start time which allows us to determine Reporting Delay.
subset = get_pings_properties(pings, ["application/channel", + "creationDate", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "payload/info/subsessionLength"]) +
p = subset.take(1)[0] +
p
+{'application/channel': u'release',
+ 'creationDate': u'2017-01-10T02:01:31.551Z',
+ 'meta/Date': u'Tue, 10 Jan 2017 02:01:31 GMT',
+ 'meta/Timestamp': 1484013691737682688L,
+ 'meta/creationTimestamp': 1.484013691551e+18,
+ 'payload/info/subsessionLength': 67}
+Quick normalization: ditch any ping that doesn’t have a subsessionLength, creationTimestamp, or Timestamp:
+subset = subset.filter(lambda p:\ + p["payload/info/subsessionLength"] is not None\ + and p["meta/Timestamp"] is not None\ + and p["meta/creationTimestamp"] is not None) +
We’ll be plotting Cumulative Distribution Functions today.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 +CHANNELS = ['release', 'beta', 'aurora', 'nightly'] +
def setup_plot(title, max_x): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 10.0, range(0, 11, 1))) + + plt.ylim(0.0, 1.0) + plt.xlim(0.0, max_x) + + plt.grid(True) + +def plot_cdf(data): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys) +
def calculate_delays(p): + reporting_delay = p["payload/info/subsessionLength"] + + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + submission_delay = (received - created - clock_skew).total_seconds() + return (reporting_delay, submission_delay) +
delays_by_chan = subset.map(lambda p: (p["application/channel"], calculate_delays(p))) +
Recording Delay is the time from when the data “happens” to the time we record it in a ping.
+The maximum value for this is the subsessionLength: the length of time from the beginning of the interval over which this ping is reporting to the end, where the ping is recorded.
setup_plot("Recording Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: d[1][0] / HOUR_IN_S if d[1][0] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="best") +
<matplotlib.legend.Legend at 0x7fa3940834d0> +
So it seems as though about 80% of recording delays on release are 2 hours or less, not much difference amongst the channels (though it is interesting that Aurora has the longer subsessions).
+Note the cliff at 24 hours. We have code that tries to ensure that our data is recorded at least every day, around local midnight. Nice to see that it appears to be working.
+Submission Delay is the delay between the data being recorded on the client and it being received by our infrastructure. It is thought to be dominated by the length of time Firefox isn’t open on a client’s computer, though retransmission attempts and throttling can also contribute.
+Here we run into a problem with clock skew. Clients’ clocks aren’t guaranteed to align with our server’s clock, so we cannot necessarily compare the two. Luckily, with bug 1144778 we introduced an HTTP Date header which tells us what time the client’s clock thinks it is when it is sending the data. Coupled with the Timestamp field recorded which is what time the server’s clock thinks it is when it receives the data, we can subtract the more egregious examples of clock skew and get values that are closer to reality.
setup_plot("Submission Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: d[1][1] / HOUR_IN_S if d[1][1] < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="lower right") +
<matplotlib.legend.Legend at 0x7fa38761a910> +
Here we see a much larger variation in delays across the channels. Nightly, as you could expect, tends to have the shortest delays. I suspect this is because its rapid update cycle tends to encourage users to restart more often. Beta is a bit of a surprise for me as having the longest delays.
+Maybe Beta users use their browsers less than other channels? But then I’d expect them to be bottom of the pile for engagement ratio, and they’re more middle of the road.
+Something to follow up on, maybe.
+Anyhoo, we get 80% of main pings from nightly within 6 hours of them being created by the client. Which is pretty awesome. From beta, we have to wait a little over 24 hours to get 80% of main pings.
+If we’re waiting for 90% the spread’s even greater with nightly getting 9 out of every 10 pings before 18 hours is up, and beta having to wait more than 96 hours.
+And, summing the delays together and graphing them we get…
+setup_plot("Combined Delay CDF", MAX_DELAY_S / HOUR_IN_S) + +for chan in CHANNELS: + plot_cdf(delays_by_chan\ + .filter(lambda d: d[0] == chan)\ + .map(lambda d: (d[1][0] + d[1][1]) / HOUR_IN_S if (d[1][0] + d[1][1]) < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect()) + +plt.legend(CHANNELS, loc="lower right") +
<matplotlib.legend.Legend at 0x7fa38743dad0> +
The 80% numbers for the combined delay again have Nightly being speediest at just over 10 hours. Beta is once again the laggiest at 27 hours.
+There is one particular client, whose client_id I’ve obscured, that seems to be sending orders of magnitude more “main” pings per day than is expected, or even possible.
I’m interested in figuring out what we can determine about this particular client to see if there are signifiers we can use to identify this anomalous use case. This identification would permit us to: + filter data from these clients out of derived datasets that aren’t relevant + identify exceptional use-cases for Firefox we don’t currently understand
+import pandas as pd +import numpy as np +import matplotlib + +from matplotlib import pyplot as plt +from moztelemetry.dataset import Dataset +from moztelemetry import get_pings_properties, get_one_ping_per_client +
Unable to parse whitelist (/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
all_pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appBuildId=lambda x: x.startswith("20161014")) \ + .where(appUpdateChannel="aurora") \ + .records(sc, sample=1) +
pings = all_pings.filter(lambda p: p['clientId'] == '<omitted for privacy>') +
submission_dates = get_pings_properties(pings, ["meta/submissionDate"]) +
from datetime import datetime +ping_counts = submission_dates.map(lambda p: (datetime.strptime(p["meta/submissionDate"], '%Y%m%d'), 1)).countByKey() +
from datetime import timedelta +
df = pd.DataFrame(ping_counts.items(), columns=["date", "count"]).set_index(["date"]) +df.plot(figsize=(17, 7)) +plt.xticks(np.arange(min(df.index), max(df.index) + timedelta(3), 3, dtype="datetime64[D]")) +plt.ylabel("ping count") +plt.xlabel("date") +plt.grid(True) +plt.show() +
Just about 100k main pings submitted by this client on a single day? (Feb 16)… that is one active client.
+Or many active clients.
+Well, since these pings all share the same clientId, they likely are sharing user profiles. This means things like profile creationDate and so forth won’t change amongst them.
However, here’s a list of things that might change in interesting ways or otherwise shed some light on the purpose of these installs.
+subset = get_pings_properties(pings, [ + "meta/geoCountry", + "meta/geoCity", + "environment/addons/activeAddons", + "environment/settings/isDefaultBrowser", + "environment/system/cpu/speedMHz", + "environment/system/os/name", + "environment/system/os/version", + "payload/info/sessionLength", + "payload/info/subsessionLength", + ]) +
subset.count() +
4571188 +
pings_with_addon = subset\ + .flatMap(lambda p: [(addon["name"], 1) for addon in filter(lambda x: "isSystem" not in x or not x["isSystem"], p["environment/addons/activeAddons"].values())])\ + .countByKey() +
sorted(pings_with_addon.items(), key=lambda x: x[1], reverse=True)[:5] +
[(u'Random Agent Spoofer', 4570618), + (u'Alexa Traffic Rank', 419985), + (u'Firefox Hotfix', 1)] +
Nearly every single ping is reporting that it has an addon called ‘Random Agent Spoofer’. Interesting.
+SESSION_MAX = 400 +
session_lengths = subset.map(lambda p: p["payload/info/sessionLength"] if p["payload/info/sessionLength"] < SESSION_MAX else SESSION_MAX).collect() +
pd.Series(session_lengths).hist(bins=250, figsize=(17, 7)) +plt.ylabel("ping count") +plt.xlabel("session length in seconds") +plt.show() +
pd.Series(session_lengths).value_counts()[:10] +
215 2756799 +135 417410 +284 273834 +27 258250 +40 257293 +64 172439 +85 160477 +25 160317 +62 30421 +63 27640 +dtype: int64 +
The session lengths for over half of all the reported pings are exactly 215 seconds long. Two minutes and 35 seconds.
+subset.map(lambda p: (p["environment/settings/isDefaultBrowser"], 1)).countByKey() +
defaultdict(int, {False: 4571188})
+No.
+MHZ_MAX = 5000 +
mhzes = subset.map(lambda p: p["environment/system/cpu/speedMHz"] if p["environment/system/cpu/speedMHz"] < MHZ_MAX else MHZ_MAX).collect() +
ds = pd.Series(mhzes) +ds.hist(bins=250, figsize=(17, 7)) +plt.ylabel("ping count (log)") +plt.xlabel("speed in MHz") +plt.yscale("log") +plt.show() +
pd.Series(mhzes).value_counts()[:10] +
3504 2796444 +2397 973539 +3503 506650 +2097 274870 +2400 4225 +2399 3324 +3495 3284 +2396 1962 +2600 1907 +2599 1540 +dtype: int64 +
There seems to be a family gathering of different hardware configurations this client is running on, most on a particular approximately-3.5GHz machine
+def major_minor(version_string): + return version_string.split('.')[0] + '.' + version_string.split('.')[1] +
pings_per_os = subset\ + .map(lambda p: (p["environment/system/os/name"] + " " + major_minor(p["environment/system/os/version"]), 1))\ + .countByKey() +
print len(pings_per_os) +sorted(pings_per_os.items(), key=lambda x: x[1], reverse=True)[:10] +
1 + + + + + + +[(u'Windows_NT 5.1', 4571188)] +
All of the pings come from Windows XP.
+pings_per_city = subset\ + .map(lambda p: (p["meta/geoCountry"] + " " + p["meta/geoCity"], 1))\ + .countByKey() +
print len(pings_per_city) +sorted(pings_per_city.items(), key=lambda x: x[1], reverse=True)[:10] +
418 + + + + + + +[(u'US Costa Mesa', 599161), + (u'US Phoenix', 449236), + (u'FR Paris', 245990), + (u'GB ??', 234012), + (u'GB London', 187256), + (u'FR ??', 183938), + (u'DE ??', 144906), + (u'US Los Angeles', 122247), + (u'US Houston', 97413), + (u'US New York', 93148)] +
These pings are coming from all over the world, mostly from countries where Firefox user share is already decent. This may just be a map of Browser use across the world’s population, which would be consistent with a profile that is inhabiting a set %ge of the browser-using population’s computers.
+None of this is concrete, but if I were invited to speculate, I’d think there’s some non-Mozilla code someplace that has embedded a particular (out-of-date) version of Firefox Developer Edition into themselves, automating it to perform a 2-minute-and-35-second task on Windows XP machines, possibly while masquerading as something completely different (using the addon).
+This could be legitimate. Firefox contains a robust networking and rendering stack so it might be desireable to embed it within, say, a video game as a fully-featured embedded browser. The user-agent-spoofing addon could very well be used to set a custom user agent to identify the video game’s browser, and of course it wouldn’t be the user’s default browser.
+However, I can’t so easily explain this client’s broad geographical presence and Windows XP focus.
+There is one particular client, whose client_id I’ve obscured, that seems to be sending orders of magnitude more “main” pings per day than is expected, or even possible.
I’m interested in figuring out what we can determine about this particular client to see if there are signifiers we can use to identify this anomalous use case. This identification would permit us to: + filter data from these clients out of derived datasets that aren’t relevant + identify exceptional use-cases for Firefox we don’t currently understand
+import pandas as pd +import numpy as np +import matplotlib + +from matplotlib import pyplot as plt +from moztelemetry.dataset import Dataset +from moztelemetry import get_pings_properties, get_one_ping_per_client +
Unable to parse whitelist (/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
all_pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appBuildId=lambda x: x.startswith("20161014")) \ + .where(appUpdateChannel="aurora") \ + .records(sc, sample=1) +
pings = all_pings.filter(lambda p: p['clientId'] == '<omitted for privacy>') +
submission_dates = get_pings_properties(pings, ["meta/submissionDate"]) +
from datetime import datetime +ping_counts = submission_dates.map(lambda p: (datetime.strptime(p["meta/submissionDate"], '%Y%m%d'), 1)).countByKey() +
from datetime import timedelta +
df = pd.DataFrame(ping_counts.items(), columns=["date", "count"]).set_index(["date"]) +df.plot(figsize=(17, 7)) +plt.xticks(np.arange(min(df.index), max(df.index) + timedelta(3), 3, dtype="datetime64[D]")) +plt.ylabel("ping count") +plt.xlabel("date") +plt.grid(True) +plt.show() +
Just about 100k main pings submitted by this client on a single day? (Feb 16)… that is one active client.
+Or many active clients.
+Well, since these pings all share the same clientId, they likely are sharing user profiles. This means things like profile creationDate and so forth won’t change amongst them.
However, here’s a list of things that might change in interesting ways or otherwise shed some light on the purpose of these installs.
+subset = get_pings_properties(pings, [ + "meta/geoCountry", + "meta/geoCity", + "environment/addons/activeAddons", + "environment/settings/isDefaultBrowser", + "environment/system/cpu/speedMHz", + "environment/system/os/name", + "environment/system/os/version", + "payload/info/sessionLength", + "payload/info/subsessionLength", + ]) +
subset.count() +
4571188 +
pings_with_addon = subset\ + .flatMap(lambda p: [(addon["name"], 1) for addon in filter(lambda x: "isSystem" not in x or not x["isSystem"], p["environment/addons/activeAddons"].values())])\ + .countByKey() +
sorted(pings_with_addon.items(), key=lambda x: x[1], reverse=True)[:5] +
[(u'Random Agent Spoofer', 4570618), + (u'Alexa Traffic Rank', 419985), + (u'Firefox Hotfix', 1)] +
Nearly every single ping is reporting that it has an addon called ‘Random Agent Spoofer’. Interesting.
+SESSION_MAX = 400 +
session_lengths = subset.map(lambda p: p["payload/info/sessionLength"] if p["payload/info/sessionLength"] < SESSION_MAX else SESSION_MAX).collect() +
pd.Series(session_lengths).hist(bins=250, figsize=(17, 7)) +plt.ylabel("ping count") +plt.xlabel("session length in seconds") +plt.show() +
pd.Series(session_lengths).value_counts()[:10] +
215 2756799 +135 417410 +284 273834 +27 258250 +40 257293 +64 172439 +85 160477 +25 160317 +62 30421 +63 27640 +dtype: int64 +
The session lengths for over half of all the reported pings are exactly 215 seconds long. Two minutes and 35 seconds.
+subset.map(lambda p: (p["environment/settings/isDefaultBrowser"], 1)).countByKey() +
defaultdict(int, {False: 4571188})
+No.
+MHZ_MAX = 5000 +
mhzes = subset.map(lambda p: p["environment/system/cpu/speedMHz"] if p["environment/system/cpu/speedMHz"] < MHZ_MAX else MHZ_MAX).collect() +
ds = pd.Series(mhzes) +ds.hist(bins=250, figsize=(17, 7)) +plt.ylabel("ping count (log)") +plt.xlabel("speed in MHz") +plt.yscale("log") +plt.show() +
pd.Series(mhzes).value_counts()[:10] +
3504 2796444 +2397 973539 +3503 506650 +2097 274870 +2400 4225 +2399 3324 +3495 3284 +2396 1962 +2600 1907 +2599 1540 +dtype: int64 +
There seems to be a family gathering of different hardware configurations this client is running on, most on a particular approximately-3.5GHz machine
+def major_minor(version_string): + return version_string.split('.')[0] + '.' + version_string.split('.')[1] +
pings_per_os = subset\ + .map(lambda p: (p["environment/system/os/name"] + " " + major_minor(p["environment/system/os/version"]), 1))\ + .countByKey() +
print len(pings_per_os) +sorted(pings_per_os.items(), key=lambda x: x[1], reverse=True)[:10] +
1 + + + + + + +[(u'Windows_NT 5.1', 4571188)] +
All of the pings come from Windows XP.
+pings_per_city = subset\ + .map(lambda p: (p["meta/geoCountry"] + " " + p["meta/geoCity"], 1))\ + .countByKey() +
print len(pings_per_city) +sorted(pings_per_city.items(), key=lambda x: x[1], reverse=True)[:10] +
418 + + + + + + +[(u'US Costa Mesa', 599161), + (u'US Phoenix', 449236), + (u'FR Paris', 245990), + (u'GB ??', 234012), + (u'GB London', 187256), + (u'FR ??', 183938), + (u'DE ??', 144906), + (u'US Los Angeles', 122247), + (u'US Houston', 97413), + (u'US New York', 93148)] +
These pings are coming from all over the world, mostly from countries where Firefox user share is already decent. This may just be a map of Browser use across the world’s population, which would be consistent with a profile that is inhabiting a set %ge of the browser-using population’s computers.
+None of this is concrete, but if I were invited to speculate, I’d think there’s some non-Mozilla code someplace that has embedded a particular (out-of-date) version of Firefox Developer Edition into themselves, automating it to perform a 2-minute-and-35-second task on Windows XP machines, possibly while masquerading as something completely different (using the addon).
+This could be legitimate. Firefox contains a robust networking and rendering stack so it might be desireable to embed it within, say, a video game as a fully-featured embedded browser. The user-agent-spoofing addon could very well be used to set a custom user agent to identify the video game’s browser, and of course it wouldn’t be the user’s default browser.
+However, I can’t so easily explain this client’s broad geographical presence and Windows XP focus.
+As previously examined there is a large volume of pings coming from a single client_id
It’s gotten much worse since then, so more investigation is needed.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
from datetime import datetime, timedelta +
pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appBuildId="20161014004013")\ + .where(appUpdateChannel="aurora") \ + .records(sc, sample=1) +
pings = pings.filter(lambda p: p["clientId"] == "omitted") +
From last time, we know there was a roughly-increasing number of pings we were receiving from this client day over day. How has this changed?
+Since the first analysis we’ve noticed that a lot of these pings are duplicates: they have the same id and everything! So while we’re here let’s skip the dupes and graph the volume of pings, counting only the first time we saw them.
+submission_dates = get_pings_properties(pings, ["id", "meta/submissionDate"]) +
ping_counts = submission_dates.map(lambda p: (datetime.strptime(p["meta/submissionDate"], '%Y%m%d'), 1)).countByKey() +
first_ping_counts = submission_dates\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: a if a["meta/submissionDate"] < b["meta/submissionDate"] else b)\ + .map(lambda p: (datetime.strptime(p[1]["meta/submissionDate"], '%Y%m%d'), 1))\ + .countByKey() +
df = pd.DataFrame(ping_counts.items(), columns=["date", "count (including duplicates)"]).set_index(["date"]) +ax = df.plot(figsize=(17, 7)) +df2 = pd.DataFrame(first_ping_counts.items(), columns=["date", "unique count"]).set_index(["date"]) +df2.plot(ax=ax) +plt.xticks(np.arange(min(df.index), max(df.index) + timedelta(3), 5, dtype="datetime64[D]")) +plt.ylabel("ping count") +plt.xlabel("date") +plt.grid(True) +plt.show() +
We are seeing an ever-increasing volume of duplicate pings. Very few are unique.
+Operating on the assumption that the duplicate pings were distributed along with the Aurora 51 binaries and profile directories, that means only the unique pings have the chance to contain accurate information about the clients.
+Yes, this means the majority of the subsession and platform analysis from the previous report is likely bunk as it didn’t filter out the duplicate pings. (To be fair, no one at the time knew the pings were duplicate and likely distributed with the binaries)
+So, looking at only the non-duplicate pings, what can we see?
+subset = get_pings_properties(pings, [ + "id", + "meta/geoCountry", + "meta/geoCity", + "environment/addons/activeAddons", + "environment/settings/isDefaultBrowser", + "environment/system/cpu/speedMHz", + "environment/system/os/name", + "environment/system/os/version", + "payload/info/sessionLength", + "payload/info/subsessionLength", + ]) +
unique_pings = subset\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: "dupe")\ + .map(lambda p: p[1])\ + .filter(lambda p: p != "dupe") +
unique_pings.count() +
312266 +
pings_with_addon = unique_pings\ + .flatMap(lambda p: [(addon["name"], 1) for addon in filter(lambda x: "isSystem" not in x or not x["isSystem"], p["environment/addons/activeAddons"].values())])\ + .countByKey() +
sorted(pings_with_addon.items(), key=lambda x: x[1], reverse=True)[:5] +
[(u'Random Agent Spoofer', 311019), + (u'Alexa Traffic Rank', 37439), + (u'RefControl', 18358), + (u'StopTube', 4831), + (u'Stop YouTube Autoplay', 2)] +
As before, every ping shows Random Agent Spoofer.
+SESSION_MAX = 250 +
session_lengths = unique_pings\ + .map(lambda p: p["payload/info/sessionLength"] if p["payload/info/sessionLength"] < SESSION_MAX else SESSION_MAX)\ + .collect() +
s = pd.Series(session_lengths) +s.hist(bins=250, figsize=(17, 7)) +plt.ylabel("ping count") +plt.xlabel("session length in seconds") +plt.xticks(np.arange(0, max(s) + 1, 5)) +plt.show() +
Not sure how this compares to “normal” session lengths on Aurora. But a peak around 1m and another around 1m15s is way too short for a human to be getting anything meaningful done.
+unique_pings.map(lambda p: (p["environment/settings/isDefaultBrowser"], 1)).countByKey() +
defaultdict(int, {None: 1, False: 312265})
+Nope. (Note that this may mean nothing at all. Some platforms (lookin’ at you, Windows 10) make it difficult to set your default browser. Also: even without it being default it can be used if the user’s workflow is to always start by opening their browser.)
+MHZ_MAX = 5000 +
mhzes = unique_pings\ + .map(lambda p: p["environment/system/cpu/speedMHz"] if p["environment/system/cpu/speedMHz"] < MHZ_MAX else MHZ_MAX)\ + .collect() +
ds = pd.Series(mhzes) +ds.hist(bins=250, figsize=(17, 7)) +plt.ylabel("ping count (log)") +plt.xlabel("speed in MHz") +plt.yscale("log") +plt.show() +
ds.value_counts()[:10] +
2400 84819 +2399 73546 +2397 44691 +3504 44401 +3503 41217 +3495 3284 +3700 2141 +2396 2108 +2299 2059 +2398 1934 +dtype: int64 +
Nowhere near as monocultural as the original analysis suspected, but still not a very broad spread.
+def major_minor(version_string): + return version_string.split('.')[0] + '.' + version_string.split('.')[1] +
pings_per_os = unique_pings\ + .map(lambda p: (p["environment/system/os/name"] + " " + major_minor(p["environment/system/os/version"]), 1))\ + .countByKey() +
print len(pings_per_os) +sorted(pings_per_os.items(), key=lambda x: x[1], reverse=True)[:10] +
1 + + + + + + +[(u'Windows_NT 5.1', 312266)] +
100% Windows XP. I should be more surprised.
+pings_per_city = unique_pings\ + .map(lambda p: (p["meta/geoCountry"] + " " + p["meta/geoCity"], 1))\ + .countByKey() +
print len(pings_per_city) +sorted(pings_per_city.items(), key=lambda x: x[1], reverse=True)[:10] +
458 + + + + + + +[(u'CA Montr\xe9al', 100792), + (u'FR ??', 62789), + (u'US Costa Mesa', 19677), + (u'US Phoenix', 14828), + (u'FR Paris', 8888), + (u'GB ??', 7155), + (u'DE ??', 5210), + (u'GB London', 5190), + (u'NZ Auckland', 5065), + (u'US Los Angeles', 4072)] +
As expected, this is little different from the analysis over the duplicated pings. The hosts submitting even the duplicated pings have come from the world over.
+That being said, Montreal’s very highly represented here. It’s a lovely city and all, but what conditions might result in its unexpected prominence?
+Well, we now know that the overwhelming majority of pings sent by this client are geographically diverse and duplicated. This suggests that these instances were packaged up with their profile data directory intact, and are submitting a pre-packaged ping as soon as they come online.
+This means the duplicated pings likely reflect the environment of the original host where the package was created: Windows XP, 3.5GHz CPU, 2m35s-long session, with Random Agent Spoofer installed.
+These pings reflect likely reflect the hosts that are actually sending the pings: geographically diverse, but still running Windows XP while spending inhumanly-short times within the browser.
+We still don’t know why this is happening. We do know that the number of initial duplicate pings we’re receiving is continuing to increase. I see no sign of this ever stopping.
+As previously examined there is a large volume of pings coming from a single client_id
It’s gotten much worse since then, so more investigation is needed.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
from datetime import datetime, timedelta +
pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appBuildId="20161014004013")\ + .where(appUpdateChannel="aurora") \ + .records(sc, sample=1) +
pings = pings.filter(lambda p: p["clientId"] == "omitted") +
From last time, we know there was a roughly-increasing number of pings we were receiving from this client day over day. How has this changed?
+Since the first analysis we’ve noticed that a lot of these pings are duplicates: they have the same id and everything! So while we’re here let’s skip the dupes and graph the volume of pings, counting only the first time we saw them.
+submission_dates = get_pings_properties(pings, ["id", "meta/submissionDate"]) +
ping_counts = submission_dates.map(lambda p: (datetime.strptime(p["meta/submissionDate"], '%Y%m%d'), 1)).countByKey() +
first_ping_counts = submission_dates\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: a if a["meta/submissionDate"] < b["meta/submissionDate"] else b)\ + .map(lambda p: (datetime.strptime(p[1]["meta/submissionDate"], '%Y%m%d'), 1))\ + .countByKey() +
df = pd.DataFrame(ping_counts.items(), columns=["date", "count (including duplicates)"]).set_index(["date"]) +ax = df.plot(figsize=(17, 7)) +df2 = pd.DataFrame(first_ping_counts.items(), columns=["date", "unique count"]).set_index(["date"]) +df2.plot(ax=ax) +plt.xticks(np.arange(min(df.index), max(df.index) + timedelta(3), 5, dtype="datetime64[D]")) +plt.ylabel("ping count") +plt.xlabel("date") +plt.grid(True) +plt.show() +
We are seeing an ever-increasing volume of duplicate pings. Very few are unique.
+Operating on the assumption that the duplicate pings were distributed along with the Aurora 51 binaries and profile directories, that means only the unique pings have the chance to contain accurate information about the clients.
+Yes, this means the majority of the subsession and platform analysis from the previous report is likely bunk as it didn’t filter out the duplicate pings. (To be fair, no one at the time knew the pings were duplicate and likely distributed with the binaries)
+So, looking at only the non-duplicate pings, what can we see?
+subset = get_pings_properties(pings, [ + "id", + "meta/geoCountry", + "meta/geoCity", + "environment/addons/activeAddons", + "environment/settings/isDefaultBrowser", + "environment/system/cpu/speedMHz", + "environment/system/os/name", + "environment/system/os/version", + "payload/info/sessionLength", + "payload/info/subsessionLength", + ]) +
unique_pings = subset\ + .map(lambda p: (p["id"], p))\ + .reduceByKey(lambda a, b: "dupe")\ + .map(lambda p: p[1])\ + .filter(lambda p: p != "dupe") +
unique_pings.count() +
312266 +
pings_with_addon = unique_pings\ + .flatMap(lambda p: [(addon["name"], 1) for addon in filter(lambda x: "isSystem" not in x or not x["isSystem"], p["environment/addons/activeAddons"].values())])\ + .countByKey() +
sorted(pings_with_addon.items(), key=lambda x: x[1], reverse=True)[:5] +
[(u'Random Agent Spoofer', 311019), + (u'Alexa Traffic Rank', 37439), + (u'RefControl', 18358), + (u'StopTube', 4831), + (u'Stop YouTube Autoplay', 2)] +
As before, every ping shows Random Agent Spoofer.
+SESSION_MAX = 250 +
session_lengths = unique_pings\ + .map(lambda p: p["payload/info/sessionLength"] if p["payload/info/sessionLength"] < SESSION_MAX else SESSION_MAX)\ + .collect() +
s = pd.Series(session_lengths) +s.hist(bins=250, figsize=(17, 7)) +plt.ylabel("ping count") +plt.xlabel("session length in seconds") +plt.xticks(np.arange(0, max(s) + 1, 5)) +plt.show() +
Not sure how this compares to “normal” session lengths on Aurora. But a peak around 1m and another around 1m15s is way too short for a human to be getting anything meaningful done.
+unique_pings.map(lambda p: (p["environment/settings/isDefaultBrowser"], 1)).countByKey() +
defaultdict(int, {None: 1, False: 312265})
+Nope. (Note that this may mean nothing at all. Some platforms (lookin’ at you, Windows 10) make it difficult to set your default browser. Also: even without it being default it can be used if the user’s workflow is to always start by opening their browser.)
+MHZ_MAX = 5000 +
mhzes = unique_pings\ + .map(lambda p: p["environment/system/cpu/speedMHz"] if p["environment/system/cpu/speedMHz"] < MHZ_MAX else MHZ_MAX)\ + .collect() +
ds = pd.Series(mhzes) +ds.hist(bins=250, figsize=(17, 7)) +plt.ylabel("ping count (log)") +plt.xlabel("speed in MHz") +plt.yscale("log") +plt.show() +
ds.value_counts()[:10] +
2400 84819 +2399 73546 +2397 44691 +3504 44401 +3503 41217 +3495 3284 +3700 2141 +2396 2108 +2299 2059 +2398 1934 +dtype: int64 +
Nowhere near as monocultural as the original analysis suspected, but still not a very broad spread.
+def major_minor(version_string): + return version_string.split('.')[0] + '.' + version_string.split('.')[1] +
pings_per_os = unique_pings\ + .map(lambda p: (p["environment/system/os/name"] + " " + major_minor(p["environment/system/os/version"]), 1))\ + .countByKey() +
print len(pings_per_os) +sorted(pings_per_os.items(), key=lambda x: x[1], reverse=True)[:10] +
1 + + + + + + +[(u'Windows_NT 5.1', 312266)] +
100% Windows XP. I should be more surprised.
+pings_per_city = unique_pings\ + .map(lambda p: (p["meta/geoCountry"] + " " + p["meta/geoCity"], 1))\ + .countByKey() +
print len(pings_per_city) +sorted(pings_per_city.items(), key=lambda x: x[1], reverse=True)[:10] +
458 + + + + + + +[(u'CA Montr\xe9al', 100792), + (u'FR ??', 62789), + (u'US Costa Mesa', 19677), + (u'US Phoenix', 14828), + (u'FR Paris', 8888), + (u'GB ??', 7155), + (u'DE ??', 5210), + (u'GB London', 5190), + (u'NZ Auckland', 5065), + (u'US Los Angeles', 4072)] +
As expected, this is little different from the analysis over the duplicated pings. The hosts submitting even the duplicated pings have come from the world over.
+That being said, Montreal’s very highly represented here. It’s a lovely city and all, but what conditions might result in its unexpected prominence?
+Well, we now know that the overwhelming majority of pings sent by this client are geographically diverse and duplicated. This suggests that these instances were packaged up with their profile data directory intact, and are submitting a pre-packaged ping as soon as they come online.
+This means the duplicated pings likely reflect the environment of the original host where the package was created: Windows XP, 3.5GHz CPU, 2m35s-long session, with Random Agent Spoofer installed.
+These pings reflect likely reflect the hosts that are actually sending the pings: geographically diverse, but still running Windows XP while spending inhumanly-short times within the browser.
+We still don’t know why this is happening. We do know that the number of initial duplicate pings we’re receiving is continuing to increase. I see no sign of this ever stopping.
+Bug 1319026 introduced logs to try and nail down what kinds of failures users experience when trying to send Telemetry pings. Let’s see what we’ve managed to collect.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appUpdateChannel='nightly') \ + .where(submissionDate=lambda x: x >= "20170429") \ + .where(appBuildId=lambda x: x >= '20170429') \ + .records(sc, sample=1) +
subset = get_pings_properties(pings, ["clientId", + "environment/system/os/name", + "payload/log"]) +
log_entries = subset\ + .flatMap(lambda p: [] if p['payload/log'] is None else [l for l in p['payload/log'] if l[0] == 'TELEMETRY_SEND_FAILURE']) +
log_entries = log_entries.cache() +
error_counts = log_entries.map(lambda l: (tuple(l[2:]), 1)).countByKey() +
entries_count = log_entries.count() +sorted(map(lambda i: ('{:.2%}'.format(1.0 * i[-1] / entries_count), i), error_counts.iteritems()), key=lambda x: x[1][1], reverse=True) +
[('72.16%', ((u'errorhandler', u'error'), 530178)),
+ ('27.04%', ((u'errorhandler', u'timeout'), 198698)),
+ ('0.73%', ((u'5xx failure', u'504'), 5327)),
+ ('0.07%', ((u'errorhandler', u'abort'), 530)),
+ ('0.00%', ((u"4xx 'failure'", u'403'), 7)),
+ ('0.00%', ((u'5xx failure', u'502'), 3))]
+Alrighty, looks like we’re mostly “error”. Not too helpful, but does narrow things down a bit.
+“timeout” is the reason for more than one in every four failures. That’s a smaller cohort than I’d originally thought.
+A few Gateway Timeouts (504) which could be server load, very few aborts, and essentially no Forbidden (403) or Bad Gateway (502).
+Bug 1319026 introduced logs to try and nail down what kinds of failures users experience when trying to send Telemetry pings. Let’s see what we’ve managed to collect.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +
Unable to parse whitelist (/mnt/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(appUpdateChannel='nightly') \ + .where(submissionDate=lambda x: x >= "20170429") \ + .where(appBuildId=lambda x: x >= '20170429') \ + .records(sc, sample=1) +
subset = get_pings_properties(pings, ["clientId", + "environment/system/os/name", + "payload/log"]) +
log_entries = subset\ + .flatMap(lambda p: [] if p['payload/log'] is None else [l for l in p['payload/log'] if l[0] == 'TELEMETRY_SEND_FAILURE']) +
log_entries = log_entries.cache() +
error_counts = log_entries.map(lambda l: (tuple(l[2:]), 1)).countByKey() +
entries_count = log_entries.count() +sorted(map(lambda i: ('{:.2%}'.format(1.0 * i[-1] / entries_count), i), error_counts.iteritems()), key=lambda x: x[1][1], reverse=True) +
[('72.16%', ((u'errorhandler', u'error'), 530178)),
+ ('27.04%', ((u'errorhandler', u'timeout'), 198698)),
+ ('0.73%', ((u'5xx failure', u'504'), 5327)),
+ ('0.07%', ((u'errorhandler', u'abort'), 530)),
+ ('0.00%', ((u"4xx 'failure'", u'403'), 7)),
+ ('0.00%', ((u'5xx failure', u'502'), 3))]
+Alrighty, looks like we’re mostly “error”. Not too helpful, but does narrow things down a bit.
+“timeout” is the reason for more than one in every four failures. That’s a smaller cohort than I’d originally thought.
+A few Gateway Timeouts (504) which could be server load, very few aborts, and essentially no Forbidden (403) or Bad Gateway (502).
+reason = ready)This analysis validates the update ping with reason = ready, which was introduced in bug 1120372 and should be sent every time an update is downloaded and ready to be applied. We verified that the ping is behaving correctly on Nightly, and we’re going to perform similar validations for Beta:
import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
The update ping landed on the Beta channel on the 8th of August, 2017 (Beta 1). The first update pings should have been sent when updating to Beta 2, which was released on the 11th of August 2017 according to the release calendar. However, with bug 1390175, the update ping started flowing to its own bucket on the 14th of August 2017. In order to have one week of data, fetch pings from the 11th to the 18th of August, 2017, both from the OTHER and the update buckets, then merge them.
BETA_BUILDID_MIN = "20170808000000" # 56.0b1 +BETA_BUILDID_MAX = "20170815999999" # 56.0b3 + +update_pings_other_bkt = Dataset.from_source("telemetry") \ + .where(docType="OTHER") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170811" <= x < "20170818") \ + .where(appBuildId=lambda x: BETA_BUILDID_MIN <= x < BETA_BUILDID_MAX) \ + .records(sc, sample=1.0) + +update_pings_update_bkt = Dataset.from_source("telemetry") \ + .where(docType="update") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170811" <= x < "20170818") \ + .where(appBuildId=lambda x: BETA_BUILDID_MIN <= x < BETA_BUILDID_MAX) \ + .records(sc, sample=1.0) +
fetching 265.33015MB in 1605 files... +fetching 248.69387MB in 819 files... +
# The 'OTHER' bucket is a miscellaneous bucket, it might contain stuff other than the update ping. +# Filter them out. +update_pings_other_bkt = update_pings_other_bkt.filter(lambda p: p.get("type") == "update") +# Then merge the pings from both buckets. +update_pings = update_pings_other_bkt.union(update_pings_update_bkt) +
def pct(a, b): + return 100.0 * a / b + +def dedupe(pings, duping_key): + return pings\ + .map(lambda p: (p[duping_key], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) +
Misc functions to plot the CDF of the submission delay.
+MAX_DELAY_S = 60 * 60 * 48.0 +HOUR_IN_S = 60 * 60.0 + +def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data, **kwargs): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys, **kwargs) + +def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
Check that the payload section contains the right entries with consistent values.
+subset = get_pings_properties(update_pings, ["id", + "clientId", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "application/buildId", + "application/channel", + "application/version", + "environment/system/os/name", + "payload/reason", + "payload/targetBuildId", + "payload/targetChannel", + "payload/targetVersion"]) + +ping_count = subset.count() +
Quantify the percentage of duplicate pings we’re receiving. We don’t expect this value to be greater than ~1%, which is the amount we usually get from main and crash: as a rule of thumb, we threat anything less than 1% as probably well behaving.
deduped_subset = dedupe(subset, "id") +deduped_count = deduped_subset.count() +print("Percentage of duplicate pings: {:.3f}".format(100.0 - pct(deduped_count, ping_count))) +
Percentage of duplicate pings: 0.271 +
The percentage of duplicate pings is within the expected range. Move on and verify the payload of the update pings.
def validate_update_payload(p): + PAYLOAD_KEYS = [ + "payload/reason", + "payload/targetBuildId", + "payload/targetChannel", + "payload/targetVersion" + ] + + # All the payload keys needs to be strings. + for k in PAYLOAD_KEYS: + if not isinstance(p.get(k), basestring): + return ("'{}' is not a string".format(k), 1) + + # We only expect "reason" = ready. + if p.get("payload/reason") != "ready": + return ("Unexpected 'reason' {}".format(p.get("payload/reason"), 1)) + + # For Beta, the target channel should be the same as the + # application channel. + if p.get("payload/targetChannel") != p.get("application/channel"): + return ("Target channel mismatch: expected {} got {}"\ + .format(p.get("payload/targetChannel"), p.get("application/channel")), 1) + + # The target buildId must be greater than the application build id. + if p.get("payload/targetBuildId") <= p.get("application/buildId"): + return ("Target buildId mismatch: {} must be more recent than {}"\ + .format(p.get("payload/targetBuildId"), p.get("application/buildId")), 1) + + return ("Ok", 1) + +validation_results = deduped_subset.map(validate_update_payload).countByKey() +for k, v in sorted(validation_results.iteritems()): + if "channel mismatch" not in k: + print("{}:\t{:.3f}%".format(k, pct(v, ping_count))) + +# We are not sure if we can or cannot disclose channel ratios. Let's be safe +# and aggregate them. +channel_mismatch_ratios = dict([(k,v) for k,v in validation_results.iteritems() if "channel mismatch" in k]) +total_channel_mismatch = pct(sum(channel_mismatch_ratios.values()), ping_count) +for k in channel_mismatch_ratios.keys(): + print("{}".format(k)) +print("\nTotal channel mismatch:\t{:.3f}%".format(total_channel_mismatch)) +
Ok: 99.454% +Target buildId mismatch: 20170808170225 must be more recent than 20170810180547: 0.002% +Target buildId mismatch: 20170810180547 must be more recent than 20170815141045: 0.004% +Target channel mismatch: expected beta-cck-mozilla14 got beta +Target channel mismatch: expected beta-cck-mozilla101 got beta +Target channel mismatch: expected beta-cck-mozilla-EMEfree got beta +Target channel mismatch: expected beta-cck-mozilla15 got beta +Target channel mismatch: expected beta-cck-yahooid got beta +Target channel mismatch: expected beta-cck-mozilla111 got beta +Target channel mismatch: expected beta-cck-mozilla19 got beta +Target channel mismatch: expected beta-cck-seznam got beta +Target channel mismatch: expected beta-cck-yahoomy got beta +Target channel mismatch: expected beta-cck-bing got beta +Target channel mismatch: expected beta-cck-mozillaonline got beta +Target channel mismatch: expected beta-cck-mozilla26 got beta +Target channel mismatch: expected beta-cck-mozilla12 got beta +Target channel mismatch: expected beta-cck-mozilla-ironsource-001 got beta +Target channel mismatch: expected beta-cck-yandex got beta +Target channel mismatch: expected beta-cck-yahoocfk got beta +Target channel mismatch: expected beta-cck-yahooca got beta +Target channel mismatch: expected beta-cck-aol got beta +Target channel mismatch: expected beta-cck-mozilla20 got beta +Target channel mismatch: expected beta-cck-euballot got beta + +Total channel mismatch: 0.269% +
The vast majority of the data in the payload seems reasonable (99.45%).
+However, a handful of update pings are reporting a targetBuildId which is older than the current build reported by the ping’s environment: this is unexpected, as the the target build id must be always greater than the current one. After discussing this with the update team, it seems like this could either be due to channel weirdness or to the customization applied by the CCK tool. Additionally, some pings are reporting a targetChannel different than the one in the environment: this is definitely due to the CCK tool, given the cck entry in the channel name. These issues do not represent a problem, as most of the data is correct and their volume is fairly low.
For each ping, build a key with the client id and the target update details. Since we expect to have exactly one ping for each update bundle marked as ready, we don’t expect duplicate keys.
+update_dupes = deduped_subset.map(lambda p: ((p.get("clientId"), + p.get("payload/targetChannel"), + p.get("payload/targetVersion"), + p.get("payload/targetBuildId")), 1)).countByKey() + +print("Percentage of pings related to the same update (for the same client):\t{:.3f}%"\ + .format(pct(sum([v for v in update_dupes.values() if v > 1]), deduped_count))) +
Percentage of pings related to the same update (for the same client): 3.598% +
We’re receiving update pings with different documentId related to the same target update bundle, for some clients. The 3.59% is slightly higher than the one we saw on Nightly, 1.74%. One possible reason for this could be users having multiple copies of Firefox installed on their machine. Let’s see if that’s the case.
clientIds_sending_dupes = [k[0] for k, v in update_dupes.iteritems() if v > 1] + +def check_same_original_build(ping_list): + # Build a "unique" identifier for the build by + # concatenating the buildId, channel and version. + unique_build_ids = [ + "{}{}{}".format(p.get("application/buildId"), p.get("application/channel"), p.get("application/version"))\ + for p in ping_list[1] + ] + + # Remove the duplicates and return True if all the pings came + # from the same build. + return len(set(unique_build_ids)) < 2 + +# Count how many duplicates come from the same builds and how many come from +# different original builds. +original_builds_same =\ + deduped_subset.filter(lambda p: p.get("clientId") in clientIds_sending_dupes)\ + .map(lambda p: ((p.get("clientId"), + p.get("payload/targetChannel"), + p.get("payload/targetVersion"), + p.get("payload/targetBuildId")), [p]))\ + .reduceByKey(lambda a, b: a + b)\ + .filter(lambda p: len(p[1]) > 1)\ + .map(check_same_original_build).countByValue() + +print("Original builds are identical:\t{:.3f}%"\ + .format(pct(original_builds_same.get(True), sum(original_builds_same.values())))) +print("Original builds are different:\t{:.3f}%"\ + .format(pct(original_builds_same.get(False), sum(original_builds_same.values())))) +
Original builds are identical: 97.293% +Original builds are different: 2.707% +
The data shows that the update pings with the same target version are not necessarily coming from the same profile being used on different Firefox builds/installation: most of them are, but not all of them. After discussing this with the update team, it turns out that this can be explained by updates failing to apply: for certain classes of errors, we download the update blob again and thus send a new update ping with the same target version. This problem shows up in the update orphaning dashboard as well but, unfortunately, it only reports Release data.
delays = deduped_subset.map(lambda p: calculate_submission_delay(p)) +
MAX_DELAY_S = 60 * 60 * 48.0 +setup_plot("'update' ('ready') ping submission delay CDF", + MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +plot_cdf(delays\ + .map(lambda d: d / HOUR_IN_S if d < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect(), label="CDF", linestyle="solid") + +plt.show() +
Almost all of the update ping are submitted within an hour from the update being ready.
update pings is reasonableThis is a tricky one. The update ping with reason = "ready" is sent as soon as an update package is downloaded, verified and deemed ready to be applied. However, nothing guarantees that the update is immediately (or ever) applied. To check if the volume of update pings is in the ballpark, we can:
main-ping for that version of Firefox.Step 1 - Get the list of client ids updating to build ‘20170810xxxxxx’
+TARGET_BUILDID_MIN = '20170810000000' +TARGET_BUILDID_MAX = '20170810999999' + +update_candidates =\ + deduped_subset.filter(lambda p: TARGET_BUILDID_MIN <= p.get("payload/targetBuildId") <= TARGET_BUILDID_MAX) +update_candidates_clientIds = dedupe(update_candidates, "clientId").map(lambda p: p.get("clientId")) +candidates_count = update_candidates_clientIds.count() +
Step 2 - Get the main-ping from that Beta build and extract the list of client ids.
updated_main_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170811" <= x < "20170818") \ + .where(appBuildId=lambda x: TARGET_BUILDID_MIN <= x <= TARGET_BUILDID_MAX) \ + .records(sc, sample=1) +
fetching 198117.80319MB in 1364 files... +
We just need the client ids and a few other fields to dedupe.
+subset_main = get_pings_properties(updated_main_pings, ["id", + "clientId", + "meta/Timestamp", + "application/buildId", + "application/channel", + "application/version"]) +
Start by deduping by document id. After that, only get a single ping per client and extract the list of client ids.
+deduped_main = dedupe(subset_main, "id") +updated_clientIds = dedupe(deduped_main, "clientId").map(lambda p: p.get("clientId")) +updated_count = updated_clientIds.count() +
Step 3 - Count how many clients that were meant to update actually updated in the following 7 days.
+matching_clientIds = update_candidates_clientIds.intersection(updated_clientIds) +matching_count = matching_clientIds.count() +
print("{:.3f}% of the clients that sent the update ping updated to the newer Beta build within a week."\ + .format(pct(matching_count, candidates_count))) +print("{:.3f}% of the clients that were seen on the newer Beta build sent an update ping."\ + .format(pct(candidates_count, updated_count))) +
91.445% of the clients that sent the update ping updated to the newer Beta build within a week. +98.819% of the clients that were seen on the newer Beta build sent an update ping. +
Roughly 98% of the clients that were seen in the new Beta build also sent the update ping. This is way higher than the 80% we saw on Nightly, but still not 100%. This could be due to a few reasons:
+update ping is sent in that case if an update is manually triggered;+
reason = ready)This analysis validates the update ping with reason = ready, which was introduced in bug 1120372 and should be sent every time an update is downloaded and ready to be applied. We verified that the ping is behaving correctly on Nightly, and we’re going to perform similar validations for Beta:
import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
The update ping landed on the Beta channel on the 8th of August, 2017 (Beta 1). The first update pings should have been sent when updating to Beta 2, which was released on the 11th of August 2017 according to the release calendar. However, with bug 1390175, the update ping started flowing to its own bucket on the 14th of August 2017. In order to have one week of data, fetch pings from the 11th to the 18th of August, 2017, both from the OTHER and the update buckets, then merge them.
BETA_BUILDID_MIN = "20170808000000" # 56.0b1 +BETA_BUILDID_MAX = "20170815999999" # 56.0b3 + +update_pings_other_bkt = Dataset.from_source("telemetry") \ + .where(docType="OTHER") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170811" <= x < "20170818") \ + .where(appBuildId=lambda x: BETA_BUILDID_MIN <= x < BETA_BUILDID_MAX) \ + .records(sc, sample=1.0) + +update_pings_update_bkt = Dataset.from_source("telemetry") \ + .where(docType="update") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170811" <= x < "20170818") \ + .where(appBuildId=lambda x: BETA_BUILDID_MIN <= x < BETA_BUILDID_MAX) \ + .records(sc, sample=1.0) +
fetching 265.33015MB in 1605 files... +fetching 248.69387MB in 819 files... +
# The 'OTHER' bucket is a miscellaneous bucket, it might contain stuff other than the update ping. +# Filter them out. +update_pings_other_bkt = update_pings_other_bkt.filter(lambda p: p.get("type") == "update") +# Then merge the pings from both buckets. +update_pings = update_pings_other_bkt.union(update_pings_update_bkt) +
def pct(a, b): + return 100.0 * a / b + +def dedupe(pings, duping_key): + return pings\ + .map(lambda p: (p[duping_key], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) +
Misc functions to plot the CDF of the submission delay.
+MAX_DELAY_S = 60 * 60 * 48.0 +HOUR_IN_S = 60 * 60.0 + +def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data, **kwargs): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys, **kwargs) + +def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
Check that the payload section contains the right entries with consistent values.
+subset = get_pings_properties(update_pings, ["id", + "clientId", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "application/buildId", + "application/channel", + "application/version", + "environment/system/os/name", + "payload/reason", + "payload/targetBuildId", + "payload/targetChannel", + "payload/targetVersion"]) + +ping_count = subset.count() +
Quantify the percentage of duplicate pings we’re receiving. We don’t expect this value to be greater than ~1%, which is the amount we usually get from main and crash: as a rule of thumb, we threat anything less than 1% as probably well behaving.
deduped_subset = dedupe(subset, "id") +deduped_count = deduped_subset.count() +print("Percentage of duplicate pings: {:.3f}".format(100.0 - pct(deduped_count, ping_count))) +
Percentage of duplicate pings: 0.271 +
The percentage of duplicate pings is within the expected range. Move on and verify the payload of the update pings.
def validate_update_payload(p): + PAYLOAD_KEYS = [ + "payload/reason", + "payload/targetBuildId", + "payload/targetChannel", + "payload/targetVersion" + ] + + # All the payload keys needs to be strings. + for k in PAYLOAD_KEYS: + if not isinstance(p.get(k), basestring): + return ("'{}' is not a string".format(k), 1) + + # We only expect "reason" = ready. + if p.get("payload/reason") != "ready": + return ("Unexpected 'reason' {}".format(p.get("payload/reason"), 1)) + + # For Beta, the target channel should be the same as the + # application channel. + if p.get("payload/targetChannel") != p.get("application/channel"): + return ("Target channel mismatch: expected {} got {}"\ + .format(p.get("payload/targetChannel"), p.get("application/channel")), 1) + + # The target buildId must be greater than the application build id. + if p.get("payload/targetBuildId") <= p.get("application/buildId"): + return ("Target buildId mismatch: {} must be more recent than {}"\ + .format(p.get("payload/targetBuildId"), p.get("application/buildId")), 1) + + return ("Ok", 1) + +validation_results = deduped_subset.map(validate_update_payload).countByKey() +for k, v in sorted(validation_results.iteritems()): + if "channel mismatch" not in k: + print("{}:\t{:.3f}%".format(k, pct(v, ping_count))) + +# We are not sure if we can or cannot disclose channel ratios. Let's be safe +# and aggregate them. +channel_mismatch_ratios = dict([(k,v) for k,v in validation_results.iteritems() if "channel mismatch" in k]) +total_channel_mismatch = pct(sum(channel_mismatch_ratios.values()), ping_count) +for k in channel_mismatch_ratios.keys(): + print("{}".format(k)) +print("\nTotal channel mismatch:\t{:.3f}%".format(total_channel_mismatch)) +
Ok: 99.454% +Target buildId mismatch: 20170808170225 must be more recent than 20170810180547: 0.002% +Target buildId mismatch: 20170810180547 must be more recent than 20170815141045: 0.004% +Target channel mismatch: expected beta-cck-mozilla14 got beta +Target channel mismatch: expected beta-cck-mozilla101 got beta +Target channel mismatch: expected beta-cck-mozilla-EMEfree got beta +Target channel mismatch: expected beta-cck-mozilla15 got beta +Target channel mismatch: expected beta-cck-yahooid got beta +Target channel mismatch: expected beta-cck-mozilla111 got beta +Target channel mismatch: expected beta-cck-mozilla19 got beta +Target channel mismatch: expected beta-cck-seznam got beta +Target channel mismatch: expected beta-cck-yahoomy got beta +Target channel mismatch: expected beta-cck-bing got beta +Target channel mismatch: expected beta-cck-mozillaonline got beta +Target channel mismatch: expected beta-cck-mozilla26 got beta +Target channel mismatch: expected beta-cck-mozilla12 got beta +Target channel mismatch: expected beta-cck-mozilla-ironsource-001 got beta +Target channel mismatch: expected beta-cck-yandex got beta +Target channel mismatch: expected beta-cck-yahoocfk got beta +Target channel mismatch: expected beta-cck-yahooca got beta +Target channel mismatch: expected beta-cck-aol got beta +Target channel mismatch: expected beta-cck-mozilla20 got beta +Target channel mismatch: expected beta-cck-euballot got beta + +Total channel mismatch: 0.269% +
The vast majority of the data in the payload seems reasonable (99.45%).
+However, a handful of update pings are reporting a targetBuildId which is older than the current build reported by the ping’s environment: this is unexpected, as the the target build id must be always greater than the current one. After discussing this with the update team, it seems like this could either be due to channel weirdness or to the customization applied by the CCK tool. Additionally, some pings are reporting a targetChannel different than the one in the environment: this is definitely due to the CCK tool, given the cck entry in the channel name. These issues do not represent a problem, as most of the data is correct and their volume is fairly low.
For each ping, build a key with the client id and the target update details. Since we expect to have exactly one ping for each update bundle marked as ready, we don’t expect duplicate keys.
+update_dupes = deduped_subset.map(lambda p: ((p.get("clientId"), + p.get("payload/targetChannel"), + p.get("payload/targetVersion"), + p.get("payload/targetBuildId")), 1)).countByKey() + +print("Percentage of pings related to the same update (for the same client):\t{:.3f}%"\ + .format(pct(sum([v for v in update_dupes.values() if v > 1]), deduped_count))) +
Percentage of pings related to the same update (for the same client): 3.598% +
We’re receiving update pings with different documentId related to the same target update bundle, for some clients. The 3.59% is slightly higher than the one we saw on Nightly, 1.74%. One possible reason for this could be users having multiple copies of Firefox installed on their machine. Let’s see if that’s the case.
clientIds_sending_dupes = [k[0] for k, v in update_dupes.iteritems() if v > 1] + +def check_same_original_build(ping_list): + # Build a "unique" identifier for the build by + # concatenating the buildId, channel and version. + unique_build_ids = [ + "{}{}{}".format(p.get("application/buildId"), p.get("application/channel"), p.get("application/version"))\ + for p in ping_list[1] + ] + + # Remove the duplicates and return True if all the pings came + # from the same build. + return len(set(unique_build_ids)) < 2 + +# Count how many duplicates come from the same builds and how many come from +# different original builds. +original_builds_same =\ + deduped_subset.filter(lambda p: p.get("clientId") in clientIds_sending_dupes)\ + .map(lambda p: ((p.get("clientId"), + p.get("payload/targetChannel"), + p.get("payload/targetVersion"), + p.get("payload/targetBuildId")), [p]))\ + .reduceByKey(lambda a, b: a + b)\ + .filter(lambda p: len(p[1]) > 1)\ + .map(check_same_original_build).countByValue() + +print("Original builds are identical:\t{:.3f}%"\ + .format(pct(original_builds_same.get(True), sum(original_builds_same.values())))) +print("Original builds are different:\t{:.3f}%"\ + .format(pct(original_builds_same.get(False), sum(original_builds_same.values())))) +
Original builds are identical: 97.293% +Original builds are different: 2.707% +
The data shows that the update pings with the same target version are not necessarily coming from the same profile being used on different Firefox builds/installation: most of them are, but not all of them. After discussing this with the update team, it turns out that this can be explained by updates failing to apply: for certain classes of errors, we download the update blob again and thus send a new update ping with the same target version. This problem shows up in the update orphaning dashboard as well but, unfortunately, it only reports Release data.
delays = deduped_subset.map(lambda p: calculate_submission_delay(p)) +
MAX_DELAY_S = 60 * 60 * 48.0 +setup_plot("'update' ('ready') ping submission delay CDF", + MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +plot_cdf(delays\ + .map(lambda d: d / HOUR_IN_S if d < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect(), label="CDF", linestyle="solid") + +plt.show() +
Almost all of the update ping are submitted within an hour from the update being ready.
update pings is reasonableThis is a tricky one. The update ping with reason = "ready" is sent as soon as an update package is downloaded, verified and deemed ready to be applied. However, nothing guarantees that the update is immediately (or ever) applied. To check if the volume of update pings is in the ballpark, we can:
main-ping for that version of Firefox.Step 1 - Get the list of client ids updating to build ‘20170810xxxxxx’
+TARGET_BUILDID_MIN = '20170810000000' +TARGET_BUILDID_MAX = '20170810999999' + +update_candidates =\ + deduped_subset.filter(lambda p: TARGET_BUILDID_MIN <= p.get("payload/targetBuildId") <= TARGET_BUILDID_MAX) +update_candidates_clientIds = dedupe(update_candidates, "clientId").map(lambda p: p.get("clientId")) +candidates_count = update_candidates_clientIds.count() +
Step 2 - Get the main-ping from that Beta build and extract the list of client ids.
updated_main_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="beta") \ + .where(submissionDate=lambda x: "20170811" <= x < "20170818") \ + .where(appBuildId=lambda x: TARGET_BUILDID_MIN <= x <= TARGET_BUILDID_MAX) \ + .records(sc, sample=1) +
fetching 198117.80319MB in 1364 files... +
We just need the client ids and a few other fields to dedupe.
+subset_main = get_pings_properties(updated_main_pings, ["id", + "clientId", + "meta/Timestamp", + "application/buildId", + "application/channel", + "application/version"]) +
Start by deduping by document id. After that, only get a single ping per client and extract the list of client ids.
+deduped_main = dedupe(subset_main, "id") +updated_clientIds = dedupe(deduped_main, "clientId").map(lambda p: p.get("clientId")) +updated_count = updated_clientIds.count() +
Step 3 - Count how many clients that were meant to update actually updated in the following 7 days.
+matching_clientIds = update_candidates_clientIds.intersection(updated_clientIds) +matching_count = matching_clientIds.count() +
print("{:.3f}% of the clients that sent the update ping updated to the newer Beta build within a week."\ + .format(pct(matching_count, candidates_count))) +print("{:.3f}% of the clients that were seen on the newer Beta build sent an update ping."\ + .format(pct(candidates_count, updated_count))) +
91.445% of the clients that sent the update ping updated to the newer Beta build within a week. +98.819% of the clients that were seen on the newer Beta build sent an update ping. +
Roughly 98% of the clients that were seen in the new Beta build also sent the update ping. This is way higher than the 80% we saw on Nightly, but still not 100%. This could be due to a few reasons:
+update ping is sent in that case if an update is manually triggered;+
reason = ready)This analysis validates the update ping with reason = ready, which was introduced in bug 1120372 and should be sent every time an update is downloaded and ready to be applied. We are going to verify that:
import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
The update ping landed on the Nightly channel on the 27th of July, 2017. However, shortly after we had merge day. Let’s try to get the first full-week of data after the merge week up to today: 6th of August to the 12th of August, 2017. Restrict to the data coming from the Nightly builds after the day the ping landed.
update_pings = Dataset.from_source("telemetry") \ + .where(docType="OTHER") \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170806" <= x < "20170813") \ + .where(appBuildId=lambda x: "20170728" <= x < "20170813") \ + .records(sc, sample=1.0) +
fetching 180.82962MB in 11757 files... +
update_pings = update_pings.filter(lambda p: p.get("type") == "update") +
def pct(a, b): + return 100.0 * a / b + +def dedupe(pings, duping_key): + return pings\ + .map(lambda p: (p[duping_key], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) +
Misc functions to plot the CDF of the submission delay.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 + +def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data, **kwargs): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys, **kwargs) + +def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
Check that the payload section contains the right entries with consistent values.
+subset = get_pings_properties(update_pings, ["id", + "clientId", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "application/buildId", + "application/channel", + "application/version", + "environment/system/os/name", + "payload/reason", + "payload/targetBuildId", + "payload/targetChannel", + "payload/targetVersion"]) + +ping_count = subset.count() +
Quantify the percentage of duplicate pings we’re receiving. We don’t expect this value to be greater than ~1%, which is the amount we usually get from main and crash: as a rule of thumb, we threat anything less than 1% as probably well behaving.
deduped_subset = dedupe(subset, "id") +deduped_count = deduped_subset.count() +print("Percentage of duplicate pings: {:.3f}".format(100.0 - pct(deduped_count, ping_count))) +
Percentage of duplicate pings: 0.236 +
The percentage of duplicate pings is within the expected range. Move on and verify the payload of the update pings.
def validate_update_payload(p): + PAYLOAD_KEYS = [ + "payload/reason", + "payload/targetBuildId", + "payload/targetChannel", + "payload/targetVersion" + ] + + # All the payload keys needs to be strings. + for k in PAYLOAD_KEYS: + if not isinstance(p.get(k), basestring): + return ("'{}' is not a string".format(k), 1) + + # We only expect "reason" = ready. + if p.get("payload/reason") != "ready": + return ("Unexpected 'reason' {}".format(p.get("payload/reason"), 1)) + + # For Nightly, the target channel should be the same as the + # application channel. + if p.get("payload/targetChannel") != p.get("application/channel"): + return ("Target channel mismatch: expected {} got {}"\ + .format(p.get("payload/targetChannel"), p.get("application/channel")), 1) + + # The target buildId must be greater than the application build id. + if p.get("payload/targetBuildId") <= p.get("application/buildId"): + return ("Target buildId mismatch: {} must be more recent than {}"\ + .format(p.get("payload/targetBuildId"), p.get("application/buildId")), 1) + + return ("Ok", 1) + +validation_results = deduped_subset.map(validate_update_payload).countByKey() +for k, v in sorted(validation_results.iteritems()): + print("{}:\t{:.3f}%".format(k, pct(v, ping_count))) +
Ok: 99.712% +Target buildId mismatch: 20170615030208 must be more recent than 20170731100325: 0.001% +Target buildId mismatch: 20170630030203 must be more recent than 20170731100325: 0.001% +Target buildId mismatch: 20170706060058 must be more recent than 20170731100325: 0.001% +Target buildId mismatch: 20170723030206 must be more recent than 20170729100254: 0.001% +Target buildId mismatch: 20170725030209 must be more recent than 20170731100325: 0.001% +Target buildId mismatch: 20170726030207 must be more recent than 20170728100358: 0.001% +Target buildId mismatch: 20170728100358 must be more recent than 20170731100325: 0.001% +Target buildId mismatch: 20170729100254 must be more recent than 20170730100307: 0.001% +Target buildId mismatch: 20170802100302 must be more recent than 20170803134456: 0.001% +Target buildId mismatch: 20170802100302 must be more recent than 20170804100354: 0.001% +Target buildId mismatch: 20170802100302 must be more recent than 20170804193726: 0.001% +Target buildId mismatch: 20170802100302 must be more recent than 20170806100257: 0.002% +Target buildId mismatch: 20170802100302 must be more recent than 20170807113452: 0.001% +Target buildId mismatch: 20170802100302 must be more recent than 20170809100326: 0.001% +Target buildId mismatch: 20170803100352 must be more recent than 20170805100334: 0.001% +Target buildId mismatch: 20170803134456 must be more recent than 20170804100354: 0.001% +Target buildId mismatch: 20170803134456 must be more recent than 20170804193726: 0.001% +Target buildId mismatch: 20170803134456 must be more recent than 20170807113452: 0.001% +Target buildId mismatch: 20170804100354 must be more recent than 20170804193726: 0.001% +Target buildId mismatch: 20170804100354 must be more recent than 20170805100334: 0.002% +Target buildId mismatch: 20170804100354 must be more recent than 20170806100257: 0.002% +Target buildId mismatch: 20170804100354 must be more recent than 20170807113452: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170805100334: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170806100257: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170807113452: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170808100224: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170809100326: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170810100255: 0.001% +Target buildId mismatch: 20170805100334 must be more recent than 20170806100257: 0.001% +Target buildId mismatch: 20170805100334 must be more recent than 20170807113452: 0.001% +Target buildId mismatch: 20170805100334 must be more recent than 20170809100326: 0.001% +Target buildId mismatch: 20170806100257 must be more recent than 20170807113452: 0.002% +Target buildId mismatch: 20170806100257 must be more recent than 20170808114032: 0.001% +Target buildId mismatch: 20170806100257 must be more recent than 20170810100255: 0.001% +Target buildId mismatch: 20170806100257 must be more recent than 20170812100345: 0.001% +Target buildId mismatch: 20170807113452 must be more recent than 20170808114032: 0.002% +Target buildId mismatch: 20170807113452 must be more recent than 20170809100326: 0.003% +Target buildId mismatch: 20170807113452 must be more recent than 20170810100255: 0.002% +Target buildId mismatch: 20170808100224 must be more recent than 20170808114032: 0.001% +Target buildId mismatch: 20170808114032 must be more recent than 20170809100326: 0.005% +Target buildId mismatch: 20170808114032 must be more recent than 20170810100255: 0.001% +Target buildId mismatch: 20170809100326 must be more recent than 20170810100255: 0.002% +Target buildId mismatch: 20170809100326 must be more recent than 20170811100330: 0.001% +Target buildId mismatch: 20170810100255 must be more recent than 20170811100330: 0.001% +Target buildId mismatch: 20170810100255 must be more recent than 20170812100345: 0.001% +Target channel mismatch: expected nightly-cck-mint got nightly: 0.005% +Target channel mismatch: expected nightly-cck-rambler got nightly: 0.002% +
The vast majority of the data in the payload seems reasonable (99.71%).
+However, a handful of update pings are reporting a targetBuildId which is older than the current build reported by the ping’s environment: this is unexpected, as the the target build id must be always greater than the current one. After discussing this with the update team, it seems like this could either be due to Nigthly channel weirdness or to the customization applied by the CCK tool. Additionally, some pings are reporting a targetChannel different than the one in the environment: this is definitely due to the CCK tool, given the cck entry in the channel name. These issues do not represent a problem, as most of the data is correct and their volume is fairly low.
For each ping, build a key with the client id and the target update details. Since we expect to have exactly one ping for each update bundle marked as ready, we don’t expect duplicate keys.
+update_dupes = deduped_subset.map(lambda p: ((p.get("clientId"), + p.get("payload/targetChannel"), + p.get("payload/targetVersion"), + p.get("payload/targetBuildId")), 1)).countByKey() + +print("Percentage of pings related to the same update (for the same client):\t{:.3f}%"\ + .format(pct(sum([v for v in update_dupes.values() if v > 1]), deduped_count))) +
Percentage of pings related to the same update (for the same client): 1.742% +
We’re receiving update pings with different documentId related to the same target update bundle, for a few clients. One possible reason for this could be users having multiple copies of Firefox installed on their machine. Let’s see if that’s the case.
clientIds_sending_dupes = [k[0] for k, v in update_dupes.iteritems() if v > 1] + +def check_same_original_build(ping_list): + # Build a "unique" identifier for the build by + # concatenating the buildId, channel and version. + unique_build_ids = [ + "{}{}{}".format(p.get("application/buildId"), p.get("application/channel"), p.get("application/version"))\ + for p in ping_list[1] + ] + + # Remove the duplicates and return True if all the pings came + # from the same build. + return len(set(unique_build_ids)) < 2 + +# Count how many duplicates come from the same builds and how many come from +# different original builds. +original_builds_same =\ + deduped_subset.filter(lambda p: p.get("clientId") in clientIds_sending_dupes)\ + .map(lambda p: ((p.get("clientId"), + p.get("payload/targetChannel"), + p.get("payload/targetVersion"), + p.get("payload/targetBuildId")), [p]))\ + .reduceByKey(lambda a, b: a + b)\ + .filter(lambda p: len(p[1]) > 1)\ + .map(check_same_original_build).countByValue() + +print("Original builds are identical:\t{:.3f}%"\ + .format(pct(original_builds_same.get(True), sum(original_builds_same.values())))) +print("Original builds are different:\t{:.3f}%"\ + .format(pct(original_builds_same.get(False), sum(original_builds_same.values())))) +
Original builds are identical: 66.219% +Original builds are different: 33.781% +
The data shows that the update pings with the same target version are not necessarily coming from the same profile being used on different Firefox builds/installation. After discussing this with the update team, it turns out that this can be explained by updates failing to apply: for certain classes of errors, we download the update blob again and thus send a new update ping with the same target version. This problem shows up in the update orphaning dashboard as well but, unfortunately, it only reports Release data.
delays = deduped_subset.map(lambda p: calculate_submission_delay(p)) +
setup_plot("'update' ('ready') ping submission delay CDF", + MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +plot_cdf(delays\ + .map(lambda d: d / HOUR_IN_S if d < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect(), label="CDF", linestyle="solid") + +plt.show() +
Almost all of the update ping are submitted within an hour from the update being ready.
update pings is reasonableThis is a tricky one. The update ping with reason = "ready" is sent as soon as an update package is downloaded, verified and deemed ready to be applied. However, nothing guarantees that the update is immediately (or ever) applied. To check if the volume of update pings is in the ballpark, we can:
main-ping for that version of Firefox.Step 1 - Get the list of client ids updating to build ‘20170809xxxxxx’
+TARGET_BUILDID_MIN = '20170809000000' +TARGET_BUILDID_MAX = '20170809999999' + +update_candidates =\ + deduped_subset.filter(lambda p: TARGET_BUILDID_MIN <= p.get("payload/targetBuildId") <= TARGET_BUILDID_MAX) +update_candidates_clientIds = dedupe(update_candidates, "clientId").map(lambda p: p.get("clientId")) +candidates_count = update_candidates_clientIds.count() +
Step 2 - Get the main-ping from that Nightly build and extract the list of client ids.
updated_main_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170809" <= x < "20170816") \ + .where(appBuildId=lambda x: TARGET_BUILDID_MIN <= x <= TARGET_BUILDID_MAX) \ + .records(sc, sample=1) +
fetching 9871.08945MB in 1135 files... +
We just need the client ids and a few other fields to dedupe.
+subset_main = get_pings_properties(updated_main_pings, ["id", + "clientId", + "meta/Timestamp", + "application/buildId", + "application/channel", + "application/version"]) +
Start by deduping by document id. After that, only get a single ping per client and extract the list of client ids.
+deduped_main = dedupe(subset_main, "id") +updated_clientIds = dedupe(deduped_main, "clientId").map(lambda p: p.get("clientId")) +updated_count = updated_clientIds.count() +
Step 3 - Count how many clients that were meant to update actually updated in the following 7 days.
+matching_clientIds = update_candidates_clientIds.intersection(updated_clientIds) +matching_count = matching_clientIds.count() +
print("{:.3f}% of the clients that sent the update ping updated to the newer Nightly build within a week."\ + .format(pct(matching_count, candidates_count))) +print("{:.3f}% of the clients that were seen on the newer Nightly build sent an update ping."\ + .format(pct(candidates_count, updated_count))) +
93.419% of the clients that sent the update ping updated to the newer Nightly build within a week. +79.678% of the clients that were seen on the newer Nightly build sent an update ping. +
Roughly 80% of the clients that were seen in the new Nightly build also sent the update ping. The 95%ile of the main-ping data from Nightly 57 reaches us with a 9.4 hour delay (see here), so most of the data should be in already. This could be due to a few reasons:
update ping is sent in that case if an update is manually triggered;+
reason = ready)This analysis validates the update ping with reason = ready, which was introduced in bug 1120372 and should be sent every time an update is downloaded and ready to be applied. We are going to verify that:
import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
The update ping landed on the Nightly channel on the 27th of July, 2017. However, shortly after we had merge day. Let’s try to get the first full-week of data after the merge week up to today: 6th of August to the 12th of August, 2017. Restrict to the data coming from the Nightly builds after the day the ping landed.
update_pings = Dataset.from_source("telemetry") \ + .where(docType="OTHER") \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170806" <= x < "20170813") \ + .where(appBuildId=lambda x: "20170728" <= x < "20170813") \ + .records(sc, sample=1.0) +
fetching 180.82962MB in 11757 files... +
update_pings = update_pings.filter(lambda p: p.get("type") == "update") +
def pct(a, b): + return 100.0 * a / b + +def dedupe(pings, duping_key): + return pings\ + .map(lambda p: (p[duping_key], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) +
Misc functions to plot the CDF of the submission delay.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 + +def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data, **kwargs): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys, **kwargs) + +def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
Check that the payload section contains the right entries with consistent values.
+subset = get_pings_properties(update_pings, ["id", + "clientId", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "application/buildId", + "application/channel", + "application/version", + "environment/system/os/name", + "payload/reason", + "payload/targetBuildId", + "payload/targetChannel", + "payload/targetVersion"]) + +ping_count = subset.count() +
Quantify the percentage of duplicate pings we’re receiving. We don’t expect this value to be greater than ~1%, which is the amount we usually get from main and crash: as a rule of thumb, we threat anything less than 1% as probably well behaving.
deduped_subset = dedupe(subset, "id") +deduped_count = deduped_subset.count() +print("Percentage of duplicate pings: {:.3f}".format(100.0 - pct(deduped_count, ping_count))) +
Percentage of duplicate pings: 0.236 +
The percentage of duplicate pings is within the expected range. Move on and verify the payload of the update pings.
def validate_update_payload(p): + PAYLOAD_KEYS = [ + "payload/reason", + "payload/targetBuildId", + "payload/targetChannel", + "payload/targetVersion" + ] + + # All the payload keys needs to be strings. + for k in PAYLOAD_KEYS: + if not isinstance(p.get(k), basestring): + return ("'{}' is not a string".format(k), 1) + + # We only expect "reason" = ready. + if p.get("payload/reason") != "ready": + return ("Unexpected 'reason' {}".format(p.get("payload/reason"), 1)) + + # For Nightly, the target channel should be the same as the + # application channel. + if p.get("payload/targetChannel") != p.get("application/channel"): + return ("Target channel mismatch: expected {} got {}"\ + .format(p.get("payload/targetChannel"), p.get("application/channel")), 1) + + # The target buildId must be greater than the application build id. + if p.get("payload/targetBuildId") <= p.get("application/buildId"): + return ("Target buildId mismatch: {} must be more recent than {}"\ + .format(p.get("payload/targetBuildId"), p.get("application/buildId")), 1) + + return ("Ok", 1) + +validation_results = deduped_subset.map(validate_update_payload).countByKey() +for k, v in sorted(validation_results.iteritems()): + print("{}:\t{:.3f}%".format(k, pct(v, ping_count))) +
Ok: 99.712% +Target buildId mismatch: 20170615030208 must be more recent than 20170731100325: 0.001% +Target buildId mismatch: 20170630030203 must be more recent than 20170731100325: 0.001% +Target buildId mismatch: 20170706060058 must be more recent than 20170731100325: 0.001% +Target buildId mismatch: 20170723030206 must be more recent than 20170729100254: 0.001% +Target buildId mismatch: 20170725030209 must be more recent than 20170731100325: 0.001% +Target buildId mismatch: 20170726030207 must be more recent than 20170728100358: 0.001% +Target buildId mismatch: 20170728100358 must be more recent than 20170731100325: 0.001% +Target buildId mismatch: 20170729100254 must be more recent than 20170730100307: 0.001% +Target buildId mismatch: 20170802100302 must be more recent than 20170803134456: 0.001% +Target buildId mismatch: 20170802100302 must be more recent than 20170804100354: 0.001% +Target buildId mismatch: 20170802100302 must be more recent than 20170804193726: 0.001% +Target buildId mismatch: 20170802100302 must be more recent than 20170806100257: 0.002% +Target buildId mismatch: 20170802100302 must be more recent than 20170807113452: 0.001% +Target buildId mismatch: 20170802100302 must be more recent than 20170809100326: 0.001% +Target buildId mismatch: 20170803100352 must be more recent than 20170805100334: 0.001% +Target buildId mismatch: 20170803134456 must be more recent than 20170804100354: 0.001% +Target buildId mismatch: 20170803134456 must be more recent than 20170804193726: 0.001% +Target buildId mismatch: 20170803134456 must be more recent than 20170807113452: 0.001% +Target buildId mismatch: 20170804100354 must be more recent than 20170804193726: 0.001% +Target buildId mismatch: 20170804100354 must be more recent than 20170805100334: 0.002% +Target buildId mismatch: 20170804100354 must be more recent than 20170806100257: 0.002% +Target buildId mismatch: 20170804100354 must be more recent than 20170807113452: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170805100334: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170806100257: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170807113452: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170808100224: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170809100326: 0.001% +Target buildId mismatch: 20170804193726 must be more recent than 20170810100255: 0.001% +Target buildId mismatch: 20170805100334 must be more recent than 20170806100257: 0.001% +Target buildId mismatch: 20170805100334 must be more recent than 20170807113452: 0.001% +Target buildId mismatch: 20170805100334 must be more recent than 20170809100326: 0.001% +Target buildId mismatch: 20170806100257 must be more recent than 20170807113452: 0.002% +Target buildId mismatch: 20170806100257 must be more recent than 20170808114032: 0.001% +Target buildId mismatch: 20170806100257 must be more recent than 20170810100255: 0.001% +Target buildId mismatch: 20170806100257 must be more recent than 20170812100345: 0.001% +Target buildId mismatch: 20170807113452 must be more recent than 20170808114032: 0.002% +Target buildId mismatch: 20170807113452 must be more recent than 20170809100326: 0.003% +Target buildId mismatch: 20170807113452 must be more recent than 20170810100255: 0.002% +Target buildId mismatch: 20170808100224 must be more recent than 20170808114032: 0.001% +Target buildId mismatch: 20170808114032 must be more recent than 20170809100326: 0.005% +Target buildId mismatch: 20170808114032 must be more recent than 20170810100255: 0.001% +Target buildId mismatch: 20170809100326 must be more recent than 20170810100255: 0.002% +Target buildId mismatch: 20170809100326 must be more recent than 20170811100330: 0.001% +Target buildId mismatch: 20170810100255 must be more recent than 20170811100330: 0.001% +Target buildId mismatch: 20170810100255 must be more recent than 20170812100345: 0.001% +Target channel mismatch: expected nightly-cck-mint got nightly: 0.005% +Target channel mismatch: expected nightly-cck-rambler got nightly: 0.002% +
The vast majority of the data in the payload seems reasonable (99.71%).
+However, a handful of update pings are reporting a targetBuildId which is older than the current build reported by the ping’s environment: this is unexpected, as the the target build id must be always greater than the current one. After discussing this with the update team, it seems like this could either be due to Nigthly channel weirdness or to the customization applied by the CCK tool. Additionally, some pings are reporting a targetChannel different than the one in the environment: this is definitely due to the CCK tool, given the cck entry in the channel name. These issues do not represent a problem, as most of the data is correct and their volume is fairly low.
For each ping, build a key with the client id and the target update details. Since we expect to have exactly one ping for each update bundle marked as ready, we don’t expect duplicate keys.
+update_dupes = deduped_subset.map(lambda p: ((p.get("clientId"), + p.get("payload/targetChannel"), + p.get("payload/targetVersion"), + p.get("payload/targetBuildId")), 1)).countByKey() + +print("Percentage of pings related to the same update (for the same client):\t{:.3f}%"\ + .format(pct(sum([v for v in update_dupes.values() if v > 1]), deduped_count))) +
Percentage of pings related to the same update (for the same client): 1.742% +
We’re receiving update pings with different documentId related to the same target update bundle, for a few clients. One possible reason for this could be users having multiple copies of Firefox installed on their machine. Let’s see if that’s the case.
clientIds_sending_dupes = [k[0] for k, v in update_dupes.iteritems() if v > 1] + +def check_same_original_build(ping_list): + # Build a "unique" identifier for the build by + # concatenating the buildId, channel and version. + unique_build_ids = [ + "{}{}{}".format(p.get("application/buildId"), p.get("application/channel"), p.get("application/version"))\ + for p in ping_list[1] + ] + + # Remove the duplicates and return True if all the pings came + # from the same build. + return len(set(unique_build_ids)) < 2 + +# Count how many duplicates come from the same builds and how many come from +# different original builds. +original_builds_same =\ + deduped_subset.filter(lambda p: p.get("clientId") in clientIds_sending_dupes)\ + .map(lambda p: ((p.get("clientId"), + p.get("payload/targetChannel"), + p.get("payload/targetVersion"), + p.get("payload/targetBuildId")), [p]))\ + .reduceByKey(lambda a, b: a + b)\ + .filter(lambda p: len(p[1]) > 1)\ + .map(check_same_original_build).countByValue() + +print("Original builds are identical:\t{:.3f}%"\ + .format(pct(original_builds_same.get(True), sum(original_builds_same.values())))) +print("Original builds are different:\t{:.3f}%"\ + .format(pct(original_builds_same.get(False), sum(original_builds_same.values())))) +
Original builds are identical: 66.219% +Original builds are different: 33.781% +
The data shows that the update pings with the same target version are not necessarily coming from the same profile being used on different Firefox builds/installation. After discussing this with the update team, it turns out that this can be explained by updates failing to apply: for certain classes of errors, we download the update blob again and thus send a new update ping with the same target version. This problem shows up in the update orphaning dashboard as well but, unfortunately, it only reports Release data.
delays = deduped_subset.map(lambda p: calculate_submission_delay(p)) +
setup_plot("'update' ('ready') ping submission delay CDF", + MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +plot_cdf(delays\ + .map(lambda d: d / HOUR_IN_S if d < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect(), label="CDF", linestyle="solid") + +plt.show() +
Almost all of the update ping are submitted within an hour from the update being ready.
update pings is reasonableThis is a tricky one. The update ping with reason = "ready" is sent as soon as an update package is downloaded, verified and deemed ready to be applied. However, nothing guarantees that the update is immediately (or ever) applied. To check if the volume of update pings is in the ballpark, we can:
main-ping for that version of Firefox.Step 1 - Get the list of client ids updating to build ‘20170809xxxxxx’
+TARGET_BUILDID_MIN = '20170809000000' +TARGET_BUILDID_MAX = '20170809999999' + +update_candidates =\ + deduped_subset.filter(lambda p: TARGET_BUILDID_MIN <= p.get("payload/targetBuildId") <= TARGET_BUILDID_MAX) +update_candidates_clientIds = dedupe(update_candidates, "clientId").map(lambda p: p.get("clientId")) +candidates_count = update_candidates_clientIds.count() +
Step 2 - Get the main-ping from that Nightly build and extract the list of client ids.
updated_main_pings = Dataset.from_source("telemetry") \ + .where(docType="main") \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: "20170809" <= x < "20170816") \ + .where(appBuildId=lambda x: TARGET_BUILDID_MIN <= x <= TARGET_BUILDID_MAX) \ + .records(sc, sample=1) +
fetching 9871.08945MB in 1135 files... +
We just need the client ids and a few other fields to dedupe.
+subset_main = get_pings_properties(updated_main_pings, ["id", + "clientId", + "meta/Timestamp", + "application/buildId", + "application/channel", + "application/version"]) +
Start by deduping by document id. After that, only get a single ping per client and extract the list of client ids.
+deduped_main = dedupe(subset_main, "id") +updated_clientIds = dedupe(deduped_main, "clientId").map(lambda p: p.get("clientId")) +updated_count = updated_clientIds.count() +
Step 3 - Count how many clients that were meant to update actually updated in the following 7 days.
+matching_clientIds = update_candidates_clientIds.intersection(updated_clientIds) +matching_count = matching_clientIds.count() +
print("{:.3f}% of the clients that sent the update ping updated to the newer Nightly build within a week."\ + .format(pct(matching_count, candidates_count))) +print("{:.3f}% of the clients that were seen on the newer Nightly build sent an update ping."\ + .format(pct(candidates_count, updated_count))) +
93.419% of the clients that sent the update ping updated to the newer Nightly build within a week. +79.678% of the clients that were seen on the newer Nightly build sent an update ping. +
Roughly 80% of the clients that were seen in the new Nightly build also sent the update ping. The 95%ile of the main-ping data from Nightly 57 reaches us with a 9.4 hour delay (see here), so most of the data should be in already. This could be due to a few reasons:
update ping is sent in that case if an update is manually triggered;+
reason = success)This analysis validates the update ping with reason = success, which was introduced in bug 1380256 and should be sent every time an update is applied after the browser is restarted. We are going to verify that:
reason = ready;import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
The update ping with reason = success landed on the Nightly channel on the 1st of September, 2017. Let’s get the first full-week of data after that date: 3rd-9th September, 2017. Restrict to the data coming from the Nightly builds after the day the ping landed.
MIN_DATE = "20170903" +MAX_DATE = "20170910" + +update_pings = Dataset.from_source("telemetry") \ + .where(docType="update") \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: MIN_DATE <= x < MAX_DATE) \ + .where(appBuildId=lambda x: MIN_DATE <= x < MAX_DATE) \ + .records(sc, sample=1.0) +
fetching 256.36621MB in 8021 files... +
def pct(a, b): + return 100.0 * a / b + +def dedupe(pings, duping_key): + return pings\ + .map(lambda p: (p[duping_key], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) +
Misc functions to plot the CDF of the submission delay.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 + +def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data, **kwargs): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys, **kwargs) + +def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
Check that the payload section contains the right entries with consistent values.
+subset = get_pings_properties(update_pings, ["id", + "clientId", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "application/buildId", + "application/channel", + "application/version", + "environment/system/os/name", + "payload/reason", + "payload/targetBuildId", + "payload/targetChannel", + "payload/targetVersion", + "payload/previousBuildId", + "payload/previousChannel", + "payload/previousVersion"]) +
ping_success = subset.filter(lambda p: p.get("payload/reason") == "success") +ping_ready = subset.filter(lambda p: p.get("payload/reason") == "ready") + +ping_success_count = ping_success.count() +ping_ready_count = ping_ready.count() +ping_count = ping_ready_count + ping_success_count +# As a safety precaution, assert that we only received the +# reasons we were looking for. +assert ping_count == subset.count() +
Quantify the percentage of duplicate pings we’re receiving. We don’t expect this value to be greater than ~1%, which is the amount we usually get from main and crash: as a rule of thumb, we threat anything less than 1% as probably well behaving.
deduped_subset = dedupe(ping_success, "id") +deduped_count = deduped_subset.count() +print("Percentage of duplicate pings: {:.3f}".format(100.0 - pct(deduped_count, ping_success_count))) +
Percentage of duplicate pings: 0.078 +
The percentage of duplicate pings is within the expected range. Move on and verify the payload of the update pings.
def validate_update_payload(p): + PAYLOAD_KEYS = [ + "payload/reason", + "payload/previousBuildId", + "payload/previousChannel", + "payload/previousVersion" + ] + + # All the payload keys needs to be strings. + for k in PAYLOAD_KEYS: + if not isinstance(p.get(k), basestring): + return ("'{}' is not a string".format(k), 1) + + # For Nightly, the previous channel should be the same as the + # application channel. + if p.get("payload/previousChannel") != p.get("application/channel"): + return ("Previous channel mismatch: expected {} got {}"\ + .format(p.get("payload/previousChannel"), p.get("application/channel")), 1) + + # The previous buildId must be smaller than the application build id. + if p.get("payload/previousBuildId") > p.get("application/buildId"): + return ("Previous buildId mismatch: {} must be older than {}"\ + .format(p.get("payload/previousBuildId"), p.get("application/buildId")), 1) + + return ("Ok", 1) + +validation_results = deduped_subset.map(validate_update_payload).countByKey() +for k, v in sorted(validation_results.iteritems()): + print("{}:\t{:.3f}%".format(k, pct(v, ping_success_count))) +
Ok: 99.875% +Previous buildId mismatch: 20170903140023 must be older than 20170903100443: 0.001% +Previous buildId mismatch: 20170904100131 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170904220027 must be older than 20170903100443: 0.001% +Previous buildId mismatch: 20170904220027 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170904220027 must be older than 20170904100131: 0.001% +Previous buildId mismatch: 20170905100117 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170905100117 must be older than 20170904220027: 0.001% +Previous buildId mismatch: 20170905220108 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170905220108 must be older than 20170904100131: 0.001% +Previous buildId mismatch: 20170905220108 must be older than 20170904220027: 0.002% +Previous buildId mismatch: 20170905220108 must be older than 20170905100117: 0.001% +Previous buildId mismatch: 20170906100107 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170906100107 must be older than 20170904100131: 0.001% +Previous buildId mismatch: 20170906100107 must be older than 20170905100117: 0.002% +Previous buildId mismatch: 20170906100107 must be older than 20170905220108: 0.001% +Previous buildId mismatch: 20170906220306 must be older than 20170903100443: 0.001% +Previous buildId mismatch: 20170906220306 must be older than 20170904220027: 0.001% +Previous buildId mismatch: 20170906220306 must be older than 20170905100117: 0.002% +Previous buildId mismatch: 20170906220306 must be older than 20170905220108: 0.002% +Previous buildId mismatch: 20170906220306 must be older than 20170906100107: 0.001% +Previous buildId mismatch: 20170907100318 must be older than 20170903100443: 0.001% +Previous buildId mismatch: 20170907100318 must be older than 20170905100117: 0.001% +Previous buildId mismatch: 20170907100318 must be older than 20170905220108: 0.002% +Previous buildId mismatch: 20170907100318 must be older than 20170906100107: 0.002% +Previous buildId mismatch: 20170907100318 must be older than 20170906220306: 0.003% +Previous buildId mismatch: 20170907194642 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170907194642 must be older than 20170905100117: 0.001% +Previous buildId mismatch: 20170907194642 must be older than 20170906100107: 0.001% +Previous buildId mismatch: 20170907194642 must be older than 20170906220306: 0.001% +Previous buildId mismatch: 20170907194642 must be older than 20170907100318: 0.001% +Previous buildId mismatch: 20170907220212 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170907220212 must be older than 20170905100117: 0.001% +Previous buildId mismatch: 20170907220212 must be older than 20170905220108: 0.001% +Previous buildId mismatch: 20170907220212 must be older than 20170906100107: 0.001% +Previous buildId mismatch: 20170907220212 must be older than 20170907100318: 0.002% +Previous buildId mismatch: 20170908100218 must be older than 20170907100318: 0.001% +Previous buildId mismatch: 20170908100218 must be older than 20170907220212: 0.001% +Previous buildId mismatch: 20170908220146 must be older than 20170907100318: 0.001% +Previous buildId mismatch: 20170908220146 must be older than 20170907220212: 0.001% +Previous buildId mismatch: 20170908220146 must be older than 20170908100218: 0.001% +Previous channel mismatch: expected nightly-cck-google got nightly: 0.001% +Previous channel mismatch: expected nightly-cck-mint got nightly: 0.003% +Previous channel mismatch: expected nightly-cck-mozillaonline got nightly: 0.001% +Previous channel mismatch: expected nightly-cck-yandex got nightly: 0.001% +
The vast majority of the data in the payload seems reasonable (99.87%).
+However, a handful of update pings are reporting a previousBuildId mismatch: this is unexpected. After discussing this with the update team, it seems like this could either be due to Nigthly channel weirdness or to the customization applied by the CCK tool. Additionally, some pings are reporting a previousChannel different than the one in the environment: this is definitely due to the CCK tool, given the cck entry in the channel name. These issues do not represent a problem, as most of the data is correct and their volume is fairly low.
For each ping, build a key with the client id and the previous build update details. Since we expect to have exactly one ping for each successfully applied update, we don’t expect duplicate keys.
+update_dupes = deduped_subset.map(lambda p: ((p.get("clientId"), + p.get("payload/previousChannel"), + p.get("payload/previousVersion"), + p.get("payload/previousBuildId")), 1)).countByKey() + +print("Percentage of pings related to the same update (for the same client):\t{:.3f}%"\ + .format(pct(sum([v for v in update_dupes.values() if v > 1]), deduped_count))) +
Percentage of pings related to the same update (for the same client): 1.318% +
We’re receiving update pings with different documentId related to the same initial build, for a few clients. One possible reason for this could be users having multiple copies of Firefox installed on their machine. Let’s see if that’s the case.
clientIds_sending_dupes = [k[0] for k, v in update_dupes.iteritems() if v > 1] + +def check_same_original_build(ping_list): + # Build a "unique" identifier for the build by + # concatenating the buildId, channel and version. + unique_build_ids = [ + "{}{}{}".format(p.get("application/buildId"), p.get("application/channel"), p.get("application/version"))\ + for p in ping_list[1] + ] + + # Remove the duplicates and return True if all the pings came + # from the same build. + return len(set(unique_build_ids)) < 2 + +# Count how many duplicates are updating to the same builds and how many are +# updating to different builds. +original_builds_same =\ + deduped_subset.filter(lambda p: p.get("clientId") in clientIds_sending_dupes)\ + .map(lambda p: ((p.get("clientId"), + p.get("payload/previousChannel"), + p.get("payload/previousVersion"), + p.get("payload/previousBuildId")), [p]))\ + .reduceByKey(lambda a, b: a + b)\ + .filter(lambda p: len(p[1]) > 1)\ + .map(check_same_original_build).countByValue() + +print("Updated builds are identical:\t{:.3f}%"\ + .format(pct(original_builds_same.get(True), sum(original_builds_same.values())))) +print("Updated builds are different:\t{:.3f}%"\ + .format(pct(original_builds_same.get(False), sum(original_builds_same.values())))) +
Updated builds are identical: 16.472% +Updated builds are different: 83.528% +
The data shows that 83.52% of the 1.31% dupes are updating from different builds. The 0.22% of all update pings that are the same client updating to the same build from the same build are, at present, unexplained (but in small enough quantities we can ignore for the moment).
The update pings with the same previous build information may be coming from the same profile, copied and then used with different versions of Firefox. Depending on when the browser is started with a specific copied profile, the downloaded update blob might be different (more recent), thus resulting in an update with reason = success being sent with the same previous build information but with different current build information.
delays = deduped_subset.map(lambda p: calculate_submission_delay(p)) +
setup_plot("'update' ('success') ping submission delay CDF", + MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +plot_cdf(delays\ + .map(lambda d: d / HOUR_IN_S if d < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect(), label="CDF", linestyle="solid") + +plt.show() +
Almost 95% of the update pings with reason = success are submitted within an hour from the ping being created. Since we know that this ping is created as soon as the update is applied we can claim that we receive 95% of these pings within an hour from the update being applied.
Check if the volume of update pings with reason = ready matches with the volume of pings with reason = success. For each ping with reason = ready, find the matching ping with reason = success.
We are considering the data within a very narrow window of time: we could see reason = success pings from users that sent a reason = ready ping before the 3rd of September and reason = ready pings from users that have sent us a reason = success after the 9th of September. Filter these edge cases out by inspecting the previousBuildId and targetBuildId.
filtered_ready = ping_ready.filter(lambda p: p.get("payload/targetBuildId") < "{}999999".format(MAX_DATE)) +filtered_success = ping_success.filter(lambda p: p.get("payload/previousBuildId") >= "{}000000".format(MIN_DATE)) +
Use the filtered RDDs to match between the different ping reasons.
+# Get an identifier that keeps in consideration both the current build +# and the target build. +ready_uuid = filtered_ready\ + .map(lambda p: (p.get("clientId"), + p.get("application/buildId"), + p.get("application/channel"), + p.get("application/version"), + p.get("payload/targetBuildId"), + p.get("payload/targetChannel"), + p.get("payload/targetVersion"))) + +# Get an identifier that considers both the prevous build info and the +# current build info. The order of the values in the tuple need to match +# the one from the previous RDD. +success_uuid = filtered_success\ + .map(lambda p: (p.get("clientId"), + p.get("payload/previousBuildId"), + p.get("payload/previousChannel"), + p.get("payload/previousVersion"), + p.get("application/buildId"), + p.get("application/channel"), + p.get("application/version"))) +
Let’s match each reason = ready ping with a reason = success one, and count them.
matching_update_pings = ready_uuid.intersection(success_uuid) +matching_update_ping_count = matching_update_pings.count() +
Finally, show up some stats.
+print("{:.3f}% of the 'update' ping with reason 'ready' have a matching ping with reason 'success'."\ + .format(pct(matching_update_ping_count, filtered_ready.count()))) +print("{:.3f}% of the 'update' ping with reason 'success' have a matching ping with reason 'ready'."\ + .format(pct(matching_update_ping_count, filtered_success.count()))) +
63.834% of the 'update' ping with reason 'ready' have a matching ping with reason 'success'. +89.428% of the 'update' ping with reason 'success' have a matching ping with reason 'ready'. +
Only ~63% of the update ping sent when an update is ready to be applied have a corrensponding ping that’s sent, for the same client and upgrade path, after the update is successfully applied. One possible explaination for this is the delay with which updates get applied after they get downloaded: unless the browser is restarted (and that can happen after days due to user suspending their machines), we won’t see the ready ping anytime soon.
+Roughly 89% of the update pings with reason = success can be traced back to an update with reason = ready. The missing ~10% matches can be due to users disabling automatic updates (see this query) and other edge cases: no update ping is sent in that case if an update is manually triggered.
+
reason = success)This analysis validates the update ping with reason = success, which was introduced in bug 1380256 and should be sent every time an update is applied after the browser is restarted. We are going to verify that:
reason = ready;import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py +import IPython + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset +from datetime import datetime, timedelta +from email.utils import parsedate_tz, mktime_tz, formatdate + +%matplotlib inline +IPython.core.pylabtools.figsize(16, 7) +
The update ping with reason = success landed on the Nightly channel on the 1st of September, 2017. Let’s get the first full-week of data after that date: 3rd-9th September, 2017. Restrict to the data coming from the Nightly builds after the day the ping landed.
MIN_DATE = "20170903" +MAX_DATE = "20170910" + +update_pings = Dataset.from_source("telemetry") \ + .where(docType="update") \ + .where(appUpdateChannel="nightly") \ + .where(submissionDate=lambda x: MIN_DATE <= x < MAX_DATE) \ + .where(appBuildId=lambda x: MIN_DATE <= x < MAX_DATE) \ + .records(sc, sample=1.0) +
fetching 256.36621MB in 8021 files... +
def pct(a, b): + return 100.0 * a / b + +def dedupe(pings, duping_key): + return pings\ + .map(lambda p: (p[duping_key], p))\ + .reduceByKey(lambda a, b: a if a["meta/Timestamp"] < b["meta/Timestamp"] else b)\ + .map(lambda pair: pair[1]) +
Misc functions to plot the CDF of the submission delay.
+MAX_DELAY_S = 60 * 60 * 96.0 +HOUR_IN_S = 60 * 60.0 + +def setup_plot(title, max_x, area_border_x=0.1, area_border_y=0.1): + plt.title(title) + plt.xlabel("Delay (hours)") + plt.ylabel("% of pings") + + plt.xticks(range(0, int(max_x) + 1, 2)) + plt.yticks(map(lambda y: y / 20.0, range(0, 21, 1))) + + plt.ylim(0.0 - area_border_y, 1.0 + area_border_y) + plt.xlim(0.0 - area_border_x, max_x + area_border_x) + + plt.grid(True) + +def plot_cdf(data, **kwargs): + sortd = np.sort(data) + ys = np.arange(len(sortd))/float(len(sortd)) + + plt.plot(sortd, ys, **kwargs) + +def calculate_submission_delay(p): + created = datetime.fromtimestamp(p["meta/creationTimestamp"] / 1000.0 / 1000.0 / 1000.0) + received = datetime.fromtimestamp(p["meta/Timestamp"] / 1000.0 / 1000.0 / 1000.0) + sent = datetime.fromtimestamp(mktime_tz(parsedate_tz(p["meta/Date"]))) if p["meta/Date"] is not None else received + clock_skew = received - sent + + return (received - created - clock_skew).total_seconds() +
Check that the payload section contains the right entries with consistent values.
+subset = get_pings_properties(update_pings, ["id", + "clientId", + "meta/creationTimestamp", + "meta/Date", + "meta/Timestamp", + "application/buildId", + "application/channel", + "application/version", + "environment/system/os/name", + "payload/reason", + "payload/targetBuildId", + "payload/targetChannel", + "payload/targetVersion", + "payload/previousBuildId", + "payload/previousChannel", + "payload/previousVersion"]) +
ping_success = subset.filter(lambda p: p.get("payload/reason") == "success") +ping_ready = subset.filter(lambda p: p.get("payload/reason") == "ready") + +ping_success_count = ping_success.count() +ping_ready_count = ping_ready.count() +ping_count = ping_ready_count + ping_success_count +# As a safety precaution, assert that we only received the +# reasons we were looking for. +assert ping_count == subset.count() +
Quantify the percentage of duplicate pings we’re receiving. We don’t expect this value to be greater than ~1%, which is the amount we usually get from main and crash: as a rule of thumb, we threat anything less than 1% as probably well behaving.
deduped_subset = dedupe(ping_success, "id") +deduped_count = deduped_subset.count() +print("Percentage of duplicate pings: {:.3f}".format(100.0 - pct(deduped_count, ping_success_count))) +
Percentage of duplicate pings: 0.078 +
The percentage of duplicate pings is within the expected range. Move on and verify the payload of the update pings.
def validate_update_payload(p): + PAYLOAD_KEYS = [ + "payload/reason", + "payload/previousBuildId", + "payload/previousChannel", + "payload/previousVersion" + ] + + # All the payload keys needs to be strings. + for k in PAYLOAD_KEYS: + if not isinstance(p.get(k), basestring): + return ("'{}' is not a string".format(k), 1) + + # For Nightly, the previous channel should be the same as the + # application channel. + if p.get("payload/previousChannel") != p.get("application/channel"): + return ("Previous channel mismatch: expected {} got {}"\ + .format(p.get("payload/previousChannel"), p.get("application/channel")), 1) + + # The previous buildId must be smaller than the application build id. + if p.get("payload/previousBuildId") > p.get("application/buildId"): + return ("Previous buildId mismatch: {} must be older than {}"\ + .format(p.get("payload/previousBuildId"), p.get("application/buildId")), 1) + + return ("Ok", 1) + +validation_results = deduped_subset.map(validate_update_payload).countByKey() +for k, v in sorted(validation_results.iteritems()): + print("{}:\t{:.3f}%".format(k, pct(v, ping_success_count))) +
Ok: 99.875% +Previous buildId mismatch: 20170903140023 must be older than 20170903100443: 0.001% +Previous buildId mismatch: 20170904100131 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170904220027 must be older than 20170903100443: 0.001% +Previous buildId mismatch: 20170904220027 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170904220027 must be older than 20170904100131: 0.001% +Previous buildId mismatch: 20170905100117 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170905100117 must be older than 20170904220027: 0.001% +Previous buildId mismatch: 20170905220108 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170905220108 must be older than 20170904100131: 0.001% +Previous buildId mismatch: 20170905220108 must be older than 20170904220027: 0.002% +Previous buildId mismatch: 20170905220108 must be older than 20170905100117: 0.001% +Previous buildId mismatch: 20170906100107 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170906100107 must be older than 20170904100131: 0.001% +Previous buildId mismatch: 20170906100107 must be older than 20170905100117: 0.002% +Previous buildId mismatch: 20170906100107 must be older than 20170905220108: 0.001% +Previous buildId mismatch: 20170906220306 must be older than 20170903100443: 0.001% +Previous buildId mismatch: 20170906220306 must be older than 20170904220027: 0.001% +Previous buildId mismatch: 20170906220306 must be older than 20170905100117: 0.002% +Previous buildId mismatch: 20170906220306 must be older than 20170905220108: 0.002% +Previous buildId mismatch: 20170906220306 must be older than 20170906100107: 0.001% +Previous buildId mismatch: 20170907100318 must be older than 20170903100443: 0.001% +Previous buildId mismatch: 20170907100318 must be older than 20170905100117: 0.001% +Previous buildId mismatch: 20170907100318 must be older than 20170905220108: 0.002% +Previous buildId mismatch: 20170907100318 must be older than 20170906100107: 0.002% +Previous buildId mismatch: 20170907100318 must be older than 20170906220306: 0.003% +Previous buildId mismatch: 20170907194642 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170907194642 must be older than 20170905100117: 0.001% +Previous buildId mismatch: 20170907194642 must be older than 20170906100107: 0.001% +Previous buildId mismatch: 20170907194642 must be older than 20170906220306: 0.001% +Previous buildId mismatch: 20170907194642 must be older than 20170907100318: 0.001% +Previous buildId mismatch: 20170907220212 must be older than 20170903220032: 0.001% +Previous buildId mismatch: 20170907220212 must be older than 20170905100117: 0.001% +Previous buildId mismatch: 20170907220212 must be older than 20170905220108: 0.001% +Previous buildId mismatch: 20170907220212 must be older than 20170906100107: 0.001% +Previous buildId mismatch: 20170907220212 must be older than 20170907100318: 0.002% +Previous buildId mismatch: 20170908100218 must be older than 20170907100318: 0.001% +Previous buildId mismatch: 20170908100218 must be older than 20170907220212: 0.001% +Previous buildId mismatch: 20170908220146 must be older than 20170907100318: 0.001% +Previous buildId mismatch: 20170908220146 must be older than 20170907220212: 0.001% +Previous buildId mismatch: 20170908220146 must be older than 20170908100218: 0.001% +Previous channel mismatch: expected nightly-cck-google got nightly: 0.001% +Previous channel mismatch: expected nightly-cck-mint got nightly: 0.003% +Previous channel mismatch: expected nightly-cck-mozillaonline got nightly: 0.001% +Previous channel mismatch: expected nightly-cck-yandex got nightly: 0.001% +
The vast majority of the data in the payload seems reasonable (99.87%).
+However, a handful of update pings are reporting a previousBuildId mismatch: this is unexpected. After discussing this with the update team, it seems like this could either be due to Nigthly channel weirdness or to the customization applied by the CCK tool. Additionally, some pings are reporting a previousChannel different than the one in the environment: this is definitely due to the CCK tool, given the cck entry in the channel name. These issues do not represent a problem, as most of the data is correct and their volume is fairly low.
For each ping, build a key with the client id and the previous build update details. Since we expect to have exactly one ping for each successfully applied update, we don’t expect duplicate keys.
+update_dupes = deduped_subset.map(lambda p: ((p.get("clientId"), + p.get("payload/previousChannel"), + p.get("payload/previousVersion"), + p.get("payload/previousBuildId")), 1)).countByKey() + +print("Percentage of pings related to the same update (for the same client):\t{:.3f}%"\ + .format(pct(sum([v for v in update_dupes.values() if v > 1]), deduped_count))) +
Percentage of pings related to the same update (for the same client): 1.318% +
We’re receiving update pings with different documentId related to the same initial build, for a few clients. One possible reason for this could be users having multiple copies of Firefox installed on their machine. Let’s see if that’s the case.
clientIds_sending_dupes = [k[0] for k, v in update_dupes.iteritems() if v > 1] + +def check_same_original_build(ping_list): + # Build a "unique" identifier for the build by + # concatenating the buildId, channel and version. + unique_build_ids = [ + "{}{}{}".format(p.get("application/buildId"), p.get("application/channel"), p.get("application/version"))\ + for p in ping_list[1] + ] + + # Remove the duplicates and return True if all the pings came + # from the same build. + return len(set(unique_build_ids)) < 2 + +# Count how many duplicates are updating to the same builds and how many are +# updating to different builds. +original_builds_same =\ + deduped_subset.filter(lambda p: p.get("clientId") in clientIds_sending_dupes)\ + .map(lambda p: ((p.get("clientId"), + p.get("payload/previousChannel"), + p.get("payload/previousVersion"), + p.get("payload/previousBuildId")), [p]))\ + .reduceByKey(lambda a, b: a + b)\ + .filter(lambda p: len(p[1]) > 1)\ + .map(check_same_original_build).countByValue() + +print("Updated builds are identical:\t{:.3f}%"\ + .format(pct(original_builds_same.get(True), sum(original_builds_same.values())))) +print("Updated builds are different:\t{:.3f}%"\ + .format(pct(original_builds_same.get(False), sum(original_builds_same.values())))) +
Updated builds are identical: 16.472% +Updated builds are different: 83.528% +
The data shows that 83.52% of the 1.31% dupes are updating from different builds. The 0.22% of all update pings that are the same client updating to the same build from the same build are, at present, unexplained (but in small enough quantities we can ignore for the moment).
The update pings with the same previous build information may be coming from the same profile, copied and then used with different versions of Firefox. Depending on when the browser is started with a specific copied profile, the downloaded update blob might be different (more recent), thus resulting in an update with reason = success being sent with the same previous build information but with different current build information.
delays = deduped_subset.map(lambda p: calculate_submission_delay(p)) +
setup_plot("'update' ('success') ping submission delay CDF", + MAX_DELAY_S / HOUR_IN_S, area_border_x=1.0) + +plot_cdf(delays\ + .map(lambda d: d / HOUR_IN_S if d < MAX_DELAY_S else MAX_DELAY_S / HOUR_IN_S)\ + .collect(), label="CDF", linestyle="solid") + +plt.show() +
Almost 95% of the update pings with reason = success are submitted within an hour from the ping being created. Since we know that this ping is created as soon as the update is applied we can claim that we receive 95% of these pings within an hour from the update being applied.
Check if the volume of update pings with reason = ready matches with the volume of pings with reason = success. For each ping with reason = ready, find the matching ping with reason = success.
We are considering the data within a very narrow window of time: we could see reason = success pings from users that sent a reason = ready ping before the 3rd of September and reason = ready pings from users that have sent us a reason = success after the 9th of September. Filter these edge cases out by inspecting the previousBuildId and targetBuildId.
filtered_ready = ping_ready.filter(lambda p: p.get("payload/targetBuildId") < "{}999999".format(MAX_DATE)) +filtered_success = ping_success.filter(lambda p: p.get("payload/previousBuildId") >= "{}000000".format(MIN_DATE)) +
Use the filtered RDDs to match between the different ping reasons.
+# Get an identifier that keeps in consideration both the current build +# and the target build. +ready_uuid = filtered_ready\ + .map(lambda p: (p.get("clientId"), + p.get("application/buildId"), + p.get("application/channel"), + p.get("application/version"), + p.get("payload/targetBuildId"), + p.get("payload/targetChannel"), + p.get("payload/targetVersion"))) + +# Get an identifier that considers both the prevous build info and the +# current build info. The order of the values in the tuple need to match +# the one from the previous RDD. +success_uuid = filtered_success\ + .map(lambda p: (p.get("clientId"), + p.get("payload/previousBuildId"), + p.get("payload/previousChannel"), + p.get("payload/previousVersion"), + p.get("application/buildId"), + p.get("application/channel"), + p.get("application/version"))) +
Let’s match each reason = ready ping with a reason = success one, and count them.
matching_update_pings = ready_uuid.intersection(success_uuid) +matching_update_ping_count = matching_update_pings.count() +
Finally, show up some stats.
+print("{:.3f}% of the 'update' ping with reason 'ready' have a matching ping with reason 'success'."\ + .format(pct(matching_update_ping_count, filtered_ready.count()))) +print("{:.3f}% of the 'update' ping with reason 'success' have a matching ping with reason 'ready'."\ + .format(pct(matching_update_ping_count, filtered_success.count()))) +
63.834% of the 'update' ping with reason 'ready' have a matching ping with reason 'success'. +89.428% of the 'update' ping with reason 'success' have a matching ping with reason 'ready'. +
Only ~63% of the update ping sent when an update is ready to be applied have a corrensponding ping that’s sent, for the same client and upgrade path, after the update is successfully applied. One possible explaination for this is the delay with which updates get applied after they get downloaded: unless the browser is restarted (and that can happen after days due to user suspending their machines), we won’t see the ready ping anytime soon.
+Roughly 89% of the update pings with reason = success can be traced back to an update with reason = ready. The missing ~10% matches can be due to users disabling automatic updates (see this query) and other edge cases: no update ping is sent in that case if an update is manually triggered.
+
import ujson as json +import datetime as dt +import os.path +import boto3 +import botocore +import calendar +import requests +import moztelemetry.standards as moz_std + +%pylab inline +
def fetch_json(uri): + """ Perform an HTTP GET on the given uri, return the results as json. + If there is an error fetching the data, raise an exception. + + Args: + uri: the string URI to fetch. + + Returns: + A JSON object with the response. + """ + data = requests.get(uri) + # Raise an exception if the fetch failed. + data.raise_for_status() + return data.json() + +def get_OS_arch(browser_arch, os_name, is_wow64): + """ Infers the OS arch from environment data. + + Args: + browser_arch: the browser architecture string (either "x86" or "x86-64"). + os_name: the operating system name. + is_wow64: on Windows, indicates if the browser process is running under WOW64. + + Returns: + 'x86' if the underlying OS is 32bit, 'x86-64' if it's a 64bit OS. + """ + + is_64bit_browser = browser_arch == 'x86-64' + # If it's a 64bit browser build, then we're on a 64bit system. + if is_64bit_browser: + return 'x86-64' + + is_windows = os_name == 'Windows_NT' + # If we're on Windows, with a 32bit browser build, and |isWow64 = true|, + # then we're on a 64 bit system. + if is_windows and is_wow64: + return 'x86-64' + + # Otherwise we're probably on a 32 bit system. + return 'x86' + +def vendor_name_from_id(id): + """ Get the string name matching the provided vendor id. + + Args: + id: A string containing the vendor id. + + Returns: + A string containing the vendor name or "(Other <ID>)" if + unknown. + """ + + # TODO: We need to make this an external resource for easier + # future updates. + vendor_map = { + '0x1013': 'Cirrus Logic', + '0x1002': 'AMD', + '0x8086': 'Intel', + '0x5333': 'S3 Graphics', + '0x1039': 'SIS', + '0x1106': 'VIA', + '0x10de': 'NVIDIA', + '0x102b': 'Matrox', + '0x15ad': 'VMWare', + '0x80ee': 'Oracle VirtualBox', + '0x1414': 'Microsoft Basic', + } + + return vendor_map.get(id, "Other") + +def get_device_family_chipset(vendor_id, device_id): + """ Get the family and chipset strings given the vendor and device ids. + + Args: + vendor_id: a string representing the vendor id (e.g. '0xabcd'). + device_id: a string representing the device id (e.g. '0xbcde'). + + Returns: + A string in the format "Device Family Name-Chipset Name". + """ + if not vendor_id in device_map: + return "Unknown" + + if not device_id in device_map[vendor_id]: + return "Unknown" + + return "-".join(device_map[vendor_id][device_id]) + +def invert_device_map(m): + """ Inverts a GPU device map fetched from the jrmuizel's Github repo. + The layout of the fetched GPU map layout is: + Vendor ID -> Device Family -> Chipset -> [Device IDs] + We should convert it to: + Vendor ID -> Device ID -> [Device Family, Chipset] + """ + device_id_map = {} + for vendor, u in m.iteritems(): + device_id_map['0x' + vendor] = {} + for family, v in u.iteritems(): + for chipset, ids in v.iteritems(): + device_id_map['0x' + vendor].update({('0x' + gfx_id): [family, chipset] for gfx_id in ids}) + return device_id_map + +def build_device_map(): + """ This function builds a dictionary that will help us mapping vendor/device ids to a + human readable device family and chipset name.""" + + intel_raw = fetch_json("https://github.com/jrmuizel/gpu-db/raw/master/intel.json") + nvidia_raw = fetch_json("https://github.com/jrmuizel/gpu-db/raw/master/nvidia.json") + amd_raw = fetch_json("https://github.com/jrmuizel/gpu-db/raw/master/amd.json") + + device_map = {} + device_map.update(invert_device_map(intel_raw)) + device_map.update(invert_device_map(nvidia_raw)) + device_map.update(invert_device_map(amd_raw)) + + return device_map + +device_map = build_device_map() +
# Reasons why the data for a client can be discarded. +REASON_INACTIVE = "inactive" +REASON_BROKEN_DATA = "broken" + +def get_valid_client_record(r, data_index): + """ Check if the referenced record is sane or contains partial/broken data. + + Args: + r: The client entry in the longitudinal dataset. + dat_index: The index of the sample within the client record. + + Returns: + An object containing the client hardware data or REASON_BROKEN_DATA if the + data is invalid. + """ + gfx_adapters = r["system_gfx"][data_index]["adapters"] + monitors = r["system_gfx"][data_index]["monitors"] + + # We should make sure to have GFX adapter. If we don't, discard this record. + if not gfx_adapters or not gfx_adapters[0]: + return REASON_BROKEN_DATA + + # Due to bug 1175005, Firefox on Linux isn't sending the screen resolution data. + # Don't discard the rest of the ping for that: just set the resolution to 0 if + # unavailable. See bug 1324014 for context. + screen_width = 0 + screen_height = 0 + if monitors and monitors[0]: + screen_width = monitors[0]["screen_width"] + screen_height = monitors[0]["screen_height"] + + # Non Windows OS do not have that property. + is_wow64 = r["system"][data_index]["is_wow64"] == True + + # At this point, we should have filtered out all the weirdness. Fetch + # the data we need. + data = { + 'browser_arch': r["build"][data_index]["architecture"], + 'os_name': r["system_os"][data_index]["name"], + 'os_version': r["system_os"][data_index]["version"], + 'memory_mb': r["system"][data_index]["memory_mb"], + 'is_wow64': is_wow64, + 'num_gfx_adapters': len(gfx_adapters), + 'gfx0_vendor_id': gfx_adapters[0]["vendor_id"], + 'gfx0_device_id': gfx_adapters[0]["device_id"], + 'screen_width': screen_width, + 'screen_height': screen_height, + 'cpu_cores': r["system_cpu"][data_index]["cores"], + 'cpu_vendor': r["system_cpu"][data_index]["vendor"], + 'cpu_speed': r["system_cpu"][data_index]["speed_mhz"], + 'has_flash': False + } + + # The plugins data can still be null or empty, account for that. + plugins = r["active_plugins"][data_index] if r["active_plugins"] else None + if plugins: + data['has_flash'] = any([True for p in plugins if p['name'] == 'Shockwave Flash']) + + return REASON_BROKEN_DATA if None in data.values() else data + +def get_latest_valid_per_client(entry): + """ Get the most recently submitted ping for a client. + + Then use this index to look up the data from the other columns (we can assume that the sizes + of these arrays match, otherwise the longitudinal dataset is broken). + Once we have the data, we make sure it's valid and return it. + + Args: + entry: The record containing all the data for a single client. + + Returns: + An object containing the valid hardware data for the client or + REASON_BROKEN_DATA if it send broken data. + + Raises: + ValueError: if the columns within the record have mismatching lengths. This + means the longitudinal dataset is corrupted. + """ + + # Some clients might be missing entire sections. If that's + # a basic section, skip them, we don't want partial data. + # Don't enforce the presence of "active_plugins", as it's not included + # by the pipeline if no plugin is reported by Firefox (see bug 1333806). + desired_sections = [ + "build", "system_os", "submission_date", "system", + "system_gfx", "system_cpu" + ] + + for field in desired_sections: + if entry[field] is None: + return REASON_BROKEN_DATA + + # All arrays in the longitudinal dataset should have the same length, for a + # single client. If that's not the case, if our index is not there, throw. + if entry[field][0] is None: + raise ValueError("Null " + field) + + return get_valid_client_record(entry, 0) +
def prepare_data(p): + """ This function prepares the data for further analyses (e.g. unit conversion, + vendor id to string, ...). """ + cpu_speed = round(p['cpu_speed'] / 1000.0, 1) + return { + 'browser_arch': p['browser_arch'], + 'cpu_cores': p['cpu_cores'], + 'cpu_cores_speed': str(p['cpu_cores']) + '_' + str(cpu_speed), + 'cpu_vendor': p['cpu_vendor'], + 'cpu_speed': cpu_speed, + 'num_gfx_adapters': p['num_gfx_adapters'], + 'gfx0_vendor_name': vendor_name_from_id(p['gfx0_vendor_id']), + 'gfx0_model': get_device_family_chipset(p['gfx0_vendor_id'], p['gfx0_device_id']), + 'resolution': str(p['screen_width']) + 'x' + str(p['screen_height']), + 'memory_gb': int(round(p['memory_mb'] / 1024.0)), + 'os': p['os_name'] + '-' + p['os_version'], + 'os_arch': get_OS_arch(p['browser_arch'], p['os_name'], p['is_wow64']), + 'has_flash': p['has_flash'] + } + +def aggregate_data(processed_data): + def seq(acc, v): + # The dimensions over which we want to aggregate the different values. + keys_to_aggregate = [ + 'browser_arch', + 'cpu_cores', + 'cpu_cores_speed', + 'cpu_vendor', + 'cpu_speed', + 'num_gfx_adapters', + 'gfx0_vendor_name', + 'gfx0_model', + 'resolution', + 'memory_gb', + 'os', + 'os_arch', + 'has_flash' + ] + + for key_name in keys_to_aggregate: + # We want to know how many users have a particular configuration (e.g. using a particular + # cpu vendor). For each dimension of interest, build a key as (hw, value) and count its + # occurrences among the user base. + acc_key = (key_name, v[key_name]) + acc[acc_key] = acc.get(acc_key, 0) + 1 + + return acc + + def cmb(v1, v2): + # Combine the counts from the two partial dictionaries. Hacky? + return { k: v1.get(k, 0) + v2.get(k, 0) for k in set(v1) | set(v2) } + + return processed_data.aggregate({}, seq, cmb) + +def collapse_buckets(aggregated_data, count_threshold): + """ Collapse uncommon configurations in generic groups to preserve privacy. + + This takes the dictionary of aggregated results from |aggregate_data| and collapses + entries with a value less than |count_threshold| in a generic bucket. + + Args: + aggregated_data: The object containing aggregated data. + count_threhold: Groups (or "configurations") containing less than this value + are collapsed in a generic bucket. + """ + + # These fields have a fixed set of values and we need to report all of them. + EXCLUSION_LIST = [ "has_flash", "browser_arch", "os_arch" ] + + collapsed_groups = {} + for k,v in aggregated_data.iteritems(): + key_type = k[0] + + # If the resolution is 0x0 (see bug 1324014), put that into the "Other" + # bucket. + if key_type == 'resolution' and k[1] == '0x0': + other_key = ('resolution', 'Other') + collapsed_groups[other_key] = collapsed_groups.get(other_key, 0) + v + continue + + # Don't clump this group into the "Other" bucket if it has enough + # users it in. + if v > count_threshold or key_type in EXCLUSION_LIST: + collapsed_groups[k] = v + continue + + # If we're here, it means that the key has not enough elements. + # Fall through the next cases and try to group things together. + new_group_key = 'Other' + + # Let's try to group similar resolutions together. + if key_type == 'resolution': + # Extract the resolution. + [w, h] = k[1].split('x') + # Round to the nearest hundred. + w = int(round(int(w), -2)) + h = int(round(int(h), -2)) + # Build up a new key. + new_group_key = '~' + str(w) + 'x' + str(h) + elif key_type == 'os': + [os, ver] = k[1].split('-', 1) + new_group_key = os + '-' + 'Other' + + # We don't have enough data for this particular group/configuration. + # Aggregate it with the data in the "Other" bucket + other_key = (k[0], new_group_key) + collapsed_groups[other_key] = collapsed_groups.get(other_key, 0) + v + + # The previous grouping might have created additional groups. Let's check again. + final_groups = {} + for k,v in collapsed_groups.iteritems(): + # Don't clump this group into the "Other" bucket if it has enough + # users it in. + if (v > count_threshold and k[1] != 'Other') or k[0] in EXCLUSION_LIST: + final_groups[k] = v + continue + + # We don't have enough data for this particular group/configuration. + # Aggregate it with the data in the "Other" bucket + other_key = (k[0], 'Other') + final_groups[other_key] = final_groups.get(other_key, 0) + v + + return final_groups + + +def finalize_data(data, sample_count, broken_ratio): + """ Finalize the aggregated data. + + Translate raw sample numbers to percentages and add the date for the reported + week along with the percentage of discarded samples due to broken data. + + Rename the keys to more human friendly names. + + Args: + data: Data in aggregated form. + sample_count: The number of samples the aggregates where generated from. + broken_ratio: The percentage of samples discarded due to broken data. + inactive_ratio: The percentage of samples discarded due to the client not sending data. + report_date: The starting day for the reported week. + + Returns: + An object containing the reported hardware statistics. + """ + + denom = float(sample_count) + + aggregated_percentages = { + 'broken': broken_ratio, + } + + keys_translation = { + 'browser_arch': 'browserArch_', + 'cpu_cores': 'cpuCores_', + 'cpu_cores_speed': 'cpuCoresSpeed_', + 'cpu_vendor': 'cpuVendor_', + 'cpu_speed': 'cpuSpeed_', + 'num_gfx_adapters': 'gpuNum_', + 'gfx0_vendor_name': 'gpuVendor_', + 'gfx0_model': 'gpuModel_', + 'resolution': 'resolution_', + 'memory_gb': 'ram_', + 'os': 'osName_', + 'os_arch': 'osArch_', + 'has_flash': 'hasFlash_' + } + + # Compute the percentages from the raw numbers. + for k, v in data.iteritems(): + # The old key is a tuple (key, value). We translate the key part and concatenate the + # value as a string. + new_key = keys_translation[k[0]] + unicode(k[1]) + aggregated_percentages[new_key] = v / denom + + return aggregated_percentages +
We compute the hardware report for users running Windows 7 or Windows 10 by taking the most recent data available.
+# Connect to the longitudinal dataset and get a subset of the columns +sqlQuery = "SELECT " +\ + "build," +\ + "client_id," +\ + "active_plugins," +\ + "system_os," +\ + "submission_date," +\ + "system," +\ + "system_gfx," +\ + "system_cpu," +\ + "normalized_channel " +\ + "FROM longitudinal" +frame = sqlContext.sql(sqlQuery)\ + .where("normalized_channel = 'release'")\ + .where("system_os is not null and system_os[0].name = 'Windows_NT'")\ + .where("build is not null and build[0].application_name = 'Firefox'") + + +# The number of all the fetched records (including inactive and broken). +records_count = frame.count() +
data = frame.rdd.map(lambda r: get_latest_valid_per_client(r)) + +# Filter out broken data. +filtered_data = data.filter(lambda r: r is not REASON_BROKEN_DATA) + +# Count the broken records +broken_count = data.filter(lambda r: r is REASON_BROKEN_DATA).count() +print("Found {} broken records.".format(broken_count)) +
This extracts the relevant information from each valid data unit returned from the previous step. Each processed_data entry represents a single user machine.
+processed_data = filtered_data.map(prepare_data) +
processed_data.first() +
Aggregate the machine configurations in a more digestible form.
+# Aggregate the data for Windows 7 (Windows NT version 6.1) +windows7_data = processed_data.filter(lambda p: p.get("os") == "Windows_NT-6.1") +aggregated_w7 = aggregate_data(windows7_data) +windows7_count = windows7_data.count() +
# Aggregate the data for Windows 10 (Windows NT version 10.0) +windows10_data = processed_data.filter(lambda p: p.get("os") == "Windows_NT-10.0") +aggregated_w10 = aggregate_data(windows10_data) +windows10_count = windows10_data.count() +
Collapse together groups that count less than 1% of our samples.
+valid_records_count = records_count - broken_count +threshold_to_collapse = int(valid_records_count * 0.01) + +print "Collapsing smaller groups into the other bucket (threshold {th})".format(th=threshold_to_collapse) +collapsed_w7 = collapse_buckets(aggregated_w7, threshold_to_collapse) +collapsed_w10 = collapse_buckets(aggregated_w10, threshold_to_collapse) +
broken_ratio = broken_count / float(records_count) + +w7_json = finalize_data(collapsed_w7, windows7_count, broken_ratio) +json_entry = json.dumps(w7_json) +with open("w7data.json", "w") as json_file: + json_file.write("[" + json_entry.encode('utf8') + "]\n") + + +w10_json = finalize_data(collapsed_w10, windows10_count, broken_ratio) +json_entry = json.dumps(w10_json) +with open("w10data.json", "w") as json_file: + json_file.write("[" + json_entry.encode('utf8') + "]\n") +
+
import ujson as json +import datetime as dt +import os.path +import boto3 +import botocore +import calendar +import requests +import moztelemetry.standards as moz_std + +%pylab inline +
def fetch_json(uri): + """ Perform an HTTP GET on the given uri, return the results as json. + If there is an error fetching the data, raise an exception. + + Args: + uri: the string URI to fetch. + + Returns: + A JSON object with the response. + """ + data = requests.get(uri) + # Raise an exception if the fetch failed. + data.raise_for_status() + return data.json() + +def get_OS_arch(browser_arch, os_name, is_wow64): + """ Infers the OS arch from environment data. + + Args: + browser_arch: the browser architecture string (either "x86" or "x86-64"). + os_name: the operating system name. + is_wow64: on Windows, indicates if the browser process is running under WOW64. + + Returns: + 'x86' if the underlying OS is 32bit, 'x86-64' if it's a 64bit OS. + """ + + is_64bit_browser = browser_arch == 'x86-64' + # If it's a 64bit browser build, then we're on a 64bit system. + if is_64bit_browser: + return 'x86-64' + + is_windows = os_name == 'Windows_NT' + # If we're on Windows, with a 32bit browser build, and |isWow64 = true|, + # then we're on a 64 bit system. + if is_windows and is_wow64: + return 'x86-64' + + # Otherwise we're probably on a 32 bit system. + return 'x86' + +def vendor_name_from_id(id): + """ Get the string name matching the provided vendor id. + + Args: + id: A string containing the vendor id. + + Returns: + A string containing the vendor name or "(Other <ID>)" if + unknown. + """ + + # TODO: We need to make this an external resource for easier + # future updates. + vendor_map = { + '0x1013': 'Cirrus Logic', + '0x1002': 'AMD', + '0x8086': 'Intel', + '0x5333': 'S3 Graphics', + '0x1039': 'SIS', + '0x1106': 'VIA', + '0x10de': 'NVIDIA', + '0x102b': 'Matrox', + '0x15ad': 'VMWare', + '0x80ee': 'Oracle VirtualBox', + '0x1414': 'Microsoft Basic', + } + + return vendor_map.get(id, "Other") + +def get_device_family_chipset(vendor_id, device_id): + """ Get the family and chipset strings given the vendor and device ids. + + Args: + vendor_id: a string representing the vendor id (e.g. '0xabcd'). + device_id: a string representing the device id (e.g. '0xbcde'). + + Returns: + A string in the format "Device Family Name-Chipset Name". + """ + if not vendor_id in device_map: + return "Unknown" + + if not device_id in device_map[vendor_id]: + return "Unknown" + + return "-".join(device_map[vendor_id][device_id]) + +def invert_device_map(m): + """ Inverts a GPU device map fetched from the jrmuizel's Github repo. + The layout of the fetched GPU map layout is: + Vendor ID -> Device Family -> Chipset -> [Device IDs] + We should convert it to: + Vendor ID -> Device ID -> [Device Family, Chipset] + """ + device_id_map = {} + for vendor, u in m.iteritems(): + device_id_map['0x' + vendor] = {} + for family, v in u.iteritems(): + for chipset, ids in v.iteritems(): + device_id_map['0x' + vendor].update({('0x' + gfx_id): [family, chipset] for gfx_id in ids}) + return device_id_map + +def build_device_map(): + """ This function builds a dictionary that will help us mapping vendor/device ids to a + human readable device family and chipset name.""" + + intel_raw = fetch_json("https://github.com/jrmuizel/gpu-db/raw/master/intel.json") + nvidia_raw = fetch_json("https://github.com/jrmuizel/gpu-db/raw/master/nvidia.json") + amd_raw = fetch_json("https://github.com/jrmuizel/gpu-db/raw/master/amd.json") + + device_map = {} + device_map.update(invert_device_map(intel_raw)) + device_map.update(invert_device_map(nvidia_raw)) + device_map.update(invert_device_map(amd_raw)) + + return device_map + +device_map = build_device_map() +
# Reasons why the data for a client can be discarded. +REASON_INACTIVE = "inactive" +REASON_BROKEN_DATA = "broken" + +def get_valid_client_record(r, data_index): + """ Check if the referenced record is sane or contains partial/broken data. + + Args: + r: The client entry in the longitudinal dataset. + dat_index: The index of the sample within the client record. + + Returns: + An object containing the client hardware data or REASON_BROKEN_DATA if the + data is invalid. + """ + gfx_adapters = r["system_gfx"][data_index]["adapters"] + monitors = r["system_gfx"][data_index]["monitors"] + + # We should make sure to have GFX adapter. If we don't, discard this record. + if not gfx_adapters or not gfx_adapters[0]: + return REASON_BROKEN_DATA + + # Due to bug 1175005, Firefox on Linux isn't sending the screen resolution data. + # Don't discard the rest of the ping for that: just set the resolution to 0 if + # unavailable. See bug 1324014 for context. + screen_width = 0 + screen_height = 0 + if monitors and monitors[0]: + screen_width = monitors[0]["screen_width"] + screen_height = monitors[0]["screen_height"] + + # Non Windows OS do not have that property. + is_wow64 = r["system"][data_index]["is_wow64"] == True + + # At this point, we should have filtered out all the weirdness. Fetch + # the data we need. + data = { + 'browser_arch': r["build"][data_index]["architecture"], + 'os_name': r["system_os"][data_index]["name"], + 'os_version': r["system_os"][data_index]["version"], + 'memory_mb': r["system"][data_index]["memory_mb"], + 'is_wow64': is_wow64, + 'num_gfx_adapters': len(gfx_adapters), + 'gfx0_vendor_id': gfx_adapters[0]["vendor_id"], + 'gfx0_device_id': gfx_adapters[0]["device_id"], + 'screen_width': screen_width, + 'screen_height': screen_height, + 'cpu_cores': r["system_cpu"][data_index]["cores"], + 'cpu_vendor': r["system_cpu"][data_index]["vendor"], + 'cpu_speed': r["system_cpu"][data_index]["speed_mhz"], + 'has_flash': False + } + + # The plugins data can still be null or empty, account for that. + plugins = r["active_plugins"][data_index] if r["active_plugins"] else None + if plugins: + data['has_flash'] = any([True for p in plugins if p['name'] == 'Shockwave Flash']) + + return REASON_BROKEN_DATA if None in data.values() else data + +def get_latest_valid_per_client(entry): + """ Get the most recently submitted ping for a client. + + Then use this index to look up the data from the other columns (we can assume that the sizes + of these arrays match, otherwise the longitudinal dataset is broken). + Once we have the data, we make sure it's valid and return it. + + Args: + entry: The record containing all the data for a single client. + + Returns: + An object containing the valid hardware data for the client or + REASON_BROKEN_DATA if it send broken data. + + Raises: + ValueError: if the columns within the record have mismatching lengths. This + means the longitudinal dataset is corrupted. + """ + + # Some clients might be missing entire sections. If that's + # a basic section, skip them, we don't want partial data. + # Don't enforce the presence of "active_plugins", as it's not included + # by the pipeline if no plugin is reported by Firefox (see bug 1333806). + desired_sections = [ + "build", "system_os", "submission_date", "system", + "system_gfx", "system_cpu" + ] + + for field in desired_sections: + if entry[field] is None: + return REASON_BROKEN_DATA + + # All arrays in the longitudinal dataset should have the same length, for a + # single client. If that's not the case, if our index is not there, throw. + if entry[field][0] is None: + raise ValueError("Null " + field) + + return get_valid_client_record(entry, 0) +
def prepare_data(p): + """ This function prepares the data for further analyses (e.g. unit conversion, + vendor id to string, ...). """ + cpu_speed = round(p['cpu_speed'] / 1000.0, 1) + return { + 'browser_arch': p['browser_arch'], + 'cpu_cores': p['cpu_cores'], + 'cpu_cores_speed': str(p['cpu_cores']) + '_' + str(cpu_speed), + 'cpu_vendor': p['cpu_vendor'], + 'cpu_speed': cpu_speed, + 'num_gfx_adapters': p['num_gfx_adapters'], + 'gfx0_vendor_name': vendor_name_from_id(p['gfx0_vendor_id']), + 'gfx0_model': get_device_family_chipset(p['gfx0_vendor_id'], p['gfx0_device_id']), + 'resolution': str(p['screen_width']) + 'x' + str(p['screen_height']), + 'memory_gb': int(round(p['memory_mb'] / 1024.0)), + 'os': p['os_name'] + '-' + p['os_version'], + 'os_arch': get_OS_arch(p['browser_arch'], p['os_name'], p['is_wow64']), + 'has_flash': p['has_flash'] + } + +def aggregate_data(processed_data): + def seq(acc, v): + # The dimensions over which we want to aggregate the different values. + keys_to_aggregate = [ + 'browser_arch', + 'cpu_cores', + 'cpu_cores_speed', + 'cpu_vendor', + 'cpu_speed', + 'num_gfx_adapters', + 'gfx0_vendor_name', + 'gfx0_model', + 'resolution', + 'memory_gb', + 'os', + 'os_arch', + 'has_flash' + ] + + for key_name in keys_to_aggregate: + # We want to know how many users have a particular configuration (e.g. using a particular + # cpu vendor). For each dimension of interest, build a key as (hw, value) and count its + # occurrences among the user base. + acc_key = (key_name, v[key_name]) + acc[acc_key] = acc.get(acc_key, 0) + 1 + + return acc + + def cmb(v1, v2): + # Combine the counts from the two partial dictionaries. Hacky? + return { k: v1.get(k, 0) + v2.get(k, 0) for k in set(v1) | set(v2) } + + return processed_data.aggregate({}, seq, cmb) + +def collapse_buckets(aggregated_data, count_threshold): + """ Collapse uncommon configurations in generic groups to preserve privacy. + + This takes the dictionary of aggregated results from |aggregate_data| and collapses + entries with a value less than |count_threshold| in a generic bucket. + + Args: + aggregated_data: The object containing aggregated data. + count_threhold: Groups (or "configurations") containing less than this value + are collapsed in a generic bucket. + """ + + # These fields have a fixed set of values and we need to report all of them. + EXCLUSION_LIST = [ "has_flash", "browser_arch", "os_arch" ] + + collapsed_groups = {} + for k,v in aggregated_data.iteritems(): + key_type = k[0] + + # If the resolution is 0x0 (see bug 1324014), put that into the "Other" + # bucket. + if key_type == 'resolution' and k[1] == '0x0': + other_key = ('resolution', 'Other') + collapsed_groups[other_key] = collapsed_groups.get(other_key, 0) + v + continue + + # Don't clump this group into the "Other" bucket if it has enough + # users it in. + if v > count_threshold or key_type in EXCLUSION_LIST: + collapsed_groups[k] = v + continue + + # If we're here, it means that the key has not enough elements. + # Fall through the next cases and try to group things together. + new_group_key = 'Other' + + # Let's try to group similar resolutions together. + if key_type == 'resolution': + # Extract the resolution. + [w, h] = k[1].split('x') + # Round to the nearest hundred. + w = int(round(int(w), -2)) + h = int(round(int(h), -2)) + # Build up a new key. + new_group_key = '~' + str(w) + 'x' + str(h) + elif key_type == 'os': + [os, ver] = k[1].split('-', 1) + new_group_key = os + '-' + 'Other' + + # We don't have enough data for this particular group/configuration. + # Aggregate it with the data in the "Other" bucket + other_key = (k[0], new_group_key) + collapsed_groups[other_key] = collapsed_groups.get(other_key, 0) + v + + # The previous grouping might have created additional groups. Let's check again. + final_groups = {} + for k,v in collapsed_groups.iteritems(): + # Don't clump this group into the "Other" bucket if it has enough + # users it in. + if (v > count_threshold and k[1] != 'Other') or k[0] in EXCLUSION_LIST: + final_groups[k] = v + continue + + # We don't have enough data for this particular group/configuration. + # Aggregate it with the data in the "Other" bucket + other_key = (k[0], 'Other') + final_groups[other_key] = final_groups.get(other_key, 0) + v + + return final_groups + + +def finalize_data(data, sample_count, broken_ratio): + """ Finalize the aggregated data. + + Translate raw sample numbers to percentages and add the date for the reported + week along with the percentage of discarded samples due to broken data. + + Rename the keys to more human friendly names. + + Args: + data: Data in aggregated form. + sample_count: The number of samples the aggregates where generated from. + broken_ratio: The percentage of samples discarded due to broken data. + inactive_ratio: The percentage of samples discarded due to the client not sending data. + report_date: The starting day for the reported week. + + Returns: + An object containing the reported hardware statistics. + """ + + denom = float(sample_count) + + aggregated_percentages = { + 'broken': broken_ratio, + } + + keys_translation = { + 'browser_arch': 'browserArch_', + 'cpu_cores': 'cpuCores_', + 'cpu_cores_speed': 'cpuCoresSpeed_', + 'cpu_vendor': 'cpuVendor_', + 'cpu_speed': 'cpuSpeed_', + 'num_gfx_adapters': 'gpuNum_', + 'gfx0_vendor_name': 'gpuVendor_', + 'gfx0_model': 'gpuModel_', + 'resolution': 'resolution_', + 'memory_gb': 'ram_', + 'os': 'osName_', + 'os_arch': 'osArch_', + 'has_flash': 'hasFlash_' + } + + # Compute the percentages from the raw numbers. + for k, v in data.iteritems(): + # The old key is a tuple (key, value). We translate the key part and concatenate the + # value as a string. + new_key = keys_translation[k[0]] + unicode(k[1]) + aggregated_percentages[new_key] = v / denom + + return aggregated_percentages +
We compute the hardware report for users running Windows 7 or Windows 10 by taking the most recent data available.
+# Connect to the longitudinal dataset and get a subset of the columns +sqlQuery = "SELECT " +\ + "build," +\ + "client_id," +\ + "active_plugins," +\ + "system_os," +\ + "submission_date," +\ + "system," +\ + "system_gfx," +\ + "system_cpu," +\ + "normalized_channel " +\ + "FROM longitudinal" +frame = sqlContext.sql(sqlQuery)\ + .where("normalized_channel = 'release'")\ + .where("system_os is not null and system_os[0].name = 'Windows_NT'")\ + .where("build is not null and build[0].application_name = 'Firefox'") + + +# The number of all the fetched records (including inactive and broken). +records_count = frame.count() +
data = frame.rdd.map(lambda r: get_latest_valid_per_client(r)) + +# Filter out broken data. +filtered_data = data.filter(lambda r: r is not REASON_BROKEN_DATA) + +# Count the broken records +broken_count = data.filter(lambda r: r is REASON_BROKEN_DATA).count() +print("Found {} broken records.".format(broken_count)) +
This extracts the relevant information from each valid data unit returned from the previous step. Each processed_data entry represents a single user machine.
+processed_data = filtered_data.map(prepare_data) +
processed_data.first() +
Aggregate the machine configurations in a more digestible form.
+# Aggregate the data for Windows 7 (Windows NT version 6.1) +windows7_data = processed_data.filter(lambda p: p.get("os") == "Windows_NT-6.1") +aggregated_w7 = aggregate_data(windows7_data) +windows7_count = windows7_data.count() +
# Aggregate the data for Windows 10 (Windows NT version 10.0) +windows10_data = processed_data.filter(lambda p: p.get("os") == "Windows_NT-10.0") +aggregated_w10 = aggregate_data(windows10_data) +windows10_count = windows10_data.count() +
Collapse together groups that count less than 1% of our samples.
+valid_records_count = records_count - broken_count +threshold_to_collapse = int(valid_records_count * 0.01) + +print "Collapsing smaller groups into the other bucket (threshold {th})".format(th=threshold_to_collapse) +collapsed_w7 = collapse_buckets(aggregated_w7, threshold_to_collapse) +collapsed_w10 = collapse_buckets(aggregated_w10, threshold_to_collapse) +
broken_ratio = broken_count / float(records_count) + +w7_json = finalize_data(collapsed_w7, windows7_count, broken_ratio) +json_entry = json.dumps(w7_json) +with open("w7data.json", "w") as json_file: + json_file.write("[" + json_entry.encode('utf8') + "]\n") + + +w10_json = finalize_data(collapsed_w10, windows10_count, broken_ratio) +json_entry = json.dumps(w10_json) +with open("w10data.json", "w") as json_file: + json_file.write("[" + json_entry.encode('utf8') + "]\n") +
+
The longitudinal dataset is logically organized as a table where rows represent profiles and columns the various metrics (e.g. startup time). Each field of the table contains a list of values, one per Telemetry submission received for that profile.
+The dataset is going to be regenerated from scratch every week, this allows us to apply non backward compatible changes to the schema and not worry about merging procedures.
+The current version of the longitudinal dataset has been build with all main pings received from 1% of profiles across all channels after mid November, which is shortly after Unified Telemetry landed. Future version will store up to 180 days of data.
+import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py + +%pylab inline +
Populating the interactive namespace from numpy and matplotlib +
sc.defaultParallelism +
32 +
The longitudinal dataset can be accessed as a Spark DataFrame, which is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R/Python.
+frame = sqlContext.sql("SELECT * FROM longitudinal") +
Number of profiles:
+frame.count() +
5421821 +
The dataset contains all histograms but it doesn’t yet include all metrics stored in the various sections of the pings. See the code that generates the dataset for a complete list of supported metrics. More metrics are going to be included in future versions of the dataset, inclusion of specific metrics can be prioritized by filing a bug.
+A Spark bug is slowing down the first and take methods on a dataframe. A way around that for now is to first convert the dataframe to a rdd and then invoke first or take, e.g.:
+first = frame.filter("normalized_channel = 'release'")\ + .select("build", + "system", + "gc_ms", + "fxa_configured", + "browser_set_default_always_check", + "browser_set_default_dialog_prompt_rawcount")\ + .rdd.first() +
As mentioned earlier on, each field of the dataframe is an array containing one value per submission per client. The submissions are chronologically sorted.
+len(first.build) +
103 +
first.build[:5] +
[Row(application_id=u'{ec8030f7-c20a-464f-9b0e-13a3a9e97384}', application_name=u'Firefox', architecture=u'x86', architectures_in_binary=None, build_id=u'20160210153822', version=u'44.0.2', vendor=u'Mozilla', platform_version=u'44.0.2', xpcom_abi=u'x86-msvc', hotfix_version=u'20160128.01'),
+ Row(application_id=u'{ec8030f7-c20a-464f-9b0e-13a3a9e97384}', application_name=u'Firefox', architecture=u'x86', architectures_in_binary=None, build_id=u'20160210153822', version=u'44.0.2', vendor=u'Mozilla', platform_version=u'44.0.2', xpcom_abi=u'x86-msvc', hotfix_version=u'20160128.01'),
+ Row(application_id=u'{ec8030f7-c20a-464f-9b0e-13a3a9e97384}', application_name=u'Firefox', architecture=u'x86', architectures_in_binary=None, build_id=u'20160210153822', version=u'44.0.2', vendor=u'Mozilla', platform_version=u'44.0.2', xpcom_abi=u'x86-msvc', hotfix_version=u'20160128.01'),
+ Row(application_id=u'{ec8030f7-c20a-464f-9b0e-13a3a9e97384}', application_name=u'Firefox', architecture=u'x86', architectures_in_binary=None, build_id=u'20160210153822', version=u'44.0.2', vendor=u'Mozilla', platform_version=u'44.0.2', xpcom_abi=u'x86-msvc', hotfix_version=u'20160128.01'),
+ Row(application_id=u'{ec8030f7-c20a-464f-9b0e-13a3a9e97384}', application_name=u'Firefox', architecture=u'x86', architectures_in_binary=None, build_id=u'20160210153822', version=u'44.0.2', vendor=u'Mozilla', platform_version=u'44.0.2', xpcom_abi=u'x86-msvc', hotfix_version=u'20160128.01')]
+Different sections of the ping are stored in different fields of the dataframe. Refer to the schema of the dataframe for a complete layout.
+first.system[0] +
Row(memory_mb=1909, virtual_max_mb=None, is_wow64=True) +
Dataframes support fields that can contain structs, maps, arrays, scalars and combination thereof. Note that in the previous example the system field is an array of Rows. You can think of a Row as a struct that allows each field to be accessed invididually.
+first.system[0].memory_mb +
1909 +
Not all profiles have all histograms. If a certain histogram, say GC_MS, is N/A for all submissions of a profile, then the field in the DataFrame will be N/A.
+first.gc_ms == None +
True +
If at least one histogram is present in the history of a profile, then all other submission that do not have that histogram will be initialized with an empty histogram.
+Flag and count “histograms” are represented as scalars.
+first.fxa_configured[:5] +
[False, False, False, False, False] +
Boolean histograms are represented with an array of two integers. Similarly, enumerated histograms are represented with an array of N integers.
+first.browser_set_default_always_check[:5] +
[[0, 1], [0, 0], [0, 0], [0, 1], [0, 1]] +
Exponential and linear histograms are represented as a struct containing an array of N integers (values field) and the sum of the entries (sum field).
+first.browser_set_default_dialog_prompt_rawcount[:5] +
[Row(values=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], sum=0), + Row(values=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], sum=0), + Row(values=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], sum=0), + Row(values=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], sum=0), + Row(values=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], sum=0)] +
Keyed histograms are stored within a map from strings to values where the values depend on the histogram types and and have the same structure as mentioned above.
+frame.select("search_counts").rdd.take(2) +
[Row(search_counts={u'amazondotcom-de.searchbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], u'google.searchbar': [0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 1, 1, 1, 1, 0, 1, 0, 1, 5, 1, 1, 5, 1, 0, 1, 1, 1, 1, 1, 0, 2, 1, 1, 1, 1, 6, 1, 0, 0, 1, 1, 3, 1, 0, 3, 2, 4, 0, 1, 1, 0, 0, 2, 2, 0, 0, 3, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 1, 2, 0], u'google.urlbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0], u'other-Bing\xae.searchbar': [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}),
+ Row(search_counts={u'yandex.urlbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 2, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], u'yandex.searchbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 2, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 1], u'yandex.contextmenu': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], u'google.searchbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], u'wikipedia-ru.searchbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]})]
+Note that the following queries are run on a single machine and have been annotated with their run-time.
+%time frame.select("system").rdd.first().system[:2] +
CPU times: user 8 ms, sys: 0 ns, total: 8 ms +Wall time: 1.53 s + + + + + + +[Row(memory_mb=1909, virtual_max_mb=None, is_wow64=True), + Row(memory_mb=1909, virtual_max_mb=None, is_wow64=True)] +
%time frame.select("system.memory_mb").rdd.first() +
CPU times: user 8 ms, sys: 4 ms, total: 12 ms +Wall time: 1.57 s + + + + + + +Row(memory_mb=[1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909]) +
%time frame.selectExpr("size(system.memory_mb) as num_submissions").rdd.take(5) +
CPU times: user 4 ms, sys: 4 ms, total: 8 ms +Wall time: 1.63 s + + + + + + +[Row(num_submissions=103), + Row(num_submissions=999), + Row(num_submissions=1), + Row(num_submissions=455), + Row(num_submissions=144)] +
%time frame.selectExpr("system_os.name[0] as os_name").rdd.take(5) +
CPU times: user 12 ms, sys: 4 ms, total: 16 ms +Wall time: 1.58 s + + + + + + +[Row(os_name=u'Windows_NT'), + Row(os_name=u'Windows_NT'), + Row(os_name=u'Windows_NT'), + Row(os_name=u'Windows_NT'), + Row(os_name=u'Windows_NT')] +
%time frame.selectExpr("system_os.name[0] as os_name").where("os_name = 'Darwin'").count() +
CPU times: user 12 ms, sys: 8 ms, total: 20 ms +Wall time: 2min 26s + + + + + + +262622 +
Note that metrics that don’t tend to change often can be “uplifted” from their nested structure for fast selection. One of such metrics is the operating system name. More metrics can be uplifed on request.
+%time frame.select("os").where("os = 'Darwin'").count() +
CPU times: user 8 ms, sys: 0 ns, total: 8 ms +Wall time: 1min 15s + + + + + + +262622 +
Dataframes can be transformed to RDDs that allow to easily apply user defined functions. In general it’s worthwhile spending some time learning the Dataframe API as operations are optimized and run entirely in the JVM which can make queries faster.
+rdd = frame.rdd +
out = rdd.map(lambda x: x.search_counts).take(2) +
Select the earliest build-id with which a profile was seen using window functions:
+from pyspark.sql.window import Window +from pyspark.sql import Row +import pyspark.sql.functions as func +
subset = frame.selectExpr("client_id", "explode(build.build_id) as build_id") +
The explode function returns a new row for each element in the given array or map. See the documentation for the complete list of functions supported by DataFrames.
+window_spec = Window.partitionBy(subset["client_id"]).orderBy(subset["build_id"]) +
min_buildid = func.min(subset["build_id"]).over(window_spec) +
%time subset.select("client_id", "build_id", min_buildid.alias("first_build_id")).count() +
CPU times: user 40 ms, sys: 8 ms, total: 48 ms +Wall time: 6min 6s + + + + + + +620203170 +
Note how individual keys can be accessed without any custom Python code.
+%time sensitive = frame.select("search_counts.`yahoo.urlbar`").map(lambda x: np.sum(x[0]) if x[0] else 0).sum() +
CPU times: user 296 ms, sys: 144 ms, total: 440 ms +Wall time: 2min 17s +
And the same operation without custom Python:
+%time sensitive = frame.selectExpr("explode(search_counts.`yahoo.urlbar`) as searches").agg({"searches": "sum"}).collect() +
CPU times: user 20 ms, sys: 0 ns, total: 20 ms +Wall time: 2min 14s +
Exploding arrays seems not to be more efficient compared to custom Python code. That said, while RDD based analyses are likely not going to improve in terms of speed over time with new Spark releases, the same isn’t true for DataFrame based ones.
+%%time + +def sum_array(x, y): + tmp = [0]*len(x) + for i in range(len(x)): + tmp[i] = x[i] + y[i] + return tmp + +histogram = frame.select("GC_MS", "settings.telemetry_enabled")\ + .where("telemetry_enabled[0] = True")\ + .flatMap(lambda x: [v.values for v in x.GC_MS] if x.GC_MS else [])\ + .reduce(lambda x, y: sum_array(x, y)) + +histogram +
CPU times: user 300 ms, sys: 128 ms, total: 428 ms +Wall time: 5min 12s +
pd.Series(histogram).plot(kind="bar") +
<matplotlib.axes._subplots.AxesSubplot at 0x7fa87155ad10> +
frame.printSchema() +
root + |-- client_id: string (nullable = true) + |-- os: string (nullable = true) + |-- normalized_channel: string (nullable = true) + |-- submission_date: array (nullable = true) + | |-- element: string (containsNull = true) + |-- sample_id: array (nullable = true) + | |-- element: double (containsNull = true) + |-- size: array (nullable = true) + | |-- element: double (containsNull = true) + |-- geo_country: array (nullable = true) + | |-- element: string (containsNull = true) + |-- geo_city: array (nullable = true) + | |-- element: string (containsNull = true) + |-- dnt_header: array (nullable = true) + | |-- element: string (containsNull = true) + |-- addons: array (nullable = true) + | |-- element: string (containsNull = true) + |-- async_plugin_init: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- flash_version: array (nullable = true) + | |-- element: string (containsNull = true) + |-- previous_build_id: array (nullable = true) + | |-- element: string (containsNull = true) + |-- previous_session_id: array (nullable = true) + | |-- element: string (containsNull = true) + |-- previous_subsession_id: array (nullable = true) + | |-- element: string (containsNull = true) + |-- profile_subsession_counter: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- profile_creation_date: array (nullable = true) + | |-- element: string (containsNull = true) + |-- profile_reset_date: array (nullable = true) + | |-- element: string (containsNull = true) + |-- reason: array (nullable = true) + | |-- element: string (containsNull = true) + |-- revision: array (nullable = true) + | |-- element: string (containsNull = true) + |-- session_id: array (nullable = true) + | |-- element: string (containsNull = true) + |-- session_length: array (nullable = true) + | |-- element: long (containsNull = true) + |-- session_start_date: array (nullable = true) + | |-- element: string (containsNull = true) + |-- subsession_counter: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- subsession_id: array (nullable = true) + | |-- element: string (containsNull = true) + |-- subsession_length: array (nullable = true) + | |-- element: long (containsNull = true) + |-- subsession_start_date: array (nullable = true) + | |-- element: string (containsNull = true) + |-- timezone_offset: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- build: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- application_id: string (nullable = true) + | | |-- application_name: string (nullable = true) + | | |-- architecture: string (nullable = true) + | | |-- architectures_in_binary: string (nullable = true) + | | |-- build_id: string (nullable = true) + | | |-- version: string (nullable = true) + | | |-- vendor: string (nullable = true) + | | |-- platform_version: string (nullable = true) + | | |-- xpcom_abi: string (nullable = true) + | | |-- hotfix_version: string (nullable = true) + |-- partner: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- distribution_id: string (nullable = true) + | | |-- distribution_version: string (nullable = true) + | | |-- partner_id: string (nullable = true) + | | |-- distributor: string (nullable = true) + | | |-- distributor_channel: string (nullable = true) + | | |-- partner_names: array (nullable = true) + | | | |-- element: string (containsNull = true) + |-- settings: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- addon_compatibility_check_enabled: boolean (nullable = true) + | | |-- blocklist_enabled: boolean (nullable = true) + | | |-- is_default_browser: boolean (nullable = true) + | | |-- default_search_engine: string (nullable = true) + | | |-- default_search_engine_data: struct (nullable = true) + | | | |-- name: string (nullable = true) + | | | |-- load_path: string (nullable = true) + | | | |-- submission_url: string (nullable = true) + | | |-- search_cohort: string (nullable = true) + | | |-- e10s_enabled: boolean (nullable = true) + | | |-- telemetry_enabled: boolean (nullable = true) + | | |-- locale: string (nullable = true) + | | |-- update: struct (nullable = true) + | | | |-- channel: string (nullable = true) + | | | |-- enabled: boolean (nullable = true) + | | | |-- auto_download: boolean (nullable = true) + | | |-- user_prefs: map (nullable = true) + | | | |-- key: string + | | | |-- value: string (valueContainsNull = true) + |-- system: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- memory_mb: integer (nullable = true) + | | |-- virtual_max_mb: string (nullable = true) + | | |-- is_wow64: boolean (nullable = true) + |-- system_cpu: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- cores: integer (nullable = true) + | | |-- count: integer (nullable = true) + | | |-- vendor: string (nullable = true) + | | |-- family: integer (nullable = true) + | | |-- model: integer (nullable = true) + | | |-- stepping: integer (nullable = true) + | | |-- l2cache_kb: integer (nullable = true) + | | |-- l3cache_kb: integer (nullable = true) + | | |-- extensions: array (nullable = true) + | | | |-- element: string (containsNull = true) + | | |-- speed_mhz: integer (nullable = true) + |-- system_device: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- model: string (nullable = true) + | | |-- manufacturer: string (nullable = true) + | | |-- hardware: string (nullable = true) + | | |-- is_tablet: boolean (nullable = true) + |-- system_os: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- name: string (nullable = true) + | | |-- version: string (nullable = true) + | | |-- kernel_version: string (nullable = true) + | | |-- service_pack_major: integer (nullable = true) + | | |-- service_pack_minor: integer (nullable = true) + | | |-- locale: string (nullable = true) + |-- system_hdd: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- profile: struct (nullable = true) + | | | |-- model: string (nullable = true) + | | | |-- revision: string (nullable = true) + | | |-- binary: struct (nullable = true) + | | | |-- model: string (nullable = true) + | | | |-- revision: string (nullable = true) + | | |-- system: struct (nullable = true) + | | | |-- model: string (nullable = true) + | | | |-- revision: string (nullable = true) + |-- system_gfx: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- d2d_enabled: boolean (nullable = true) + | | |-- d_write_enabled: boolean (nullable = true) + | | |-- adapters: array (nullable = true) + | | | |-- element: struct (containsNull = true) + | | | | |-- description: string (nullable = true) + | | | | |-- vendor_id: string (nullable = true) + | | | | |-- device_id: string (nullable = true) + | | | | |-- subsys_id: string (nullable = true) + | | | | |-- ram: integer (nullable = true) + | | | | |-- driver: string (nullable = true) + | | | | |-- driver_version: string (nullable = true) + | | | | |-- driver_date: string (nullable = true) + | | | | |-- gpu_active: boolean (nullable = true) + | | |-- monitors: array (nullable = true) + | | | |-- element: struct (containsNull = true) + | | | | |-- screen_width: integer (nullable = true) + | | | | |-- screen_height: integer (nullable = true) + | | | | |-- refresh_rate: string (nullable = true) + | | | | |-- pseudo_display: boolean (nullable = true) + | | | | |-- scale: double (nullable = true) + |-- active_addons: array (nullable = true) + | |-- element: map (containsNull = true) + | | |-- key: string + | | |-- value: struct (valueContainsNull = true) + | | | |-- blocklisted: boolean (nullable = true) + | | | |-- description: string (nullable = true) + | | | |-- name: string (nullable = true) + | | | |-- user_disabled: boolean (nullable = true) + | | | |-- app_disabled: boolean (nullable = true) + | | | |-- version: string (nullable = true) + | | | |-- scope: integer (nullable = true) + | | | |-- type: string (nullable = true) + | | | |-- foreign_install: boolean (nullable = true) + | | | |-- has_binary_components: boolean (nullable = true) + | | | |-- install_day: long (nullable = true) + | | | |-- update_day: long (nullable = true) + | | | |-- signed_state: integer (nullable = true) + |-- theme: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- id: string (nullable = true) + | | |-- blocklisted: boolean (nullable = true) + | | |-- description: string (nullable = true) + | | |-- name: string (nullable = true) + | | |-- user_disabled: boolean (nullable = true) + | | |-- app_disabled: boolean (nullable = true) + | | |-- version: string (nullable = true) + | | |-- scope: integer (nullable = true) + | | |-- foreign_install: boolean (nullable = true) + | | |-- has_binary_components: boolean (nullable = true) + | | |-- install_day: long (nullable = true) + | | |-- update_day: long (nullable = true) + |-- active_plugins: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- name: string (nullable = true) + | | | |-- version: string (nullable = true) + | | | |-- description: string (nullable = true) + | | | |-- blocklisted: boolean (nullable = true) + | | | |-- disabled: boolean (nullable = true) + | | | |-- clicktoplay: boolean (nullable = true) + | | | |-- mime_types: array (nullable = true) + | | | | |-- element: string (containsNull = true) + | | | |-- update_day: long (nullable = true) + |-- active_gmp_plugins: array (nullable = true) + | |-- element: map (containsNull = true) + | | |-- key: string + | | |-- value: struct (valueContainsNull = true) + | | | |-- version: string (nullable = true) + | | | |-- user_disabled: boolean (nullable = true) + | | | |-- apply_background_updates: integer (nullable = true) + |-- active_experiment: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- id: string (nullable = true) + | | |-- branch: string (nullable = true) + |-- persona: array (nullable = true) + | |-- element: string (containsNull = true) + |-- thread_hang_activity: array (nullable = true) + | |-- element: map (containsNull = true) + | | |-- key: string + | | |-- value: struct (valueContainsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- thread_hang_stacks: array (nullable = true) + | |-- element: map (containsNull = true) + | | |-- key: string + | | |-- value: map (valueContainsNull = true) + | | | |-- key: string + | | | |-- value: struct (valueContainsNull = true) + | | | | |-- values: array (nullable = true) + | | | | | |-- element: integer (containsNull = true) + | | | | |-- sum: long (nullable = true) + |-- simple_measurements: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- active_ticks: long (nullable = true) + | | |-- profile_before_change: long (nullable = true) + | | |-- select_profile: long (nullable = true) + | | |-- session_restore_init: long (nullable = true) + | | |-- first_load_uri: long (nullable = true) + | | |-- uptime: long (nullable = true) + | | |-- total_time: long (nullable = true) + | | |-- saved_pings: long (nullable = true) + | | |-- start: long (nullable = true) + | | |-- startup_session_restore_read_bytes: long (nullable = true) + | | |-- pings_overdue: long (nullable = true) + | | |-- first_paint: long (nullable = true) + | | |-- shutdown_duration: long (nullable = true) + | | |-- session_restored: long (nullable = true) + | | |-- startup_window_visible_write_bytes: long (nullable = true) + | | |-- startup_crash_detection_end: long (nullable = true) + | | |-- startup_session_restore_write_bytes: long (nullable = true) + | | |-- startup_crash_detection_begin: long (nullable = true) + | | |-- startup_interrupted: long (nullable = true) + | | |-- after_profile_locked: long (nullable = true) + | | |-- delayed_startup_started: long (nullable = true) + | | |-- main: long (nullable = true) + | | |-- create_top_level_window: long (nullable = true) + | | |-- session_restore_initialized: long (nullable = true) + | | |-- maximal_number_of_concurrent_threads: long (nullable = true) + | | |-- startup_window_visible_read_bytes: long (nullable = true) + |-- spdy_npn_connect: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_sync_number_of_syncs_failed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_canvasdebugger_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- telemetry_scheduler_tick_exception: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- moz_sqlite_open_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- disk_cache_revalidation_safe: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_corrupt_file: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_copy_panel_actions: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_sessions: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_predict_work_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_truncate_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_cache_entry_reuse_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_keyed_count_init_no_record: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- prconnectcontinue_blocking_time_connectivity_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- history_lastvisited_tree_query_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_thumbnails_bg_capture_queue_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_reflow_duration: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- fennec_load_saved_page: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webfont_download_time_after_start: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- httpconnmgr_used_speculative_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_quota_reset_to: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnect_fail_blocking_time_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_sitesettings: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_not_pref_update_service_enabled_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- places_annos_pages_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gradient_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_syn_ratio: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_perftools_recording_import_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_debugger_rdp_local_navigateto_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- defective_permissions_sql_removed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- loop_max_audio_receive_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_subscribe_failed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_opencacheentry: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_country_timeout: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_developertoolbar_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_threaddetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- paint_build_displaylist_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- social_enabled_on_session: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- urlclassifier_update_remote_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_discarded_content_pings_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- fx_session_restore_send_update_caused_oom: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cycle_collector_time_between: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- social_toolbar_buttons: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- push_api_permission_granted: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_video_quality_inbound_packetloss_rate: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_release_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_menu_eyedropper_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- ssl_reasons_for_not_false_starting: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- predictor_wait_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_distribution_code_category: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_isstorageenabledforpolicy: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- html_background_reflow_ms_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- device_reset_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_favicon_ico_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_country_fetch_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_sub_open_to_first_from_cache_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_hit_rate_per_cache_size: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_validation_success_by_ca: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_gesture_install_snapshot_of_page: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_blackbox_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_worker_visited_gced: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_disk_cache_shutdown_clear_private: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_window_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cert_pinning_moz_test_results_by_host: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_audio_quality_inbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_decoder_framerate_10x_std_dev_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_memory_reporter_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_v1_miss_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_clientevaluate_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_handshake_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_ping_count_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- newtab_page_blocked_sites_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- process_crash_submit_success: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- fx_thumbnails_bg_queue_size_on_capture: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getlastfetched: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_cl_update_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_tabdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_notify_registration_lost: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- scroll_input_methods: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_startup_time_mediaenumerated: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_hud_app_memory_navigationloaded_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- cache_disk_search_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fullscreen_change_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_storage_sqlite: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_has_no_keys_when_unlocked: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- network_cache_metadata_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- audiostream_first_open_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- browser_set_default_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macromanian: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- early_gluestartup_read_transfer: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- aboutcrashes_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_video_quality_outbound_rtt: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_audio_quality_outbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_test_count_init_no_record: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- url_path_ends_in_exclamation: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_thumbnails_bg_capture_canvas_draw_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_threaddetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_country_fetch_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_truncate_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_datachannel_negotiated: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- httpconnmgr_unused_speculative_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gluestartup_read_transfer: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- onbeforeunload_prompt_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- about_accounts_content_server_load_started_count: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_form_autofill_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_canvasdebugger_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_decoder_framerate_avg_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_prototypesandproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_quota_expiration_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_session_restore_dom_storage_size_estimate_chars: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_eme_request_success_latency_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_slice_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_settings_dl_bw: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_project_editor_save_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ntlm_module_used_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_ice_success_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_isstreambased: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_window_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_subscribe_http2_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_koi8r: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- plugin_called_directly: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- e10s_addons_blocker_ran: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- url_path_contains_exclamation_double_slash: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_write_file_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_fontinspector_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ipc_message_size: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_workerdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_migration_errors: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- plugin_hang_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- bucket_order_errors: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- localdomstorage_getkey_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_getcurrentposition_secure_origin: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_shadereditor_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- keygen_generated_key_type: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- predictor_predict_time_to_action: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_responsive_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- web_notification_request_permission_callback: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- newtab_page_shown: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_archive_scan_ping_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_audio_quality_outbound_packetloss_rate: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_call_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_video_decoder_bitrate_std_dev_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- localdomstorage_value_size_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_configured_master_password: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- prclose_tcp_blocking_time_connectivity_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugins_notification_user_action: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_new_project_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_close: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_download_code_partial: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_options_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_debugger_rdp_local_threadgrips_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- stumbler_observations_per_day: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_metadata_first_read_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gradient_retention_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_slow_phase: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_last_notify_interval_days_external: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- bad_fallback_font: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_scope_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_open_to_first_from_cache_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_cl_check_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_app_startup_time_contentinteractive: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- ssl_cert_error_overrides: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- composite_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- csp_documents_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macarabic: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_picker_eyedropper_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_computedview_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fennec_reader_view_cache_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sts_poll_and_event_the_last_cycle: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_auth_ecdsa_curve_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- weave_start_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_inspector_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- xul_background_reflow_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sts_poll_block_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_ping: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- flash_plugin_states: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fullscreen_transition_black_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_onprofileshutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_memory_diff_census: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_disk_cache_streamio_close_main_thread: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_detach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sts_number_of_pending_events_in_the_last_cycle: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_import_project_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- moz_storage_async_requests_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getsecurityinfo: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsdiskcachedevicedeactivateentryevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_write_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_win8_source_is_mls: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- xhr_in_worker: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- gc_reason_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- weave_fxa_key_fetch_auth_errors: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- geolocation_watchposition_secure_origin: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- translated_characters: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_invalid_lastupdatetime_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_shutdown_database_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_custom_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- rejected_message_manager_message: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- sts_number_of_pending_events: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_unique: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_search_loader_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tracking_protection_pbm_disabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gfx_crash: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- media_ogg_loaded_is_chained: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_jsdebugger_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webconsole_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webrtc_load_state_normal_short: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tab_switch_cache_position: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_most_recent_expired_visit_days: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_lc_completions: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_call_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_backups_tojson_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_reload_addon_reload_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_unload_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- reader_mode_download_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- subprocess_abnormal_abort: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- cookie_scheme_security: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_touch_used: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- checkerboard_severity: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- about_accounts_content_server_loaded_time_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- telemetry_archive_directories_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_cache_miss_halflife_experiment_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_audio_quality_outbound_rtt: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- link_icon_sizes_attr_dimension: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_shadereditor_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_09_info: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugin_hang_ui_response_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- popup_notification_dismissal_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- a11y_iatable_usage_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_thumbnails_hit_or_miss: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- slow_script_notify_delay: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_sync11_migrations_failed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_getlength_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tracking_protection_shield: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_cache_entry_alive_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_urlbar_selected_result_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_paintflashing_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_browserconsole_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_reload_addon_installed_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webfont_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_warnings: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_downloads: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- low_memory_events_commit_space: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_resume_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_hang_notice_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_pending_load_failure_parse: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- spdy_server_initiated_streams: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_memory_export_snapshot_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- decoder_instantiated_maccyrillic: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webrtc_video_decoder_framerate_avg_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- url_path_contains_exclamation_slash: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_content_crash_presented: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- idle_notify_idle_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_options_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_canplaytype_h264_constraint_set_flag: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_not_pref_update_enabled_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- stumbler_upload_cell_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- social_sidebar_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_memory_navigationinteractive_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- telemetry_archive_evicting_dirs_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_tcp_connection: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_jsbrowserdebugger_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- pwmgr_login_page_safety: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_openoutputstream: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webfont_size_per_page: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_settings_cwnd: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- crash_store_compressed_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnectcontinue_blocking_time_link_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_cache_disposition_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cookies_3rdparty_num_sites_accepted: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_encoder_bitrate_avg_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webaudioeditor_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_call_count_2: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_sessiondata_failed_parse: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- prconnect_fail_blocking_time_link_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_formdata: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_responsive_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- predictor_prefetch_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_subresource_degradation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_resumed_session: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_request_per_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_listaddons_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_username_present: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_ice_on_time_trickle_arrival_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_reload_addon_installed_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- sqlitebridge_provider_home_locked: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- flash_plugin_area: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fontlist_initfacenamelists: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getexpirationtime: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_listprocesses_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_predictions: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_permission_requested: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- charset_override_situation: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_startup_init_session_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_parameternames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_discarded_pending_pings_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_document_generator: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- browser_set_default_always_check: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_tab_switch_total_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- link_icon_sizes_attr_usage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_save_heap_snapshot_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_capabilities_seccomp_tsync: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- network_disk_cache_revalidation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_favicon_gif_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setofflinecachecapacity: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_minor_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- about_accounts_content_server_failure_time_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_hud_app_startup_time_scanend: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- predictor_total_preconnects_used: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_check_no_update_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_reconfigurethread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_manage_deleted_all: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- prconnect_fail_blocking_time_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setstoragepolicy: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_mft_output_null_samples: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- shutdown_ok: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_favicon_bmp_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_history_library_search_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- idle_notify_idle_listeners: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_permanent_cert_error_overrides: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_identity_popup_open_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_call_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_images_content_used_uncompressed: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tabs_pinned_peak_linear: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_font_types: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscompressoutputstreamwrapper_release: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_latency_us: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_cache_disposition_2_v2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_migration_usage: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- fx_session_restore_collect_all_windows_data_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_cookies_write_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_sync11_migrations_succeeded: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- ssl_auth_algorithm_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_ice_late_trickle_arrival_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_key_exchange_algorithm_resumed: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_dominator_tree_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- graphics_sanity_test: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_vp9_benchmark_fps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_developertoolbar_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- startup_measurement_errors: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_settings_iw: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_budget_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_app_memory_scanend_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- graphics_sanity_test_os_snapshot: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_property_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- display_scaling_linux: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tilt_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- remote_jar_protocol_used: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- predictor_predict_attempts: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- web_notification_menu: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_addondetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_not_pref_update_staging_enabled_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- update_service_installed_notify: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_activity_counter: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_video_encoder_dropped_frames_per_call_fpm: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_total: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_lc_prefixes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webide_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- print_preview_simplify_page_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- image_decode_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_listtabs_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_learn_full_queue: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- slow_script_page_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_ruleview_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- web_notification_senders: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- places_database_pagesize_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_cookies_write_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_frames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cookies_3rdparty_num_attempts_blocked: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_rust_mp4parse_track_match_video: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_ice_add_candidate_errors_given_failure: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_audio_quality_inbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_assign_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_favicon_svg_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_not_pref_update_enabled_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_reconfigurethread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_proxy_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_display_source_remote_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_decoder_framerate_10x_std_dev_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_symmetric_cipher_resumed: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- prconnectcontinue_blocking_time_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cookies_3rdparty_num_sites_blocked: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_uss: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- gc_minor_reason_long: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_check_code_external: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_reload_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- alerts_service_dnd_supported_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_jsprofiler_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webrtc_video_quality_outbound_packetloss_rate: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_disk_cache_trashrename: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_developertoolbar_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load_net_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_places_write_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- browser_is_user_default: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dnt_usage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setdiskcacheenabled: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- requests_of_original_content: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webide_wifi_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_video_recovery_before_error_per_min: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_migration_homepage_imported: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- fennec_topsites_loader_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_state_code_unknown_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_onprofilechanged: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_learn_work_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_first_sent_to_last_received: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- top_level_content_documents_destroyed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- total_count_high_errors: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_listserviceworkerregistrations_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_time_to_view_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_print: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- early_gluestartup_read_ops: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_engine_apply_new_failures: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getpredicteddatasize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_get_user_media_secure_origin: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- popup_notification_stats: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- graphics_driver_startup_test: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_plugins: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsasyncdoomevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_quality_outbound_jitter: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_bytes_before_cert_callback: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_sync11_migration_sentinels_seen: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- loop_max_video_send_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- eventloop_ui_activity_exp_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- disk_cache_invalidation_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_startup_external_content_handler: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_response_status_code: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_frames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_iso_8859_5: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- search_reset_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_chain_key_size_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_listtabs_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_non_incremental: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_video_decoder_discarded_packets_per_call_ppm: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getstoragepolicy: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- d3d11_sync_handle_failure: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- moz_sqlite_cookies_read_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- database_locked_exception: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_cookies_read_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_app_memory_mediaenumerated_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- cert_pinning_moz_results: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_connected_runtime_platform_version: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- update_not_pref_update_auto_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- prclose_tcp_blocking_time_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_detailed_dropped_frames_proportion: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- push_api_notification_received: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_screen_resolution_enumerated_per_user: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gc_mmu_50: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- disk_cache_corrupt_details: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_backups_daysfromlast: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugin_startup_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- flash_plugin_height: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_kea_ecdhe_curve_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_archive_oldest_directory_age: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_transaction_use_altsvc: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- search_service_init_sync: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_storage_async_requests_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- urlclassifier_ps_fallocate_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_size_full_fat: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gdi_initfontlist_total: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_property_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_cipher_suite_resumed: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- subject_principal_accessed_without_script_on_stack: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- shutdown_phase_duration_ticks_xpcom_will_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_reconfiguretab_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_get_user_media_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_autocomplete_1st_result_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_delete_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_tls13_intolerance_reason_pre: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_cache_v1_hit_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_memory_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tap_to_load_image_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prclose_udp_blocking_time_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_session_restore_file_size_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_sweep_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- idle_notify_back_listeners: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- word_cache_misses_content: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_ice_success_rate: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- transaction_wait_time_http: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_codec_used: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsoutputstreamwrapper_closeinternal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_requestdatasizechange: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_embed: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getlastmodified: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_memory_search_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- newtab_page_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_stoptrace_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- word_cache_hits_chrome: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_complete_load_cached: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_ice_late_trickle_arrival_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_os_is_64_bits_per_user: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_settings_rtt: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_engine_sync_errors: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- memory_js_main_runtime_temporary_peak: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_other_write_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setdiskcachecapacity: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- localdomstorage_getallkeys_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_export_tohtml_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- flash_plugin_width: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_jank: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_bindings_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_eme_play_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- sts_poll_and_events_cycle: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_tracerdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_app_startup_time_fullyloaded: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_heap_snapshot_edge_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- stumbler_upload_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_capabilities_seccomp_bpf: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- reader_mode_worker_parse_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_favicon_other_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_cipher_suite_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- thunderbird_gloda_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_jsbrowserdebugger_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fxa_configured: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- message_manager_message_size: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- http_transaction_use_altsvc_oe: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_can_create_h264_decoder: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_status_error_code_partial_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_import_project_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_layoutview_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- onbeforeunload_prompt_action: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_diskdeviceheapsize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_vsize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_scc_sweep_max_pause_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_hash_stats: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- memory_resident_fast: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_os_enumerated_per_user: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_toolbox_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_total: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_worker: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- addon_manager_upgrade_ui_shown: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- gc_mark_gray_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- deferred_finalize_async: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tracking_protection_events: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_client_call_url_requests_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_offlineapps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_chunks: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_discarded_archived_pings_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- disk_cache_smart_size_using_old_max: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_session_at_900fd: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_validation_http_request_failed_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_fs_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_ice_checking_rate: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- font_cache_hit: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_collect_data_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_pending_pings_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_globalhistory_visited_build_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_cache_read_time_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_initial_failed_cert_validation_time_mozillapkix: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_load_state_stressed: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_fallback_shown: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_unloaded_flash: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dns_failed_lookup_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- update_pref_service_errors_external: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_koi8u: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_auth_dsa_key_size_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- perf_monitoring_test_cpu_rescheduling_proportion_moved: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_us_country_mismatched_platform_osx: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_ev_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_archive_session_ping_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- newtab_page_life_span_suggested: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load_cached_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_minor_us: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webfont_srctype: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_annos_bookmarks_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gluestartup_hard_faults: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_build_cache_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_processrequest: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugins_infobar_shown: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_syn_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_encoder_bitrate_avg_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_prompt_remember_action: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_visitmetadata: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_fastseek_used: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- blocked_on_plugin_module_init_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- places_keywords_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_ps_construct_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_num_saved_passwords: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_encoder_framerate_avg_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_iso2022jp: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webrtc_load_state_relaxed_short: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_aboutdebugging_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_ruleview_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- transaction_wait_time_spdy: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_perftools_recording_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- places_favicon_jpeg_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_substring_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_bookmarks_toolbar_init_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_doom: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_clientevaluate_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_browserconsole_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_project_editor_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- web_notification_exceptions_opened: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_storage_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webide_other_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_tab_click_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_worker_collected: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_compress: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- changes_of_target_language: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- pwmgr_prompt_update_action: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- js_define_getter_setter_this_null_undefined: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsdiskcachestreamio_closeoutputstream: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_addondetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_simulator_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_menu_eyedropper_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- video_openh264_gmp_disappeared: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- page_faults_hard: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_decode_error_time_permille: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_urlbar_selected_result_index: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_aboutdebugging_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_ping_evicted_for_server_errors: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- idle_notify_back_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- thunderbird_indexing_rate_msg_per_s: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_revalidation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_archive_checking_over_quota_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_offline_cache_document_load: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_sync_skippable: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_disk_cache_streamio_close: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_canplaytype_h264_level: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- sqlitebridge_provider_forms_locked: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- audiostream_later_open_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_stoptrace_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_subscribe_succeeded: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macgurmukhi: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_npn_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setofflinecacheenabled: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_computedview_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_pref_update_cancelations_notify: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_cache_v2_miss_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_keyed_flag: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: boolean (containsNull = true) + |-- webfont_download_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_play_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tabs_open_average_linear: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_cookies_read_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_recovery_after_error_per_min: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_quality_outbound_rtt: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tracking_protection_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_ocsp_stapling: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webcrypto_extractable_sig: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_sub_dns_lookup_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_scope_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getfile: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_predict_full_queue: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_anim_open_preview_frame_interval_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_session_restore_collect_cookies_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_jsbrowserdebugger_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- subprocess_kill_hard: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- flash_plugin_instances_on_page: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_test_keyed_count: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsevictdiskcacheentriesevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_ice_failure_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- service_worker_registrations: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- system_font_fallback_first: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_encoder_framerate_10x_std_dev_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnectcontinue_blocking_time_offline: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webaudioeditor_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- telemetry_pending_pings_evicted_over_quota: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_getcacheiotarget: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_prototypeandproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setpredicteddatasize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_device_search_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_lm_inconsistent: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webconsole_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- forced_device_reset_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- shutdown_phase_duration_ticks_profile_before_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_audio_quality_outbound_rtt: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_preresolves: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- newtab_page_life_span: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_storage_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- network_session_at_256fd: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugins_notification_shown: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_startup_time_visuallyloaded: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_evictentriesforclient: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_decoder_bitrate_std_dev_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsdiskcachestreamio_write: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_startup_migration_browser_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_startup_time_navigationloaded: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- places_database_size_per_page_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_picker_eyedropper_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- prclose_udp_blocking_time_connectivity_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_unsubscribe_succeeded: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- update_ping_count_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- update_check_code_notify: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- canvas_2d_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- disk_cache_reduction_trial: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- weave_complete_success_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_errors: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_video_decoder_bitrate_avg_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_unsubscribe_attempt: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- spdy_settings_max_streams: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- service_worker_spawn_gets_queued: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_layoutview_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webcrypto_extractable_enc: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_max_audio_receive_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- backgroundfilesaver_thread_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_cookies_sync_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_stream_types: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_canvasdebugger_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- weave_device_count_desktop: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_sync_number_of_syncs_failed_backoff: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- total_count_low_errors: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_listworkers_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- canvas_webgl_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- browser_set_default_time_to_completion_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sts_poll_cycle: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- translation_opportunities_by_language: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- network_cache_metadata_first_read_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_read_heap_snapshot_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_ice_final_connection_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_prototypesandproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_metadata_second_read_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load_net: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- family_safety: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_visited_ref_counted: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_incremental_disabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_disk_cache_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_ocsp_may_fetch: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_manual_restore_duration_until_eager_tabs_restored_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_capabilities_user_namespaces: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- places_bookmarks_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_listworkers_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_global_degradation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_connected_runtime_type: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- loop_two_way_media_conn_length_1: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_browserconsole_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_webide_local_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- long_reflow_interruptible: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dns_lookup_method2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_protocoldescription_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dwritefont_delayedinitfontlist_total: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- low_memory_events_virtual: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_ownpropertynames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_version_fallback_inappropriate: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_startup_onload_initial_window_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_scheme_upgrade: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_syn_reply_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_keyed_release_optin: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: boolean (containsNull = true) + |-- js_telemetry_addon_exceptions: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_was_spawned: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- moz_sqlite_places_read_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_revalidation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- shutdown_phase_duration_ticks_quit_application: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- mixed_content_unblock_counter: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_decoded_h264_sps_level: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macturkish: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fennec_sync_number_of_syncs_completed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_cache_entry_reload_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_dns_issue_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- permissions_remigration_comparison: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_transaction_is_ssl: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_places_read_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- canvas_webgl_failure_id: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- memory_heap_allocated: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load_cached: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_unable_to_apply_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- js_deprecated_language_extensions_in_addons: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gc_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_prototype_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_visited_gced: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- youtube_rewritable_embed_seen: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cycle_collector_full: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getmetadataelement: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_reader_view_button: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- search_service_us_country_mismatched_platform_win: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_homepanels_custom: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- security_ui: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_project_editor_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- check_java_enabled: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_adobe_gmp_disappeared: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_hud_app_memory_visuallyloaded_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- alerts_service_dnd_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugins_infobar_block: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_restore_window_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsdiskcachebinding_destructor: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_netmonitor_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- html_foreground_reflow_ms_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_cookies: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webfont_compression_woff: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_archive_evicting_over_quota_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- content_response_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_room_create: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_listaddons_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tabs_pinned_average_linear: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_assign_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_load_state_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_other_read_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_nonus_country_mismatched_platform_osx: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_places_write_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_eventlisteners_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_connected_runtime_id: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- loop_video_encoder_framerate_10x_std_dev_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_max_pause_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- translated_pages_by_language: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_inverted_census: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setmemorycachemaxentrysize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_complete_load_cached_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_kea_rsa_key_size_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_audio_quality_outbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_unblackbox_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_status_error_code_partial_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- e10s_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_webapps_read_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_call_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_preconnects: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsinputstreamwrapper_release: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_rust_mp4parse_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_eventlisteners_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_v2_hit_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_places_read_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- transaction_wait_time_http_pipelines: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_discarded_send_pings_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_subscribe_ws_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_read_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_sync_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_oom: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_not_pref_update_auto_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- graphics_sanity_test_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_manage_sorted: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- geolocation_getcurrentposition_visible: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- br_9_2_1_subject_alt_names: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_bookmarks_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_cache_read_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_speed_gif: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_tls11_intolerance_reason_pre: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- weave_device_count_mobile: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_other_read_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_gesture_take_snapshot_of_page: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_keyed_release_optout: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: boolean (containsNull = true) + |-- composite_frame_roundtrip_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_displaystring_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- canvas_webgl2_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- e10s_blocked_from_running: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_async_snow_white_freeing: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_has_icon_updates: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_gesture_compress_snapshot_of_page: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_quality_inbound_jitter: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_encoder_bitrate_std_dev_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_js_compartments_user: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_quality_outbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_renegotiations: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- thunderbird_conversations_time_to_2nd_gloda_query_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_perftools_recording_export_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- video_mse_buffering_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- canvas_webgl_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_client_call_url_shared: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_auth_dialog_stats: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_audio_quality_outbound_jitter: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sqlitebridge_provider_passwords_locked: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_jsprofiler_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_complete_load_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_disk_cache_open: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- startup_crash_detected: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_tls10_intolerance_reason_post: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_version2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dns_lookup_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_document_version: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_visitentries: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_sources_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_configured: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_idle_frecency_decay_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_decoder_discarded_packets_per_call_ppm: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_interrupt_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_accuracy_exponential: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_anim_open_frame_interval_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_audio_quality_inbound_packetloss_rate: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_document_size_kb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- js_deprecated_language_extensions_in_content: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gc_mark_roots_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_browser_fullscreen_used: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- changes_of_detected_language: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_jsdebugger_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dwritefont_delayedinitfontlist_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_complete_timeout: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ipc_reply_size: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- ssl_key_exchange_algorithm_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- process_crash_submit_attempt: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_aboutdebugging_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_disk_cache_deletedir_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- view_source_external_result_boolean: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_sessiondata_failed_load: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_pref_service_errors_notify: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getclientid: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getcacheelement: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_pinning_results: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_kea_dhe_key_size_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_sub_open_to_first_from_cache: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- social_panel_clicks: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_collect_data_longest_op_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_netmonitor_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_can_create_aac_decoder: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- media_hls_decoder_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- search_counts: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- prconnect_blocking_time_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- a11y_isimpledom_usage_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setsecurityinfo: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_animation_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_tabdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dwritefont_init_problem: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_send: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_activation_count: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- urlclassifier_ps_failure: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_weak_ciphers_fallback: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_was_killed: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_download_code_complete: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dns_cleanup_age: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_avsync_when_video_lags_audio_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_expired: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_thumbnails_bg_capture_service_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_request_per_page_from_cache: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_refresh_driver_chrome_frame_delay_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webcrypto_extractable_import: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugins_infobar_allow: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_decoded_h264_sps_profile: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- youtube_nonrewritable_embed_seen: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- geolocation_watchposition_visible: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_listprocesses_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- slow_addon_warning_response_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- subprocess_crashes_with_dump: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_disk_cache_overhead: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_new_project_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- mac_initfontlist_total: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_parallel_streams: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_styleeditor_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- fx_sanitize_cache: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsdecompressinputstreamwrapper_release: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- e10s_window: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_tracking_protection_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- newtab_page_enhanced: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_video_recovery_before_error_per_min: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsdiskcachemap_revalidation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_globalhistory_update_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_listserviceworkerregistrations_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- checkerboard_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- csp_unsafe_eval_documents_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_resume_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- localdomstorage_removekey_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- newtab_page_site_clicked: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- system_font_fallback_script: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_favicon_png_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_invalid_lastupdatetime_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- content_documents_destroyed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_storage_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- stumbler_volume_bytes_uploaded_per_sec: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_cookies_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- stumbler_time_between_uploads_sec: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_autocomplete_urlinline_domain_query_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_macgujarati: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- browserprovider_xul_import_history: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_status_error_code_complete_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_probe_maxcount: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_heap_committed_unused: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_mark_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_form_action_effect: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- addon_shim_usage: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- webrtc_stun_rate_limit_exceeded_by_type_given_failure: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- predictor_confidence: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_cookies_sync_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_open_to_first_sent: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- xul_initial_frame_construction: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_navigateto_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getstoragedatasize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnect_blocking_time_offline: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- osfile_writeatomic_jank_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- startup_cache_age_hours: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_read_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- input_event_response_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- browserprovider_xul_import_bookmarks: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- message_manager_message_size2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- moz_sqlite_places_sync_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- audio_mft_output_null_samples: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- loop_video_decode_error_time_permille: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_check_extended_error_external: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_startup_time_javaui: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_validation_http_request_succeeded_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_speed_jpeg: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_tiny_content: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- ssl_server_auth_eku: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_worker_need_gc: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setexpirationtime: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_maccroatian: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_thumbnails_store_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_anim_close_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_audio_quality_inbound_jitter: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dom_range_detached: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_startup_time_navigationinteractive: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- places_pages_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_avsync_when_audio_lags_video_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tilt_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- moz_sqlite_cookies_write_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_cannot_stage_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getkey: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_content_crash_dump_unavailable: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- search_service_engine_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_pending_evicting_over_quota_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_max_video_receive_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_closeallstreams: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_tabqueue_prompt_enable_yes: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- web_notification_permission_removed: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- push_api_used: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- loop_room_delete: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_breakdown_census_count: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_sessiondata_failed_validation: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsinputstreamwrapper_lazyinit: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prclose_udp_blocking_time_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_need_gc: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dns_renewal_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_base_confidence: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_v2_input_stream_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_mse_play_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_response_version: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_pinning_test_results: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_sync_number_of_syncs_started: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cycle_collector_collected: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_cert_verification_errors: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- web_notification_permissions: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_interrupt_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- application_reputation_local: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_other_write_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_ruleview_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- search_service_nonus_country_mismatched_platform_win: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_ice_add_candidate_errors_given_failure: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_service_installed_external: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_number_of_tabs_restored: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_openh264_gmp_missing_files: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_starttrace_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_complete_load: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getfetchcount: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- data_storage_entries: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_request_granted: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- check_addons_modified_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_infobar_action_buttons: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_perftools_recording_duration_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- display_scaling_mswin: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setmetadataelement: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_decoded_h264_sps_constraint_set_flag: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_not_pref_update_service_enabled_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- places_backups_bookmarkstree_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_adobe_gmp_missing_files: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_renegotiations: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_goaway_local: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ipc_transaction_cancel: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_last_notify_interval_days_notify: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_refresh_driver_content_frame_delay_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_connection_play_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_handshake_version: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_page_load_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_ping_size_exceeded_archived: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- battery_status_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_pageload_is_ssl: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_refresh_driver_sync_scroll_frame_delay_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_memory_import_snapshot_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_sessiononly_preload_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- touch_enabled_device: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- predictor_learn_attempts: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_startup_migration_automated_import_succeeded: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_disk_cache_disposition_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugin_load_metadata: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_wmf_decode_error: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_request_per_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- permissions_migration_7_error: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_tab_switch_spinner_visible_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dns_renewal_time_for_ttl: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- localdomstorage_clear_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_offline_search_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_reflows: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_computedview_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- translation_opportunities: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_controlled_documents: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_ice_failure_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_state_code_unknown_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_thumbnails_bg_capture_page_load_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- blocked_on_plugin_stream_init_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_releasemany_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_v2_output_stream_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_state_code_partial_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_manage_visibility_toggled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- checkerboard_peak: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_tls12_intolerance_reason_pre: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- osfile_worker_ready_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- xul_cache_disabled: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_tls10_intolerance_reason_pre: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_page_dns_issue_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_goaway_peer: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_service_manually_uninstalled_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_getvalue_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- checkerboard_potential_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_app_memory_contentinteractive_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- localdomstorage_key_size_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_kbread_per_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_workerdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_subscribe_attempt: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_load_state_relaxed: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_cannot_stage_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webide_connection_time_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_blackbox_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_heap_snapshot_node_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- web_notification_shown: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_custom_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- print_preview_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_sub_complete_load_net_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- service_worker_updated: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_sub_cache_read_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_memory_filter_census: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gluestartup_read_ops: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_perftools_console_recording_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_sub_tcp_connection: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_switch_total_e10s_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- range_checksum_errors: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_idle_maintenance_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_ibm866: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- telemetry_invalid_ping_type_submitted: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- find_plugins: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_open_to_first_received: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_status_error_code_unknown_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_validation_http_request_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- websockets_handshake_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fxa_hawk_errors: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_eme_request_failure_latency_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_complete_remote_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gfx_content_failed_to_acquire_device: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_pinning_moz_results_by_host: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setcacheelement: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_unblackbox_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- a11y_consumers: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_symmetric_cipher_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugin_hang_ui_user_response: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_finish_igc: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- web_notification_clicked: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- network_disk_cache_deletedir: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_scheduler_send_daily: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_preload_pending_on_first_access: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_styleeditor_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webaudioeditor_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- permissions_sql_corrupted: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_pending_pings_age: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_num_httpauth_passwords: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_capabilities_user_namespaces_privileged: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_state_code_partial_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- prclose_tcp_blocking_time_link_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ipc_same_process_message_copy_oom_kb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_parameternames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_hls_canplay_requested: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_connected_runtime_app_type: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- fennec_globalhistory_add_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_options_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- search_service_init_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_netmonitor_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- http_saw_quic_alt_protocol: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_reload_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_wiz_last_page_code: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsprocessrequestevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_heap_overhead_fraction: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- application_reputation_server_verdict: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_spawn_attempts: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- fx_new_window_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_session_restore_auto_restore_duration_until_eager_tabs_restored_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_num_passwords_per_hostname: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_history: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_open_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_us_timezone_mismatched_country: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- search_service_has_updates: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- image_max_decode_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_sorted_bookmarks_perc: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_v1_truncate_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_places_sync_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- about_accounts_content_server_loaded_rate: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- stumbler_upload_observation_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_state_code_complete_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pdf_viewer_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- perf_monitoring_slow_addon_jank_us: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- webrtc_ice_on_time_trickle_arrival_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_paintflashing_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_enumproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_eyedropper_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- word_cache_misses_chrome: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_memory_cache_disposition_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- master_password_enabled: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_session_restore_read_file_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getdatasize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- total_content_page_load_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- components_shim_accessed_by_content: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_debugger_display_source_local_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_login_last_used_days: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_time_until_handshake_finished: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnect_fail_blocking_time_connectivity_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- application_reputation_should_block: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_project_editor_save_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webide_connected_runtime_os: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- places_autocomplete_6_first_results_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_pending_load_failure_read: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_tls11_intolerance_reason_post: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- push_api_permission_denied: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_prototypeandproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webcrypto_extractable_generate: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsblockoncachethreadevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_disk_cache2_shutdown_clear_private: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_maintenance_daysfromlast: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_ping_size_exceeded_send: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- dwritefont_delayedinitfontlist_collect: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_doomandfailpendingrequests: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- blocked_on_plugin_instance_init_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- webrtc_datachannel_negotiated: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fips_enabled: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- application_reputation_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_take_snapshot_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_archive_evicted_old_dirs: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_sync_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_dropped_frames_proportion: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_places_write_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- reader_mode_download_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_ping_size_exceeded_pending: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- ssl_auth_rsa_key_size_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- prclose_udp_blocking_time_link_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tabletmode_page_load: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_animationinspector_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- word_cache_hits_content: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_predictions_calculated: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_unload_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_registration_loading: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_prefetches_used: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- shared_worker_spawn_gets_queued: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- ghost_windows: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cookies_3rdparty_num_attempts_accepted: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_get_executable_lines_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_encoder_dropped_frames_per_call_fpm: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_content_encoding: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- charset_override_used: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_debugger_rdp_remote_get_executable_lines_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_thumbnails_capture_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_releasemany_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- application_reputation_server: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsoutputstreamwrapper_release: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_total_top_visits: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- system_font_fallback: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_max_video_receive_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webfont_compression_woff2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsinputstreamwrapper_closeinternal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dom_timers_recently_set: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_expiration_steps_to_clean2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_worker_oom: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- http_offline_cache_disposition_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_id: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- paint_rasterize_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_preconnects_unused: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_prefetches: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_assemble_payload_exception: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- sandbox_capabilities_enabled_media: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setmemorycache: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- browser_set_default_dialog_prompt_rawcount: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_encoder_framerate_avg_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_bindings_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_settings_retrans: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- newtab_page_pinned_sites_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_max_pause: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_recovery_after_error_per_min: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_macicelandic: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- view_source_in_browser_opened_boolean: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_mse_join_latency_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_state_code_complete_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_paintflashing_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- perf_monitoring_slow_addon_cpow_us: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- reader_mode_serialize_dom_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_scc_sweep_total_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_scheduler_wakeup: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_jsdebugger_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- br_9_2_2_subject_common_name: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- blocklist_sync_file_load: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_chain_sha1_policy_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_usb_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- push_api_unsubscribe_failed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- browser_shim_usage_blocked: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- moz_sqlite_other_read_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_eyedropper_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- prconnect_blocking_time_connectivity_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_speed_png: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_error_recovery_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_shadereditor_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_time_until_ready: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_broker_initialized: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_connection_debug_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_observed_end_entity_certificate_lifetime: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- stumbler_time_between_start_sec: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_perftools_selected_view_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- video_mse_unload_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- sts_number_of_onsocketready_calls: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_call_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webfont_per_page: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_fontinspector_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_subitem_open_latency_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- startup_cache_invalid: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fennec_restricted_profile_restrictions: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- update_unable_to_apply_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_toolbox_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_session_restore_number_of_windows_restored: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_sync11_migration_notifications_offered: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_rust_mp4parse_track_match_audio: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_inspector_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- should_translation_ui_appear: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getdeviceid: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- social_sidebar_open_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- shutdown_phase_duration_ticks_profile_change_teardown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- subprocess_launch_failure: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugin_drawing_model: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsoutputstreamwrapper_lazyinit: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- e10s_still_accepted_from_prompt: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_max_video_send_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_other_write_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_tabqueue_prompt_enable_no: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fetch_is_mainthread: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_restoring_activity: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- canvas_webgl_accl_failure_id: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_eme_adobe_hidden_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- prconnect_blocking_time_link_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_capabilities_enabled_content: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_perftools_recording_features_used: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macce: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- gc_is_compartmental: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- a11y_update_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setdisksmartsize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_custom_homepage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- image_decode_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_fontinspector_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_delete_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_tls13_intolerance_reason_post: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_animationinspector_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webrtc_max_audio_send_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_tags_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_check_extended_error_notify: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- fontlist_initotherfamilynames: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_displaystring_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_custom_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- http_page_dns_lookup_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_pinning_failures_by_ca: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ipv4_and_ipv6_address_connectivity: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_tabqueue_queuesize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_enumproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_notification_received_but_did_not_notify: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- popup_notification_main_action_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- webrtc_stun_rate_limit_exceeded_by_type_given_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_stringify: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- enable_privilege_ever_called: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- http_page_open_to_first_from_cache: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_validation_http_request_canceled_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_cache_read_time_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tls_error_report_ui: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_security_category: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- update_check_no_update_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- gc_compact_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_decoder_bitrate_avg_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- browser_is_assist_default: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- memory_vsize_max_contiguous: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_ocsp_required: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- predictor_predict_time_to_inaction: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- reader_mode_parse_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_reload_addon_reload_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_service_manually_uninstalled_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- scroll_linked_effect_found: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- display_scaling_osx: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_openwindows: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_migration_entry_point: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_content_collect_data_longest_op_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_blocked_for_stability: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- stumbler_time_between_received_locations_sec: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_switch_update_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_markvalid: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_macgreek: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_status_error_code_complete_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_inspector_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- browserprovider_xul_import_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_machebrew: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- pdf_viewer_form: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dns_blacklist_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_room_session_withchat: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- dom_window_showmodaldialog_used: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- prclose_tcp_blocking_time_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_detach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_get_user_media_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_other_sync_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_sessiondata_failed_save: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- plugins_notification_plugin_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_sub_complete_load_net: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_sources_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- view_source_in_window_opened_boolean: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_test_release_optin: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- localdomstorage_init_database_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- stumbler_upload_wifi_ap_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setdatasize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_content_crash_not_submitted: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- csp_unsafe_inline_documents_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- osfile_worker_launch_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_saving_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_styleeditor_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cert_pinning_moz_test_results: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_tls12_intolerance_reason_post: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_has_pref_url_override_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- sessiondomstorage_key_size_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_h264_sps_max_num_ref_frames: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- slow_script_notice_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_auth_type_stats: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- early_gluestartup_hard_faults: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- slow_addon_warning_states: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_sharing_room_url: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_webapps_write_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_load_state_stressed_short: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_npn_join: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macdevanagari: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- a11y_instantiated_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- http_request_per_page: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_distribution_download_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_reset: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_tracerdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnectcontinue_blocking_time_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webcrypto_resolved: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dom_timers_fired_per_native_timeout: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_anim_any_frame_paint_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_ice_add_candidate_errors_given_success: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_hang_ui_dont_ask: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_sync11_migrations_completed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webcrypto_alg: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- zoomed_view_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_release_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_kbread_per_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_osx_source_is_mls: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsdoomevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webconsole_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- loop_ice_success_rate: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_toolbox_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_memory_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_substring_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_other_sync_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- denied_translation_offers: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_pending_checking_over_quota_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nssetdisksmartsizecallback_notify: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_on_draw_latency: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_shutdown_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sync_worker_operation: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- video_eme_adobe_install_failed_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_connection_entry_cache_hit_1: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_prototype_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_not_pref_update_staging_enabled_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- update_has_pref_url_override_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_archive_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prclose_tcp_blocking_time_offline: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_project_editor_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- network_cache_hit_miss_stat_per_cache_size: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- database_successful_unlock: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- blocked_on_plugin_instance_destroy_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- urlclassifier_ps_fileload_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_free_purged_pages_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- auto_rejected_translation_offers: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- sessiondomstorage_value_size_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tilt_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_rust_mp4parse_error_code: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- innerwindows_with_mutation_listeners: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_tabqueue_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_cookies_open_readahead_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_ownpropertynames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_canplaytype_h264_profile: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_disk_cache_shutdown_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- forget_skippable_max: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_distribution_referrer_invalid: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setdiskcachemaxentrysize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_ice_add_candidate_errors_given_success: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webcrypto_method: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_page_open_to_first_sent: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_error_recovery_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_thumbnails_bg_capture_done_reason_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macfarsi: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_session_restore_all_files_corrupt: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_request_passthrough: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_connected_runtime_version: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- refresh_driver_tick: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_password_input_in_form: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- browser_set_default_error: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- media_decoder_backend_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_breakdown_dominator_tree_count: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_life_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_ocsp_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_serialize_data_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_reading_list_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prclose_udp_blocking_time_offline: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_subitem_first_byte_latency_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_status_error_code_unknown_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_jsprofiler_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- xmlhttprequest_async_or_sync: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- mixed_content_hsts: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_memory_fullyloaded_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_openinputstream: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- low_memory_events_physical: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_preconnects_created: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_encoder_bitrate_std_dev_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_javascript_error_displayed: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- loop_video_quality_inbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_js_compartments_system: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- localdomstorage_setvalue_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_startup_time_geckoready: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_tagged_bookmarks_perc: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- mixed_content_page_load: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_ice_final_connection_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_tab_anim_open_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_anim_any_frame_interval_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_syn_reply_ratio: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_starttrace_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- mixed_content_object_subrequest: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_blocklist_num_sites: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_release_optout: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- memory_js_gc_heap: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_country_fetch_caused_sync_init: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_video_quality_outbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_eme_adobe_unsupported_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_number_of_eager_tabs_restored: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_manage_copied_password: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cycle_collector_worker_visited_ref_counted: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_window_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_responsive_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- prconnect_fail_blocking_time_offline: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_pref_update_cancelations_external: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_remote_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_succesful_cert_validation_time_mozillapkix: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_layoutview_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_manage_deleted: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- network_autodial: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- should_auto_detect_language: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_animationinspector_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webfont_fonttype: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_loaded_flash: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_open_to_first_received: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_fallback_limit_reached: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_manage_opened: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- browser_is_user_default_error: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fxa_secure_credentials_save_with_mp_locked: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_tabs_open_peak_linear: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnect_blocking_time_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tap_to_load_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_page_first_sent_to_last_received: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_threadgrips_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_engine_apply_failures: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_manage_copied_username: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_protocoldescription_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_ice_success_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- blocked_on_pluginasyncsurrogate_waitforinit_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- httpconnmgr_total_speculative_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_write_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_database_filesize_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_quality_inbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_us_country_mismatched_timezone: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- safe_mode_usage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- xul_foreground_reflow_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_migration_source_browser: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- disk_cache_revalidation_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- translated_pages: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- loop_max_audio_send_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_lookup_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_chunk_recvd: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_reconfiguretab_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_settings_ul_bw: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_error: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- telemetry_archive_evicted_over_quota: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_startup_migration_existing_default_browser: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_connected_runtime_processor: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) +
+
The longitudinal dataset is logically organized as a table where rows represent profiles and columns the various metrics (e.g. startup time). Each field of the table contains a list of values, one per Telemetry submission received for that profile.
+The dataset is going to be regenerated from scratch every week, this allows us to apply non backward compatible changes to the schema and not worry about merging procedures.
+The current version of the longitudinal dataset has been build with all main pings received from 1% of profiles across all channels after mid November, which is shortly after Unified Telemetry landed. Future version will store up to 180 days of data.
+import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py + +%pylab inline +
Populating the interactive namespace from numpy and matplotlib +
sc.defaultParallelism +
32 +
The longitudinal dataset can be accessed as a Spark DataFrame, which is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R/Python.
+frame = sqlContext.sql("SELECT * FROM longitudinal") +
Number of profiles:
+frame.count() +
5421821 +
The dataset contains all histograms but it doesn’t yet include all metrics stored in the various sections of the pings. See the code that generates the dataset for a complete list of supported metrics. More metrics are going to be included in future versions of the dataset, inclusion of specific metrics can be prioritized by filing a bug.
+A Spark bug is slowing down the first and take methods on a dataframe. A way around that for now is to first convert the dataframe to a rdd and then invoke first or take, e.g.:
+first = frame.filter("normalized_channel = 'release'")\ + .select("build", + "system", + "gc_ms", + "fxa_configured", + "browser_set_default_always_check", + "browser_set_default_dialog_prompt_rawcount")\ + .rdd.first() +
As mentioned earlier on, each field of the dataframe is an array containing one value per submission per client. The submissions are chronologically sorted.
+len(first.build) +
103 +
first.build[:5] +
[Row(application_id=u'{ec8030f7-c20a-464f-9b0e-13a3a9e97384}', application_name=u'Firefox', architecture=u'x86', architectures_in_binary=None, build_id=u'20160210153822', version=u'44.0.2', vendor=u'Mozilla', platform_version=u'44.0.2', xpcom_abi=u'x86-msvc', hotfix_version=u'20160128.01'),
+ Row(application_id=u'{ec8030f7-c20a-464f-9b0e-13a3a9e97384}', application_name=u'Firefox', architecture=u'x86', architectures_in_binary=None, build_id=u'20160210153822', version=u'44.0.2', vendor=u'Mozilla', platform_version=u'44.0.2', xpcom_abi=u'x86-msvc', hotfix_version=u'20160128.01'),
+ Row(application_id=u'{ec8030f7-c20a-464f-9b0e-13a3a9e97384}', application_name=u'Firefox', architecture=u'x86', architectures_in_binary=None, build_id=u'20160210153822', version=u'44.0.2', vendor=u'Mozilla', platform_version=u'44.0.2', xpcom_abi=u'x86-msvc', hotfix_version=u'20160128.01'),
+ Row(application_id=u'{ec8030f7-c20a-464f-9b0e-13a3a9e97384}', application_name=u'Firefox', architecture=u'x86', architectures_in_binary=None, build_id=u'20160210153822', version=u'44.0.2', vendor=u'Mozilla', platform_version=u'44.0.2', xpcom_abi=u'x86-msvc', hotfix_version=u'20160128.01'),
+ Row(application_id=u'{ec8030f7-c20a-464f-9b0e-13a3a9e97384}', application_name=u'Firefox', architecture=u'x86', architectures_in_binary=None, build_id=u'20160210153822', version=u'44.0.2', vendor=u'Mozilla', platform_version=u'44.0.2', xpcom_abi=u'x86-msvc', hotfix_version=u'20160128.01')]
+Different sections of the ping are stored in different fields of the dataframe. Refer to the schema of the dataframe for a complete layout.
+first.system[0] +
Row(memory_mb=1909, virtual_max_mb=None, is_wow64=True) +
Dataframes support fields that can contain structs, maps, arrays, scalars and combination thereof. Note that in the previous example the system field is an array of Rows. You can think of a Row as a struct that allows each field to be accessed invididually.
+first.system[0].memory_mb +
1909 +
Not all profiles have all histograms. If a certain histogram, say GC_MS, is N/A for all submissions of a profile, then the field in the DataFrame will be N/A.
+first.gc_ms == None +
True +
If at least one histogram is present in the history of a profile, then all other submission that do not have that histogram will be initialized with an empty histogram.
+Flag and count “histograms” are represented as scalars.
+first.fxa_configured[:5] +
[False, False, False, False, False] +
Boolean histograms are represented with an array of two integers. Similarly, enumerated histograms are represented with an array of N integers.
+first.browser_set_default_always_check[:5] +
[[0, 1], [0, 0], [0, 0], [0, 1], [0, 1]] +
Exponential and linear histograms are represented as a struct containing an array of N integers (values field) and the sum of the entries (sum field).
+first.browser_set_default_dialog_prompt_rawcount[:5] +
[Row(values=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], sum=0), + Row(values=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], sum=0), + Row(values=[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], sum=0), + Row(values=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], sum=0), + Row(values=[1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], sum=0)] +
Keyed histograms are stored within a map from strings to values where the values depend on the histogram types and and have the same structure as mentioned above.
+frame.select("search_counts").rdd.take(2) +
[Row(search_counts={u'amazondotcom-de.searchbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], u'google.searchbar': [0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 1, 1, 1, 1, 0, 1, 0, 1, 5, 1, 1, 5, 1, 0, 1, 1, 1, 1, 1, 0, 2, 1, 1, 1, 1, 6, 1, 0, 0, 1, 1, 3, 1, 0, 3, 2, 4, 0, 1, 1, 0, 0, 2, 2, 0, 0, 3, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 1, 2, 0], u'google.urlbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0], u'other-Bing\xae.searchbar': [0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]}),
+ Row(search_counts={u'yandex.urlbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 2, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], u'yandex.searchbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 2, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 3, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 1], u'yandex.contextmenu': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], u'google.searchbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], u'wikipedia-ru.searchbar': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]})]
+Note that the following queries are run on a single machine and have been annotated with their run-time.
+%time frame.select("system").rdd.first().system[:2] +
CPU times: user 8 ms, sys: 0 ns, total: 8 ms +Wall time: 1.53 s + + + + + + +[Row(memory_mb=1909, virtual_max_mb=None, is_wow64=True), + Row(memory_mb=1909, virtual_max_mb=None, is_wow64=True)] +
%time frame.select("system.memory_mb").rdd.first() +
CPU times: user 8 ms, sys: 4 ms, total: 12 ms +Wall time: 1.57 s + + + + + + +Row(memory_mb=[1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909, 1909]) +
%time frame.selectExpr("size(system.memory_mb) as num_submissions").rdd.take(5) +
CPU times: user 4 ms, sys: 4 ms, total: 8 ms +Wall time: 1.63 s + + + + + + +[Row(num_submissions=103), + Row(num_submissions=999), + Row(num_submissions=1), + Row(num_submissions=455), + Row(num_submissions=144)] +
%time frame.selectExpr("system_os.name[0] as os_name").rdd.take(5) +
CPU times: user 12 ms, sys: 4 ms, total: 16 ms +Wall time: 1.58 s + + + + + + +[Row(os_name=u'Windows_NT'), + Row(os_name=u'Windows_NT'), + Row(os_name=u'Windows_NT'), + Row(os_name=u'Windows_NT'), + Row(os_name=u'Windows_NT')] +
%time frame.selectExpr("system_os.name[0] as os_name").where("os_name = 'Darwin'").count() +
CPU times: user 12 ms, sys: 8 ms, total: 20 ms +Wall time: 2min 26s + + + + + + +262622 +
Note that metrics that don’t tend to change often can be “uplifted” from their nested structure for fast selection. One of such metrics is the operating system name. More metrics can be uplifed on request.
+%time frame.select("os").where("os = 'Darwin'").count() +
CPU times: user 8 ms, sys: 0 ns, total: 8 ms +Wall time: 1min 15s + + + + + + +262622 +
Dataframes can be transformed to RDDs that allow to easily apply user defined functions. In general it’s worthwhile spending some time learning the Dataframe API as operations are optimized and run entirely in the JVM which can make queries faster.
+rdd = frame.rdd +
out = rdd.map(lambda x: x.search_counts).take(2) +
Select the earliest build-id with which a profile was seen using window functions:
+from pyspark.sql.window import Window +from pyspark.sql import Row +import pyspark.sql.functions as func +
subset = frame.selectExpr("client_id", "explode(build.build_id) as build_id") +
The explode function returns a new row for each element in the given array or map. See the documentation for the complete list of functions supported by DataFrames.
+window_spec = Window.partitionBy(subset["client_id"]).orderBy(subset["build_id"]) +
min_buildid = func.min(subset["build_id"]).over(window_spec) +
%time subset.select("client_id", "build_id", min_buildid.alias("first_build_id")).count() +
CPU times: user 40 ms, sys: 8 ms, total: 48 ms +Wall time: 6min 6s + + + + + + +620203170 +
Note how individual keys can be accessed without any custom Python code.
+%time sensitive = frame.select("search_counts.`yahoo.urlbar`").map(lambda x: np.sum(x[0]) if x[0] else 0).sum() +
CPU times: user 296 ms, sys: 144 ms, total: 440 ms +Wall time: 2min 17s +
And the same operation without custom Python:
+%time sensitive = frame.selectExpr("explode(search_counts.`yahoo.urlbar`) as searches").agg({"searches": "sum"}).collect() +
CPU times: user 20 ms, sys: 0 ns, total: 20 ms +Wall time: 2min 14s +
Exploding arrays seems not to be more efficient compared to custom Python code. That said, while RDD based analyses are likely not going to improve in terms of speed over time with new Spark releases, the same isn’t true for DataFrame based ones.
+%%time + +def sum_array(x, y): + tmp = [0]*len(x) + for i in range(len(x)): + tmp[i] = x[i] + y[i] + return tmp + +histogram = frame.select("GC_MS", "settings.telemetry_enabled")\ + .where("telemetry_enabled[0] = True")\ + .flatMap(lambda x: [v.values for v in x.GC_MS] if x.GC_MS else [])\ + .reduce(lambda x, y: sum_array(x, y)) + +histogram +
CPU times: user 300 ms, sys: 128 ms, total: 428 ms +Wall time: 5min 12s +
pd.Series(histogram).plot(kind="bar") +
<matplotlib.axes._subplots.AxesSubplot at 0x7fa87155ad10> +
frame.printSchema() +
root + |-- client_id: string (nullable = true) + |-- os: string (nullable = true) + |-- normalized_channel: string (nullable = true) + |-- submission_date: array (nullable = true) + | |-- element: string (containsNull = true) + |-- sample_id: array (nullable = true) + | |-- element: double (containsNull = true) + |-- size: array (nullable = true) + | |-- element: double (containsNull = true) + |-- geo_country: array (nullable = true) + | |-- element: string (containsNull = true) + |-- geo_city: array (nullable = true) + | |-- element: string (containsNull = true) + |-- dnt_header: array (nullable = true) + | |-- element: string (containsNull = true) + |-- addons: array (nullable = true) + | |-- element: string (containsNull = true) + |-- async_plugin_init: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- flash_version: array (nullable = true) + | |-- element: string (containsNull = true) + |-- previous_build_id: array (nullable = true) + | |-- element: string (containsNull = true) + |-- previous_session_id: array (nullable = true) + | |-- element: string (containsNull = true) + |-- previous_subsession_id: array (nullable = true) + | |-- element: string (containsNull = true) + |-- profile_subsession_counter: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- profile_creation_date: array (nullable = true) + | |-- element: string (containsNull = true) + |-- profile_reset_date: array (nullable = true) + | |-- element: string (containsNull = true) + |-- reason: array (nullable = true) + | |-- element: string (containsNull = true) + |-- revision: array (nullable = true) + | |-- element: string (containsNull = true) + |-- session_id: array (nullable = true) + | |-- element: string (containsNull = true) + |-- session_length: array (nullable = true) + | |-- element: long (containsNull = true) + |-- session_start_date: array (nullable = true) + | |-- element: string (containsNull = true) + |-- subsession_counter: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- subsession_id: array (nullable = true) + | |-- element: string (containsNull = true) + |-- subsession_length: array (nullable = true) + | |-- element: long (containsNull = true) + |-- subsession_start_date: array (nullable = true) + | |-- element: string (containsNull = true) + |-- timezone_offset: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- build: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- application_id: string (nullable = true) + | | |-- application_name: string (nullable = true) + | | |-- architecture: string (nullable = true) + | | |-- architectures_in_binary: string (nullable = true) + | | |-- build_id: string (nullable = true) + | | |-- version: string (nullable = true) + | | |-- vendor: string (nullable = true) + | | |-- platform_version: string (nullable = true) + | | |-- xpcom_abi: string (nullable = true) + | | |-- hotfix_version: string (nullable = true) + |-- partner: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- distribution_id: string (nullable = true) + | | |-- distribution_version: string (nullable = true) + | | |-- partner_id: string (nullable = true) + | | |-- distributor: string (nullable = true) + | | |-- distributor_channel: string (nullable = true) + | | |-- partner_names: array (nullable = true) + | | | |-- element: string (containsNull = true) + |-- settings: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- addon_compatibility_check_enabled: boolean (nullable = true) + | | |-- blocklist_enabled: boolean (nullable = true) + | | |-- is_default_browser: boolean (nullable = true) + | | |-- default_search_engine: string (nullable = true) + | | |-- default_search_engine_data: struct (nullable = true) + | | | |-- name: string (nullable = true) + | | | |-- load_path: string (nullable = true) + | | | |-- submission_url: string (nullable = true) + | | |-- search_cohort: string (nullable = true) + | | |-- e10s_enabled: boolean (nullable = true) + | | |-- telemetry_enabled: boolean (nullable = true) + | | |-- locale: string (nullable = true) + | | |-- update: struct (nullable = true) + | | | |-- channel: string (nullable = true) + | | | |-- enabled: boolean (nullable = true) + | | | |-- auto_download: boolean (nullable = true) + | | |-- user_prefs: map (nullable = true) + | | | |-- key: string + | | | |-- value: string (valueContainsNull = true) + |-- system: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- memory_mb: integer (nullable = true) + | | |-- virtual_max_mb: string (nullable = true) + | | |-- is_wow64: boolean (nullable = true) + |-- system_cpu: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- cores: integer (nullable = true) + | | |-- count: integer (nullable = true) + | | |-- vendor: string (nullable = true) + | | |-- family: integer (nullable = true) + | | |-- model: integer (nullable = true) + | | |-- stepping: integer (nullable = true) + | | |-- l2cache_kb: integer (nullable = true) + | | |-- l3cache_kb: integer (nullable = true) + | | |-- extensions: array (nullable = true) + | | | |-- element: string (containsNull = true) + | | |-- speed_mhz: integer (nullable = true) + |-- system_device: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- model: string (nullable = true) + | | |-- manufacturer: string (nullable = true) + | | |-- hardware: string (nullable = true) + | | |-- is_tablet: boolean (nullable = true) + |-- system_os: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- name: string (nullable = true) + | | |-- version: string (nullable = true) + | | |-- kernel_version: string (nullable = true) + | | |-- service_pack_major: integer (nullable = true) + | | |-- service_pack_minor: integer (nullable = true) + | | |-- locale: string (nullable = true) + |-- system_hdd: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- profile: struct (nullable = true) + | | | |-- model: string (nullable = true) + | | | |-- revision: string (nullable = true) + | | |-- binary: struct (nullable = true) + | | | |-- model: string (nullable = true) + | | | |-- revision: string (nullable = true) + | | |-- system: struct (nullable = true) + | | | |-- model: string (nullable = true) + | | | |-- revision: string (nullable = true) + |-- system_gfx: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- d2d_enabled: boolean (nullable = true) + | | |-- d_write_enabled: boolean (nullable = true) + | | |-- adapters: array (nullable = true) + | | | |-- element: struct (containsNull = true) + | | | | |-- description: string (nullable = true) + | | | | |-- vendor_id: string (nullable = true) + | | | | |-- device_id: string (nullable = true) + | | | | |-- subsys_id: string (nullable = true) + | | | | |-- ram: integer (nullable = true) + | | | | |-- driver: string (nullable = true) + | | | | |-- driver_version: string (nullable = true) + | | | | |-- driver_date: string (nullable = true) + | | | | |-- gpu_active: boolean (nullable = true) + | | |-- monitors: array (nullable = true) + | | | |-- element: struct (containsNull = true) + | | | | |-- screen_width: integer (nullable = true) + | | | | |-- screen_height: integer (nullable = true) + | | | | |-- refresh_rate: string (nullable = true) + | | | | |-- pseudo_display: boolean (nullable = true) + | | | | |-- scale: double (nullable = true) + |-- active_addons: array (nullable = true) + | |-- element: map (containsNull = true) + | | |-- key: string + | | |-- value: struct (valueContainsNull = true) + | | | |-- blocklisted: boolean (nullable = true) + | | | |-- description: string (nullable = true) + | | | |-- name: string (nullable = true) + | | | |-- user_disabled: boolean (nullable = true) + | | | |-- app_disabled: boolean (nullable = true) + | | | |-- version: string (nullable = true) + | | | |-- scope: integer (nullable = true) + | | | |-- type: string (nullable = true) + | | | |-- foreign_install: boolean (nullable = true) + | | | |-- has_binary_components: boolean (nullable = true) + | | | |-- install_day: long (nullable = true) + | | | |-- update_day: long (nullable = true) + | | | |-- signed_state: integer (nullable = true) + |-- theme: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- id: string (nullable = true) + | | |-- blocklisted: boolean (nullable = true) + | | |-- description: string (nullable = true) + | | |-- name: string (nullable = true) + | | |-- user_disabled: boolean (nullable = true) + | | |-- app_disabled: boolean (nullable = true) + | | |-- version: string (nullable = true) + | | |-- scope: integer (nullable = true) + | | |-- foreign_install: boolean (nullable = true) + | | |-- has_binary_components: boolean (nullable = true) + | | |-- install_day: long (nullable = true) + | | |-- update_day: long (nullable = true) + |-- active_plugins: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- name: string (nullable = true) + | | | |-- version: string (nullable = true) + | | | |-- description: string (nullable = true) + | | | |-- blocklisted: boolean (nullable = true) + | | | |-- disabled: boolean (nullable = true) + | | | |-- clicktoplay: boolean (nullable = true) + | | | |-- mime_types: array (nullable = true) + | | | | |-- element: string (containsNull = true) + | | | |-- update_day: long (nullable = true) + |-- active_gmp_plugins: array (nullable = true) + | |-- element: map (containsNull = true) + | | |-- key: string + | | |-- value: struct (valueContainsNull = true) + | | | |-- version: string (nullable = true) + | | | |-- user_disabled: boolean (nullable = true) + | | | |-- apply_background_updates: integer (nullable = true) + |-- active_experiment: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- id: string (nullable = true) + | | |-- branch: string (nullable = true) + |-- persona: array (nullable = true) + | |-- element: string (containsNull = true) + |-- thread_hang_activity: array (nullable = true) + | |-- element: map (containsNull = true) + | | |-- key: string + | | |-- value: struct (valueContainsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- thread_hang_stacks: array (nullable = true) + | |-- element: map (containsNull = true) + | | |-- key: string + | | |-- value: map (valueContainsNull = true) + | | | |-- key: string + | | | |-- value: struct (valueContainsNull = true) + | | | | |-- values: array (nullable = true) + | | | | | |-- element: integer (containsNull = true) + | | | | |-- sum: long (nullable = true) + |-- simple_measurements: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- active_ticks: long (nullable = true) + | | |-- profile_before_change: long (nullable = true) + | | |-- select_profile: long (nullable = true) + | | |-- session_restore_init: long (nullable = true) + | | |-- first_load_uri: long (nullable = true) + | | |-- uptime: long (nullable = true) + | | |-- total_time: long (nullable = true) + | | |-- saved_pings: long (nullable = true) + | | |-- start: long (nullable = true) + | | |-- startup_session_restore_read_bytes: long (nullable = true) + | | |-- pings_overdue: long (nullable = true) + | | |-- first_paint: long (nullable = true) + | | |-- shutdown_duration: long (nullable = true) + | | |-- session_restored: long (nullable = true) + | | |-- startup_window_visible_write_bytes: long (nullable = true) + | | |-- startup_crash_detection_end: long (nullable = true) + | | |-- startup_session_restore_write_bytes: long (nullable = true) + | | |-- startup_crash_detection_begin: long (nullable = true) + | | |-- startup_interrupted: long (nullable = true) + | | |-- after_profile_locked: long (nullable = true) + | | |-- delayed_startup_started: long (nullable = true) + | | |-- main: long (nullable = true) + | | |-- create_top_level_window: long (nullable = true) + | | |-- session_restore_initialized: long (nullable = true) + | | |-- maximal_number_of_concurrent_threads: long (nullable = true) + | | |-- startup_window_visible_read_bytes: long (nullable = true) + |-- spdy_npn_connect: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_sync_number_of_syncs_failed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_canvasdebugger_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- telemetry_scheduler_tick_exception: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- moz_sqlite_open_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- disk_cache_revalidation_safe: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_corrupt_file: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_copy_panel_actions: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_sessions: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_predict_work_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_truncate_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_cache_entry_reuse_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_keyed_count_init_no_record: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- prconnectcontinue_blocking_time_connectivity_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- history_lastvisited_tree_query_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_thumbnails_bg_capture_queue_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_reflow_duration: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- fennec_load_saved_page: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webfont_download_time_after_start: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- httpconnmgr_used_speculative_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_quota_reset_to: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnect_fail_blocking_time_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_sitesettings: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_not_pref_update_service_enabled_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- places_annos_pages_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gradient_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_syn_ratio: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_perftools_recording_import_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_debugger_rdp_local_navigateto_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- defective_permissions_sql_removed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- loop_max_audio_receive_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_subscribe_failed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_opencacheentry: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_country_timeout: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_developertoolbar_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_threaddetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- paint_build_displaylist_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- social_enabled_on_session: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- urlclassifier_update_remote_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_discarded_content_pings_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- fx_session_restore_send_update_caused_oom: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cycle_collector_time_between: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- social_toolbar_buttons: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- push_api_permission_granted: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_video_quality_inbound_packetloss_rate: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_release_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_menu_eyedropper_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- ssl_reasons_for_not_false_starting: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- predictor_wait_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_distribution_code_category: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_isstorageenabledforpolicy: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- html_background_reflow_ms_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- device_reset_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_favicon_ico_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_country_fetch_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_sub_open_to_first_from_cache_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_hit_rate_per_cache_size: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_validation_success_by_ca: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_gesture_install_snapshot_of_page: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_blackbox_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_worker_visited_gced: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_disk_cache_shutdown_clear_private: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_window_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cert_pinning_moz_test_results_by_host: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_audio_quality_inbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_decoder_framerate_10x_std_dev_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_memory_reporter_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_v1_miss_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_clientevaluate_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_handshake_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_ping_count_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- newtab_page_blocked_sites_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- process_crash_submit_success: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- fx_thumbnails_bg_queue_size_on_capture: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getlastfetched: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_cl_update_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_tabdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_notify_registration_lost: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- scroll_input_methods: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_startup_time_mediaenumerated: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_hud_app_memory_navigationloaded_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- cache_disk_search_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fullscreen_change_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_storage_sqlite: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_has_no_keys_when_unlocked: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- network_cache_metadata_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- audiostream_first_open_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- browser_set_default_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macromanian: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- early_gluestartup_read_transfer: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- aboutcrashes_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_video_quality_outbound_rtt: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_audio_quality_outbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_test_count_init_no_record: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- url_path_ends_in_exclamation: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_thumbnails_bg_capture_canvas_draw_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_threaddetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_country_fetch_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_truncate_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_datachannel_negotiated: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- httpconnmgr_unused_speculative_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gluestartup_read_transfer: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- onbeforeunload_prompt_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- about_accounts_content_server_load_started_count: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_form_autofill_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_canvasdebugger_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_decoder_framerate_avg_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_prototypesandproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_quota_expiration_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_session_restore_dom_storage_size_estimate_chars: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_eme_request_success_latency_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_slice_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_settings_dl_bw: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_project_editor_save_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ntlm_module_used_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_ice_success_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_isstreambased: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_window_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_subscribe_http2_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_koi8r: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- plugin_called_directly: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- e10s_addons_blocker_ran: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- url_path_contains_exclamation_double_slash: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_write_file_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_fontinspector_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ipc_message_size: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_workerdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_migration_errors: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- plugin_hang_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- bucket_order_errors: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- localdomstorage_getkey_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_getcurrentposition_secure_origin: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_shadereditor_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- keygen_generated_key_type: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- predictor_predict_time_to_action: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_responsive_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- web_notification_request_permission_callback: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- newtab_page_shown: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_archive_scan_ping_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_audio_quality_outbound_packetloss_rate: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_call_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_video_decoder_bitrate_std_dev_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- localdomstorage_value_size_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_configured_master_password: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- prclose_tcp_blocking_time_connectivity_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugins_notification_user_action: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_new_project_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_close: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_download_code_partial: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_options_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_debugger_rdp_local_threadgrips_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- stumbler_observations_per_day: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_metadata_first_read_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gradient_retention_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_slow_phase: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_last_notify_interval_days_external: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- bad_fallback_font: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_scope_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_open_to_first_from_cache_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_cl_check_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_app_startup_time_contentinteractive: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- ssl_cert_error_overrides: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- composite_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- csp_documents_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macarabic: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_picker_eyedropper_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_computedview_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fennec_reader_view_cache_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sts_poll_and_event_the_last_cycle: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_auth_ecdsa_curve_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- weave_start_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_inspector_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- xul_background_reflow_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sts_poll_block_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_ping: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- flash_plugin_states: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fullscreen_transition_black_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_onprofileshutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_memory_diff_census: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_disk_cache_streamio_close_main_thread: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_detach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sts_number_of_pending_events_in_the_last_cycle: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_import_project_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- moz_storage_async_requests_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getsecurityinfo: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsdiskcachedevicedeactivateentryevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_write_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_win8_source_is_mls: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- xhr_in_worker: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- gc_reason_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- weave_fxa_key_fetch_auth_errors: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- geolocation_watchposition_secure_origin: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- translated_characters: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_invalid_lastupdatetime_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_shutdown_database_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_custom_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- rejected_message_manager_message: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- sts_number_of_pending_events: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_unique: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_search_loader_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tracking_protection_pbm_disabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gfx_crash: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- media_ogg_loaded_is_chained: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_jsdebugger_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webconsole_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webrtc_load_state_normal_short: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tab_switch_cache_position: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_most_recent_expired_visit_days: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_lc_completions: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_call_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_backups_tojson_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_reload_addon_reload_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_unload_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- reader_mode_download_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- subprocess_abnormal_abort: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- cookie_scheme_security: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_touch_used: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- checkerboard_severity: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- about_accounts_content_server_loaded_time_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- telemetry_archive_directories_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_cache_miss_halflife_experiment_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_audio_quality_outbound_rtt: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- link_icon_sizes_attr_dimension: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_shadereditor_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_09_info: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugin_hang_ui_response_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- popup_notification_dismissal_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- a11y_iatable_usage_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_thumbnails_hit_or_miss: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- slow_script_notify_delay: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_sync11_migrations_failed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_getlength_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tracking_protection_shield: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_cache_entry_alive_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_urlbar_selected_result_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_paintflashing_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_browserconsole_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_reload_addon_installed_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webfont_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_warnings: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_downloads: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- low_memory_events_commit_space: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_resume_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_hang_notice_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_pending_load_failure_parse: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- spdy_server_initiated_streams: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_memory_export_snapshot_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- decoder_instantiated_maccyrillic: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webrtc_video_decoder_framerate_avg_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- url_path_contains_exclamation_slash: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_content_crash_presented: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- idle_notify_idle_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_options_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_canplaytype_h264_constraint_set_flag: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_not_pref_update_enabled_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- stumbler_upload_cell_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- social_sidebar_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_memory_navigationinteractive_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- telemetry_archive_evicting_dirs_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_tcp_connection: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_jsbrowserdebugger_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- pwmgr_login_page_safety: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_openoutputstream: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webfont_size_per_page: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_settings_cwnd: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- crash_store_compressed_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnectcontinue_blocking_time_link_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_cache_disposition_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cookies_3rdparty_num_sites_accepted: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_encoder_bitrate_avg_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webaudioeditor_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_call_count_2: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_sessiondata_failed_parse: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- prconnect_fail_blocking_time_link_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_formdata: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_responsive_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- predictor_prefetch_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_subresource_degradation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_resumed_session: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_request_per_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_listaddons_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_username_present: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_ice_on_time_trickle_arrival_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_reload_addon_installed_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- sqlitebridge_provider_home_locked: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- flash_plugin_area: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fontlist_initfacenamelists: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getexpirationtime: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_listprocesses_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_predictions: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_permission_requested: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- charset_override_situation: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_startup_init_session_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_parameternames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_discarded_pending_pings_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_document_generator: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- browser_set_default_always_check: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_tab_switch_total_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- link_icon_sizes_attr_usage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_save_heap_snapshot_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_capabilities_seccomp_tsync: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- network_disk_cache_revalidation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_favicon_gif_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setofflinecachecapacity: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_minor_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- about_accounts_content_server_failure_time_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_hud_app_startup_time_scanend: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- predictor_total_preconnects_used: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_check_no_update_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_reconfigurethread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_manage_deleted_all: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- prconnect_fail_blocking_time_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setstoragepolicy: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_mft_output_null_samples: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- shutdown_ok: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_favicon_bmp_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_history_library_search_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- idle_notify_idle_listeners: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_permanent_cert_error_overrides: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_identity_popup_open_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_call_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_images_content_used_uncompressed: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tabs_pinned_peak_linear: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_font_types: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscompressoutputstreamwrapper_release: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_latency_us: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_cache_disposition_2_v2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_migration_usage: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- fx_session_restore_collect_all_windows_data_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_cookies_write_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_sync11_migrations_succeeded: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- ssl_auth_algorithm_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_ice_late_trickle_arrival_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_key_exchange_algorithm_resumed: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_dominator_tree_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- graphics_sanity_test: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_vp9_benchmark_fps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_developertoolbar_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- startup_measurement_errors: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_settings_iw: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_budget_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_app_memory_scanend_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- graphics_sanity_test_os_snapshot: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_property_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- display_scaling_linux: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tilt_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- remote_jar_protocol_used: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- predictor_predict_attempts: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- web_notification_menu: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_addondetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_not_pref_update_staging_enabled_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- update_service_installed_notify: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_activity_counter: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_video_encoder_dropped_frames_per_call_fpm: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_total: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_lc_prefixes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webide_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- print_preview_simplify_page_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- image_decode_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_listtabs_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_learn_full_queue: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- slow_script_page_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_ruleview_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- web_notification_senders: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- places_database_pagesize_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_cookies_write_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_frames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cookies_3rdparty_num_attempts_blocked: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_rust_mp4parse_track_match_video: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_ice_add_candidate_errors_given_failure: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_audio_quality_inbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_assign_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_favicon_svg_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_not_pref_update_enabled_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_reconfigurethread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_proxy_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_display_source_remote_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_decoder_framerate_10x_std_dev_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_symmetric_cipher_resumed: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- prconnectcontinue_blocking_time_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cookies_3rdparty_num_sites_blocked: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_uss: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- gc_minor_reason_long: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_check_code_external: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_reload_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- alerts_service_dnd_supported_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_jsprofiler_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webrtc_video_quality_outbound_packetloss_rate: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_disk_cache_trashrename: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_developertoolbar_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load_net_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_places_write_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- browser_is_user_default: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dnt_usage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setdiskcacheenabled: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- requests_of_original_content: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webide_wifi_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_video_recovery_before_error_per_min: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_migration_homepage_imported: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- fennec_topsites_loader_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_state_code_unknown_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_onprofilechanged: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_learn_work_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_first_sent_to_last_received: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- top_level_content_documents_destroyed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- total_count_high_errors: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_listserviceworkerregistrations_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_time_to_view_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_print: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- early_gluestartup_read_ops: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_engine_apply_new_failures: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getpredicteddatasize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_get_user_media_secure_origin: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- popup_notification_stats: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- graphics_driver_startup_test: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_plugins: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsasyncdoomevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_quality_outbound_jitter: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_bytes_before_cert_callback: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_sync11_migration_sentinels_seen: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- loop_max_video_send_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- eventloop_ui_activity_exp_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- disk_cache_invalidation_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_startup_external_content_handler: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_response_status_code: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_frames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_iso_8859_5: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- search_reset_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_chain_key_size_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_listtabs_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_non_incremental: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_video_decoder_discarded_packets_per_call_ppm: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getstoragepolicy: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- d3d11_sync_handle_failure: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- moz_sqlite_cookies_read_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- database_locked_exception: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_cookies_read_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_app_memory_mediaenumerated_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- cert_pinning_moz_results: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_connected_runtime_platform_version: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- update_not_pref_update_auto_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- prclose_tcp_blocking_time_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_detailed_dropped_frames_proportion: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- push_api_notification_received: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_screen_resolution_enumerated_per_user: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gc_mmu_50: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- disk_cache_corrupt_details: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_backups_daysfromlast: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugin_startup_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- flash_plugin_height: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_kea_ecdhe_curve_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_archive_oldest_directory_age: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_transaction_use_altsvc: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- search_service_init_sync: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_storage_async_requests_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- urlclassifier_ps_fallocate_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_size_full_fat: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gdi_initfontlist_total: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_property_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_cipher_suite_resumed: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- subject_principal_accessed_without_script_on_stack: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- shutdown_phase_duration_ticks_xpcom_will_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_reconfiguretab_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_get_user_media_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_autocomplete_1st_result_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_delete_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_tls13_intolerance_reason_pre: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_cache_v1_hit_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_memory_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tap_to_load_image_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prclose_udp_blocking_time_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_session_restore_file_size_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_sweep_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- idle_notify_back_listeners: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- word_cache_misses_content: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_ice_success_rate: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- transaction_wait_time_http: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_codec_used: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsoutputstreamwrapper_closeinternal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_requestdatasizechange: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_embed: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getlastmodified: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_memory_search_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- newtab_page_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_stoptrace_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- word_cache_hits_chrome: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_complete_load_cached: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_ice_late_trickle_arrival_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_os_is_64_bits_per_user: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_settings_rtt: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_engine_sync_errors: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- memory_js_main_runtime_temporary_peak: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_other_write_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setdiskcachecapacity: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- localdomstorage_getallkeys_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_export_tohtml_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- flash_plugin_width: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_jank: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_bindings_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_eme_play_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- sts_poll_and_events_cycle: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_tracerdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_app_startup_time_fullyloaded: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_heap_snapshot_edge_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- stumbler_upload_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_capabilities_seccomp_bpf: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- reader_mode_worker_parse_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_favicon_other_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_cipher_suite_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- thunderbird_gloda_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_jsbrowserdebugger_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fxa_configured: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- message_manager_message_size: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- http_transaction_use_altsvc_oe: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_can_create_h264_decoder: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_status_error_code_partial_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_import_project_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_layoutview_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- onbeforeunload_prompt_action: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_diskdeviceheapsize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_vsize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_scc_sweep_max_pause_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_hash_stats: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- memory_resident_fast: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_os_enumerated_per_user: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_toolbox_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_total: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_worker: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- addon_manager_upgrade_ui_shown: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- gc_mark_gray_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- deferred_finalize_async: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tracking_protection_events: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_client_call_url_requests_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_offlineapps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_chunks: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_discarded_archived_pings_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- disk_cache_smart_size_using_old_max: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_session_at_900fd: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_validation_http_request_failed_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_fs_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_ice_checking_rate: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- font_cache_hit: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_collect_data_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_pending_pings_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_globalhistory_visited_build_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_cache_read_time_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_initial_failed_cert_validation_time_mozillapkix: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_load_state_stressed: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_fallback_shown: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_unloaded_flash: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dns_failed_lookup_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- update_pref_service_errors_external: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_koi8u: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_auth_dsa_key_size_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- perf_monitoring_test_cpu_rescheduling_proportion_moved: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_us_country_mismatched_platform_osx: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_ev_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_archive_session_ping_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- newtab_page_life_span_suggested: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load_cached_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_minor_us: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webfont_srctype: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_annos_bookmarks_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gluestartup_hard_faults: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_build_cache_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_processrequest: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugins_infobar_shown: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_syn_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_encoder_bitrate_avg_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_prompt_remember_action: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_visitmetadata: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_fastseek_used: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- blocked_on_plugin_module_init_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- places_keywords_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_ps_construct_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_num_saved_passwords: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_encoder_framerate_avg_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_iso2022jp: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webrtc_load_state_relaxed_short: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_aboutdebugging_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_ruleview_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- transaction_wait_time_spdy: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_perftools_recording_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- places_favicon_jpeg_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_substring_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_bookmarks_toolbar_init_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_doom: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_clientevaluate_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_browserconsole_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_project_editor_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- web_notification_exceptions_opened: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_storage_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webide_other_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_tab_click_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_worker_collected: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_compress: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- changes_of_target_language: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- pwmgr_prompt_update_action: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- js_define_getter_setter_this_null_undefined: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsdiskcachestreamio_closeoutputstream: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_addondetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_simulator_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_menu_eyedropper_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- video_openh264_gmp_disappeared: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- page_faults_hard: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_decode_error_time_permille: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_urlbar_selected_result_index: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_aboutdebugging_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_ping_evicted_for_server_errors: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- idle_notify_back_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- thunderbird_indexing_rate_msg_per_s: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_revalidation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_archive_checking_over_quota_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_offline_cache_document_load: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_sync_skippable: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_disk_cache_streamio_close: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_canplaytype_h264_level: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- sqlitebridge_provider_forms_locked: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- audiostream_later_open_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_stoptrace_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_subscribe_succeeded: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macgurmukhi: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_npn_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setofflinecacheenabled: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_computedview_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_pref_update_cancelations_notify: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_cache_v2_miss_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_keyed_flag: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: boolean (containsNull = true) + |-- webfont_download_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_play_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tabs_open_average_linear: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_cookies_read_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_recovery_after_error_per_min: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_quality_outbound_rtt: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tracking_protection_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_ocsp_stapling: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webcrypto_extractable_sig: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_sub_dns_lookup_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_scope_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getfile: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_predict_full_queue: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_anim_open_preview_frame_interval_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_session_restore_collect_cookies_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_jsbrowserdebugger_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- subprocess_kill_hard: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- flash_plugin_instances_on_page: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_test_keyed_count: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsevictdiskcacheentriesevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_ice_failure_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- service_worker_registrations: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- system_font_fallback_first: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_encoder_framerate_10x_std_dev_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnectcontinue_blocking_time_offline: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webaudioeditor_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- telemetry_pending_pings_evicted_over_quota: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_getcacheiotarget: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_prototypeandproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setpredicteddatasize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_device_search_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_lm_inconsistent: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webconsole_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- forced_device_reset_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- shutdown_phase_duration_ticks_profile_before_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_audio_quality_outbound_rtt: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_preresolves: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- newtab_page_life_span: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_storage_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- network_session_at_256fd: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugins_notification_shown: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_startup_time_visuallyloaded: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_evictentriesforclient: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_decoder_bitrate_std_dev_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsdiskcachestreamio_write: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_startup_migration_browser_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_startup_time_navigationloaded: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- places_database_size_per_page_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_picker_eyedropper_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- prclose_udp_blocking_time_connectivity_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_unsubscribe_succeeded: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- update_ping_count_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- update_check_code_notify: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- canvas_2d_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- disk_cache_reduction_trial: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- weave_complete_success_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_errors: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_video_decoder_bitrate_avg_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_unsubscribe_attempt: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- spdy_settings_max_streams: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- service_worker_spawn_gets_queued: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_layoutview_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webcrypto_extractable_enc: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_max_audio_receive_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- backgroundfilesaver_thread_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_cookies_sync_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_stream_types: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_canvasdebugger_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- weave_device_count_desktop: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_sync_number_of_syncs_failed_backoff: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- total_count_low_errors: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_listworkers_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- canvas_webgl_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- browser_set_default_time_to_completion_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sts_poll_cycle: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- translation_opportunities_by_language: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- network_cache_metadata_first_read_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_read_heap_snapshot_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_ice_final_connection_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_prototypesandproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_metadata_second_read_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load_net: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- family_safety: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_visited_ref_counted: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_incremental_disabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_disk_cache_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_ocsp_may_fetch: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_manual_restore_duration_until_eager_tabs_restored_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_capabilities_user_namespaces: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- places_bookmarks_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_listworkers_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_global_degradation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_connected_runtime_type: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- loop_two_way_media_conn_length_1: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_browserconsole_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_webide_local_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- long_reflow_interruptible: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dns_lookup_method2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_protocoldescription_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dwritefont_delayedinitfontlist_total: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- low_memory_events_virtual: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_ownpropertynames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_version_fallback_inappropriate: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_startup_onload_initial_window_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_scheme_upgrade: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_syn_reply_size: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_keyed_release_optin: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: boolean (containsNull = true) + |-- js_telemetry_addon_exceptions: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_was_spawned: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- moz_sqlite_places_read_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_revalidation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- shutdown_phase_duration_ticks_quit_application: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- mixed_content_unblock_counter: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_decoded_h264_sps_level: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macturkish: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fennec_sync_number_of_syncs_completed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_cache_entry_reload_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_dns_issue_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- permissions_remigration_comparison: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_transaction_is_ssl: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_places_read_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- canvas_webgl_failure_id: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- memory_heap_allocated: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load_cached: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_unable_to_apply_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- js_deprecated_language_extensions_in_addons: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gc_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_prototype_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_visited_gced: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- youtube_rewritable_embed_seen: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cycle_collector_full: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getmetadataelement: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_reader_view_button: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- search_service_us_country_mismatched_platform_win: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_homepanels_custom: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- security_ui: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_project_editor_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- check_java_enabled: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_adobe_gmp_disappeared: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_hud_app_memory_visuallyloaded_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- alerts_service_dnd_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugins_infobar_block: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_restore_window_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsdiskcachebinding_destructor: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_netmonitor_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- html_foreground_reflow_ms_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_cookies: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webfont_compression_woff: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_archive_evicting_over_quota_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- content_response_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_room_create: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_listaddons_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tabs_pinned_average_linear: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_assign_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_load_state_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_other_read_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_nonus_country_mismatched_platform_osx: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_places_write_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_eventlisteners_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_connected_runtime_id: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- loop_video_encoder_framerate_10x_std_dev_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_max_pause_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- translated_pages_by_language: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_inverted_census: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setmemorycachemaxentrysize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_complete_load_cached_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_kea_rsa_key_size_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_audio_quality_outbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_unblackbox_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_status_error_code_partial_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- e10s_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_webapps_read_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_call_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_preconnects: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsinputstreamwrapper_release: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_rust_mp4parse_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_eventlisteners_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_v2_hit_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_places_read_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- transaction_wait_time_http_pipelines: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_discarded_send_pings_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_subscribe_ws_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_read_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_sync_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_oom: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_not_pref_update_auto_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- graphics_sanity_test_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_manage_sorted: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- geolocation_getcurrentposition_visible: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- br_9_2_1_subject_alt_names: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_bookmarks_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_cache_read_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_speed_gif: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_tls11_intolerance_reason_pre: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- weave_device_count_mobile: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_other_read_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_gesture_take_snapshot_of_page: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_keyed_release_optout: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: boolean (containsNull = true) + |-- composite_frame_roundtrip_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_displaystring_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- canvas_webgl2_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- e10s_blocked_from_running: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_async_snow_white_freeing: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_has_icon_updates: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_gesture_compress_snapshot_of_page: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_quality_inbound_jitter: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_encoder_bitrate_std_dev_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_js_compartments_user: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_quality_outbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_renegotiations: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- thunderbird_conversations_time_to_2nd_gloda_query_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_perftools_recording_export_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- video_mse_buffering_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- canvas_webgl_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_client_call_url_shared: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_auth_dialog_stats: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_audio_quality_outbound_jitter: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sqlitebridge_provider_passwords_locked: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_jsprofiler_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_complete_load_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_disk_cache_open: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- startup_crash_detected: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_tls10_intolerance_reason_post: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_version2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dns_lookup_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_document_version: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_visitentries: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_sources_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_configured: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_idle_frecency_decay_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_decoder_discarded_packets_per_call_ppm: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_interrupt_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_accuracy_exponential: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_anim_open_frame_interval_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_audio_quality_inbound_packetloss_rate: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pdf_viewer_document_size_kb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- js_deprecated_language_extensions_in_content: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gc_mark_roots_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_browser_fullscreen_used: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- changes_of_detected_language: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_jsdebugger_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dwritefont_delayedinitfontlist_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_complete_timeout: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ipc_reply_size: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- ssl_key_exchange_algorithm_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- process_crash_submit_attempt: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_aboutdebugging_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_disk_cache_deletedir_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- view_source_external_result_boolean: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_sessiondata_failed_load: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_pref_service_errors_notify: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getclientid: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getcacheelement: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_pinning_results: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_kea_dhe_key_size_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_sub_open_to_first_from_cache: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- social_panel_clicks: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_collect_data_longest_op_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_netmonitor_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_can_create_aac_decoder: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- media_hls_decoder_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- search_counts: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- prconnect_blocking_time_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- a11y_isimpledom_usage_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setsecurityinfo: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_animation_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_tabdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dwritefont_init_problem: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_send: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_activation_count: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- urlclassifier_ps_failure: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_weak_ciphers_fallback: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_was_killed: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_download_code_complete: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dns_cleanup_age: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_avsync_when_video_lags_audio_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_expired: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_thumbnails_bg_capture_service_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_request_per_page_from_cache: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_refresh_driver_chrome_frame_delay_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webcrypto_extractable_import: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugins_infobar_allow: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_decoded_h264_sps_profile: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- youtube_nonrewritable_embed_seen: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- geolocation_watchposition_visible: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_listprocesses_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- slow_addon_warning_response_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- subprocess_crashes_with_dump: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_disk_cache_overhead: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_new_project_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- mac_initfontlist_total: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_parallel_streams: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_styleeditor_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- fx_sanitize_cache: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsdecompressinputstreamwrapper_release: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- e10s_window: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_tracking_protection_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- newtab_page_enhanced: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_video_recovery_before_error_per_min: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nsdiskcachemap_revalidation: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_globalhistory_update_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_listserviceworkerregistrations_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- checkerboard_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- csp_unsafe_eval_documents_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_resume_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- localdomstorage_removekey_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- newtab_page_site_clicked: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- system_font_fallback_script: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- places_favicon_png_sizes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_invalid_lastupdatetime_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- content_documents_destroyed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_storage_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- stumbler_volume_bytes_uploaded_per_sec: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_cookies_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- stumbler_time_between_uploads_sec: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_autocomplete_urlinline_domain_query_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_macgujarati: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- browserprovider_xul_import_history: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_status_error_code_complete_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_probe_maxcount: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_heap_committed_unused: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_mark_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_form_action_effect: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- addon_shim_usage: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- webrtc_stun_rate_limit_exceeded_by_type_given_failure: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- predictor_confidence: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_cookies_sync_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_open_to_first_sent: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- xul_initial_frame_construction: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_navigateto_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getstoragedatasize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnect_blocking_time_offline: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- osfile_writeatomic_jank_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- startup_cache_age_hours: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_read_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- input_event_response_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- browserprovider_xul_import_bookmarks: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- message_manager_message_size2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- moz_sqlite_places_sync_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- audio_mft_output_null_samples: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- loop_video_decode_error_time_permille: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_check_extended_error_external: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_startup_time_javaui: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_validation_http_request_succeeded_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_speed_jpeg: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_tiny_content: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- ssl_server_auth_eku: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_worker_need_gc: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setexpirationtime: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_maccroatian: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_thumbnails_store_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_anim_close_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_audio_quality_inbound_jitter: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dom_range_detached: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_startup_time_navigationinteractive: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- places_pages_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_avsync_when_audio_lags_video_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tilt_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- moz_sqlite_cookies_write_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_cannot_stage_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getkey: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_content_crash_dump_unavailable: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- search_service_engine_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_pending_evicting_over_quota_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_max_video_receive_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_closeallstreams: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_tabqueue_prompt_enable_yes: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- web_notification_permission_removed: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- push_api_used: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- loop_room_delete: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_breakdown_census_count: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_sessiondata_failed_validation: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsinputstreamwrapper_lazyinit: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prclose_udp_blocking_time_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cycle_collector_need_gc: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dns_renewal_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_base_confidence: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_v2_input_stream_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_mse_play_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_response_version: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_pinning_test_results: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_sync_number_of_syncs_started: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cycle_collector_collected: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_cert_verification_errors: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- web_notification_permissions: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_interrupt_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- application_reputation_local: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_other_write_b: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_ruleview_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- search_service_nonus_country_mismatched_platform_win: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_ice_add_candidate_errors_given_failure: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_service_installed_external: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_number_of_tabs_restored: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_openh264_gmp_missing_files: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_starttrace_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_complete_load: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getfetchcount: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- data_storage_entries: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_request_granted: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- check_addons_modified_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_infobar_action_buttons: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_perftools_recording_duration_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- display_scaling_mswin: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setmetadataelement: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_decoded_h264_sps_constraint_set_flag: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_not_pref_update_service_enabled_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- places_backups_bookmarkstree_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_adobe_gmp_missing_files: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_renegotiations: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_goaway_local: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ipc_transaction_cancel: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_last_notify_interval_days_notify: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_refresh_driver_content_frame_delay_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_connection_play_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_handshake_version: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_page_load_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_ping_size_exceeded_archived: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- battery_status_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_pageload_is_ssl: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_refresh_driver_sync_scroll_frame_delay_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_memory_import_snapshot_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_sessiononly_preload_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- touch_enabled_device: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- predictor_learn_attempts: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_startup_migration_automated_import_succeeded: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_disk_cache_disposition_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugin_load_metadata: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_wmf_decode_error: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- spdy_request_per_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- permissions_migration_7_error: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_tab_switch_spinner_visible_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dns_renewal_time_for_ttl: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- localdomstorage_clear_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_offline_search_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_reflows: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_computedview_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- translation_opportunities: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_controlled_documents: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_ice_failure_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_state_code_unknown_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_thumbnails_bg_capture_page_load_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- blocked_on_plugin_stream_init_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_releasemany_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_v2_output_stream_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_state_code_partial_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_manage_visibility_toggled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- checkerboard_peak: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_tls12_intolerance_reason_pre: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- osfile_worker_ready_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- xul_cache_disabled: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_tls10_intolerance_reason_pre: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_page_dns_issue_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_goaway_peer: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_service_manually_uninstalled_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_getvalue_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- checkerboard_potential_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_hud_app_memory_contentinteractive_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- localdomstorage_key_size_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_kbread_per_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_workerdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_subscribe_attempt: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webrtc_load_state_relaxed: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_cannot_stage_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webide_connection_time_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_blackbox_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_heap_snapshot_node_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- web_notification_shown: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_custom_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- print_preview_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_sub_complete_load_net_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- service_worker_updated: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_sub_cache_read_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_memory_filter_census: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gluestartup_read_ops: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_perftools_console_recording_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_sub_tcp_connection: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_switch_total_e10s_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- range_checksum_errors: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_idle_maintenance_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_ibm866: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- telemetry_invalid_ping_type_submitted: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- find_plugins: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_open_to_first_received: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_status_error_code_unknown_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_validation_http_request_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- websockets_handshake_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fxa_hawk_errors: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_eme_request_failure_latency_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_complete_remote_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- gfx_content_failed_to_acquire_device: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_pinning_moz_results_by_host: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setcacheelement: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_unblackbox_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- a11y_consumers: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_symmetric_cipher_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugin_hang_ui_user_response: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_finish_igc: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- web_notification_clicked: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- network_disk_cache_deletedir: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_scheduler_send_daily: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- localdomstorage_preload_pending_on_first_access: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_styleeditor_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webaudioeditor_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- permissions_sql_corrupted: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_pending_pings_age: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_num_httpauth_passwords: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_capabilities_user_namespaces_privileged: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_state_code_partial_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- prclose_tcp_blocking_time_link_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ipc_same_process_message_copy_oom_kb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_parameternames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_hls_canplay_requested: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_connected_runtime_app_type: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- fennec_globalhistory_add_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_options_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- search_service_init_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_netmonitor_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- http_saw_quic_alt_protocol: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_reload_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_wiz_last_page_code: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsprocessrequestevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_heap_overhead_fraction: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- application_reputation_server_verdict: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_spawn_attempts: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- fx_new_window_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_session_restore_auto_restore_duration_until_eager_tabs_restored_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_num_passwords_per_hostname: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_history: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_open_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_us_timezone_mismatched_country: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- search_service_has_updates: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- image_max_decode_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_sorted_bookmarks_perc: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_cache_v1_truncate_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_places_sync_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- about_accounts_content_server_loaded_rate: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- stumbler_upload_observation_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_state_code_complete_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pdf_viewer_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- perf_monitoring_slow_addon_jank_us: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- webrtc_ice_on_time_trickle_arrival_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_paintflashing_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_enumproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_eyedropper_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- word_cache_misses_chrome: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_memory_cache_disposition_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- master_password_enabled: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_session_restore_read_file_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getdatasize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- total_content_page_load_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- components_shim_accessed_by_content: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_debugger_display_source_local_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_login_last_used_days: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_time_until_handshake_finished: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnect_fail_blocking_time_connectivity_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- application_reputation_should_block: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_project_editor_save_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_webide_connected_runtime_os: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- places_autocomplete_6_first_results_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_pending_load_failure_read: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_tls11_intolerance_reason_post: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- push_api_permission_denied: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_prototypeandproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webcrypto_extractable_generate: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsblockoncachethreadevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- network_disk_cache2_shutdown_clear_private: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_maintenance_daysfromlast: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_ping_size_exceeded_send: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- dwritefont_delayedinitfontlist_collect: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_doomandfailpendingrequests: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- blocked_on_plugin_instance_init_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- webrtc_datachannel_negotiated: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fips_enabled: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- application_reputation_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_take_snapshot_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_archive_evicted_old_dirs: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_sync_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_dropped_frames_proportion: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_places_write_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- reader_mode_download_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_ping_size_exceeded_pending: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- ssl_auth_rsa_key_size_full: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- prclose_udp_blocking_time_link_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tabletmode_page_load: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- devtools_animationinspector_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- word_cache_hits_content: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_predictions_calculated: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_unload_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_registration_loading: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_prefetches_used: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- shared_worker_spawn_gets_queued: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- ghost_windows: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cookies_3rdparty_num_attempts_accepted: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_get_executable_lines_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_encoder_dropped_frames_per_call_fpm: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_content_encoding: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- charset_override_used: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_debugger_rdp_remote_get_executable_lines_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_thumbnails_capture_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_releasemany_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- application_reputation_server: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsoutputstreamwrapper_release: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_total_top_visits: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- system_font_fallback: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_max_video_receive_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webfont_compression_woff2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsinputstreamwrapper_closeinternal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- dom_timers_recently_set: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_expiration_steps_to_clean2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_worker_oom: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- http_offline_cache_disposition_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_id: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- paint_rasterize_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_preconnects_unused: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_prefetches: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_assemble_payload_exception: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- sandbox_capabilities_enabled_media: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setmemorycache: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- browser_set_default_dialog_prompt_rawcount: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_encoder_framerate_avg_per_call: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_bindings_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_settings_retrans: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- newtab_page_pinned_sites_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cycle_collector_max_pause: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_recovery_after_error_per_min: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_macicelandic: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- view_source_in_browser_opened_boolean: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_mse_join_latency_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_state_code_complete_stage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_paintflashing_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- perf_monitoring_slow_addon_cpow_us: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- reader_mode_serialize_dom_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_scc_sweep_total_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_scheduler_wakeup: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_jsdebugger_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- br_9_2_2_subject_common_name: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- blocklist_sync_file_load: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cert_chain_sha1_policy_status: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_usb_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- push_api_unsubscribe_failed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- browser_shim_usage_blocked: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- moz_sqlite_other_read_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_eyedropper_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- prconnect_blocking_time_connectivity_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_speed_png: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_error_recovery_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_shadereditor_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- ssl_time_until_ready: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_broker_initialized: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_connection_debug_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_observed_end_entity_certificate_lifetime: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- stumbler_time_between_start_sec: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_perftools_selected_view_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- video_mse_unload_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- sts_number_of_onsocketready_calls: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_call_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webfont_per_page: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_fontinspector_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_subitem_open_latency_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- startup_cache_invalid: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fennec_restricted_profile_restrictions: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- update_unable_to_apply_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_toolbox_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_session_restore_number_of_windows_restored: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_sync11_migration_notifications_offered: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_rust_mp4parse_track_match_audio: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_inspector_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- should_translation_ui_appear: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_getdeviceid: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- social_sidebar_open_duration: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- shutdown_phase_duration_ticks_profile_change_teardown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- subprocess_launch_failure: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- plugin_drawing_model: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsoutputstreamwrapper_lazyinit: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- e10s_still_accepted_from_prompt: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- webrtc_max_video_send_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_other_write_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_tabqueue_prompt_enable_no: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fetch_is_mainthread: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_restoring_activity: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- canvas_webgl_accl_failure_id: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- video_eme_adobe_hidden_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- prconnect_blocking_time_link_change: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sandbox_capabilities_enabled_content: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_perftools_recording_features_used: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macce: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- gc_is_compartmental: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- a11y_update_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setdisksmartsize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_custom_homepage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- image_decode_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_fontinspector_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_delete_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_tls13_intolerance_reason_post: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_animationinspector_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- webrtc_max_audio_send_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_tags_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_check_extended_error_notify: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- fontlist_initotherfamilynames: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_displaystring_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_custom_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- http_page_dns_lookup_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_pinning_failures_by_ca: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ipv4_and_ipv6_address_connectivity: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_tabqueue_queuesize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_enumproperties_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- push_api_notification_received_but_did_not_notify: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- popup_notification_main_action_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- webrtc_stun_rate_limit_exceeded_by_type_given_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_stringify: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- enable_privilege_ever_called: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- http_page_open_to_first_from_cache: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_validation_http_request_canceled_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_cache_read_time_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tls_error_report_ui: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_security_category: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- update_check_no_update_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- gc_compact_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_decoder_bitrate_avg_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- browser_is_assist_default: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- memory_vsize_max_contiguous: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_ocsp_required: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- predictor_predict_time_to_inaction: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- reader_mode_parse_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_reload_addon_reload_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_service_manually_uninstalled_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- scroll_linked_effect_found: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- display_scaling_osx: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_sanitize_openwindows: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_migration_entry_point: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_content_collect_data_longest_op_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_blocked_for_stability: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- stumbler_time_between_received_locations_sec: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_switch_update_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_markvalid: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_macgreek: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- update_status_error_code_complete_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_inspector_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- browserprovider_xul_import_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- decoder_instantiated_machebrew: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- pdf_viewer_form: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dns_blacklist_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_room_session_withchat: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- dom_window_showmodaldialog_used: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- prclose_tcp_blocking_time_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_detach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_get_user_media_type: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_other_sync_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_sessiondata_failed_save: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- plugins_notification_plugin_count: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_sub_complete_load_net: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_sources_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- view_source_in_window_opened_boolean: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- telemetry_test_release_optin: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- localdomstorage_init_database_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- stumbler_upload_wifi_ap_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_setdatasize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_content_crash_not_submitted: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- csp_unsafe_inline_documents_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- osfile_worker_launch_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_saving_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_styleeditor_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cert_pinning_moz_test_results: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_tls12_intolerance_reason_post: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- update_has_pref_url_override_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- sessiondomstorage_key_size_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_h264_sps_max_num_ref_frames: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- slow_script_notice_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- http_auth_type_stats: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- early_gluestartup_hard_faults: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- slow_addon_warning_states: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_sharing_room_url: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_webapps_write_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_load_state_stressed_short: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_npn_join: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macdevanagari: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- a11y_instantiated_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- http_request_per_page: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_distribution_download_time_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- gc_reset: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_local_tracerdetach_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnectcontinue_blocking_time_shutdown: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webcrypto_resolved: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- dom_timers_fired_per_native_timeout: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_anim_any_frame_paint_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_ice_add_candidate_errors_given_success: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_hang_ui_dont_ask: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_sync11_migrations_completed: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webcrypto_alg: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- zoomed_view_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_release_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_kbread_per_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_osx_source_is_mls: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- cache_service_lock_wait_mainthread_nsdoomevent_run: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webconsole_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- loop_ice_success_rate: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_toolbox_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_memory_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_substring_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_other_sync_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- denied_translation_offers: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_pending_checking_over_quota_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nssetdisksmartsizecallback_notify: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- image_decode_on_draw_latency: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- plugin_shutdown_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_page_complete_load: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- sync_worker_operation: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- video_eme_adobe_install_failed_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_connection_entry_cache_hit_1: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_prototype_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_not_pref_update_staging_enabled_notify: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- update_has_pref_url_override_external: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- telemetry_archive_size_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prclose_tcp_blocking_time_offline: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_webide_project_editor_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- network_cache_hit_miss_stat_per_cache_size: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- database_successful_unlock: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- blocked_on_plugin_instance_destroy_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- urlclassifier_ps_fileload_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_free_purged_pages_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- auto_rejected_translation_offers: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- sessiondomstorage_value_size_bytes: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_tilt_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- media_rust_mp4parse_error_code: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- innerwindows_with_mutation_listeners: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fennec_tabqueue_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- moz_sqlite_cookies_open_readahead_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_ownpropertynames_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_canplaytype_h264_profile: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- network_disk_cache_shutdown_v2: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- forget_skippable_max: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_distribution_referrer_invalid: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- cache_service_lock_wait_mainthread_nscacheservice_setdiskcachemaxentrysize: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_ice_add_candidate_errors_given_success: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webcrypto_method: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_page_open_to_first_sent: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_error_recovery_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_thumbnails_bg_capture_done_reason_2: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- decoder_instantiated_macfarsi: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- fx_session_restore_all_files_corrupt: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_request_passthrough: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_connected_runtime_version: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- refresh_driver_tick: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_password_input_in_form: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- browser_set_default_error: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- media_decoder_backend_used: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_memory_breakdown_dominator_tree_count: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- service_worker_life_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- cert_ocsp_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_serialize_data_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_reading_list_count: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prclose_udp_blocking_time_offline: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_subitem_first_byte_latency_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_status_error_code_unknown_startup: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_jsprofiler_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- xmlhttprequest_async_or_sync: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- mixed_content_hsts: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_hud_app_memory_fullyloaded_v2: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- cache_service_lock_wait_mainthread_nscacheentrydescriptor_openinputstream: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- low_memory_events_physical: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- predictor_total_preconnects_created: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_video_encoder_bitrate_std_dev_per_call_kbps: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_javascript_error_displayed: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) + |-- loop_video_quality_inbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- memory_js_compartments_system: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- localdomstorage_setvalue_blocking_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fennec_startup_time_geckoready: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_tagged_bookmarks_perc: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- mixed_content_page_load: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_ice_final_connection_state: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_tab_anim_open_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_tab_anim_any_frame_interval_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_syn_reply_ratio: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_local_starttrace_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- mixed_content_object_subrequest: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_blocklist_num_sites: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- telemetry_test_release_optout: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- memory_js_gc_heap: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_country_fetch_caused_sync_init: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- loop_video_quality_outbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- video_eme_adobe_unsupported_reason: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_session_restore_number_of_eager_tabs_restored: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_manage_copied_password: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- cycle_collector_worker_visited_ref_counted: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_scratchpad_window_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_responsive_opened_per_user_flag: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- prconnect_fail_blocking_time_offline: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- update_pref_update_cancelations_external: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_remote_connection_result: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- ssl_succesful_cert_validation_time_mozillapkix: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_layoutview_time_active_seconds: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- pwmgr_manage_deleted: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- network_autodial: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- should_auto_detect_language: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- devtools_animationinspector_opened_count: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- webfont_fonttype: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fx_sanitize_loaded_flash: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- http_sub_open_to_first_received: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- ssl_fallback_limit_reached: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_manage_opened: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- browser_is_user_default_error: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- fxa_secure_credentials_save_with_mp_locked: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_tabs_open_peak_linear: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- prconnect_blocking_time_normal: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- tap_to_load_enabled: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- http_page_first_sent_to_last_received: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_threadgrips_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- weave_engine_apply_failures: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: integer (containsNull = true) + |-- pwmgr_manage_copied_username: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- devtools_debugger_rdp_remote_protocoldescription_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- loop_ice_success_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- blocked_on_pluginasyncsurrogate_waitforinit_ms: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: struct (containsNull = true) + | | | |-- values: array (nullable = true) + | | | | |-- element: integer (containsNull = true) + | | | |-- sum: long (nullable = true) + |-- httpconnmgr_total_speculative_conn: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- moz_sqlite_webapps_write_main_thread_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- places_database_filesize_mb: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- webrtc_video_quality_inbound_bandwidth_kbits: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- search_service_us_country_mismatched_timezone: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- safe_mode_usage: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- xul_foreground_reflow_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_migration_source_browser: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- disk_cache_revalidation_success: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- translated_pages: array (nullable = true) + | |-- element: integer (containsNull = true) + |-- loop_max_audio_send_track: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- urlclassifier_lookup_time: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_chunk_recvd: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- devtools_debugger_rdp_remote_reconfiguretab_ms: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- spdy_settings_ul_bw: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- geolocation_error: array (nullable = true) + | |-- element: boolean (containsNull = true) + |-- telemetry_archive_evicted_over_quota: array (nullable = true) + | |-- element: struct (containsNull = true) + | | |-- values: array (nullable = true) + | | | |-- element: integer (containsNull = true) + | | |-- sum: long (nullable = true) + |-- fx_startup_migration_existing_default_browser: array (nullable = true) + | |-- element: array (containsNull = true) + | | |-- element: integer (containsNull = true) + |-- devtools_webide_connected_runtime_processor: map (nullable = true) + | |-- key: string + | |-- value: array (valueContainsNull = true) + | | |-- element: array (containsNull = true) + | | | |-- element: integer (containsNull = true) +
+
This is a very a brief introduction to Spark and Telemetry in Python. You should have a look at the tutorial in Scala and the associated talk if you are interested to learn more about Spark.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +
Unable to parse whitelist (/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
The goal of this example is to plot the startup distribution for each OS. Let’s see how many parallel workers we have at our disposal:
+sc.defaultParallelism +
32 +
Let’s fetch 10% of Telemetry submissions for a given submission date…
+Dataset.from_source("telemetry").schema +
[u'submissionDate', + u'sourceName', + u'sourceVersion', + u'docType', + u'appName', + u'appUpdateChannel', + u'appVersion', + u'appBuildId'] +
pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(submissionDate="20161101") \ + .where(appUpdateChannel="nightly") \ + .records(sc, sample=0.1) +
… and extract only the attributes we need from the Telemetry submissions:
+subset = get_pings_properties(pings, ["clientId", + "environment/system/os/name", + "payload/simpleMeasurements/firstPaint"]) +
Let’s filter out submissions with an invalid startup time:
+subset = subset.filter(lambda p: p.get("payload/simpleMeasurements/firstPaint", -1) >= 0) +
To prevent pseudoreplication, let’s consider only a single submission for each client. As this step requires a distributed shuffle, it should always be run only after extracting the attributes of interest with get_pings_properties.
+subset = get_one_ping_per_client(subset) +
Caching is fundamental as it allows for an iterative, real-time development workflow:
+cached = subset.cache() +
How many pings are we looking at?
+cached.count() +
7132 +
Let’s group the startup timings by OS:
+grouped = cached.map(lambda p: (p["environment/system/os/name"], p["payload/simpleMeasurements/firstPaint"])).groupByKey().collectAsMap() +
And finally plot the data:
+frame = pd.DataFrame({x: np.log10(pd.Series(list(y))) for x, y in grouped.items()}) +plt.figure(figsize=(17, 7)) +frame.boxplot(return_type="axes") +plt.ylabel("log10(firstPaint)") +plt.show() +
You can also create interactive plots with plotly:
+fig = plt.figure(figsize=(18, 7)) +frame["Windows_NT"].plot(kind="hist", bins=50) +plt.title("startup distribution for Windows") +plt.ylabel("count") +plt.xlabel("log10(firstPaint)") +py.iplot_mpl(fig, strip_style=True) +
Let’s extract a histogram from the submissions:
+histograms = get_pings_properties(pings, "payload/histograms/GC_MARK_MS", with_processes=True) +
The API returns three distinct histograms for each submission: +- a histogram for the parent process (GC_MARK_MS_parent) +- an aggregated histogram for the child processes (GC_MARK_MS_children) +- the aggregate of the parent and child histograms (GC_MARK)
+Let’s aggregate the histogram over all submissions and plot it:
+def aggregate_arrays(xs, ys): + if xs is None: + return ys + + if ys is None: + return xs + + return xs + ys + +aggregate = histograms.map(lambda p: p["payload/histograms/GC_MARK_MS"]).reduce(aggregate_arrays) +aggregate.plot(kind="bar", figsize=(15, 7)) +
<matplotlib.axes._subplots.AxesSubplot at 0x7f1cea8b49d0> +
Keyed histograms follow a similar pattern. To extract a keyed histogram for which we know the key/label we are interested in:
+histograms = get_pings_properties(pings, "payload/keyedHistograms/SUBPROCESS_ABNORMAL_ABORT/plugin", with_processes=True) +
List all keys/labels for a keyed histogram:
+keys = pings.flatMap(lambda p: p["payload"].get("keyedHistograms", {}).get("MISBEHAVING_ADDONS_JANK_LEVEL", {}).keys()) +keys = keys.distinct().collect() +
keys[:5] +
[u'firefox@zenmate.com', + u'jid1-f3mYMbCpz2AZYl@jetpack', + u'jid0-SQnwtgW1b8BsMB5PLV5WScEDWOjw@jetpack', + u'light_plugin_ACF0E80077C511E59DED005056C00008@kaspersky.com', + u'netvideohunter@netvideohunter.com'] +
Retrieve the histograms for a set of labels:
+properties = map(lambda k: "payload/keyedHistograms/{}/{}".format("MISBEHAVING_ADDONS_JANK_LEVEL", k), keys[:5]) +
histograms = get_pings_properties(pings, properties, with_processes=True) +
This is a very a brief introduction to Spark and Telemetry in Python. You should have a look at the tutorial in Scala and the associated talk if you are interested to learn more about Spark.
+import ujson as json +import matplotlib.pyplot as plt +import pandas as pd +import numpy as np +import plotly.plotly as py + +from plotly.graph_objs import * +from moztelemetry import get_pings_properties, get_one_ping_per_client +from moztelemetry.dataset import Dataset + +%matplotlib inline +
Unable to parse whitelist (/home/hadoop/anaconda2/lib/python2.7/site-packages/moztelemetry/histogram-whitelists.json). Assuming all histograms are acceptable. +
The goal of this example is to plot the startup distribution for each OS. Let’s see how many parallel workers we have at our disposal:
+sc.defaultParallelism +
32 +
Let’s fetch 10% of Telemetry submissions for a given submission date…
+Dataset.from_source("telemetry").schema +
[u'submissionDate', + u'sourceName', + u'sourceVersion', + u'docType', + u'appName', + u'appUpdateChannel', + u'appVersion', + u'appBuildId'] +
pings = Dataset.from_source("telemetry") \ + .where(docType='main') \ + .where(submissionDate="20161101") \ + .where(appUpdateChannel="nightly") \ + .records(sc, sample=0.1) +
… and extract only the attributes we need from the Telemetry submissions:
+subset = get_pings_properties(pings, ["clientId", + "environment/system/os/name", + "payload/simpleMeasurements/firstPaint"]) +
Let’s filter out submissions with an invalid startup time:
+subset = subset.filter(lambda p: p.get("payload/simpleMeasurements/firstPaint", -1) >= 0) +
To prevent pseudoreplication, let’s consider only a single submission for each client. As this step requires a distributed shuffle, it should always be run only after extracting the attributes of interest with get_pings_properties.
+subset = get_one_ping_per_client(subset) +
Caching is fundamental as it allows for an iterative, real-time development workflow:
+cached = subset.cache() +
How many pings are we looking at?
+cached.count() +
7132 +
Let’s group the startup timings by OS:
+grouped = cached.map(lambda p: (p["environment/system/os/name"], p["payload/simpleMeasurements/firstPaint"])).groupByKey().collectAsMap() +
And finally plot the data:
+frame = pd.DataFrame({x: np.log10(pd.Series(list(y))) for x, y in grouped.items()}) +plt.figure(figsize=(17, 7)) +frame.boxplot(return_type="axes") +plt.ylabel("log10(firstPaint)") +plt.show() +
You can also create interactive plots with plotly:
+fig = plt.figure(figsize=(18, 7)) +frame["Windows_NT"].plot(kind="hist", bins=50) +plt.title("startup distribution for Windows") +plt.ylabel("count") +plt.xlabel("log10(firstPaint)") +py.iplot_mpl(fig, strip_style=True) +
Let’s extract a histogram from the submissions:
+histograms = get_pings_properties(pings, "payload/histograms/GC_MARK_MS", with_processes=True) +
The API returns three distinct histograms for each submission: +- a histogram for the parent process (GC_MARK_MS_parent) +- an aggregated histogram for the child processes (GC_MARK_MS_children) +- the aggregate of the parent and child histograms (GC_MARK)
+Let’s aggregate the histogram over all submissions and plot it:
+def aggregate_arrays(xs, ys): + if xs is None: + return ys + + if ys is None: + return xs + + return xs + ys + +aggregate = histograms.map(lambda p: p["payload/histograms/GC_MARK_MS"]).reduce(aggregate_arrays) +aggregate.plot(kind="bar", figsize=(15, 7)) +
<matplotlib.axes._subplots.AxesSubplot at 0x7f1cea8b49d0> +
Keyed histograms follow a similar pattern. To extract a keyed histogram for which we know the key/label we are interested in:
+histograms = get_pings_properties(pings, "payload/keyedHistograms/SUBPROCESS_ABNORMAL_ABORT/plugin", with_processes=True) +
List all keys/labels for a keyed histogram:
+keys = pings.flatMap(lambda p: p["payload"].get("keyedHistograms", {}).get("MISBEHAVING_ADDONS_JANK_LEVEL", {}).keys()) +keys = keys.distinct().collect() +
keys[:5] +
[u'firefox@zenmate.com', + u'jid1-f3mYMbCpz2AZYl@jetpack', + u'jid0-SQnwtgW1b8BsMB5PLV5WScEDWOjw@jetpack', + u'light_plugin_ACF0E80077C511E59DED005056C00008@kaspersky.com', + u'netvideohunter@netvideohunter.com'] +
Retrieve the histograms for a set of labels:
+properties = map(lambda k: "payload/keyedHistograms/{}/{}".format("MISBEHAVING_ADDONS_JANK_LEVEL", k), keys[:5]) +
histograms = get_pings_properties(pings, properties, with_processes=True) +