Error if a JSON validation file does not match the naming convention.
Clean up some unnecessary files and validation examples, including the validation files for org-mozilla-samples-glean since they exercise the same schema as the "glean" validation examples and
are identical.
Fix the launcher-process-failure parquet module schema.
* Make branch required
* Make branch non-required in Online News v2 Schemas
This fixes schema validation issues where we were receiving some
leftover log entries from the v1 study with v2 pings
* Add optional displayVersion field to environment schema (#145)
This information is redundant, but we're adding the field for now to support
https://bugzilla.mozilla.org/show_bug.cgi?id=1452745. This just adds a little
bit of extra validation on the server side to make sure that the field
is properly specified if it is present (obviously older versions will not have
the field).
* Update crash ping schema (#146)
Add `metadata.StartupCrash` field, specify required fields.
* Bug 1414352 - Add mobile metrics ping (#147)
* Add mobile metrics ping
* Add JSON Schema for mobile metrics ping
* Add createdDate and createdTimestamp for mobile events pings
* Add experiments for mobile events pings
* Request changes
* Add ipc_channel_error to crash ping schema
* Add processType to crash ping schema (#151)
* Move mobile-event 'os' to partitions
* Change core ping partitions
* Bug 1453376 - Add mercurial hgpush event ping (#153)
* Bug 1453376 - Add mercurial hgpush event ping
- Add a new eng-workflow namespace
- Add a ping for mercurial repository push events
* Fix review feedback
- Change reviewSystemUsed enum to a plain string with documentation
- Document pushDate integer timestamp and set a minimum value
- Punctuation, misc. docstring fixups
* Fix hgpush schema path
* Bug 1454739 - Add geo_subdivision to telemetry_new_profile_parquet
* Add optional displayVersion field to core ping (#156)
See https://bugzilla.mozilla.org/show_bug.cgi?id=1461772
* Update heartbeat ping validation
* Update compiled new-profile parquet schema
* Issue 141 - adding "accessibilityServices" property to core schema. (#142)
* Add downgrade ping schemas and validation test. (#160)
* Add the downgrade ping for bug 1463198.
* Add instructions for adding a new schema. (#163)
* Add instructions for adding a new schema.
* Add a .dockerignore file to skip the release dir. (#164)
* bug 1460595 - Add an event ping
Fixes#161
* Make shield-study Date header metadata optional
* Issue 166 - Fix new profile json / parquet schema mismatch
* Bug 1468498 - add landingSystem to hgpush events (#169)
* Bug 1468498 - add landingSystem to hgpush events
Add an optional 'landingSystem' field to hgpush events so that we can
track commits that came through Lando, our automated code landing system.
* Update the existing v1 schema instead of adding a v2 schema.
* Add frecency_update templates
* Add example for frecency_update
* Add compiled schema for frecency_update
* Add frecency_update parquetmr with proper spacing
* Use dash instead of underscore for frecency-update
* Order alphabetically and use client id include
* Add clientId macros to frecency-update
* Allow hgpush landingSystem to be null (#173)
Allow the hgpush ping schema's landingSystem field to be null.
* Add study_variation, frecency_scores and four new weights to frecency-update (#175)
* Add 4 more weights and a field for treatment/control
* Use version 4 for frecency-update
* Add frecency_scores field
* Bug 1473129 - Add appVersion field to core parquet. (#177)
* Add a couple more histogram validation examples. (#176)
* Add JSONSchema for initial edge-validator report
* Add parquet schema for edge-validator
* Add data and generated schema
* Move edge-validator into its own namespace
* Rewrite the error-report schema
* Automatically remove the test container when it exits
The created container is only used for a single test run and never reused. It however persists on the user's machine until its deleted. Let's just remove it immediately after running the tests.
* Bug 1473582 - add a diffstat field to the hgpush ping (#178)
Add a diffstat field to the hgpush ping
* Bug 1445921 - Add schema for optout ping (#180)
* Add DecodeErrorDetail to heka datasets
* Add field Timestamp to frecency-update (#182)
* Use int64 for frecency-update Timestamp field (#183)
* Bugzilla 1474598 - Verify schema naming convention
* Update environment (#186)
Add a few missing fields/types to the environment.
* Make error-report validation more granular
* add mozdata schemas
* Bug 1237610: add schemas for build system telemetry (#191)
This commit adds a schema for build system telemetry data
collection in both json-schema and Parquet formats. The
schema is added under the eng-workflow namespace. The
datareview+ for this new ping was received in bug 1291053.
* add JSON schema and validation tests.
* add attributes `interact` and `responseTime` in `promptResponse` object.
* Use CircleCI instead of TravisCI
This gives more granular control over the CI process
* Bug 1458734 - Add integrate step that compares against base commit
* Use `[0-9]` instead of `\d` in histograms
* Mobile experiments (#195)
* Don't try to compile hidden files
* Add experiments block to mobile parquet output
The experiments block was included in validation, but not
in the parquet output
Additionally, this change pulls together the mostly similar
focus-event and mobile-event parquet schemas
* Remove trailing comma
* Remove focus-event trailing comma
* Bug 1487578 - Add schemas for "coverage" ping (#198)
* Change coverage ping telemetryEnabled to integer type
* Bug 1453613 - Add schema for new installer ping
* Bug 1453613 - Fix profile_cleanup_requested type in parquet
* Bug 1453613 - Typo fix
* Add the firstUseDate and isStubProfile properties to the profile in the environment.
* Update the location of sampled data
* Bug 1502439: Add bug_1501329_affected to core ping (#207)
* Add shield study schemas for pioneer
* Add generated schemas
* Avoid duplication by using templating
* Add an early return for forked pull requests
* Document a command for integration reporting via edge-validator
* Fix invalid circleci config
* Update CircleCI to 2.1
* Add first revision of glean ping schema (#210)
* First import of glean ping schema
* Fix $schema values
* Reorganize to create three docTypes
* Only include one docType for now
* Remove generated files
* Add UUID metric type (#214)
* Remove now unneeded hasBinary flag from downgrade ping. (#205)
* Add basic docs for writing pioneer schemas. (#212)
Add basic docs for writing pioneer schemas.
* Include type for nested objects
* Generate new schemas for keyedHistograms
* [glean] Add place to store information about active experiments (#217)
* Add place to store active experiments
* Make experiments an object rather than an array
* Bug 1435827: Adding JSON and Parquet schemas (#215)
* Bug 1435827: Adding JSON and Parquet schemas
This adds the necessary validation and ingestion schemas for the new "Untrusted Modules" ping introduced in Bug 1435827, along with basic test payloads.
* Bug 1435827: Fixing comment syntax
Now using "description" and "examples" instead of "$comment" for JSON schemas.
* Bug 1435827: Fixing required / optional fields in the environment block
Some fields in the environment block were marked as required when the JSON schema does not require them. Fixed to synchronize the schemas.
* 1507788: Support dots in category part of dotted notation
* Add maxLength to category name
* Store events as we do on desktop
* 1510308: experiments should be in the ping_info section
* Bug 1505874 - Add parquet glean schema (#219)
* Add parquet glean schema
* Make glean histogram fields required
* Address review feedback
* Add missing semicolon
* Add missing type for end_time
* Update schema
* Move experiments into ping_info
* Fix typo
* Fix typos; add channel
* Make host and docversion optional
* Bug 1510704 - remove UINT_64 type annotations (#223)
* Bug 1510704 - remove UINT_64 type annotations
* Rename untrusted-modules to untrustedModules
* Allow -1 value for untrustedModules / moduleIndex field (#226) (#227)
* Allow -1 value for untrustedModules / moduleIndex field (#226)
The previous schema specified 0 as the minimum value for this field, but -1
should also pass validation.
A value of -1 indicates that the address does not fall in any known module.
See also:
* Documentation: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/untrusted-modules-ping.html#payload-combinedstacks
* The value of -1 is generated here: https://searchfox.org/mozilla-central/source/toolkit/components/telemetry/other/CombinedStacks.cpp#206
* And the underlying logic for this value comes from: and https://searchfox.org/mozilla-central/rev/e22c0d152060f4f8d4ca8904094f15f65a1b6f93/toolkit/components/telemetry/other/ProcessedStack.cpp#141
* Adding schema validation test for untrustedModules unknown module index
Now that the schema supports a module index of -1 (issue #226), this adds
a schema validation test for coverage of this case.
* Add parquet schema beautification
* Add glean events docType
* Add activity-stream impression-stats schema (#228)
Implements a schema matching the existing ping-centre payloads defined in
https://github.com/mozilla/activity-stream/blob/master/docs/v2-system-addon/data_events.md#top-story-pings
Should allow double-publishing of payloads via ping-centre and via telemetry.
* Bug 1503256 Add pull request template with reviewer instructions (#231)
Also adds caching of the data from S3, which brings runtime for the
integration test down to ~60 seconds.
* Pull down parquetfmt with cmake
* Bug 1515029: Fix untrusted modules ping environment block nesting
For the untrusted modules ping, data under environment/system is not coming
through in redash. This corrects the nesting of objects under environment/system
in order to fix this.
* Bug 1513590: Add Pocket Fire-TV Events Ping (#236)
* Fix empty integration reports by updating edge-validator image (#238)
* Update edge-validator image
* Bump integrate cache to v2
* Bug 1515103 - Add LowEndMachine field (#241)
* Bug 1515103 - Add LowEndMachine field
* Add rendered schemas
* Add org-mozilla-reference-browser location; add parquet for events (#234)
* Add schema for core ping version 10
Version was increased as part of
https://hg.mozilla.org/mozilla-central/rev/5cb4e28e1eb8
* Rename the `glean` namespace to `org-mozilla-samples-glean` (#242)
* Issue 240 - allow null scope for addons (#244)
* Add the "app_display_version" to the glean pings
* Bug 1513552 add modules ping parquet schema (#239)
* Bug 1513552 Part 1/2: Create include file for ping environment structure Parquet schema
As many pings include a common "environment" structure, I have moved the
environment structure from the untrusted modules schema into its own file as
/templates/include/telemetry/environment.1.parquetmr.txt
This is a Parquet companion to "environment.1.schema.json", and can be included
in ping Parquet schemas as `@TELEMETRY_ENVIRONMENT_1_TXT@`
This results in no schema changes to the untrusted modules ping.
* Bug 1513552 Part 2/2: Add modules ping Parquet schema
Adding this schema so the data can be made accessible on redash.
* Bug 1518490: Add pipeline schemas for 2 new timing fields in untrusted modules ping
This modifies the untrusted modules ping Parquet and JSON schemas to add 2 new
fields:
- "xulLoadDurationMS" (optional, number) at the payload root
- "loadDurationMS" (optional, number) per module object
And adds these fields to validation schema: untrustedModules.4.sample.pass.json
These fields are marked optional in order to be compatible with the previous
schema.
The code that generates these fields landed in
https://hg.mozilla.org/mozilla-central/rev/1b1fe5815dff
* Bug 1518490: Adding minimum value to untrusted modules timing fields
* Implement glean metrics ping
Adding schema and validation for &-browser and samples-glean
* Remove extra spaces from validation files
* Add easier instructions for MacOS (#252)
Addresses #251
* 1516527: Add labeled metrics to glean
* Remove uses of allOf
* Fix the pattern for labels
* Bug 1502921 - Add new 'intl' fields to environment data schema (#248)
* Bug 1502921 - Add new 'intl' fields to environment data schema
* Accept null for regionalPrefsLocales and systemLocales, add validation tests
* Generate schemas from template
* Add parquet content
* Bug 1524291 Add certSubject to modules ping schemas and corresponding validation tests (#255)
This adds the additional optional "certSubject" field to the modules ping
schemas, and updates the Windows validation test accordingly.
The "certSubject" field was already present in the payload sent from clients,
but it was not available in redash. This patch aims to correct that.
* 1525045: Simplify datetime to just have value, not unit
* Fix printed integration branch location
* Add a CHANGELOG
* Add engagement type to heartbeat schema (#225)
* Add engagement type to heartbeat schema
Fixes#224
* Fix typo star -> stars
* Remove extraneous newline (#256)
* Add some old changelog entries
* Update $schema value
* Mention updating the Glean changelog
* Don't allow additionalProperties in ping_info
* Add WebPageTest ping schema
* Remove Uuid from fire-tv metadata
* Fix indentation nit (#265)
* Fix Webpagetest parquet schema (#264)
* Bug 1517639: Add launcherProcessState to telemetry environment schema (#267)
* env change
* Update schemas to reflect env change
* Add first_run_date to ping_info
* clientId -> client_id
* Fix tests
* Add first_run_date to parquet schemas
* Loosen label regex
* Remove UUID from glean metadata (#271)
* Implement the new events API schema
* Fix#272: Move everything in v2 back to v1
* Rename some validation files
* Bug 1383111 - Add build step to validate if all generated schemas are committed (#273)
* Bug 1383111 - Add build step to validate if all generated schemas are committed
* Update .circleci/config.yml
Co-Authored-By: akkomar <akkomar@users.noreply.github.com>
* Catch also directly modified schemas
* Fix inconsistencies between template and builds
* Bug 1460433: Add schema for launcher-process-failure ping (#266)
* Updated frecency-update to match the v2 implementation (#277)
* Update frecency-update to match the v2.1.0 implementation
* Update frecency-update to match the v2.1.1 implementation
* NULL timespan checks (#280)
* Make environment an object (#281)
* Fix PR #280 - Skip the injection on failure instead of aborting the test (#282)
* Error if a JSON validation file does not match the naming convention (#284)
Error if a JSON validation file does not match the naming convention.
Clean up some unnecessary files and validation examples, including the validation files for org-mozilla-samples-glean since they exercise the same schema as the "glean" validation examples and
are identical.
Fix the launcher-process-failure parquet module schema.
* Bug 1533954 - Add Pioneer metadata to parquet schemas
* Add `sum` to glean histogram and timing_distribution schemas (#287)
* Move pioneer metadata template to its own directory
* Add better definitions for array types missing items (#290)
* Add items for gcItems (#292)
* Add trailing newlines to validation files where missing. (#295)
Used this command:
for f in $(find . -name "*.json"); do
tail -c1 $f | read -r _ || echo >> $f
done
* Split some fields out of gleans'`ping_info` to `client_info`
This additionally changes fields in `ping_info` and `client_info`
to use the glean templates, where available, as done with other
metrics.
* Add the "app_channel" to glean pings' `client_info`
* Bug 1526072 - Add fields for clobber builds and resource utilization to build telemetry (#294)
* Bug 1530740 - Add "loaded" to activity-stream.impression-stats
* Add the optional `android_sdk_version` to glean pings
* Fix docs to explain how to install parquetfmt from git (#299)
* Add explicit types to Glean pings (#302)
* Add explicit types to Glean pings
* Update CHANGELOG
* Make client_info required in glean pings
* Avoid incompatible array and object union type (#291)
* Avoid incompatible array and object union type
* Revert changes to payload.log
* Convert glean event schema from array to object
* Add Mozilla Code of Conduct file (#304)
Fixes#303.
_(Message COC002)_
* Add schema template for "prio" ping. (#298)
Docs: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/prio-ping.html
bug 1538744
* Add a full schema for payload.logs in main pings (#293)
Add schema for .payload.log
* Add Fenix schemas :( (#308)
* Bug 1534730 - Schema for the pre-account ecosystem ping (#286)
* Bug 1534730 - Schema for the pre-account ecosystem ping
* Bug 1534730 - Add minimal parquet schema for pre-account ping
* Require creation date & id
* Specify nested additional properties for scalars
* Use include
* Add schema and passing examples for BMO simple ping (#309)
* Add schema and passing examples for BMO simple ping (called bmobugs here)
* add copy of schema to the template per review feedback
* reformat from cmake
* Add `Headless` to Telemetry Environment. (#310)
Fixes bug 1536175
Might be nice to know if the ping's coming from a headless Firefox.
* Improve schema/template inconsistency check
By deleting schemas directory before generating them we can catch those written by hand without a template.
* 1529226: Make client_id optional
* Bug 1535717 - Add SPOCS fill telemetry for Activity Stream
* Add a new enum entry to activity stream spocs fill
* Add schema for the Fenix activation ping
* Add schemas for ping metadata in the GCP pipeline (#312)
* Rename spocs-fills to spoc-fills for Activity Stream
* Make submission_timestamp datetime formatted (#329)
* Make submission_timestamp datetime formatted
* Remove non-failing datetime test
In RapidJSON, formats are optionally enforced. The expected
failing test was then not failing. We're not as concerned with
validating these fields, so we're removing the test.
* Add schema for Firefox for FireTV (#330)
* Add mobile/activation ping
As described in the [Fennece Mobile Activation Ping design doc](https://docs.google.com/document/d/1D-0m3TRvIAOzGtkBNjaddyVr7baE7uEhiBUFyVQr2pg)
* Match required fields in the client implementation
As seen in https://phabricator.services.mozilla.com/D29668
* Integer tz for mobile/activation
This makes mobile/activation consistent with existing pings (core
and mobile events) that have an integer "tz" field given in minutes.
* Add a new item called `type` to `hdd` object (#326)
* Add a new item called `type` to `hdd` object
* Add two more test files for hdd.type
* Fix the problem that null value causes the test to fail
* Glean: allow dotted snake-case for labeled metric labels (#335)
* Add placeholder schemas for most known docTypes
* Add stub ping specific fields (#336)
* Add stub ping specific fields
and limit `installer_type` to valid values
* Consider speed and others as optional in parquet output (#314)
Fixes#313
* 1556684: Fix label length so it can support any metric name (#338)
* 1556684: Fix label length so it can support any metric name
* Add glean CHANGELOG
* Update payload section for the Prio ping (fx69.0a1) (#340)
* Add a reference prio ping generated by Firefox 69 nightly
* Update prio payload version to a string enum
* Add generated schema for Prio
* Allow `_` in ping schema names.
Fenix sends two new Glean pings (`history_sync` and `bookmarks_sync`),
but `_` wasn't allowed in schema names.
* Add schemas for Glean bookmarks and history sync pings.
* Add missing "string" type. (#343)
* Add missing string types for some enum fields. (#344)
* Update fenix sync pings per #344 (#345)
* Allow null values for environment/addons/theme/updateDay (#346)
* Increase number of phases in GC telemetry (#339)
See https://bugzilla.mozilla.org/show_bug.cgi?id=1555101
* Fix incompatible types in glean schemas (#348)
Fix incompatible types in glean schemas
cc @whd for deploy
* Remove non v4 main pings (#349)
Mark Reid
Mike Trinkala
Anthony Miyaguchi
Frank Bertsch