In the aftermath of the Oct 4 streaming outage we discovered the default
checkpoint location for StreamingJobBase was the same location for each concrete
descendent. This change adds a required JobName val and uses that as
part of the checkpoint directory.
As suggested by @fbertsch in
https://github.com/mozilla/telemetry-streaming/pull/176#discussion_r212410688
this PR simplifies EventsToAmplitude by unifying the classes for
Amplitude's two different endpoints.
Along the way, we also add a method to the HttpSink base class that allows
our Amplitude sink to specify how to split a single call into multiple
requests in the case that we get a PayloadTooLarge response.
Finally, we reduce the number of times that we duplicate explicit defaults for
boring parameters like timeouts by refactoring them to an HttpSink.Config
case class.
* Throw on schema validation errors in EventsToAmplitude
* Add devtools config to schema validation test
* Update focus event config to conform to newest schema schema
* Update meta-schema to conform with what's in the code
* Update meta-schema per review suggestions
* Fix focus events schema EventGroup labels
The "Meta - session split v2" events we backfilled were
rejected for recent days due to having identical insert_id
to the v1 events.
We change the logic so that future event name changes will
not get deduplicated, and we bump the session split event name again.
This time, I'll backfill and verify _before_ merging the PR in case
we encounter another hiccup.
As discussed in https://bugzilla.mozilla.org/show_bug.cgi?id=1482924
we want to backfill just session split events in order to pick up the
new active-ticks logic. We can only accomplish that in Amplitude
by sending events with a different name.
Once this is merged, I'll proceed with backfilling.
Telemetry-batch-view is significantly alterning the deploy.sh
script as that repo moves to CircleCI-based builds in
https://github.com/mozilla/telemetry-batch-view/pull/440
We will move this project to a similar style in a future change,
but for now we're pinning the deploy.sh url to the last tag
of t-b-v with Travis-based deploys.
Jeff Klukas
Sunah Suh
Arkadiusz Komarzewski
Sunah Suh
Ben Wu
Anthony Miyaguchi
Daniel Thorn