Граф коммитов

13 Коммитов

Автор SHA1 Сообщение Дата
Anna Scholtz 72a60c3bb3 Fix tests 2020-05-28 14:12:24 -07:00
Anna Scholtz ac6cb9ddaa Create tasks with validation 2020-05-28 14:12:24 -07:00
Anna Scholtz 1116be16b6 Pull in telemetry-airflow 2020-05-28 14:12:24 -07:00
Daniel Thorn a08fa06b69
update dependencies (#994)
also fix unregistered marker warnings
also fix new lint errors
2020-05-20 15:55:17 -07:00
dependabot-preview[bot] 38c1947bdc Bump pytest from 5.3.5 to 5.4.1
Bumps [pytest](https://github.com/pytest-dev/pytest) from 5.3.5 to 5.4.1.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/5.3.5...5.4.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-05-08 16:39:32 -04:00
dependabot-preview[bot] 9f1655972c Bump sqlparse from 0.3.0 to 0.3.1
Bumps [sqlparse](https://github.com/andialbrecht/sqlparse) from 0.3.0 to 0.3.1.
- [Release notes](https://github.com/andialbrecht/sqlparse/releases)
- [Changelog](https://github.com/andialbrecht/sqlparse/blob/master/CHANGELOG)
- [Commits](https://github.com/andialbrecht/sqlparse/compare/0.3.0...0.3.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-05-08 15:51:15 -04:00
Anna Scholtz 74cf1a4cbd Use streaming for converting ndjson to json 2020-03-31 13:28:45 -07:00
Anna Scholtz 438b87a6da Add tests for publishing json 2020-03-31 13:28:45 -07:00
Anna Scholtz 825b6ccf0e Update entrypoint and format ndjson to json 2020-03-31 13:28:45 -07:00
Anthony Miyaguchi 4f0080559a
Add histograms to fenix glam etl (#766)
* Add initial template for histogram aggregates

* Factor out common functions and get all distributions

* Add viable query for histogram aggregates

* Add more efficient aggregation

* Update header and update comment

* Add code to generate clients daily histograms

* Add queries for generated sql

* Return non-zero exit code when histograms not found

* Delete empty queries to reduce data scanned

* Add non-zero exit code for scalars if probes are not found

* Sort histograms for stable output

* Add view for histogram aggregates

* Add initial sql for histogram aggregates

* Format template scripts

* Add mostly reformatted sql for aggregates

* Update histogram aggregates before adding statements

* Fix up details for daily aggregation

* Add completed histograms template

* Add code to generate clients histogram aggregates

* Add init for clients histogram aggregates

* Remove sample_id from set of attributes

* Add sections to run generated sql

* Add generated sql

* Remove extra latest_version columns

* Fix many small issues during first draft of sql

* Fix clients histogram aggregates

* Add initial modification to probe counts

* Add histogram bucket counts

* Add option to generate histogram probe counts

* Update generated_fenix_sql for histograms

* Add generated sql

* Update run_fenix_sql

* Fix bucket counts

* Update source table for probe counts

* Add missing ping_type to histograms

* Add first,last,num buckets

* Update probe counts so it succeeds

* Add mozilla_schema_generator to dependencies

* Add metadata from probe-info for custom distributions

* Update probe counts with metadata for custom distributions

* Add UDF for generating functional buckets

* Add proper bucketing by including range_max of measures

* Format histogram udfs

* Add updated templates to skip

* Add new queries to dryrun ignore

* Add view to the publish ignore list

* Fix python linting

* Remove old comments from probe counts

* Do not count metadata from custom distributions twice

* Remove sum from histogram aggregates

* Add generated SQL

* Add sample_id to histograms earlier in pipeline

* Add generated SQL

* Add comments to functional bucketing for metrics
2020-03-18 13:53:28 -07:00
dependabot-preview[bot] 907a860b97
Bump pytest-black from 0.3.7 to 0.3.8 (#742)
Bumps [pytest-black](https://github.com/shopkeep/pytest-black) from 0.3.7 to 0.3.8.
- [Release notes](https://github.com/shopkeep/pytest-black/releases)
- [Commits](https://github.com/shopkeep/pytest-black/compare/0.3.7...0.3.8)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: Anthony Miyaguchi <acmiyaguchi@gmail.com>
2020-02-13 13:42:36 -08:00
dependabot-preview[bot] 4ed18beab1
Bump pytest from 5.3.2 to 5.3.5 (#712)
Bumps [pytest](https://github.com/pytest-dev/pytest) from 5.3.2 to 5.3.5.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/5.3.2...5.3.5)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-02-12 20:35:08 +01:00
Daniel Thorn f3e11b4255
Use pip-compile instead of constraints.txt (#736)
* Fix docstyle check

* Use pip-compile instead of constraints.txt
2020-02-12 20:26:24 +01:00