Граф коммитов

478 Коммитов

Автор SHA1 Сообщение Дата
Eduardo Filho 78218d3274 DENG-1025: gfx_telemetry r/w to GCS instead of S3 2023-08-10 10:39:03 -04:00
Alekhya dc0be49a84
Merge pull request #387 from mozilla/repoint_s3togcs_bhr
DENG 1022 - Update bhr_collection DAG to write to GCS instead of S3
2023-08-09 10:00:43 -04:00
Alekhya Kommasani 3adc5cee8c Fix flake8 issue 2023-08-09 08:55:42 -04:00
Alekhya Kommasani 0a39d766be Add generic comment for better readability 2023-08-09 08:39:52 -04:00
Alekhya Kommasani bea58beae8 repoint to prod bucket 2023-08-07 11:20:20 -04:00
Alekhya Kommasani d196e34b11 Fix black format issue 2023-07-28 10:57:33 -04:00
Alekhya Kommasani 23e8eb6160 Repoint end point to GCS from S3 2023-07-28 10:48:03 -04:00
kik-kik 8bc7c92191
Merge pull request #386 from mozilla/fix/DENG-1146/fixing-install-dependencies
fix(DENG-1146): fixing install dependencies ci step
2023-07-26 12:22:51 +02:00
kik-kik 6b8bf21dd7
bubmped up docker image used for ci steps 2023-07-11 12:17:25 +02:00
Will Kahn-Greene 781c017962 update symbols server url
bhr_collection is accessing the public symbols AWS S3 bucket directly rather than going through the Mozilla Symbols Server and getting redirected. We're planning to migrate to GCP and after we migrate, there won't be any AWS S3 bucket.

This fixes bhr_collection so that it works with our current setup as well as the setup we'll have after the GCP migration.

Details: https://bugzilla.mozilla.org/show_bug.cgi?id=1831952
2023-06-01 08:35:49 -04:00
Alexander 5296f5a3d0
Pin flake8 and protobuf to fix lint and py37 CI jobs (#379)
* Bump flake to 4.0.0
* Pin protobuf to 3.20.3
* Add explanations to pinned dependencies
Co-authored-by: Will Kahn-Greene <willkg@users.noreply.github.com>
Co-authored-by: Will Kahn-Greene <willkg@users.noreply.github.com>
2023-01-18 16:19:59 -05:00
kik-kik 069ed0f5e1
Merge pull request #378 from mozilla/bug/1809624/graphics_telemetry-dataproc_cluster-init-failure
bug(1809624): tweaked package requirements to use git for installing the moztelemetry package
2023-01-13 14:50:56 +01:00
kik-kik 423a62c6b4
tweaked package requirements to use git for installing the moztelemetry package 2023-01-12 14:47:09 +01:00
Will Kahn-Greene a8dbc07338 add README for symbolication scripts 2022-11-22 14:02:44 -05:00
Will Kahn-Greene 6a9e4e435d bug 1775444: drop top signatures from 250 to 200
We're having memory issues and it looks like the total number of crash
reports for the top signatures is increasing over time. This drops the
number of top signatures we look at which will drop the number of crash
reports which should reduce the memory used back to a workable number.
2022-06-22 10:40:39 -04:00
Will Kahn-Greene eb32d5a5d3 bug 1774415: reduce top signatures to 250
The top_signatures_correlations task is getting killed off for using too
much memory. This drops the top signatures we're looking at from 300 to
250 as a temporary attempt to get the job succeeding again.
2022-06-15 15:15:45 -04:00
Jeff Klukas 21b8003666
Update bhr_collection.py (#371) 2022-05-09 13:46:35 -04:00
Doug Thayer c70b5c8c6d
Allow empty symbols in breakpad files for BHR (#370)
The bhr_collection job is currently failing because we are now
seeing a FUNC line in a breakpad file which omits a symbol. I'm
not entirely sure why we're seeing this, but I don't see any
problem with just letting it through annotated as an empty symbol.
2022-05-09 13:38:16 -04:00
Will Kahn-Greene 20ab1aaa76
Revert to 300 signatures for correlations (#368) (#369)
We recently raised the number of signatures to look at to 1000, but this
caused the container to run out of memory when working on correlations
for the release channel.

This reverts back to the previous value of 300.
2022-03-24 09:17:00 -04:00
Marco Castelluccio c04a09ec9e
Generate correlations for the top 1000 signatures instead of top 300 (#367) 2022-03-22 09:26:46 -04:00
= af6d1ab2a7 Merge branch 'main' of https://github.com/mozilla/python_mozetl 2022-01-21 15:13:40 +01:00
= 81777e520e syncing changes from FirefoxGraphics/telemetry pull#14 2022-01-21 15:13:33 +01:00
kik-kik fc0fafedf4
Merge pull request #365 from mozilla/update-key
Update CircleCI SSH Key DSRE-413
2022-01-21 13:10:10 +01:00
Jason Thomas 7ccabe99c9
Update CircleCI SSH Key DSRE-413 2022-01-20 14:02:51 -05:00
kik-kik cd90abcf91
Merge pull request #364 from mozilla/bug/1737588/graphics_telemetry-none-not-subscriptable-error
Also catching TypeError in graphics_telemetry_dashboard.py
2022-01-11 18:12:26 +01:00
= 8ea63753ce because nothing ever is easy in life 2022-01-11 17:56:23 +01:00
= 37c9e5b997 sphinx and m2r extension changed to m2r2 due to no support 2022-01-11 17:50:13 +01:00
= 06cd6e1556 now also catching TypeError 2022-01-11 16:59:46 +01:00
Will Kahn-Greene 2172bb3dba
Merge pull request #362 from willkg/bug-1746874-missing-symbols
bug 1746874: fix case where version is None
2021-12-20 13:09:10 -05:00
Will Kahn-Greene 12bb1f6fbf
bug 1746874: fix case where version is None
Socorro switched stackwalkers last week and the new one emits a null
instead of an empty string for the module version. This fixes
is_old_firefox_module to work with null (which is None).
2021-12-20 11:46:21 -05:00
Ben Wu 946fbd50f4
Fix error on missing media decoders in graphics job (#361) 2021-08-18 14:36:21 -04:00
Doug Thayer 4a11103d5b
Output hang annotation information in the bhr job (#360) 2021-05-10 11:24:20 -04:00
Marco Castelluccio f72b778131
Generate correlations for the top 300 signatures instead of top 200 (#359) 2021-04-15 18:02:32 -04:00
Ben Wu 024582554a
Change graphics jobs to use mozdata for temp tables (#358) 2021-02-19 14:52:19 -05:00
Ben Wu 3e9cfbfabb
Mod 7 weekday arg in symbolication jobs (#357) 2020-12-21 15:21:49 -05:00
Ben Wu c183d901e8
Port bhr collection from databricks (#356) 2020-12-17 18:01:18 -05:00
Ben Wu 43ffb961db
Change default branch to main (#355) 2020-12-17 11:26:35 -05:00
Ben Wu e42d0a9d23
Update dependencies to fix ci (#354) 2020-12-16 11:51:45 -05:00
Ben Wu 5756722cd5
Port symbolication jobs from databricks (#353) 2020-12-15 12:23:57 -05:00
Ben Wu 96e9933ad4
Port graphics trends from databricks to dataproc (#351) 2020-12-11 18:16:26 -05:00
dependabot[bot] 3d1ca45f74 Bump pyspark from 2.3.1 to 2.3.2
Bumps [pyspark](https://github.com/apache/spark) from 2.3.1 to 2.3.2.
- [Release notes](https://github.com/apache/spark/releases)
- [Commits](https://github.com/apache/spark/compare/v2.3.1...v2.3.2)

Signed-off-by: dependabot[bot] <support@github.com>
2019-11-08 09:58:56 -05:00
Mark Reid 9c2715a67e Remove maudau tests and docs. 2019-11-08 09:58:38 -05:00
Mark Reid ee5f306d67 Remove the ETL code for the "engagement_ratio" job.
This is an ancient thing that was used by the dashboard at:
https://metrics.services.mozilla.com/firefox-dashboard/

which has been decommissioned per Bug 1571454.
2019-11-08 09:58:38 -05:00
William Lachance 8755e0c4e9
Bug 1589118 - Remove tab spinner job (#344) 2019-10-30 10:12:50 -04:00
William Lachance 58dce245ce
Fix undercounting of tab spinners (#343)
We would consider something "unaffected" for a short duration if we had no "short" tab spinner histogram OR "long" tab spinner histogramWe would consider something "unaffected" for a short duration if we had no "short" tab spinner histogram OR "long" tab spinner histograms. It should actually be "no short AND no long".
2019-10-22 09:47:14 -04:00
Victor Ng 1838654310 added tests and code to exercise `get_addons_per_client` in taar_lite_guidguid 2019-09-30 12:00:19 -07:00
Victor Ng 86eb9a9a44 The `get_addons_per_client` call had a bug where `active_addons` in the longitudinal table was structured as a dictionary of addon_guid -> addon_metadata, but the clients_daily table models the active_addons as a list of just addon_metadata.
This patch also unrolls the `extract_telemetry` function from using any nested functions to aid clarity.
2019-09-30 12:00:19 -07:00
Victor Ng 9efff104db updated taarlite guidranking tests to use mock clients_daily 2019-09-17 13:09:10 -07:00
Victor Ng 4d8b313eda Ported taar_lite ETL jobs to use clients_daily to prepare for GCP migration. 2019-09-17 13:09:10 -07:00
Victor Ng c377fc0fc5 Fixed a bad logger.error call when an exception occurs for taar_amodump 2019-09-17 13:09:10 -07:00