* added existing fxa tables to shredder config
* Apply suggestions from code review
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* removing some of the fxa tables from the config as suggested by srose
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Add skip-existing option to ./bqetl query initialize
* Handle initialization exceptions and refactor skip-existing check
* Refactoring of ./bqetl initialization
* Add --force option to ./bqetl initialize
* Update bigquery_etl/cli/query.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Update bigquery_etl/cli/query.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Update bigquery_etl/cli/query.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Update bigquery_etl/cli/query.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Update bigquery_etl/cli/query.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Handle Jinja whitespace control characters in `bqetl format`.
* Use default formatting for Jinja in `bigquery_usage_v2` ETL.
* Reformat `sql_generators/active_users/templates/mobile_checks.sql`.
* DAG docs - fix broken links and add tags to docs
* change tests
* remove empty line
* fix typo
* fix second test template
* add if case for private-bigquery-etl
* Define `event_monitoring_live_v1` views in `view.sql` files.
So they get automatically deployed by the `bqetl_artifact_deployment.publish_views` Airflow task.
* Support materialized views in view naming validation.
* Handle `IF NOT EXISTS` in view naming validation.
* Use regular expression to extract view ID in view naming validation.
This simplifies the logic and avoids a sqlparse bug where it doesn't recognize the `MATERIALIZED` keyword.
* Update other view regular expressions to allow for materialized views.
* Skip backfills for queries without metadata.yaml
* Support date_partition_offset
* Fixed exclude, modified exception
* Add test for offset backfill
* Apply suggestions from code review
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Formatting
---------
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* added firefox_ios_derived.clients_activation_v1 and corresponding view
* fixing a missing seperator in firefox_ios_derived.clients_activation_v1 checks
* adding firefox_ios_derived.clients_activation_v1 to shredder configuration
* removed is_suspicious_device_client as it should not be there, thanks bani for pointing this out
* fixed black formatting error inside shredder/config.py
* applied bqetl formatting
* minor styling tweak as suggested by bani in PR#4631
* deleting fenix_derived/firefox_android_clients_v2, v1 will remain the active model
* removed fenix_derived.firefox_android_clients_v2 from shredder config
* Add support for assigning Airflow tasks to task groups
* Generate separate Airflow tasks for glean_usage
* Remove Airflow dependencies from old glean_usage tasks
* Glam - fix legacy windows & release probes' sample count going fwd
* Glam FOG accounts for sampling when calculating total_sample for windows & release probes
* fog - fix client count and sample count
* Add channel filtering for fog
* Fix checks to filter on partitions
* Don't print "missing checks file" on success
Previously, the statement that checks.sql files
were missing was printed on any execution of the for
statement. ("else" clauses after "for"s execute after
completion of the "for" clause).
Instead, we want to print only when there are no files.
* Fill empty description
* Assign a friendly name if the table doesn't have one
* Update metadata tests
* Update bigquery_etl/metadata/parse_metadata.py
Co-authored-by: Alexander <anicholson@mozilla.com>
* update test again
---------
Co-authored-by: Alexander <anicholson@mozilla.com>
* Generate normal task dependencies from `depends_on` if the task is in the same DAG.
* Update `metadata.yaml` files to use `depends_on` rather than `upstream_dependencies`.
* DS-3054. Create functions to support running an initialization query for all sample_ids in parallel.
* DS-3054. Update _run_query function.
* DS-3054. Use _run_query and mapped values for initialization in parallel.
* DS-3054. Unify initialization to run in parallel and get sample_id range from metadata.
* DS-3054. Minimize formatting of query template and remove need to modify existing initialization queries. Validate if a query should use parallelized or regular update.
* DS-3054. Adding link to caveats.
* DS-3054. Update sample_id range for initialization.
* DS-3054. Use current implementation of run_query.
* DS-3054. Update using a parameter instead of initialization in metadata.
* DS-3054. DAG update with new parameter.
* Pass parameters before calling _run_query().
* Use --append_tablein favour of INSERT INTO.
* DS-3054 Separate parallel and non parallel init, plus some improvements.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
* Put assert UDFs in `mozfun` project.
* Tweak syntax in `assert.array_equals()` to avoid SQLGlot parsing error.
https://github.com/tobymao/sqlglot/issues/2348
* Fix SQL syntax error in `assert.struct_equals()` tests.
* Fix UDF dependency file path logic when deploying to stage.
* Change regular expressions in `parse_routine` module to allow quotes around routines' dataset and name.
* Respect sql_dir in dryrun skip
* Update bigquery_etl/dryrun.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Update bigquery_etl/dryrun.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Set sql_dir when using Schema.from_query_file()
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Fix publishing udfs that use backticks in identifiers
* Update bigquery_etl/routine/parse_routine.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* DENG-1314 Implement changes to bqetl and create default DAG.
* DENG-1314. Update Documentation.
* DENG-1314. Dummy query to enable generating DAG and run tests.
* DENG-1314. Update tests.
* Update bigquery_etl/cli/query.py
Raise exception when scheduling information is missing.
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* DENG-1314. Update tests.
* DS-3054. Update query creation to set bqetl_default as default value for --dag. Update tests.
* Default task and tests update.
* Default task and tests update.
* 3650 - Remove default DAG option, update DAG template comment & tests.
* 3650 - Condition for DAG warning.
* 3650 - Update docs.
* Clarification on sql/moz-fx-data-shared-prod/analysis/bqetl_default_task_v1/metadata.yaml
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update docs/cookbooks/creating_a_derived_dataset.md
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Small tweaks made to the cli cmds comments / help display for data checks
* added usage docs to data_checks reference docs
* Apply suggestions from code review provided by scholtzan
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* added render subcommand to the bqetl check command
* added a dry_run flag to bqqetl check run command
* added a test to make sure run command exists with status code 0
* added test for check render subcommand
* fixing linter checks
* attempting using an alternative way of testing the render command
* fixing render test by testing the _render() directly rather than the render cli wrapper
* removed dead test
* Apply suggestions from code review by ascholtz
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* fixed black and mypy errors
* fixed app_store_funnel_v1 check formatting
* reformatted tests checks
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* added history and bookmarks fields
* adding automated corrections
* some auto schema updates but perhaps not all
* Update schemas
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Bug 1741487 - Rename url2 and related fields in stable views
This removes the following unpopulated fields from Glean views: `metrics.url`, `metrics.text`, `metrics.jwe`, and `metrics.labeled_rate`. If any of these metrics exist in the source table under `2`-suffixed name, it is also aliased to its original name (`url2` to `url` and so on).
Suffixed fields are still preserved until view consumers migrate.
* Remove redundant comma from generated sql
* Ignore missing fields in views if any of them were removed
* added a todo comment
* Added additional context around why we are excluding some of the non-suffixed fields and why alising to remove suffix 2 from some fields
---------
Co-authored-by: Arkadiusz Komarzewski <akomarzewski@mozilla.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Add `synced_at` column to `stripe_subscriptions_changelog_v1`.
* Tweak `stripe_subscriptions_changelog_v1` tax rate and discount joins to only include those that existed when the change happened.
* Parse subscription metadata in `stripe_subscriptions_changelog_v1`.
* Add `stripe_external.invoice_line_item_v1` ETL.
* Add `stripe_subscriptions_revised_changelog_v1` ETL.
* Add `stripe_subscriptions_history_v2` ETL.
* CAccomodate dq checks in dag generation
* Modify the tests to include dq check
* Generate dags to include bigquery_dq_check
* rename destination to source for dq check
* Add DQ check to download attribution dag
* Update bigquery_etl/query_scheduling/templates/airflow_dag.j2
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update bigquery_etl/query_scheduling/generate_airflow_dags.py
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Set upstream check dependencies using upstream_dependencies
* Change bigquery_dq_check as per gcp.py utils
* remove sql_file_path in airflow jinja
* Fix download attribution dag
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* added a new table for the new nonprod fxa backend events and a fxa_all_events_nonprod view to simulate the process we will need to follow for prod
* added date filters to the nonprod_fxa_all_events view as requested by akkomar and updated the metadata
* added the new nonprod_fxa_server_events_v1 table to dry run skip due to permissions
* improved the comment about deleting a view as requested by akkomar
* tweaked date filtering as requested by srose
* pulled nonprod_fxa schema from DENG-1006-fxa-log-fields
* added schema.yaml for nonprod_fxa_server_events_v1
* deleted init.sql and added clustering config to metadata.yaml instead
* added AS as requested by srose
* fixed yaml lint errors
* added the ability to pass end_date param into Airflow task
* updated nonprod_fxa queries and schema for fxa_server_events_v1 as requested by srose, this query also pulls data for stout which now has end date
* regenerated bqetl_fxa_events DAG
* renamed fxa_log to fxa_server as agreed on with srose
* reverted merging of the stdout and server event etls due to incompatible schemas
* removed changes related to task level end_date
* removed date filter for stdout events
* undoing test changes
* added country to fxa_server_events_v1 schema
* tweaked selected ordering as requested by srose and updated comments and metadata.yaml
* Speed up schema update
* Speed up schema update
* Sort and update schemas in parallel
* Update sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_v6/metadata.yaml
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
---------
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* Create new Fenix attributable_clients table
Further updates to attributable clients
- Handle clients who were only _activated_ on that day
- Separate facts/dimensions
- Rename some things
- Add metadata about why a client is present
- Limit new_activations to just activated clients
- Rename client_count field
- Include submission_date in activation join
- Move to v2
* Add DAG
* Add schema file
* Move some joins to view; add initialization
1. Move attribution & activation joins to the view. This lets
us immediately access updates to those tables, rather than
re-materializing this table on changes there.
2. Add the capability to init from a query file. This uses
an `is_init` jinja function, which is only set to True
when run from `bqetl query initialize`.
* Use dict for default template vars
* Add default for addl_templates
* Reformat files
* Update view
* Regenerate DAG
* Keep metadata field in view
* glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch)
* glam: add description and eol to init
* glam: Partition clients_histogram_aggregates by sample_id (has been running like this since April 3 from a different branch)
* glam: add description and eol to init
* add init.sql to missing tbls
* Add schema.yaml
* increase ci output timeout to 30m
* remove init.sql to prevent ci from trying to derive schema from it and break
* Fix schema.yaml files
* Revert output timeout to default
* initial impl
* Updated based on PR feedback
* Moved check from query to separate command
* Expanded from --partition option to generic --parameter option
* Removed `query check` command (check moved to new command)
* Update bigquery_etl/cli/check.py
remove date param format check
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Removed 'parameter' parameter, everything is passed through ctx.args and then converted to a dict for Jinja rendering. There are no restrictions on ctx.args values.
* Merge error
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
When using `bqetl query schema update` to create a new `schema.yaml` file, BigQuery returns the column schema properties in a sensible order (`name`, `type`, `mode`, `fields`), but our `schema.yaml` output has been sorting those properties alphabetically which makes it much less readable.
Also, when using `bqetl query schema update` to update an existing `schema.yaml` file, this will now preserve whatever order the column schema properties were in.
* added support for --log-level to bqetl query command and updated print statements to be log statements
* now --log-level flag is a bqetl global flag
* fixing linter errors
* Update bigquery_etl/cli/__init__.py
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Update bigquery_etl/cli/__init__.py
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* fixed indentation of --log-level option
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Fix `bqetl stage` ID quoting.
Quoting the entire table ID breaks cases where an unaliased table name is used to qualify a column reference.
* Have `bqetl stage` preserve fully quoted references.
* Simplify regular expressions for fully quoted references.
* Compile all reference replacement regular expressions for performance.
* Save current SubPlat ETL views logic in versioned ETLs (DENG-973).
* Add `incremental` labels to the new tables.
* List all CJMS ETLs to dryrun-skip rather than using `glob`.
The `glob` approach doesn't currently work well with the CI staging process.
* DENG-970 Only Glean in Focus Android view.
* DENG-970 Only Glean in Focus Android view.
* DENG-970 Only Glean in Focus Android view.
* DENG-970 Only Glean in Focus Android view.
* DENG-970 Only Glean in Focus Android view.
* DENG-970 Only Glean in Focus Android view.
* DENG-970 CI fix
* DENG-970 CI failure fix. Related to issue 3889.
* Fix UDF dependencies deploy on stage
* DENG-970 Revert specific calling to dataset for UDF.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
Co-authored-by: Brad Ochocki <brad.ochocki@gmail.com>
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Bug 1823627 - Normalize the channel based on probeinfo data in UNIONized views
* Handle fenix channel normalization for app pings
* Parallelize stage schema deploys
* Fix schema field order
---------
Co-authored-by: Jan-Erik Rediger <jrediger@mozilla.com>
* added apple_ads_derived for copying over apple_ad data from the fivetran dataset, and apple_ads views now read from it
* added bqetl_fivetran_apple_ads.py DAG responsible for copying apple_ads data from the fivetran project over to moz-fx-data-shared-prod
* now dryrun skips apple_ads_derived instead of apple_ads as the query now accesses restricted dataset
* added schema files for apple_ads_derived datasets
* added descriptions to schema.yaml files for apple_ads_derived namespace
* added dataset_metadata for apple_ads_derived to include a link to the dbt transformations
* fixed apple_ads view definitions
* removed application label and referenced_tables section inside metadata.yaml for apple_ads as requsted by srose in PR#3847
* corrected source project for apple_ads views
* renamed apple_ads_derived to apple_ads_external
* added * to apple_ads_external namespace name to skip in the dryrun due to integration test deployment
* made tweaks to apple_ads and apple_ads_external datasets/namespaces as requested by whd
* updated apple_ads_external skip rule to the way it is meant to be defined, this will work once a fix is rolled out for dryrun
* fixed dag bqetl_fivetran_apple_ads description and updated the schedule to run once a day
* RS-722 Remove task_name from dag generation when it is not available.
* RS-722 Reformat files.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
* DENG-847 Remove the metadata validation for sql_generated files, because the validation needs to be implemented using a CODEOWNERS file in the main branch as well ans sql files. Currently files exist in the generated-sql branch.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
* DENG-775 Added session_id to JOIN between GA data and stub_attr.stdout. Also expanded date range on GA session data to [download_date - 2 days, download_date + 1 day]
* Updated query to handle missing GA download_session_id. It effectively applies V1 logic to the MISSING_GA_CLIENT dl_tokens.
* DENG-774 Add change control to active_users_aggregates and test.
* DENG-774 Add test coverage.
---------
Co-authored-by: Lucia Vargas <lvargas@mozilla.com>
* Update domain metadata dag.
* Remove from triage with tags
* Remove telemetry-alerts email
* Add date formatting for monthly partition id
* Add support for `table_partition_format` in dag generation
* Don't add partition format if there's already a destination table
* use the correct name
* Add partition templates for all time partitioning types
* lint fixes
* more docs
* update all dags to include `table_partition_template` parameter.
* don't set if we have a partition offset
* don't add the parameter for the default 'day' partitioning scheme
* Initial table definitions for dl_token processing. Includes update to sql pytest_plugin to account for tablenames with date suffixes.
* Removed cluster reference and shortened description
* Added sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql to dryrun skip
* Added time_on_site
* Moved country_names sample test data file.
* Update bigquery_etl/pytest_plugin/sql.py
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Update sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Updated based on PR feedback. Added LEFT JOIN to ensure sessions without pageviews are not dropped.
* Set has_ga_download_event = null if exception=GA_UNRESOLVABLE
* Standardized logic for time_on_site
* - Added test for multiple downloads for 1 session
- Added detailed description of table.
* Updated to use mode_last_retain_nulls instead of ANY_VALUE
* Set pageviews, unique_pageviews = 0 if null.
* Added boolean additional_download_occurred to indicate if another download occurred in the same session.
---------
Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Fix stage identifiers - only encapsulate project name in "`"
* Update bigquery_etl/cli/stage.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Update bigquery_etl/cli/stage.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Update bigquery_etl/cli/stage.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* Update bigquery_etl/cli/stage.py
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
---------
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* add bootstrap_percentile_ci UDF
* updates
Accidentally pushed an old version of the code.
* add qbinom.js
* remove trailing spaces
* Update sql/moz-fx-data-shared-prod/udf_js/bootstrap_percentile_ci/udf.sql
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* remove `numericSort` on `full_histogram`
The sorting step is computationally expensive on histograms with large numbers of elements, and is also unnecessary if we can ensure that the histograms themselves are sorted. Sorting a histogram is much cheaper.
* performance enhancements
A previous version of the code was running into memory issues with large histograms.
This commit aims to be more memory efficient.
- Instead of sorting arrays, it only ever sorts arrays, which are orders of magnitude smaller.
- It only expands a single histogram (the histogram of binomial samples) into an array. This array has a fixed-width (10k elements) for every input histogram, so it should have consistent memory performance across different metrics.
* debug typos
* remove unnecessary code
* improve readability
Change a some variable names in an attempt to make the code less confusing to read.
* handle histograms with zero counts
A few changes in this commit:
- Handle histograms with zero counts. I think this was implicitly handled in the previous version of the algorithm, but there wasn't a test for that case, so I didn't catch that the more space-efficient version of the algorithm wasn't handling it properly.
- Add a test case for histograms that have zero observations but are validly constructed.
- Minor formatting changes for readability.
* Formatting
* Fix tests
* make UDF work with normalized histograms
- Adds a new `scale` constant to simulate continuous values drawn from the discrete binomial distribution.
- Other changes to support working with non-integer count values while maintaining constant space/time complexity.
* fix failing tests
- use `assert.array_empty` instead of `assert.null` for empty array tests
- normalize all histogram values
- revert the histogram property `VALUES -> values` in the function definition so that the property is can be found.
* Reformat
---------
Co-authored-by: Anna Scholtz <anna@scholtzan.net>
* Indent `WHEN` and `ELSE` clauses one level more than `CASE`.
* Indent `THEN` clauses one level more than the corresponding `WHEN` clause.
* Have the content of `WHEN`, `THEN`, and `ELSE` clauses start on the same line as the clause keyword.
* Allow an alias, comma, or dot right after a `CASE` statement's `END`.
* Add and incrementally populate a table for google ads campaign cost metrics
* Register dag in dags.yaml
* make two strings match that apparently have to match
* Consistentify another thing in the dag
* Reformat a sql file
* Update sql/moz-fx-data-shared-prod/fenix_derived/google_ads_campaign_cost_breakdowns_v1/metadata.yaml
Add update dependency on upstream table
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
* Small adjustments apropos of review
* Attempt to fix verify-dags-up-to-date
* Evidently there are too many blank lines in the metadata file
* Stop dry running an access denied table
Also raise the line length limit on the linter, which otherwise prevented the needed change
* represent micros more accurately
* Update dags.yaml to include Frank as maintainer
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* rename ad_clicks
* Update sql/moz-fx-data-shared-prod/fenix_derived/google_ads_campaign_cost_breakdowns_v1/query.sql
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* Add conversions to columns of output table
* Group stats table by campaign id and date
* Add revenue generating ad clicks and disambiguate that number from marketing ad clicks
* SO IT TURNS OUT, campaigns with the same ID can have different names over time. This associates the appropriate names with each id so we can sum up metrics by campaign rather than by NAME of campaign.
* ./bqetl format /Users/chelseatroy/mozilla/bigquery-etl/sql/moz-fx-data-shared-prod/fenix_derived/google_ads_campaign_cost_breakdowns_v1/query.sql
* Document the fenix campaign identifier
* ./bqetl format
* ./bqetl dag generate bqetl_campaign_cost_breakdowns
* ./bqetl dag generate bqetl_org_mozilla_firefox_derived
Co-authored-by: Lucia <30448600+lucia-vargas-a@users.noreply.github.com>
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
* relay dataset and data tables are created
* added country_name columns to all tables
* reformat query files for circle ci error fixing
* Schema edited and dry run list updated
* removing status column in multiple places
* matching date_partition_offset setting with the table
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* keep the consistency of friendly_name
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* removing unnecessary comment
* subscriptions query updated
* renaming a CTE
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
* subscriptions_v1 schema editted
* add "relay-phones" to be included for filtering
Co-authored-by: Sean Rose <1994030+sean-rose@users.noreply.github.com>
Using a poke interval of 5 seconds means only 7 Fivetran sensors polling for an hour would exceed Fivetran's rate limit of 5,000 API requests per hour. Increasing the poke interval to 30 seconds will allow for up to 41 Fivetran sensors polling for an hour.
And use the `airflow-provider-fivetran` package's new feature to pass the return value from the Fivetran operator to the Fivetran sensor via XCom so the sensor doesn't miss syncs that finish before it can check.
Alexander
Sean Rose
kik-kik
Anna Scholtz
Leli
Jan-Erik Rediger
Daniel Thorn
Eduardo Filho
Lucia
Frank Bertsch
Linh Nguyen
akkomar
Mike Williams
Curtis Morales
Alekhya
Winnie Chan
betling
Glenda Leonard
skahmann3
Marlene Hirose
richard baffour
wil stuckey
Brad Ochocki
Chelsea Troy
Yeonjoo Yoo
kik-kik
haroldwoo
whd