mozilla
/
bigquery-etl
зеркало из https://github.com/mozilla/bigquery-etl.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

DENG-658 - Initial table definitions for dl_token processing. (#3644) 

* Initial table definitions for dl_token processing.  Includes update to sql pytest_plugin to account for tablenames with date suffixes.

* Removed cluster reference and shortened description

* Added sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql to dryrun skip

* Added time_on_site

* Moved country_names sample test data file.

* Update bigquery_etl/pytest_plugin/sql.py

Co-authored-by: Daniel Thorn <dthorn@mozilla.com>

* Update sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql

Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>

* Update sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql

Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>

* Updated based on PR feedback.  Added LEFT JOIN to ensure sessions without pageviews are not dropped.

* Set has_ga_download_event = null if exception=GA_UNRESOLVABLE

* Standardized logic for time_on_site

* - Added test for multiple downloads for 1 session
- Added detailed description of table.

* Updated to use mode_last_retain_nulls instead of ANY_VALUE

* Set pageviews, unique_pageviews = 0 if null.

* Added boolean additional_download_occurred to indicate if another download occurred in the same session.

---------

Co-authored-by: Daniel Thorn <dthorn@mozilla.com>
Co-authored-by: Frank Bertsch <frank.bertsch@gmail.com>
This commit is contained in:
Glenda Leonard 2023-03-23 16:45:58 -04:00 коммит произвёл GitHub
Родитель 100df65bb1
Коммит b13f45bc63
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
13 изменённых файлов: 846 добавлений и 0 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

1
bigquery_etl/dryrun.py
Просмотреть файл

@ -184,6 +184,7 @@ SKIP = {
"sql/moz-fx-data-shared-prod/**/client_deduplication*/*.sql",
recursive=True,
),
"sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql",
# Materialized views
"sql/moz-fx-data-shared-prod/telemetry_derived/experiment_search_events_live_v1/init.sql", # noqa E501
"sql/moz-fx-data-shared-prod/telemetry_derived/experiment_events_live_v1/init.sql", # noqa E501

16
bigquery_etl/pytest_plugin/sql.py
Просмотреть файл

@ -1,5 +1,6 @@
"""PyTest plugin for running sql tests."""
import datetime
import json
import os.path
@ -123,6 +124,21 @@ class SqlTest(pytest.Item, pytest.File):
table_name.replace(".", "_").replace("-", "_"),
)
query = query.replace(original, table_name)
# second check for tablename tweaks.
# if the tablename ends with a date then need to replace that date with '*' for the
# query text substitution to work.
# e.g. see moz-fx-data-marketing-prod.65789850.ga_sessions_20230214
# A query using that table uses moz-fx-data-marketing-prod.65789850.ga_sessions_*
# with the date appended to allow for daily processing.
try:
datetime.datetime.strptime(table_name[-8:], "%Y%m%d")
except ValueError:
pass
else:
generic_table_name = table_name[:-8] + "*"
generic_original = original[:-8] + "*"
query = query.replace(generic_original, generic_table_name)
tables[table_name] = Table(table_name, source_format, source_path)
print(f"Initialized {table_name}")
elif extension == "sql":

19
dags/bqetl_google_analytics_derived.py
Просмотреть файл

@ -104,6 +104,21 @@ with DAG(
depends_on_past=False,
)
ga_derived__downloads_with_attribution__v1 = bigquery_etl_query(
task_id="ga_derived__downloads_with_attribution__v1",
destination_table="downloads_with_attribution_v1",
dataset_id="ga_derived",
project_id="moz-fx-data-marketing-prod",
owner="gleonard@mozilla.com",
email=[
"ascholtz@mozilla.com",
"gleonard@mozilla.com",
"telemetry-alerts@mozilla.com",
],
date_partition_parameter="submission_date",
depends_on_past=False,
)
ga_derived__firefox_whatsnew_summary__v1 = bigquery_etl_query(
task_id="ga_derived__firefox_whatsnew_summary__v1",
destination_table="firefox_whatsnew_summary_v1",
@ -212,6 +227,10 @@ with DAG(
ga_derived__blogs_sessions__v1.set_upstream(ga_derived__blogs_empty_check__v1)
ga_derived__downloads_with_attribution__v1.set_upstream(
ga_derived__www_site_empty_check__v1
)
ga_derived__firefox_whatsnew_summary__v1.set_upstream(ga_derived__www_site_hits__v1)
ga_derived__www_site_downloads__v1.set_upstream(ga_derived__www_site_hits__v1)

45
sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/metadata.yaml Normal file
Просмотреть файл

@ -0,0 +1,45 @@
friendly_name: Downloads With Attribution
description: |-
A daily joining of Firefix download tokens with Google Analytics attribution data.
1. Each row in this table should be a unique dltoken.
2. A single GA session can have multiple dltokens (a session resulted in multiple downloads).
3. If the stub_visit_id is '', '(not set)', or 'something', the exception column will be:
* DOWNLOAD_SESSION_ID_EMPTY
* DOWNLOAD_SESSION_ID_VALUE_NOT_SET or
* DOWNLOAD_SESSION_ID_VALUE_SOMETHING
* All GA derived values will be null
4. If the stub_visit_id is valid but not found in that day's GA sessions:
* exception column will be MISSING_GA_CLIENT
* All GA derived values will be null
5. A GA session which does not have a dltoken with a corresponding stub_visit_id/clientId match is excluded from this dataset.
6. The stub_visit_id has a clientId match but that clientId has multiple GA sessions that day:
* exception value will be GA_UNRESOLVABLE
* All GA derived values will be null
7. A single dltoken has multiple records
* count_dltoken_duplicates will be >= 1.
* A value of 0 means there are no duplicates (expected state).
* A value of 1 means there was 1 other row with the dltoken value, etc.
owners:
- gleonard@mozilla.com
labels:
incremental: true
schedule: daily
scheduling:
dag_name: bqetl_google_analytics_derived
referenced_tables:
- ['moz-fx-data-marketing-prod', 'ga_derived', 'www_site_empty_check_v1']
bigquery:
time_partitioning:
type: day
field: download_date
require_partition_filter: false
expiration_days: null
references: {}

345
sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/query.sql Normal file
Просмотреть файл

@ -0,0 +1,345 @@
-- Query for ga_derived.downloads_with_attribution_v1
CREATE TEMP FUNCTION normalize_browser(browser STRING) AS (
CASE
WHEN `moz-fx-data-shared-prod.udf.ga_is_mozilla_browser`(browser)
THEN 'Firefox'
WHEN browser IN ('Internet Explorer')
THEN 'MSIE'
WHEN browser IN ('Edge')
THEN 'Edge'
WHEN browser IN ('Chrome')
THEN 'Chrome'
WHEN browser IN ('Safari')
THEN 'Safari'
WHEN browser IN ('(not set)')
THEN NULL
WHEN browser IS NULL
THEN NULL
ELSE 'Other'
END
);
WITH all_hits AS (
SELECT
clientId AS client_id,
visitId AS visit_id,
hit.appInfo.landingScreenName AS landing_page,
hit.page.pagePath AS pagepath,
CASE
WHEN (
hit.type = 'EVENT'
AND hit.eventInfo.eventAction = 'Firefox Download'
AND hit.eventInfo.eventCategory IS NOT NULL
AND hit.eventInfo.eventLabel LIKE 'Firefox for Desktop%'
)
THEN TRUE
ELSE FALSE
END AS has_ga_download_event,
hit.type AS hit_type
FROM
`moz-fx-data-marketing-prod.65789850.ga_sessions_*` AS ga
CROSS JOIN
UNNEST(hits) AS hit
WHERE
_TABLE_SUFFIX = FORMAT_DATE('%Y%m%d', @submission_date)
),
pageviews AS (
SELECT
client_id AS client_id,
visit_id AS visit_id,
COUNT(pagePath) AS pageviews,
COUNT(DISTINCT pagePath) AS unique_pageviews,
FROM
all_hits
WHERE
hit_type = 'PAGE'
GROUP BY
client_id,
visit_id
),
dl_events AS (
SELECT
client_id AS client_id,
visit_id AS visit_id,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(landing_page)) AS landing_page,
LOGICAL_OR(has_ga_download_event) AS has_ga_download_event
FROM
all_hits
GROUP BY
client_id,
visit_id
),
ga_sessions AS (
SELECT
clientId AS client_id,
visitId AS visit_id,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(geoNetwork.country)) AS country,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(trafficSource.adContent)) AS ad_content,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(trafficSource.campaign)) AS campaign,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(trafficSource.medium)) AS medium,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(trafficSource.source)) AS source,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(device.deviceCategory)) AS device_category,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(device.operatingSystem)) AS os,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(device.browser)) AS browser,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(device.browserVersion)) AS browser_version,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(device.language)) AS `language`,
IFNULL(mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(totals.timeOnSite)), 0) AS time_on_site
FROM
`moz-fx-data-marketing-prod.65789850.ga_sessions_*` AS ga
WHERE
_TABLE_SUFFIX = FORMAT_DATE('%Y%m%d', @submission_date)
GROUP BY
client_id,
visit_id
),
ga_sessions_with_hits AS (
SELECT
* EXCEPT (pageviews, unique_pageviews),
IFNULL(pageviews, 0) AS pageviews,
IFNULL(unique_pageviews, 0) AS unique_pageviews,
FROM
ga_sessions ga
LEFT JOIN
pageviews p
USING
(client_id, visit_id)
JOIN
dl_events
USING
(client_id, visit_id)
),
stub_dl AS (
SELECT
s.jsonPayload.fields.visit_id AS stub_visit_id,
s.jsonPayload.fields.dltoken AS dltoken,
(COUNT(*) - 1) AS count_dltoken_duplicates,
DATE(s.timestamp) AS download_date
FROM
`moz-fx-stubattribut-prod-32a5.stubattribution_prod.stdout` s
WHERE
DATE(s.timestamp) = @submission_date
AND s.jsonPayload.fields.log_type = 'download_started'
GROUP BY
stub_visit_id,
dltoken,
DATE(s.timestamp)
),
stub_other_dl AS (
SELECT
s.jsonPayload.fields.visit_id AS stub_visit_id,
CASE
WHEN (COUNT(*) > 1)
THEN TRUE
ELSE FALSE
END AS additional_download_occurred
FROM
`moz-fx-stubattribut-prod-32a5.stubattribution_prod.stdout` s
WHERE
DATE(s.timestamp) = @submission_date
AND s.jsonPayload.fields.log_type = 'download_started'
GROUP BY
stub_visit_id
),
stub AS (
SELECT
stub_visit_id,
dltoken,
count_dltoken_duplicates,
additional_download_occurred,
download_date
FROM
stub_dl
JOIN
stub_other_dl
USING
(stub_visit_id)
),
-- This will drop all the ga_sessions w/o a DLtoken but keep DLtoken without a GA session.
-- This will also result in multiple rows as the ga.client_id is not unique for the day
-- since this visit_id is missing from the stub.
downloads_and_ga_session AS (
SELECT
gs.client_id AS client_id,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(country)) AS country,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(ad_content)) AS ad_content,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(campaign)) AS campaign,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(medium)) AS medium,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(source)) AS source,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(device_category)) AS device_category,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(os)) AS os,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(browser)) AS browser,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(browser_version)) AS browser_version,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(`language`)) AS `language`,
s.stub_visit_id AS stub_visit_id,
s.dltoken AS dltoken,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(landing_page)) AS landing_page,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(pageviews)) AS pageviews,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(unique_pageviews)) AS unique_pageviews,
LOGICAL_OR(
has_ga_download_event
) AS has_ga_download_event, -- this will be ignored if nrows >1
mozfun.stats.mode_last_retain_nulls(
ARRAY_AGG(count_dltoken_duplicates)
) AS count_dltoken_duplicates,
LOGICAL_OR(additional_download_occurred) AS additional_download_occurred,
COUNT(*) AS nrows,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(s.download_date)) AS download_date,
mozfun.stats.mode_last_retain_nulls(ARRAY_AGG(time_on_site)) AS time_on_site
FROM
ga_sessions_with_hits gs
RIGHT JOIN
stub s
ON
gs.client_id = s.stub_visit_id
GROUP BY
gs.client_id,
dltoken,
stub_visit_id
)
SELECT
dltoken,
download_date,
CASE
WHEN nrows > 1
THEN NULL
ELSE time_on_site
END
time_on_site,
CASE
WHEN nrows > 1
THEN NULL
ELSE ad_content
END
ad_content,
CASE
WHEN nrows > 1
THEN NULL
ELSE campaign
END
campaign,
CASE
WHEN nrows > 1
THEN NULL
ELSE medium
END
medium,
CASE
WHEN nrows > 1
THEN NULL
ELSE source
END
source,
CASE
WHEN nrows > 1
THEN NULL
ELSE landing_page
END
landing_page,
CASE
WHEN nrows > 1
THEN NULL
ELSE country
END
country,
CASE
WHEN nrows > 1
THEN NULL
ELSE cn.code
END
normalized_country_code,
CASE
WHEN nrows > 1
THEN NULL
ELSE device_category
END
device_category,
CASE
WHEN nrows > 1
THEN NULL
ELSE os
END
os,
CASE
WHEN nrows > 1
THEN NULL
WHEN os IS NULL
THEN NULL
WHEN os LIKE 'Macintosh%'
THEN 'Mac' -- these values are coming from GA.
ELSE mozfun.norm.os(os)
END
normalized_os,
CASE
WHEN nrows > 1
THEN NULL
ELSE browser
END
browser,
CASE
WHEN nrows > 1
THEN NULL
ELSE normalize_browser(browser)
END
normalized_browser,
CASE
WHEN nrows > 1
THEN NULL
ELSE browser_version
END
browser_version,
-- only setting browser major version since that is the only value used in
-- moz-fx-data-shared-prod.firefox_installer.install
CASE
WHEN nrows > 1
THEN NULL
ELSE CAST(mozfun.norm.extract_version(browser_version, 'major') AS INTEGER)
END
browser_major_version,
CASE
WHEN nrows > 1
THEN NULL
ELSE `language`
END
`language`,
CASE
WHEN nrows > 1
THEN NULL
ELSE pageviews
END
pageviews,
CASE
WHEN nrows > 1
THEN NULL
ELSE unique_pageviews
END
unique_pageviews,
CASE
WHEN nrows > 1
THEN NULL
ELSE has_ga_download_event
END
has_ga_download_event,
count_dltoken_duplicates,
additional_download_occurred,
CASE
WHEN stub_visit_id = ''
THEN 'DOWNLOAD_SESSION_ID_EMPTY'
WHEN stub_visit_id = '(not set)'
THEN 'DOWNLOAD_SESSION_ID_VALUE_NOT_SET'
WHEN stub_visit_id = 'something'
THEN 'DOWNLOAD_SESSION_ID_VALUE_SOMETHING'
WHEN client_id IS NULL
THEN 'MISSING_GA_CLIENT'
WHEN dltoken IS NULL
THEN 'MISSING_DL_TOKEN'
WHEN nrows > 1
THEN 'GA_UNRESOLVABLE'
ELSE NULL
END
`exception`
FROM
downloads_and_ga_session
LEFT JOIN
`moz-fx-data-shared-prod.static.country_names_v1` AS cn
ON
cn.name = country

158
sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/schema.yaml Normal file
Просмотреть файл

@ -0,0 +1,158 @@
fields:
- description: "Unique token generated for the Firefox download. \nValue should\
\ never be NULL.\n"
mode: NULLABLE
name: dltoken
type: STRING
- description: "Time visitor on site"
mode: NULLABLE
name: time_on_site
type: NUMERIC
- description: "A description of the advertising or promotional copy. \nValue will\
\ be null if the Firefox download has no corresponding Google Analytics session.\n"
mode: NULLABLE
name: ad_content
type: STRING
- description: "The name of a marketing campaign. \nValue will be null if the Firefox\
\ download has no corresponding Google Analytics session.\n"
mode: NULLABLE
name: campaign
type: STRING
- description: "The type of referrals (e.g. referral, organic, email). \nValue will\
\ be null if the Firefox download has no corresponding Google Analytics session.\n"
mode: NULLABLE
name: medium
type: STRING
- description: "The source of referrals (e.g. bing). \nValue will be null if the\
\ Firefox download has no corresponding Google Analytics session.\n"
mode: NULLABLE
name: source
type: STRING
- description: "Landing page for the Google Analytics session corresponding to the\
\ Firefox download. \nValue will be null if the Firefox download has no corresponding\
\ Google Analytics session.\n"
mode: NULLABLE
name: landing_page
type: STRING
- description: "Full country name. \nValue will be null if the Firefox download has\
\ no corresponding Google Analytics session.\n"
mode: NULLABLE
name: country
type: STRING
- description: "2-char country code. \nValue will be null if the Firefox download\
\ has no corresponding Google Analytics session.\n"
mode: NULLABLE
name: normalized_country_code
type: STRING
- description: "e.g. desktop, tablet, mobile. \nValue will be null if the Firefox\
\ download has no corresponding Google Analytics session.\n"
mode: NULLABLE
name: device_category
type: STRING
- description: "OS used during download. \nValue will be null if the Firefox download\
\ has no corresponding Google Analytics session.\n"
mode: NULLABLE
name: os
type: STRING
- description: "OS values grouped into (Windows, Android, Mac, Linux, iOS and Other).\
\ \nValue will be null if the Firefox download has no corresponding Google Analytics\
\ session.\n"
mode: NULLABLE
name: normalized_os
type: STRING
- description: "Browser used to download Firefox (e.g. Chrome, Edge). \nValue will\
\ be null if the Firefox download has no corresponding Google Analytics session.\n"
mode: NULLABLE
name: browser
type: STRING
- description: "Browser values grouped into (Firefox, MSIE, Edge, Chrome, Safari,\
\ Other, NULL). \nValue will be null if the Firefox download has no corresponding\
\ Google Analytics session.\n"
mode: NULLABLE
name: normalized_browser
type: STRING
- description: "Complete version label of browser used to download Firefox\n(note\
\ this is NOT the version of Firefox which was downloaded). \nValue will be null\
\ if the Firefox download has no corresponding Google Analytics session.\n"
mode: NULLABLE
name: browser_version
type: STRING
- description: "Major version of the browser used to download Firefox. This is numeric\
\ to facilitate querying. \ne.g where browser='Edge' and browser_major_version\
\ >= 11. This is NOT the version of Firefox which was downloaded.\nValue will\
\ be null if the Firefox download has no corresponding Google Analytics session.\n"
mode: NULLABLE
name: browser_major_version
type: NUMERIC
- description: 'Device language, may be 2 char language code (en) or 4 char locale
(en-gb).
Value will be null if the Firefox download has no corresponding Google Analytics
session.
'
mode: NULLABLE
name: language
type: STRING
- description: "Total count of pages hit for the Google Analytics session associated\
\ with the Firefox download. \nValue will be null if the Firefox download has\
\ no corresponding Google Analytics session.\n"
mode: NULLABLE
name: pageviews
type: INTEGER
- description: "Total count of unique pages hit for the Google Analytics session associated\
\ with the Firefox download. \nValue will be null if the Firefox download has\
\ no corresponding Google Analytics session.\n \n"
mode: NULLABLE
name: unique_pageviews
type: INTEGER
- description: 'If has_ga_download_event = True then a corresponding Google Analytics
''Download Event'' has been found.
If has_ga_download_event = False, while there may be a corresponding Google Analytics
session for the Firefox download, no ''Download Event'' was captured for that
session.
'
mode: NULLABLE
name: has_ga_download_event
type: BOOLEAN
- description: "This field is intended for diagnostics.\nThe dltoken should be unique\
\ however occasionally duplicates may be present in the raw data. \nIt has been\
\ confirmed that duplicate dltokens are true duplicates in that all the other\
\ column values for the record \nare also duplicated. This table does not contain\
\ duplicate dltokens however count_dltoken_duplicates will indicate \nif duplicate\
\ dltokens existed in the raw data.a\ncount_dltoken_duplicates = 0 indicates that\
\ there was no duplicate dltoken in the raw data.\ncount_dltoken_duplicates =\
\ 1 indicates that there was 1 other row with the same dltoken.\n"
mode: NULLABLE
name: count_dltoken_duplicates
type: INTEGER
- description: "This column represents if there was at least one other download completed during the same\
\ session as this download. "
mode: NULLABLE
name: additional_download_occurred
type: BOOLEAN
- description: "Indicates an exception state when determining the GA\
\ session associated with the Firefox download\naction. Values are limited to:\n\
NULL: unique GA session matches the Firefox download. This is\
\ the normal state.\nGA_UNRESOLVABLE: This indicates that more than one Google\
\ Analytics session matches the Firefox download. \nThe occurrences of this value\
\ will decrease in the future. \nDOWNLOAD_SESSION_ID_EMPTY: The raw download\
\ record did not contain any GA session identifier.\nDOWNLOAD_SESSION_ID_VALUE_NOT_SET:\
\ The raw download record contained GA session identifier with\
\ value\n\u2018(not set)\u2019 and could not be associated with a GA\
\ session.\nDOWNLOAD_SESSION_ID_VALUE_SOMETHING: The raw download record contained\
\ GA session identifier with value\n\u2018something\u2019 and could\
\ not be associated with a GA session.\nMISSING_GA_CLIENT: The\
\ raw download record contained a valid GA session identifier however\
\ there was \nno corresponding GA session found. This value is\
\ used to identify why the GA sourced \ncolumns are NULL.\n"
mode: NULLABLE
name: exception
type: STRING
- description: The date of download, sourced from download logs.
mode: NULLABLE
name: download_date
type: DATE

158
tests/sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/moz-fx-data-marketing-prod.65789850.ga_sessions_20230214.schema.json Normal file
Просмотреть файл

@ -0,0 +1,158 @@
[
{
"mode": "NULLABLE",
"name": "visitId",
"type": "INTEGER"
},
{
"mode": "NULLABLE",
"name": "date",
"type": "STRING"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "timeOnSite",
"type": "INTEGER"
}
],
"mode": "NULLABLE",
"name": "totals",
"type": "RECORD"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "campaign",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "source",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "medium",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "adContent",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "trafficSource",
"type": "RECORD"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "browser",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "browserVersion",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "operatingSystem",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "language",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "deviceCategory",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "device",
"type": "RECORD"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "country",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "geoNetwork",
"type": "RECORD"
},
{
"fields": [
{
"fields": [
{
"mode": "NULLABLE",
"name": "pagePath",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "page",
"type": "RECORD"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "landingScreenName",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "appInfo",
"type": "RECORD"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "eventCategory",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "eventAction",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "eventLabel",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "eventInfo",
"type": "RECORD"
},
{
"mode": "NULLABLE",
"name": "type",
"type": "STRING"
}
],
"mode": "REPEATED",
"name": "hits",
"type": "RECORD"
},
{
"mode": "NULLABLE",
"name": "clientId",
"type": "STRING"
}
]

41
tests/sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/moz-fx-stubattribut-prod-32a5.stubattribution_prod.stdout.schema.json Normal file
Просмотреть файл

@ -0,0 +1,41 @@
[
{
"fields": [
{
"fields": [
{
"mode": "NULLABLE",
"name": "log_type",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "visit_id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "dltoken",
"type": "STRING"
}
],
"mode": "NULLABLE",
"name": "fields",
"type": "RECORD"
},
{
"mode": "NULLABLE",
"name": "timestamp",
"type": "FLOAT"
}
],
"mode": "NULLABLE",
"name": "jsonPayload",
"type": "RECORD"
},
{
"mode": "NULLABLE",
"name": "timestamp",
"type": "TIMESTAMP"
}
]

20
tests/sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/test_exception/expect.ndjson Normal file
Просмотреть файл

@ -0,0 +1,20 @@
{"dltoken": "ga_unresolve", "download_date": "2023-02-14", "count_dltoken_duplicates": 0, "additional_download_occurred": 0, "exception": "GA_UNRESOLVABLE"}
{"dltoken": "normal", "download_date": "2023-02-14", "time_on_site": 0, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "Internet Explorer", "normalized_browser": "MSIE" ,"browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 1, "unique_pageviews": 1, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": true, "count_dltoken_duplicates": 0, "additional_download_occurred": 0}
{"dltoken": "missing_ga_client", "download_date": "2023-02-14", "count_dltoken_duplicates": 0, "additional_download_occurred": 0, "exception": "MISSING_GA_CLIENT"}
{"dltoken": "duplicate_token", "download_date": "2023-02-14", "time_on_site": 0, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "Safari", "normalized_browser": "Safari" ,"browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 1, "unique_pageviews": 1, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": true, "count_dltoken_duplicates": 1, "additional_download_occurred": 1}
{"dltoken": "dl_session_id_empty", "download_date": "2023-02-14", "count_dltoken_duplicates": 0, "additional_download_occurred": 0, "exception": "DOWNLOAD_SESSION_ID_EMPTY"}
{"dltoken": "dl_session_not_set", "download_date": "2023-02-14", "count_dltoken_duplicates": 0, "additional_download_occurred": 0, "exception": "DOWNLOAD_SESSION_ID_VALUE_NOT_SET"}
{"dltoken": "dl_session_something", "download_date": "2023-02-14", "count_dltoken_duplicates": 0, "additional_download_occurred": 0, "exception": "DOWNLOAD_SESSION_ID_VALUE_SOMETHING"}
{"dltoken": "no_dl_event", "download_date": "2023-02-14", "time_on_site": 0, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "Edge", "normalized_browser": "Edge" ,"browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 1, "unique_pageviews": 1, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": false, "count_dltoken_duplicates": 0, "additional_download_occurred": 0}
{"dltoken": "pageviews_test", "download_date": "2023-02-14", "time_on_site": 0, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "Chrome", "normalized_browser": "Chrome" ,"browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 3, "unique_pageviews": 2, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": false, "count_dltoken_duplicates": 0, "additional_download_occurred": 0}
{"dltoken": "browser_firefox", "download_date": "2023-02-14", "time_on_site": 0, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "Firefox", "normalized_browser": "Firefox" ,"browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 1, "unique_pageviews": 1, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": true, "count_dltoken_duplicates": 0, "additional_download_occurred": 0}
{"dltoken": "browser_null", "download_date": "2023-02-14", "time_on_site": 0, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "(not set)", "browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 1, "unique_pageviews": 1, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": true, "count_dltoken_duplicates": 0, "additional_download_occurred": 0}
{"dltoken": "browser_not_set", "download_date": "2023-02-14", "time_on_site": 0, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 1, "unique_pageviews": 1, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": true, "count_dltoken_duplicates": 0, "additional_download_occurred": 0}
{"dltoken": "browser_other", "download_date": "2023-02-14", "time_on_site": 0, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "Fake browser", "normalized_browser": "Other" ,"browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 1, "unique_pageviews": 1, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": true, "count_dltoken_duplicates": 0, "additional_download_occurred": 0}
{"dltoken": "null_time_on_site", "download_date": "2023-02-14", "time_on_site": 0, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "Internet Explorer", "normalized_browser": "MSIE" ,"browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 1, "unique_pageviews": 1, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": true, "count_dltoken_duplicates": 0, "additional_download_occurred": 0}
{"dltoken": "valid_time_on_site", "download_date": "2023-02-14", "time_on_site": 112, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "Internet Explorer", "normalized_browser": "MSIE" ,"browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 1, "unique_pageviews": 1, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": true, "count_dltoken_duplicates": 0, "additional_download_occurred": 0}
{"dltoken": "missing_hit_type_page", "download_date": "2023-02-14", "time_on_site": 112, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "Internet Explorer", "normalized_browser": "MSIE" ,"browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 0, "unique_pageviews": 0, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": true, "count_dltoken_duplicates": 0, "additional_download_occurred": 0}
{"dltoken": "multi_download_1", "download_date": "2023-02-14", "time_on_site": 0, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "Internet Explorer", "normalized_browser": "MSIE" ,"browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 1, "unique_pageviews": 1, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": true, "count_dltoken_duplicates": 0, "additional_download_occurred": 1}
{"dltoken": "multi_download_2", "download_date": "2023-02-14", "time_on_site": 0, "country": "Canada", "normalized_country_code": "CA", "ad_content": "(not set)", "campaign": "(not set)", "medium": "(none)", "source": "(direct)", "device_category": "desktop", "os": "Windows", "normalized_os": "Windows", "browser": "Internet Explorer", "normalized_browser": "MSIE" ,"browser_version": "16.1", "browser_major_version": 16, "language": "en-ca", "pageviews": 1, "unique_pageviews": 1, "landing_page": "www.mozilla.org/en-GB/firefox/", "has_ga_download_event": true, "count_dltoken_duplicates": 0, "additional_download_occurred": 1}
{"dltoken": "ga_unresolve_multi_download_1", "download_date": "2023-02-14", "count_dltoken_duplicates": 0, "additional_download_occurred": 1, "exception": "GA_UNRESOLVABLE"}
{"dltoken": "ga_unresolve_multi_download_2", "download_date": "2023-02-14", "count_dltoken_duplicates": 0, "additional_download_occurred": 1, "exception": "GA_UNRESOLVABLE"}

17
tests/sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/test_exception/moz-fx-data-marketing-prod.65789850.ga_sessions_20230214.ndjson Normal file
Просмотреть файл

@ -0,0 +1,17 @@
{"clientId": "client100", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Safari", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client100", "visitId": "2","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Safari", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client101", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Internet Explorer", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client103", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Safari", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client104", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Edge", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client105", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Chrome", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}, {"page": {"pagePath": "/en-GB/firefox/download", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client106", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Firefox", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client107", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "(not set)", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client108", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": null, "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client109", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Fake browser", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client110", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Fake browser", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client111", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Internet Explorer", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "totals": { "timeOnSite": null, }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client112", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Internet Explorer", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "totals": { "timeOnSite": 112, }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client113", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Internet Explorer", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "totals": { "timeOnSite": 112, }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}]}
{"clientId": "client114", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Internet Explorer", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client115", "visitId": "1","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Safari", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}
{"clientId": "client115", "visitId": "2","date": "20230214","trafficSource": {"campaign": "(not set)", "source": "(direct)","medium": "(none)", "adContent": "(not set)",}, "device": {"browser": "Safari", "browserVersion": "16.1", "operatingSystem": "Windows", "language": "en-ca", "deviceCategory": "desktop"}, "geoNetwork": { "country": "Canada", }, "hits": [{"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop"}, "type": "EVENT"}, {"page": {"pagePath": "/en-GB/firefox/", }, "appInfo": { "landingScreenName": "www.mozilla.org/en-GB/firefox/", }, "eventInfo": { "eventCategory": "/firefox/ Interactions", "eventAction": "Firefox Download", "eventLabel": "Firefox for Desktop", }, "type": "PAGE"}]}

1
tests/sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/test_exception/moz-fx-data-shared-prod.static.country_names_v1.ndjson Normal file
Просмотреть файл

@ -0,0 +1 @@
{"name": "Canada", "code": "CA"}

21
tests/sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/test_exception/moz-fx-stubattribut-prod-32a5.stubattribution_prod.stdout.ndjson Normal file
Просмотреть файл

@ -0,0 +1,21 @@
{"jsonPayload":{ "fields": {"visit_id": "client100", "dltoken": "ga_unresolve", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client101", "dltoken": "normal", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client102", "dltoken": "missing_ga_client", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client103", "dltoken": "duplicate_token", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client103", "dltoken": "duplicate_token", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "", "dltoken": "dl_session_id_empty", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "(not set)", "dltoken": "dl_session_not_set", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "something", "dltoken": "dl_session_something", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client104", "dltoken": "no_dl_event", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client105", "dltoken": "pageviews_test", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client106", "dltoken": "browser_firefox", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client107", "dltoken": "browser_null", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client108", "dltoken": "browser_not_set", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client109", "dltoken": "browser_other", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client111", "dltoken": "null_time_on_site", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client112", "dltoken": "valid_time_on_site", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client113", "dltoken": "missing_hit_type_page", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client114", "dltoken": "multi_download_1", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client114", "dltoken": "multi_download_2", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client115", "dltoken": "ga_unresolve_multi_download_1", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}
{"jsonPayload":{ "fields": {"visit_id": "client115", "dltoken": "ga_unresolve_multi_download_2", "log_type": "download_started" }},"timestamp": "2023-02-14 01:16:43.101135 UTC"}

4
tests/sql/moz-fx-data-marketing-prod/ga_derived/downloads_with_attribution_v1/test_exception/query_params.yaml Normal file
Просмотреть файл

@ -0,0 +1,4 @@
---
- name: submission_date
type: DATE
value: 2023-02-14