Add script/export_incline_dash
The logic for incline exports in getting too complex to handle via Airflow operations (see https://github.com/mozilla/telemetry-airflow/pull/935). Instead, we put the logic into this script and will invoke via docker.
This commit is contained in:
Родитель
86350db0d4
Коммит
98a8b83b0c
|
@ -0,0 +1,26 @@
|
|||
#!/bin/bash
|
||||
|
||||
## Export subsets of the incline_executive_dashboard table to objects in GCS.
|
||||
|
||||
set -eo pipefail
|
||||
|
||||
if [ "$#" -ne 1 ]; then
|
||||
echo "ERROR: Exactly one parameter is required to specify the date of processing"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
DS=$1
|
||||
BUCKET="gs://moz-fx-data-prod-analysis"
|
||||
|
||||
for country in US CA DE FR GB IN CN IR BR IE ID tier-1 non-tier-1; do
|
||||
dest_latest="$BUCKET/incline/executive_dash/latest/${country}.csv.gz"
|
||||
echo "Populating $dest_latest from query"
|
||||
bq query --nouse_legacy_sql --project_id=moz-fx-data-shared-prod \
|
||||
--max_rows=100000 --format=csv -q \
|
||||
"SELECT * FROM org_mozilla_firefox_derived.incline_executive_v1 where country = '$country'" \
|
||||
| gzip \
|
||||
| gsutil -q cp - "$dest_latest"
|
||||
dest_ds="${dest_latest/latest/$DS}"
|
||||
echo "Copying ${dest_latest} to ${dest_ds}"
|
||||
gsutil -q cp "${dest_latest}" "${dest_ds}"
|
||||
done
|
Загрузка…
Ссылка в новой задаче