Add script/export_incline_dash
The logic for incline exports in getting too complex to handle via Airflow operations (see https://github.com/mozilla/telemetry-airflow/pull/935). Instead, we put the logic into this script and will invoke via docker.
This commit is contained in:
Родитель
86350db0d4
Коммит
98a8b83b0c
|
@ -0,0 +1,26 @@
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
## Export subsets of the incline_executive_dashboard table to objects in GCS.
|
||||||
|
|
||||||
|
set -eo pipefail
|
||||||
|
|
||||||
|
if [ "$#" -ne 1 ]; then
|
||||||
|
echo "ERROR: Exactly one parameter is required to specify the date of processing"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
DS=$1
|
||||||
|
BUCKET="gs://moz-fx-data-prod-analysis"
|
||||||
|
|
||||||
|
for country in US CA DE FR GB IN CN IR BR IE ID tier-1 non-tier-1; do
|
||||||
|
dest_latest="$BUCKET/incline/executive_dash/latest/${country}.csv.gz"
|
||||||
|
echo "Populating $dest_latest from query"
|
||||||
|
bq query --nouse_legacy_sql --project_id=moz-fx-data-shared-prod \
|
||||||
|
--max_rows=100000 --format=csv -q \
|
||||||
|
"SELECT * FROM org_mozilla_firefox_derived.incline_executive_v1 where country = '$country'" \
|
||||||
|
| gzip \
|
||||||
|
| gsutil -q cp - "$dest_latest"
|
||||||
|
dest_ds="${dest_latest/latest/$DS}"
|
||||||
|
echo "Copying ${dest_latest} to ${dest_ds}"
|
||||||
|
gsutil -q cp "${dest_latest}" "${dest_ds}"
|
||||||
|
done
|
Загрузка…
Ссылка в новой задаче