This commit is contained in:
Nan Jiang 2020-02-20 13:40:45 -05:00
Коммит 6b5413d431
5 изменённых файлов: 270 добавлений и 0 удалений

2
.gitignore поставляемый Normal file
Просмотреть файл

@ -0,0 +1,2 @@
venv/*
*.pyc

66
README.md Normal file
Просмотреть файл

@ -0,0 +1,66 @@
Auto generate dashboard on Redash for User Journey
## Install
```sh
# This utility requires Python 3, highly recommended to use the virtual environment
$ virtualenv venv
$ source venv/bin/activate
# Install dependencies
$ pip install -r requirements.txt
# Make sure get your Redash API key, it's available on Redash `Edit Profile -> Settings -> API Key`
# You can add it to your shell profile, such as `export REDASH_API_KEY="your_api_key"`.
```
## Usage
### Dashboard Template
Each dashboard is defined by a dashboard template file (YAML), which, in turn, is comprised of
a title and a list of charts.
#### Define Chart
Each dashboard consists of multiple charts, and each chart can be defined as follows:
```yaml
title: "WNP 72: Badge Impression&Clicks"
type: line
query: |
SELECT EXTRACT(date from submission_timestamp) as date,
event,
count(*) as count,
FROM messaging_system.cfr
WHERE EXTRACT(date FROM submission_timestamp) BETWEEN '2020-01-07' AND '2020-02-10'
AND release_channel = 'release'
AND version LIKE '72%'
AND event in ('IMPRESSION', 'CLICK')
AND message_id = 'WHATS_NEW_BADGE_72'
GROUP BY 1, 2
x_axis: date
y_axis: count
group_by: event
```
Most properties are self-explained,
* The `query`, defined by the author, provides the input of this chart
* The `type` sets the chart type, it could be "line", "area", "bar", "pie", "scatter", "bubble", "box", "pivot", and "table"
* The `x_axis` and `y_axis` set the axes, they should be the columns from the SELECT statement in the `query`
* The x axis could be further broken down by the `group_by`.
### Generate Dashboard
Once the template is defined, you can generate a dashboard with the generator.
For instance, to generate a dashboard for the "What's New Panel".
```sh
$ ./generate-wnp.py templates/whats-new-panel.yaml
```
Then you can go to Redash, click on `Dashboards` on the top-left, you should find the new dashboard (already published) there.
Note that for each chart, there should be a new query crated on Redash, though it's not published. You can find them in the `Queries` section on Redash.

89
generate-wnp.py Executable file
Просмотреть файл

@ -0,0 +1,89 @@
#!/usr/bin/env python3
import os
import sys
import yaml
from redashAPI import RedashAPIClient
USAGE = """
Usage: ./generate-wnp path/to/template.yaml
"""
STMO = "https://sql.telemetry.mozilla.org"
BIGQUERY_SOURCE = 63
def main():
if len(sys.argv) < 2:
print(USAGE)
sys.exit(1)
with open(sys.argv[1], "r") as f:
template = yaml.load(f, yaml.FullLoader)
rd = RedashAPIClient(os.environ['REDASH_API_KEY'], STMO)
# Create the dashboard
print(f'Creating dashboard {template["title"]}...')
dashboard = rd.create_dashboard(template["title"]).json()
dashboard_id = dashboard["id"]
# Create charts for each query and corresponding graph
for index, chart in enumerate(template["charts"]):
print(f'Creating chart {chart["title"]}...')
query = \
rd.create_query(
BIGQUERY_SOURCE,
chart["title"],
chart["query"]).json()
graph = \
rd.create_visualization(
query["id"],
chart["type"],
"Chart",
x_axis=chart["x_axis"],
y_axis=[{
"name": chart["y_axis"],
"label": "actions"
}],
group_by=chart["group_by"],
custom_options={
"xAxis": {
"type": "Auto Detect",
"labels": {"enable": True}
}
}
).json()
rd.add_widget(
dashboard_id,
vs_id=graph["id"],
position=calc_position(index)
)
# Publish dashboard
print("Publishing dashboard...")
rd.publish_dashboard(dashboard_id)
print("Done")
def calc_position(index):
""" Need to manually calculate the position as it's broken in `add_widget`.
"""
position = {
"col": 0,
"row": 0,
"sizeX": 3,
"sizeY": 8
}
row, col = divmod(index, 2)
position['row'] = row * 8
position['col'] = col * 3
return position
if __name__ == '__main__':
main()

2
requirements.txt Normal file
Просмотреть файл

@ -0,0 +1,2 @@
redash-api-client==0.2.7
PyYAML==5.3

Просмотреть файл

@ -0,0 +1,111 @@
---
# Redash dashboard template for Whats-new-panel.
#
#
# Dashboard Layout
# ---------------------------------------------------------------
# |
# |
# Badge: Impression/Clicks | What's New Toolbar button CTR
# |
# |
# ---------------------------------------------------------------
# |
# |
# What's New Access Path | Panel Message Clicks
# |
# |
# ---------------------------------------------------------------
title: "(AUTO) Messaging System: What's New Panel (Firefox 72)"
charts:
- title: "WNP 72: Badge Impression&Clicks"
type: line
query: |
SELECT EXTRACT(date from submission_timestamp) as date,
event,
count(*) as count,
FROM messaging_system.cfr
WHERE EXTRACT(date FROM submission_timestamp) BETWEEN '2020-01-07' AND '2020-02-10'
AND release_channel = 'release'
AND version LIKE '72%'
AND event in ('IMPRESSION', 'CLICK')
AND message_id = 'WHATS_NEW_BADGE_72'
GROUP BY 1, 2
x_axis: date
y_axis: count
group_by: event
- title: "WNP 72: Badge CTR"
type: line
query: |
WITH clicks AS (
SELECT
EXTRACT(date from submission_timestamp) as date,
count(*) as clicks
FROM
messaging_system.cfr
WHERE EXTRACT(date FROM submission_timestamp) BETWEEN '2020-01-07' AND '2020-02-10'
AND release_channel = 'release'
AND REGEXP_CONTAINS(version, r'7[2-3].*')
AND message_id = 'WHATS_NEW_BADGE_72'
AND event = 'CLICK'
GROUP BY
1
),
impressions AS (
SELECT
EXTRACT(date from submission_timestamp) as date,
count(*) as impressions
FROM
messaging_system.cfr
WHERE EXTRACT(date FROM submission_timestamp) BETWEEN '2020-01-07' AND '2020-02-10'
AND release_channel = 'release'
AND VERSION LIKE '72%'
AND message_id = 'WHATS_NEW_BADGE_72'
AND event = 'IMPRESSION'
GROUP BY
1
)
SELECT
clicks.*,
impressions.impressions,
cast(clicks.clicks as float64) / impressions.impressions * 100 as CTR
FROM
clicks
JOIN impressions ON clicks.date = impressions.date
x_axis: date
y_axis: CTR
group_by: null
- title: "WNP 72: Access Path"
type: bar
query: |
SELECT EXTRACT(date FROM submission_timestamp) AS date,
event_context,
count(*) as count
FROM messaging_system.cfr
WHERE EXTRACT(date FROM submission_timestamp) BETWEEN '2020-01-07' AND '2020-02-10'
AND message_id = 'WHATS_NEW_FINGERPRINTER_COUNTER_ALT,WHATS_NEW_PERMISSION_PROMPT_72,WHATS_NEW_PIP_72'
GROUP BY 1, 2;
x_axis: date
y_axis: count
group_by: event_context
- title: "WNP 72: Message Clicks"
type: line
query: |
SELECT EXTRACT(date FROM submission_timestamp) AS date,
message_id,
count(*) AS count,
FROM messaging_system.cfr
WHERE EXTRACT(date FROM submission_timestamp) BETWEEN '2020-01-07' AND '2020-02-10'
AND release_channel = 'release'
AND version LIKE '72%'
AND message_id IN ( 'WHATS_NEW_FINGERPRINTER_COUNTER_72',
'WHATS_NEW_FINGERPRINTER_COUNTER_ALT',
'WHATS_NEW_PERMISSION_PROMPT_72',
'WHATS_NEW_PIP_72')
GROUP BY 1, 2
ORDER BY 3 DESC
x_axis: date
y_axis: count
group_by: message_id