Collection of dockerized ETL jobs managed by data engineering.

Перейти к файлу

Anthony Miyaguchi 559d429708 Factor network view out of the main app		2020-12-23 17:12:15 -08:00
etl-graph	Write data to public/data	2020-12-23 16:14:57 -08:00
public	Upgrade packages and fix all warnings	2020-12-23 17:01:18 -08:00
src	Factor network view out of the main app	2020-12-23 17:12:15 -08:00
.gitignore	Add skeleton for sveltified app	2020-12-23 16:16:28 -08:00
NOTES.md	Update notes	2020-06-18 17:23:47 -07:00
README.md	Add skeleton for sveltified app	2020-12-23 16:16:28 -08:00
deploy.sh	Remove setting the project	2020-12-23 16:06:06 -08:00
package-lock.json	Upgrade packages and fix all warnings	2020-12-23 17:01:18 -08:00
package.json	Upgrade packages and fix all warnings	2020-12-23 17:01:18 -08:00
requirements.in	Add command line interface via click	2020-06-16 10:56:21 -07:00
requirements.txt	Add command line interface via click	2020-06-16 10:56:21 -07:00
rollup.config.js	Add skeleton for sveltified app	2020-12-23 16:16:28 -08:00

README.md

etl-graph

Quickstart

Run the crawler.

# optional: virtualenv
python3 -m venv venv
source venv/bin/activate

# generate table entities and resolve view references
python -m etl-graph crawl

# generate edgelist from query logs
python -m etl-graph query-logs

# generate final index
python -m etl-graph index

Start the web client for visualization.

npm run dev

Deploy to hosting.

./deploy.sh