This is motivated by
[a discussion doc](https://docs.google.com/document/d/1NwY1iReuxcbvxCi9jRUOYiymCUOoSq3cnShCI-OiW3c/edit#heading=h.c6fp0v2ffacr)
that focused on the problem of implicit dependencies between repos.
emr-bootstrap-spark is one of the projects where changes can cause unanticipated
problems for running code that lives elsewhere, and it can be difficult to
track down the source of the problem when an emr-bootstrap-spark deploy goes
wrong.

This will be followed up by documentation changes to other repos that depend
on this one, making the dependency more explicit.
This commit is contained in:
Jeff Klukas 2018-12-14 15:00:31 -05:00
Родитель a72d3c83cf
Коммит cfc762f9d4
2 изменённых файлов: 27 добавлений и 1 удалений

11
.github/pull_request_template.md поставляемый Normal file
Просмотреть файл

@ -0,0 +1,11 @@
------------------------
Testing checklist:
- [ ] Deploy your changes to the staging location ([see README](https://github.com/mozilla/emr-bootstrap-spark#deploy-to-aws-via-ansible))
- [ ] Launch an [ATMO stage](https://atmo.stage.mozaws.net/) cluster and check for startup failures
- [ ] Ensure that the [Telemetry Hello World notebook](https://github.com/mozilla/mozilla-reports/blob/master/tutorials/telemetry_hello_world.kp/orig_src/Telemetry%20Hello%20World.ipynb) can run on your ATMO cluster without errors
The above checks are not exhaustive, nor are they strictly required before merging,
but they are useful to catch many common error cases
such as incompatibilities when updating python library versions.

Просмотреть файл

@ -1,7 +1,10 @@
emr-bootstrap-spark
===================
This packages contains the AWS bootstrap scripts for Mozilla's flavoured Spark setup.
This package contains the AWS bootstrap scripts for Mozilla's flavoured Spark setup.
The deployed scripts in S3 are referenced by
[ATMO clusters](https://github.com/mozilla/telemetry-analysis-service) and
[Airflow jobs](https://github.com/mozilla/telemetry-airflow).
## Interactive job
```bash
@ -43,6 +46,18 @@ aws emr create-cluster \
```
## Deploy to AWS via ansible
To deploy to the staging location:
```bash
ansible-playbook ansible/deploy.yml -e '@ansible/envs/stage.yml' -i ansible/inventory
```
Once deployed, you can see the effects in action by launching a cluster via
[ATMO stage](https://atmo.stage.mozaws.net/).
To deploy for production clusters:
```bash
ansible-playbook ansible/deploy.yml -e '@ansible/envs/production.yml' -i ansible/inventory
```