This commit is contained in:
Hamel Husain 2020-06-16 12:32:37 -07:00
Родитель 907015b391
Коммит abcab3dc5e
5 изменённых файлов: 77 добавлений и 1 удалений

Просмотреть файл

@ -0,0 +1,13 @@
---
title: Experiment-Tracking
order_n: 4
has_children: true
permalink: /docs/experiment-tracking
---
Experiment tracking for machine learning involves logging of metrics and artificacts associated with different training runs. This is essential for effective mlops as it allows you to track your performance metrics and promotes reproduceability in a transparent, reuseable way.
There are several experiment tracking systems available as third-party solutions that GitHub Actions can integrate with. For example, the below example illustrates how results of a training run can be fetched from [Weights and Biases](https://www.wandb.com/) and dropped into a pull request:
<img src='https://raw.githubusercontent.com/machine-learning-apps/actions-ml-cicd/master/images/mlops.png'></img>

Просмотреть файл

@ -0,0 +1,61 @@
---
title: Weights & Biases
order_n: 3
permalink: /docs/wandb
parent: Experiment Tracking
---
## Fetch Runs From Weights & Biases
[Weights and Biases](https://www.wandb.com/) is a system for experiment tracking, model optimization, and dataset versioning.
The [Weights and Biases (W&B) Action](https://github.com/machine-learning-apps/wandb-action) can help you fetch runs from W&B for reporting in your GitHub workflows.
**This Action saves a csv file called `wandb_report.csv` into the path specified by the [default environment variable](https://help.github.com/en/articles/virtual-environments-for-github-actions#environment-variables) `GITHUB_WORKSPACE` set for you in GitHub Actions**, which allows this data to be accessed by subsequent Actions. Information in this CSV can be displayed in a variety of ways, such as a markdown formatted comment in a pull request or via the [GitHub Checks](https://developer.github.com/v3/checks/) API.
This csv file always has the following fields:
- `run.url`: the url for the run in the W&B api.
- `run.name`: the name of the run. This is automatically set by wandb if not specified by the user.
- `run.tags`: a list with all of the tags assigned to the run.
- `run.id`: the id associated with the run. This corresponds to the input `RUN_ID`
- `run.entity`: this name of the entity that contains the project the run can be found in. This is similar to an org in GitHub.
- `run.project`: the name of the project that contains the run. This is simlar to a repo in GitHub.
- `github_sha`: the config variable `github_sha`.
- `__eval.category`: this field will contain either the value `candiate` or `baseline`, depending on how the run was queried.
In addition to the above fields the user can specify additional fields.
Below is an example of how this Action can be used to fetch model runs:
{% raw %}
```yaml
name: Get WandB Runs
on: [issue_comment]
jobs:
get-runs:
if: (github.event.issue.pull_request != null) && contains(github.event.comment.body, '/get-runs')
runs-on: ubuntu-latest
steps:
- name: Get the latest SHA for the PR that was commented on
id: chatops
uses: machine-learning-apps/actions-chatops@master
with:
TRIGGER_PHRASE: "/get-runs"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Get Runs Using SHA
uses: machine-learning-apps/wandb-action@master
with:
PROJECT_NAME: ${{ format('{0}/{1}', secrets.WANDB_ENTITY, secrets.WANDB_PROJECT) }}
FILTER_GITHUB_SHA: ${{ steps.chatops.outputs.SHA }}
BASELINE_TAGS: "['baseline', 'reference']"
DISPLAY_METRICS: "['accuracy', 'loss', 'best_val_acc', 'best_val_loss', '_runtime']"
WANDB_API_KEY: ${{ secrets.WANDB_API_KEY }}
DEBUG: 'true'
```
{% endraw %}
See [this project](https://github.com/machine-learning-apps/wandb-action) for more information.

Просмотреть файл

@ -1,6 +1,7 @@
--- ---
title: fastpages title: fastpages
parent: Jupyter parent: Jupyter
order_n: 2
permalink: /docs/fastpages permalink: /docs/fastpages
--- ---

Просмотреть файл

@ -1,11 +1,11 @@
--- ---
title: papermill title: papermill
parent: Jupyter parent: Jupyter
order_n: 5
permalink: /docs/papermill permalink: /docs/papermill
--- ---
## Refresh Jupyter Notebooks With [Papermill](https://github.com/nteract/papermill) ## Refresh Jupyter Notebooks With [Papermill](https://github.com/nteract/papermill)
[papermill](https://github.com/nteract/papermill) is a great project for running notebooks programatically. You can pass parameters to notebooks, use different kernels, etc. [papermill](https://github.com/nteract/papermill) is a great project for running notebooks programatically. You can pass parameters to notebooks, use different kernels, etc.

Просмотреть файл

@ -1,6 +1,7 @@
--- ---
title: repo2docker title: repo2docker
parent: Jupyter parent: Jupyter
order_n: 1
permalink: /docs/repo2docker permalink: /docs/repo2docker
--- ---