5.0 KiB

Исходник Ответственный История

Experimenter

1. Design 2. Launch 3. Analyze

Experimenter is a platform for managing experiments in Mozilla Firefox.

What is an experiment?

An experiment is a way to measure how a change to your product affects how people use it.

An experiment has three parts:

A new feature that can be selectively enabled
A group of users to test the new feature
Telemetry to measure how people interact with the new feature

How do I run an experiment?

Build a new feature behind a pref flag
Define an experiment for that feature in Experimenter
Send it to Shield
After Shield reviews and approves it, it is sent to Firefox
Firefox clients check whether they should enroll in the experiment and configure themselves accordingly
Telemetry about the experiment is collected
Dashboards are created to visualize the telemetry
Analyze and collect the results to understand how the new feature impacted users
Do it again!

Installation

Install docker on your machine
Clone the repo
```
 git clone <your fork>
```
Copy the sample env file
```
 cp .env.sample .env
```
Create a new secret key and put it in .env
```
 make secretkey
```
Run tests
```
 make test
```
Run database migrations
```
 make migrate
```
Make a local user
```
 make createuser
```
Run a dev instance
```
 make up
```

Done!

Usage

Experimenter uses docker for all development, testing, and deployment.

The following helpful commands have been provided via a Makefile:

build

Build the application container by executing the build script

compose_build

Build the supporting services (nginx, postgresql) defined in the compose file

up

Start a dev server listening on port 80 using the Django runserver

test

Run the Django test suite with code coverage

lint

Run flake8 against the code

check

Run both test and lint

migrate

Apply all django migrations

createuser

Create an admin user in the local dev instance

shell

Start an ipython shell inside the container (this lets you import and test code, interact with the db, etc)

bash

Start a bash shell inside the container (this lets you interact with the containerized filesystem)

API

GET /api/v1/experiments/

List all of the started experiments.

Optional Query Parameters

project__slug - Return only the experiments for a given project, an invalid slug will raise 404 status - Return only the experiments with the given status, options are:

'Created'
'Pending'
'Accepted'
'Launched'
'Complete'
'Rejected'

Example: GET /api/v1/experiments/?project__slug=project-slug&status=Pending

    [
      {
        "active": true,
        "name": "New Feature",
        "slug": "new-feature",
        "pref_key": "new.feature",
        "addon_versions": [
          "1.0",
          "2.0"
        ],
        "start_date": 1493928948000.0,
        "end_date": 1495138548000.0,
        "variant": {
          "slug": "enabled",
          "experiment_variant_slug": "new-feature:enabled",
          "value": true,
          "threshold": 0.2
        },
        "control": {
          "slug": "disabled",
          "experiment_variant_slug": "new-feature:disabled",
          "value": false
        }
      }
    ]

PATCH /api/v1/experiments/<experiment_slug>/accept

    Body: None

Set the status of a Pending experiment to Accepted.

Example: PATCH /api/v1/experiments/my-first-experiment/accept

PATCH /api/v1/experiments/<experiment_slug>/reject

    content-type: application/json
    Body: {message: "This experiment was rejected for reasons."}

Set the status of a Pending experiment to Rejected.