Experimenter

Experimenter is a Python/Django application for managing experiments in Mozilla projects, such as Activity Stream.

What is an experiment?

An experiment is a way to measure how a change to an application will affect its usage and behaviour. For example you could change the way your application processes data in a way that affects its performance characteristics, as well as how it presents that data to the user. To make that change into an experiment, you could create a flag which optionally enables the new code path, and then enable it for a subset of users and measure how it affects things like performance and engagement.

Installation

1. Install docker on your machine

2. Clone the repo

    git clone <your fork>
   

3. Copy the sample env file

    cp .env.sample .env
   

4. Create a new secret key and put it in .env

    make secretkey
   

5. Run tests

    make test
   

6. Run database migrations

    make migrate
   

7. Make a local user

    make createuser
   

8. Run a dev instance

    make up
   

Done!

Usage

Experimenter uses docker for all development, testing, and deployment.

The following helpful commands have been provided via a Makefile:

build

Build the application container by executing the build script

compose_build

Build the supporting services (nginx, postgresql) defined in the compose file

up

Start a dev server listening on port 80 using the Django runserver

test

Run the Django test suite with code coverage

lint

Run flake8 against the code

check

Run both test and lint

migrate

Apply all django migrations

createuser

Create an admin user in the local dev instance

shell

Start an ipython shell inside the container (this lets you import and test code, interact with the db, etc)

bash

Start a bash shell inside the container (this lets you interact with the containerized filesystem)

API

GET /api/v1/<project_slug>/experiments.json

List all of the active experiments for a project.

Example:

    [
      {
        "name": "New Feature",
        "slug": "new-feature",
        "addon_versions": [
          "1.0",
          "2.0"
        ],
        "start_date": 1493928948000.0,
        "end_date": 1495138548000.0,
        "variant": {
          "slug": "enabled",
          "experiment_variant_slug": "new-feature:enabled",
          "value": "true",
          "threshold": 10
        },
        "control": {
          "slug": "disabled",
          "experiment_variant_slug": "new-feature:disabled",
          "value": "false"
        }
      }
    ]

Contributing

1. Fork it!
2. Create your feature branch: git checkout -b my-new-feature
3. Commit your changes: git commit -am 'Add some feature'
4. Push to the branch: git push origin my-new-feature
5. Submit a pull request :D

License

Experimenter uses the Mozilla Public License