major reorganization of the sidebar (#208)
* major reorganization of the sidebar * update caniuse-list per instructions * fix broken links to a few docs
This commit is contained in:
Janet Dragojevic
GitHub
Родитель
eddd706d69
Коммит
a099f6481e
|
|
@ -44,7 +44,7 @@ Each experiment begins with an enrollment period with fixed start and end dates
|
|||
|
||||
For each client, experiment metrics are analyzed over a defined period of time from enrollment. We report results for an analysis period (e.g. the first day after enrollment) after all clients have had a chance to experience the treatment for that duration. The [Jetstream overview](jetstream/jetstream.md#analysis-paradigm) describes the analysis paradigm in more depth and how it relates to the length of an experiment.
|
||||
|
||||
For more nuances about sampling, enrollment and exposure (whether or not the client actually saw the treatment or control), see [the experiment lifecycle overview](client-sdk-states-and-lifecycle.mdx).
|
||||
For more nuances about sampling, enrollment and exposure (whether or not the client actually saw the treatment or control), see [the experiment lifecycle overview](deep-dives/specifications/client-sdk-states-and-lifecycle.mdx).
|
||||
|
||||
## Sample size recommendations
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
id: bucketing
|
||||
title: Bucketing
|
||||
slug: /Bucketing
|
||||
slug: /bucketing
|
||||
---
|
||||
|
||||
**Bucketing** is the process of randomly assigning users to experiment branches. When a user is “bucketed” into an experiment, it means that the configuration in one of its branches (such as a change to part of the UI) can be activated, and that any interactions we record from that moment on can be associated with the experiment and branch identifier.
|
||||
|
|
@ -1,3 +1,9 @@
|
|||
---
|
||||
id: mobile-feature-api
|
||||
title: Mobile Feature API
|
||||
slug: /mobile-feature-api
|
||||
---
|
||||
|
||||
# Feature Variables and Me
|
||||
|
||||
## About this document
|
||||
|
|
@ -1,3 +1,8 @@
|
|||
---
|
||||
id: fm-unimplemented-spec
|
||||
title: Proposed changes for Feature Manifest Language
|
||||
slug: /fm-unimplemented-spec
|
||||
---
|
||||
## Future specifications
|
||||
|
||||
This document houses a number of proposals not ready for comment. They are not linked to from the spec.
|
||||
|
|
@ -42,7 +42,7 @@ The "Feature Manifest Language" is definition language for writing feature manif
|
|||
|
||||
Features are configured remotely by sending JSON objects to the Nimbus SDK on each device. It is the job of the `nimbus-fml`, and the code that it generates to unpack that JSON, validate it, coerce it into values usable by the app and recover if anything goes wrong.
|
||||
|
||||
[feature-api]: feature-variables-and-me.md
|
||||
[feature-api]: mobile-feature-api
|
||||
|
||||
### JSON Merge Patch
|
||||
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
---
|
||||
id: fenix-experiments-getting-started
|
||||
title: Implementing an experiment
|
||||
slug: /fenix-engineers
|
||||
---
|
||||
|
||||
## Client configuration prechecks
|
||||
|
||||
✅ Get a copy of the experiment brief for your experiment from the [Experiments Repository](https://mana.mozilla.org/wiki/x/FaXNBw) mana page.
|
||||
|
||||
✅ Assure that your experiment brief contains values for:
|
||||
|
||||
- experiment name
|
||||
- featureID
|
||||
- branch names (for a basic a/b experiment, prefer _control_ and _treatment_)
|
||||
|
||||
✅ Confirm the **featureID** is added to the experimenter list of console values.
|
||||
|
||||
> NOTE: featureID setup access is controlled via experimenter Django admin.
|
||||
>
|
||||
> Please ask for help in the `#ask-experimenter` channel
|
||||
|
||||
## Fenix setup
|
||||
|
||||
Preparing your app for experimentation is now about getting your app features to get its configuration from the Nimbus Feature API.
|
||||
|
||||
1. [Instrument your application feature with the Feature API][feature-api].
|
||||
1. Once your feature is instrumented, you should add the feature-id to Experimenter.
|
||||
1. Now Experimenter knows about the feature, you can configure your application features with an experiment.
|
||||
|
||||
[feature-api]: feature-variables-and-me
|
||||
## Local Testing
|
||||
|
||||
1. Setup your local Fenix build to point to the experimenter
|
||||
[staging url](https://github.com/mozilla-mobile/fenix#using-nimbus-servers-during-local-development).
|
||||
1. Build Fenix with the updated remote-settings configuration.
|
||||
1. Configure your experiment in the [experimenter nimbus staging site](https://stage.experimenter.nonprod.dataops.mozgcp.net/nimbus/).
|
||||
1. Run Fenix and check you are able to see your expected behavior.
|
||||
|
|
@ -1,5 +1,4 @@
|
|||
---
|
||||
id: access
|
||||
title: Getting Access
|
||||
slug: /access
|
||||
---
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
id: desktop-feature-api
|
||||
title: Nimbus Feature API (JS and C++)
|
||||
title: Desktop Feature API (JS and C++)
|
||||
slug: /desktop-feature-api
|
||||
---
|
||||
|
||||
|
|
@ -1,9 +1,10 @@
|
|||
---
|
||||
id: desktop-migration-guide
|
||||
title: Migration Guide (JS)
|
||||
title: Desktop Migration Guide (JS)
|
||||
slug: /desktop-migration-guide
|
||||
---
|
||||
|
||||
## To migrate front-end normandy prefs to Nimbus controlled experiments or rollouts
|
||||
This guide will help you migrate your Desktop front-end code to run experiments with Nimbus, while still being able to use preferences for default and user-override values
|
||||
|
||||
Prerequisites:
|
||||
|
|
@ -0,0 +1,14 @@
|
|||
---
|
||||
id: for-engineers
|
||||
title: Engineers
|
||||
slug: /for-engineers
|
||||
---
|
||||
|
||||
Engineers are typically tasked with implementing an experiment on a new surface with an experiment that has been designed by their product manager in partnership with a data scientist.
|
||||
|
||||
## To get started with implementation
|
||||
|
||||
* Review the experiment design document.
|
||||
* Determine if you need to implement any additional telemetry collection
|
||||
* Check if you need to implement a custom audience
|
||||
* Confirm the name of the feature and follow the guide to [adding a new feature to the manifest](http://localhost:3000/feature-definition#to-define-your-feature-in-the-feature-manifest-file)
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
id: for-leadership
|
||||
title: Leaders
|
||||
slug: /for-leadership
|
||||
---
|
||||
|
||||
## How to start implementing an experiment
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
id: for-product
|
||||
title: Product Managers
|
||||
slug: /for-product
|
||||
---
|
||||
|
||||
## How to start implementing an experiment
|
||||
|
|
@ -1,32 +0,0 @@
|
|||
---
|
||||
id: intro
|
||||
title: Getting Started
|
||||
sidebar_label: Introduction
|
||||
slug: /
|
||||
---
|
||||
|
||||
:::tip
|
||||
Need help? Reach out in [`#ask-experimenter`](https://mozilla.slack.com/archives/CF94YGE03) on Slack or join a [lightning advisory session](https://docs.google.com/document/d/1fj4BqG0N-cN_Fx5sq-AozdiOdKO7asaDWR4B7sH88F0).
|
||||
:::
|
||||
|
||||
Welcome to the [Experimenter](https://experimenter.services.mozilla.com/nimbus/) documentation hub, your central resource for A/B experiments in Firefox Mobile and Desktop. Whether you are an engineer working on a feature, a product manager with a hypothesis, or a data scientist helping design and interpret results, we want to give you the tools to get started!
|
||||
|
||||
## Experimentation Workflow
|
||||
|
||||
To view a recorded training for the Experimentation Workflow, [click here](https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=4d337632-a0bd-4be7-bee8-ae5b017134ae&start=0)
|
||||
|
||||
<div class="miro-container">
|
||||
<iframe class="responsive-iframe" src="https://miro.com/app/live-embed/uXjVOJ3IYRA=/?moveToViewport=-2130,380,5949,3025&embedAutoplay=true" frameBorder="0" scrolling="no"></iframe>
|
||||
</div>
|
||||
|
||||
## Find an existing experiment
|
||||
|
||||
If the experiment was run with Nimbus, check out the [directory of live and completed experiments](https://experimenter.services.mozilla.com/nimbus/). Normandy experiments can be found [here](https://experimenter.services.mozilla.com/).
|
||||
|
||||
If the experiment is not yet live, you will probably be able to find a relevant Jira ticket on this [Data Science Experimentation Collaboration Board](https://mozilla-hub.atlassian.net/jira/software/c/projects/DS/boards/258)
|
||||
|
||||
## About these docs
|
||||
|
||||
This website is built using [Docusaurus](https://v2.docusaurus.io/).
|
||||
|
||||
Want to edit one of these docs, add a new one, or modify the sidebar? Check out the [Contributing](/contributing) page.
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
---
|
||||
id: capabilities
|
||||
title: Nimbus Features
|
||||
sidebar_label: Features
|
||||
slug: /capabilities
|
||||
---
|
||||
|
||||
Nimbus is a full-featured experimentation platform that provides configuration, analysis and client libraries for both experiments and rollouts.
|
||||
|
||||
## Experimenter Console
|
||||
- Simple experiment configuration UI
|
||||
- Pre-launch testing support
|
||||
- Multifeature experiments
|
||||
- Simple targeting
|
||||
- Advanced customizable targeting
|
||||
- Remote settings integration
|
||||
- Experiment Review / Approval Workflow
|
||||
- Dashboards
|
||||
- Monitoring and Analysis
|
||||
|
||||
## Experiment Types
|
||||
- A/A validation
|
||||
- A/B experiments
|
||||
- Holdback experiments
|
||||
- Messaging experiments
|
||||
- Feature Rollouts
|
||||
- Winning branch promotion
|
||||
|
||||
## Multiple langauage Client integrations
|
||||
- Android (Kotlin)
|
||||
- iOS (Swift)
|
||||
- Firefox Desktop Frontend (JS)
|
||||
- Firefox Platform (C++)
|
||||
|
||||
## Client-side functionality
|
||||
- Consumer opt-out
|
||||
- Integration with console preview testing
|
||||
- View experiments (about:studies)
|
||||
|
||||
## Current client integrations
|
||||
- Firefox Desktop
|
||||
- Frontend via JS SDK
|
||||
- Platform
|
||||
- Windows installer
|
||||
- Firefox Mobile via nimbus Rust Component
|
||||
- Fenix
|
||||
- Firefox iOS
|
||||
- Focus Andriod
|
||||
-Focus iOS
|
||||
|
||||
## Requesting feature support
|
||||
If you aren't sure we have what you need, pop into #ask-experimenter with your questions or [file an issue](https://mozilla-hub.atlassian.net/secure/CreateIssueDetails!init.jspa?pid=10203&issuetype=10097)
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
---
|
||||
id: finding-help
|
||||
title: Finding Help
|
||||
sidebar_label: Finding Help
|
||||
slug: /help
|
||||
---
|
||||
|
||||
:::tip
|
||||
Need help? Not sure where to start?
|
||||
Reach out in [`#ask-experimenter`](https://mozilla.slack.com/archives/CF94YGE03) on Slack or join a [lightning advisory session](https://docs.google.com/document/d/1fj4BqG0N-cN_Fx5sq-AozdiOdKO7asaDWR4B7sH88F0).
|
||||
:::
|
||||
|
|
@ -0,0 +1,35 @@
|
|||
---
|
||||
id: intro
|
||||
title: Welcome
|
||||
sidebar_label: Introduction
|
||||
slug: /
|
||||
---
|
||||
|
||||
|
||||
Welcome to the [Experimenter](https://experimenter.services.mozilla.com/nimbus/) documentation hub, your central resource for A/B experiments and feature rollouts in Firefox Mobile and Desktop. You should find these documents helpful if you are:
|
||||
|
||||
* A **product manager** and you have a hypothesis you want to test or a feature you'd like to safely release.
|
||||
* An **engineer** who needs to implement an experiment or feature rollout in your client
|
||||
* A **data scientist** helping to design and interpret experiment results
|
||||
* A **Mozilla leader** who would like insight into completed and ongoing experiments and rollouts.
|
||||
|
||||
|
||||
## What is covered here?
|
||||
|
||||
These documents are specific to the Mozilla experimentation program know as **Nimbus**. Nimbus experimentation and support is currently available for our Firefox Desktop and Mobile browser applications.
|
||||
|
||||
Our platform code is all open-source however, we do not offer support or services to 3rd party consumers. Some documentation links may only be available to Mozilla employees and NDA contributors.
|
||||
|
||||
## What other tools exist
|
||||
|
||||
The following projects have bespoke tools for experimentation. These are not included as part of this documentation with the exception of a migration guide from Normandy to Nimbus for engineers.
|
||||
|
||||
* Firefox Desktop (legacy experimentation): [Normandy](https://wiki.mozilla.org/Firefox/Normandy/PreferenceRollout)
|
||||
* Mozilla Websites (a/b experiments): [Bedrock](https://github.com/mozilla/bedrock)
|
||||
* Firefox Accounts (limited content server): [Fxa docs](https://mozilla.github.io/ecosystem-platform/reference/experiments-ab-testing)
|
||||
|
||||
|
||||
|
||||
## About these docs
|
||||
|
||||
This website is built using [Docusaurus](https://v2.docusaurus.io/). If you'd like to edit or add to them, check out the [Contributing](/contributing) page.
|
||||
|
|
@ -391,4 +391,4 @@ window_end = "analysis_window_end"
|
|||
Results for exposure based metrics are currently not visualized in Experimenter. To access results, the BigQuery tables need to be queried directly.
|
||||
|
||||
|
||||
[Nimbus exposure event]: feature-variables-and-me.md#recording-exposure-events
|
||||
[Nimbus exposure event]: ../mobile-feature-api#recording-exposure-events
|
||||
|
|
|
|||
|
|
@ -80,7 +80,7 @@ Any telemetry collection can be used as an exposure event, though events are oft
|
|||
Many Nimbus features will send a [Nimbus exposure event] automatically when the feature configuration is consulted;
|
||||
these are `normandy#expose` events on desktop and `nimbus_events.exposure` events in Glean.
|
||||
|
||||
[Nimbus exposure event]: feature-variables-and-me.md#recording-exposure-events
|
||||
[Nimbus exposure event]: ../mobile-feature-api#recording-exposure-events
|
||||
|
||||
## Analysis steps
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
id: mobile-messaging
|
||||
title: Using the Experimenter to send messages to Mobile clients
|
||||
title: Mobile Messaging
|
||||
slug: /mobile-messaging
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,24 @@
|
|||
---
|
||||
title: A/B Experiments
|
||||
slug: /a-b-experiments
|
||||
---
|
||||
|
||||
## To implement an A/A or A/B experiment on Desktop
|
||||
|
||||
|
||||
## To implement an A/A or A/B experiment on Mobile
|
||||
|
||||
## To implement a holdback experiment
|
||||
* Desktop
|
||||
* Mobile
|
||||
|
||||
|
||||
## To implement custom telemetry collection
|
||||
If you need telemetry beyond what is already specfied for your mobile client as part of your implementation process you will need to add the additional telemetry to your client.
|
||||
|
||||
* Desktop
|
||||
* Mobile
|
||||
|
||||
## To locally validate your changes
|
||||
* Desktop
|
||||
* Mobile
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
id: analyzing
|
||||
title: Experiment Analysis
|
||||
sidebar_label: Analyzing
|
||||
slug: /analyzing
|
||||
---
|
||||
|
|
@ -4,7 +4,7 @@ title: Adding new targeting attributes to Android
|
|||
---
|
||||
# Adding new targeting attributes to Android
|
||||
This page demonstrates how to add new targeting attributes to Android, enabling experiment creators more specific targeting.
|
||||
For more general documentation on targeting custom audiences, check out [the custom audiences docs](experiment-owners/custom-audiences.md)
|
||||
For more general documentation on targeting custom audiences, check out [the custom audiences docs](custom-audiences)
|
||||
|
||||
## Adding the attribute to the application
|
||||
The Nimbus SDK exposes a new `customTargetingAttributes` parameter in its initializer that is a `Map<String, String>` map. We can take advantage of this parameter to pass in new targeting attributes without modifying the Nimbus SDK at all.
|
||||
|
|
@ -28,7 +28,7 @@ val appInfo = NimbusAppInfo(
|
|||
Note that since we need to add the targeting attributes on the client code, the attribute changes will have to ride the trains before they are available for targeting.
|
||||
|
||||
## Adding the attribute on experimenter
|
||||
After the targeting attribute is ready on the app, you will need to modify experimenter to allow creating experiments that target the attribute you created. Follow the instructions on [the custom audiences page](experiment-owners/custom-audiences.md#how-to-add-a-new-custom-audience) to add the new targeting on experimenter.
|
||||
After the targeting attribute is ready on the app, you will need to modify experimenter to allow creating experiments that target the attribute you created. Follow the instructions on [the custom audiences page](custom-audiences#how-to-add-a-new-custom-audience) to add the new targeting on experimenter.
|
||||
:::warning
|
||||
The targeting `JEXL` expression on experimenter **must** use the same name as the key given to the SDK. For example, if the app defines a key-value pair, with key `isFirstRun`. experimenter expression must use the same name (i.e `isFirstRun`).
|
||||
:::
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
id: configuring
|
||||
title: Experiment Configuration
|
||||
sidebar_label: Configuring
|
||||
slug: /configuring
|
||||
---
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
---
|
||||
id: designing
|
||||
title: Designing Your Experiment
|
||||
sidebar_label: Designing
|
||||
slug: /designing
|
||||
---
|
||||
|
||||
**Draft content**
|
||||
|
||||
In the designing phase of the experimentation workflow it is important to consult with a data scientist in order to:
|
||||
* do this
|
||||
* do that
|
||||
* something something audience
|
||||
* determine targeting needs (custom or default)
|
||||
|
||||
## Types of experiments
|
||||
|
||||
|
||||
| | Branches | Control | Co-existence / Avoidance | Enrollment | Analysis | Good for? | Permanent Change |
|
||||
|--------------------------------------------|---------------------------------------------------------------------------|--------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------|----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|
|
||||
| Experiment | A/B - equal sizes. Size fixed at launch - cannot grow. | Y | One per feature at a time. Avoids other live experiments on a specific feature. Keep as small as possible to learn. | Fixed timeframe. 7 day typically. Analysis only AFTER enrollment ends | Automated for default metrics. Can select outcomes or create new custom | Stat sig differences from control + absolute counts. | No - all changes delivered through nimbus reverted. |
|
||||
| Message experiment | Same AND each branch is given a messageID that the messaging system uses. | Same as experiment | Same as experiment AND message system throttles max # per time frame | Same as experiment | Same as experiment | Same as experiment | N Same as experiment |
|
||||
| Multi-feature experiment | Same as experiment | Same as experiment | Avoids other live experiment with EITHER FEATURE - **restricted audience | Same as experiment | Same as experiment | Same as experiment | N Same as experiment |
|
||||
| Holdback Experiment | Go right to end stage of rollout - 90% treatment | Y - 10% | Same as experiment AND no other feature can be tested in this feature for the duration. | Same as experiment | Same as experiment | Decision made to deploy quickly - but want data. A brief step before shipping as default in tree. | No - all changes delivered through nimbus reverted. Fast follow shipping in tree. |
|
||||
| Rollout Today | Size increases 5%, 25%, 50%, 100% | N | Unrestricted co-existence Does not avoid other rollouts or experiments. | N/A | None - teams can make and monitor their own dashboards - but no “stat sig analysis” offered. | Risk mitigation for explosive failures (looking for crash, bugs, media issues) | Yes. Changes do not revert on targeted versions. Teams ship as default in tree in the release at 100%. |
|
||||
| Branch Promotion | 1 - 100% | N | On an experiment with a winning branch - you can select “promote to rollout” which clones the ticket with the winning configuration filled in | N | N | Get winning branch to users quicker than trains | Same as Rollout Today |
|
||||
| Do No Harm Monitoring Platform Experiment | Same as experiment | Same as experiment | Same as experiment. | No fixed timeframe means there is no automatic analysis trigger. | Not automated. Requires OpMon configured or team using their own custom dashboards. | Code changing/fixes landing during. Risk mitigation for quick major breaking stability / perf changes. Looking at adding search metrics as well. | N Same as experiment |
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
id: ending
|
||||
title: Ending Your Experiment
|
||||
sidebar_label: Ending
|
||||
slug: /ending
|
||||
---
|
||||
|
||||
|
|
@ -0,0 +1,18 @@
|
|||
---
|
||||
title: Feature Definition
|
||||
sidebar_label: Feature Definition
|
||||
slug: /feature-definition
|
||||
---
|
||||
|
||||
In the experimentation ecosystem, experiment surfaces are described as features. A feature is an area of code instrumented with telemetry and accessible for remote configuration. It can be as small as a single function or as complex as a whole page. Some examples:
|
||||
|
||||
aboutwelcome: The about:welcome page in Desktop
|
||||
homescreen: The homescreen page in Fenix
|
||||
tabTrayFeature: The tab tray in Firefox iOS
|
||||
|
||||
|
||||
Features are defined in a Feature Manifest file for the application, and the client code uses the Nimbus SDK to access variables associated with those features.
|
||||
|
||||
## To define your feature in the feature manifest file
|
||||
* Desktop
|
||||
* Mobile
|
||||
|
|
@ -4,7 +4,7 @@ title: Adding new targeting attributes to iOS
|
|||
---
|
||||
# Adding new targeting attributes to iOS
|
||||
This page demonstrates how to add new targeting attributes to iOS, enabling experiment creators more specific targeting.
|
||||
For more general documentation on targeting custom audiences, check out [the custom audiences docs](experiment-owners/custom-audiences.md)
|
||||
For more general documentation on targeting custom audiences, check out [the custom audiences docs](custom-audiences)
|
||||
|
||||
## Adding the attribute to the application
|
||||
The Nimbus SDK exposes a new `customTargetingAttributes` parameter in its initializer that is a `[String:String]` map. We can take advantage of this parameter to pass in new targeting attributes without modifying the Nimbus SDK at all.
|
||||
|
|
@ -22,7 +22,7 @@ Experiments.customTargetingAttributes = ["isFirstRun": "\(isFirstRun)", "newTar
|
|||
```
|
||||
|
||||
## Adding the attribute on experimenter
|
||||
After the targeting attribute is ready on the app, you will need to modify experimenter to allow creating experiments that target the attribute you created. Follow the instructions on [the custom audiences page](experiment-owners/custom-audiences.md#how-to-add-a-new-custom-audience) to add the new targeting on experimenter.
|
||||
After the targeting attribute is ready on the app, you will need to modify experimenter to allow creating experiments that target the attribute you created. Follow the instructions on [the custom audiences page](custom-audiences#how-to-add-a-new-custom-audience) to add the new targeting on experimenter.
|
||||
:::warning
|
||||
The targeting `JEXL` expression on experimenter **must** use the same name as the key given to the SDK. For example, for targeting users on their first run, the app defines a key-value pair, with key `isFirstRun`. The experimenter expression must use the same name (i.e `isFirstRun`)
|
||||
:::
|
||||
|
|
@ -11,7 +11,7 @@ slug: /ios-preview-testing
|
|||
The first step to testing the preview flow is to launch an experiment
|
||||
to preview on the stage nimbus server (we should not using the production server for testing). Go to the [stage experimenter](https://stage.experimenter.nonprod.dataops.mozgcp.net/nimbus/) and create your experiment there.
|
||||
|
||||
Follow the first few instructions on [the preview docs](experiment-owners/preview.md) to get an experiment to preview. (note that the experiment should be set to target iOS, look at the [FAQ](#faq) section at the bottom)
|
||||
Follow the first few instructions on [the preview docs](preview.md) to get an experiment to preview. (note that the experiment should be set to target iOS, look at the [FAQ](#faq) section at the bottom)
|
||||
|
||||
## Pointing a local iOS build to the stage server
|
||||
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
id: launching
|
||||
title: Launching Your Experiment
|
||||
sidebar_label: Launching
|
||||
slug: /launching
|
||||
---
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
id: monitoring
|
||||
title: Monitoring Your Experiment
|
||||
sidebar_label: Monitoring
|
||||
slug: /monitoring
|
||||
---
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
---
|
||||
id: overview
|
||||
title: Workflow
|
||||
sidebar_label: Overview
|
||||
slug: overview
|
||||
---
|
||||
|
||||
## Experimentation Workflow
|
||||
|
||||
To view a recorded training for the Experimentation Workflow, [click here](https://mozilla.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=4d337632-a0bd-4be7-bee8-ae5b017134ae&start=0)
|
||||
|
||||
<div class="miro-container">
|
||||
<iframe class="responsive-iframe" src="https://miro.com/app/live-embed/uXjVOJ3IYRA=/?moveToViewport=-2130,380,5949,3025&embedAutoplay=true" frameBorder="0" scrolling="no"></iframe>
|
||||
</div>
|
||||
|
||||
## Find an existing experiment
|
||||
|
||||
If the experiment was run with Nimbus, check out the [directory of live and completed experiments](https://experimenter.services.mozilla.com/nimbus/). Normandy experiments can be found [here](https://experimenter.services.mozilla.com/).
|
||||
|
||||
If the experiment is not yet live, you will probably be able to find a relevant Jira ticket on this [Data Science Experimentation Collaboration Board](https://mozilla-hub.atlassian.net/jira/software/c/projects/DS/boards/258)
|
||||
|
|
@ -1,10 +1,10 @@
|
|||
---
|
||||
id: desktop-rollouts
|
||||
title: Desktop Rollouts
|
||||
slug: /desktop-rollouts
|
||||
id: rollouts
|
||||
title: Rollouts
|
||||
slug: /rollouts
|
||||
---
|
||||
|
||||
If you want to set configurations for a feature remotely _for non-experiment users_, you can do so with Nimbus Rollouts.
|
||||
If you want to set configurations for a feature remotely _for non-experiment users_, you can do so with a **Rollout**.
|
||||
|
||||
## FAQS
|
||||
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
id: testing
|
||||
title: Testing Experiments
|
||||
sidebar_label: Testing
|
||||
slug: /testing
|
||||
---
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
module.exports = {
|
||||
title: "Experimenter Docs",
|
||||
tagline: "Documentation hub for Experimenter",
|
||||
title: "Experimenter Docs",
|
||||
tagline: "Documentation souce for Data scientists, Product Managers and Engineers",
|
||||
url: "https://experimenter.info",
|
||||
baseUrl: "/",
|
||||
onBrokenLinks: "throw",
|
||||
|
|
@ -19,7 +19,7 @@ module.exports = {
|
|||
respectPrefersColorScheme: true,
|
||||
},
|
||||
navbar: {
|
||||
title: "Experimenter Docs",
|
||||
title: "Mozilla Experimentation and Feature Delivery",
|
||||
logo: {
|
||||
alt: "Experimenter Logo",
|
||||
src: "img/logo.svg",
|
||||
|
|
|
|||
265
sidebars.js
265
sidebars.js
|
|
@ -1,117 +1,168 @@
|
|||
module.exports = {
|
||||
sidebar: [
|
||||
{
|
||||
type: "category",
|
||||
label: "Getting Started",
|
||||
items: ["getting-started/intro", "getting-started/contributing"],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Experiment Owners",
|
||||
items: [
|
||||
"experiment-owners/access",
|
||||
"experiment-owners/custom-audiences",
|
||||
"experiment-owners/preview",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "The Nimbus Cookbook",
|
||||
items: [
|
||||
{
|
||||
type: "link",
|
||||
label: "Feature Manifest Language Specification",
|
||||
href: "/fml-spec",
|
||||
},
|
||||
"cookbook/fml/index",
|
||||
],
|
||||
},
|
||||
{
|
||||
{
|
||||
type: "category",
|
||||
label: "Welcome",
|
||||
items: [
|
||||
"homepage/intro",
|
||||
"homepage/finding-help",
|
||||
"homepage/capabilities",
|
||||
]
|
||||
},
|
||||
{
|
||||
"Getting Started":
|
||||
[
|
||||
"getting-started/access",
|
||||
{
|
||||
type: "category",
|
||||
label: "Data Scientists",
|
||||
items: [
|
||||
"data-scientists/data-scientists-root",
|
||||
"data-scientists/telemetry",
|
||||
"data-scientists/bucketing",
|
||||
"data-scientists/validating-experiments",
|
||||
],
|
||||
},
|
||||
|
||||
{
|
||||
type: "category",
|
||||
label: "Firefox for Android",
|
||||
label: "Engineers",
|
||||
items: [
|
||||
"fenix-experiments-getting-started",
|
||||
{
|
||||
type: "link",
|
||||
label: "Feature Variables API",
|
||||
href: "/feature-variables-and-me",
|
||||
},
|
||||
"android-frontend-testing",
|
||||
{
|
||||
type: "link",
|
||||
label: "Adding custom targeting attributes",
|
||||
href: "/android-custom-targeting",
|
||||
},
|
||||
{
|
||||
type: "link",
|
||||
label: "Feature Manifest Language Specification",
|
||||
href: "/fml-spec",
|
||||
},
|
||||
"getting-started/for-engineers",
|
||||
"getting-started/desktop-migration-guide",
|
||||
"getting-started/desktop-feature-api",
|
||||
"deep-dives/mobile/mobile-feature-api"
|
||||
],
|
||||
},
|
||||
"getting-started/for-product",
|
||||
"getting-started/for-leadership"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Experimentation Workflow": [
|
||||
"workflow/overview",
|
||||
"workflow/designing",
|
||||
{
|
||||
"type": "category",
|
||||
"label": "Implementing",
|
||||
"items": [
|
||||
"workflow/feature-definition",
|
||||
"workflow/a-b-experiments",
|
||||
"workflow/custom-audiences",
|
||||
"workflow/android-custom-targeting",
|
||||
"workflow/ios-custom-targeting",
|
||||
{
|
||||
"type": "category",
|
||||
"label": "Messaging Experiments",
|
||||
items: [
|
||||
"messaging/messaging-surfaces",
|
||||
"messaging/display-logic",
|
||||
{
|
||||
type: "link",
|
||||
label: "Triggers & User Actions",
|
||||
href: "https://firefox-source-docs.mozilla.org/toolkit/components/messaging-system/docs/",
|
||||
},
|
||||
"messaging/frequency-cap",
|
||||
{
|
||||
type: "link",
|
||||
label: "Available Targeting",
|
||||
href: "https://firefox-source-docs.mozilla.org/browser/components/newtab/content-src/asrouter/docs/targeting-attributes.html",
|
||||
},
|
||||
"messaging/mobile-messaging"
|
||||
]
|
||||
},
|
||||
],
|
||||
},
|
||||
"workflow/configuring",
|
||||
{
|
||||
"type": "category",
|
||||
"label": "Testing",
|
||||
"items": [
|
||||
"workflow/testing",
|
||||
"workflow/preview",
|
||||
"workflow/ios-preview-testing",
|
||||
"workflow/android-frontend-testing",
|
||||
"workflow/desktop-feature-api-testing"
|
||||
]
|
||||
},
|
||||
"workflow/launching",
|
||||
"workflow/monitoring",
|
||||
"workflow/analyzing",
|
||||
"workflow/ending"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Deep Dives": [
|
||||
{
|
||||
type: "category",
|
||||
label: "Jetstream",
|
||||
items: [
|
||||
"jetstream/jetstream",
|
||||
"jetstream/metrics",
|
||||
"jetstream/statistics",
|
||||
"jetstream/outcomes",
|
||||
"jetstream/configuration",
|
||||
"jetstream/data-products",
|
||||
"jetstream/operations",
|
||||
"jetstream/troubleshooting",
|
||||
{
|
||||
type: "link",
|
||||
label: "GitHub Repo",
|
||||
href: "https://github.com/mozilla/jetstream",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
"type": "category",
|
||||
"label": "Data topics",
|
||||
"items": [
|
||||
{
|
||||
type: 'autogenerated',
|
||||
dirName: "deep-dives/data", // Source folder to generate the sidebar slice from (relative to docs)
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "category",
|
||||
"label": "Desktop topics",
|
||||
"items": [
|
||||
{
|
||||
type: 'autogenerated',
|
||||
dirName: "deep-dives/desktop", // Source folder to generate the sidebar slice from (relative to docs)
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "category",
|
||||
"label": "Mobile topics",
|
||||
"items": [
|
||||
{
|
||||
type: 'autogenerated',
|
||||
dirName: "deep-dives/mobile", // Source folder to generate the sidebar slice from (relative to docs)
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "category",
|
||||
"label": "Specifications",
|
||||
"items": [
|
||||
{
|
||||
type: 'autogenerated',
|
||||
dirName: "deep-dives/specifications", // Source folder to generate the sidebar slice from (relative to docs)
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Experimentation Cookbook",
|
||||
items: [
|
||||
"cookbook/fml/index",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Firefox for iOS",
|
||||
label: "Additional Links",
|
||||
items: [
|
||||
{
|
||||
type: "link",
|
||||
label: "Feature Variables API",
|
||||
href: "/feature-variables-and-me",
|
||||
},
|
||||
{
|
||||
type: "link",
|
||||
label: "Adding custom targeting attributes",
|
||||
href: "/ios-custom-targeting",
|
||||
},
|
||||
"ios-preview-testing",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Firefox for Desktop",
|
||||
items: [
|
||||
"desktop-feature-api",
|
||||
"desktop-migration-guide",
|
||||
"desktop-rollouts",
|
||||
"desktop-feature-api-testing",
|
||||
"desktop-targeting-debug",
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Jetstream",
|
||||
items: [
|
||||
"jetstream/jetstream",
|
||||
"jetstream/metrics",
|
||||
"jetstream/statistics",
|
||||
"jetstream/outcomes",
|
||||
"jetstream/configuration",
|
||||
"jetstream/data-products",
|
||||
"jetstream/operations",
|
||||
"jetstream/troubleshooting",
|
||||
{
|
||||
type: "link",
|
||||
label: "GitHub Repo",
|
||||
href: "https://github.com/mozilla/jetstream",
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Core Nimbus Team",
|
||||
items: [
|
||||
"client-sdk-states-and-lifecycle",
|
||||
"integration-tests",
|
||||
{
|
||||
type: "link",
|
||||
|
|
@ -140,25 +191,5 @@ module.exports = {
|
|||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
type: "category",
|
||||
label: "Messaging System",
|
||||
items: [
|
||||
"messaging/messaging-surfaces",
|
||||
"messaging/display-logic",
|
||||
{
|
||||
type: "link",
|
||||
label: "Triggers & User Actions",
|
||||
href: "https://firefox-source-docs.mozilla.org/toolkit/components/messaging-system/docs/",
|
||||
},
|
||||
"messaging/frequency-cap",
|
||||
{
|
||||
type: "link",
|
||||
label: "Available Targeting",
|
||||
href: "https://firefox-source-docs.mozilla.org/browser/components/newtab/content-src/asrouter/docs/targeting-attributes.html",
|
||||
},
|
||||
"messaging/mobile-messaging"
|
||||
],
|
||||
},
|
||||
],
|
||||
]
|
||||
};
|
||||
|
|
|
|||
23
yarn.lock
23
yarn.lock
|
|
@ -2686,25 +2686,10 @@ caniuse-api@^3.0.0:
|
|||
lodash.memoize "^4.1.2"
|
||||
lodash.uniq "^4.5.0"
|
||||
|
||||
caniuse-lite@^1.0.0, caniuse-lite@^1.0.30001181:
|
||||
version "1.0.30001192"
|
||||
resolved "https://registry.yarnpkg.com/caniuse-lite/-/caniuse-lite-1.0.30001192.tgz#b848ebc0ab230cf313d194a4775a30155d50ae40"
|
||||
integrity sha512-63OrUnwJj5T1rUmoyqYTdRWBqFFxZFlyZnRRjDR8NSUQFB6A+j/uBORU/SyJ5WzDLg4SPiZH40hQCBNdZ/jmAw==
|
||||
|
||||
caniuse-lite@^1.0.30001125:
|
||||
version "1.0.30001200"
|
||||
resolved "https://registry.yarnpkg.com/caniuse-lite/-/caniuse-lite-1.0.30001200.tgz#25435af6ba907c2a9c86d21ce84950d4824e6620"
|
||||
integrity sha512-ic/jXfa6tgiPBAISWk16jRI2q8YfjxHnSG7ddSL1ptrIP8Uy11SayFrjXRAk3NumHpDb21fdTkbTxb/hOrFrnQ==
|
||||
|
||||
caniuse-lite@^1.0.30001219:
|
||||
version "1.0.30001220"
|
||||
resolved "https://registry.yarnpkg.com/caniuse-lite/-/caniuse-lite-1.0.30001220.tgz#c080e1c8eefb99f6cc9685da6313840bdbaf4c36"
|
||||
integrity sha512-pjC2T4DIDyGAKTL4dMvGUQaMUHRmhvPpAgNNTa14jaBWHu+bLQgvpFqElxh9L4829Fdx0PlKiMp3wnYldRtECA==
|
||||
|
||||
caniuse-lite@^1.0.30001230:
|
||||
version "1.0.30001242"
|
||||
resolved "https://registry.yarnpkg.com/caniuse-lite/-/caniuse-lite-1.0.30001242.tgz#04201627abcd60dc89211f22cbe2347306cda46b"
|
||||
integrity sha512-KvNuZ/duufelMB3w2xtf9gEWCSxJwUgoxOx5b6ScLXC4kPc9xsczUVCPrQU26j5kOsHM4pSUL54tAZt5THQKug==
|
||||
caniuse-lite@^1.0.0, caniuse-lite@^1.0.30001125, caniuse-lite@^1.0.30001181, caniuse-lite@^1.0.30001219, caniuse-lite@^1.0.30001230:
|
||||
version "1.0.30001338"
|
||||
resolved "https://registry.npmjs.org/caniuse-lite/-/caniuse-lite-1.0.30001338.tgz"
|
||||
integrity sha512-1gLHWyfVoRDsHieO+CaeYe7jSo/MT7D7lhaXUiwwbuR5BwQxORs0f1tAwUSQr3YbxRXJvxHM/PA5FfPQRnsPeQ==
|
||||
|
||||
ccount@^1.0.0, ccount@^1.0.3:
|
||||
version "1.1.0"
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче