mozilla
/
experimenter
зеркало из https://github.com/mozilla/experimenter.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

chore(nimbus): only run integration tests for a single app (#9762) 

Because

* We run many integration tests against many applications for coverage
* As we add more applications, we don't want to disrupt the existing
tests
* Some integration tests were specifying applications by excluding other
applications becuase IOS and FOCUS_IOS would both match the same string
IOS
* This was resulting in some test runs running against multiple
applications which is slowing down CI and PRs

This commit

* Renames applications to have unique names
* Updates all the config references to use those names

Fixes #9718
This commit is contained in:
Jared Lockhart 2023-11-23 15:48:41 -05:00 коммит произвёл GitHub
Родитель 0def0132ae
Коммит 3985384ade
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
15 изменённых файлов: 150 добавлений и 299 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

14
.circleci/config.yml
Просмотреть файл

@ -214,7 +214,7 @@ jobs:
working_directory: ~/experimenter
environment:
FIREFOX_VERSION: nimbus-firefox-release
PYTEST_ARGS: -k FENIX -m remote_settings --reruns 1
PYTEST_ARGS: -k FIREFOX_FENIX -m remote_settings --reruns 1
steps:
- checkout
- check_file_paths:
@ -235,7 +235,7 @@ jobs:
working_directory: ~/experimenter
environment:
FIREFOX_VERSION: nimbus-firefox-release
PYTEST_ARGS: -k 'not (FOCUS_IOS or DESKTOP or FENIX or FOCUS_ANDROID)' -m remote_settings --reruns 1
PYTEST_ARGS: -k FIREFOX_IOS -m remote_settings --reruns 1
steps:
- checkout
- check_file_paths:
@ -321,7 +321,7 @@ jobs:
working_directory: ~/experimenter
environment:
FIREFOX_VERSION: nimbus-firefox-release
PYTEST_ARGS: -k FIREFOX_DESKTOP -m nimbus_ui -n 2 --reruns 1
PYTEST_ARGS: -m nimbus_ui -n 2 --reruns 1
steps:
- checkout
- check_file_paths:
@ -340,6 +340,8 @@ jobs:
docker_layer_caching: true
resource_class: medium
working_directory: ~/experimenter
environment:
PYTEST_ARGS: -k FIREFOX_FENIX
steps:
- checkout
- check_file_paths:
@ -353,7 +355,7 @@ jobs:
- store_artifacts:
path: ~/experimenter/experimenter/tests/integration/test-reports/report.htm
integration_nimbus_demo_app:
integration_nimbus_cirrus:
machine:
image: ubuntu-2004:2023.10.1
docker_layer_caching: true
@ -361,7 +363,7 @@ jobs:
working_directory: ~/experimenter
environment:
FIREFOX_VERSION: nimbus-firefox-release
PYTEST_ARGS: -k 'not (FOCUS_IOS or DESKTOP or FENIX or FOCUS_ANDROID or IOS)' -m demo_app --reruns 1
PYTEST_ARGS: -k DEMO_APP -m cirrus_enrollment --reruns 1
steps:
- checkout
- check_file_paths:
@ -729,7 +731,7 @@ workflows:
branches:
ignore:
- main
- integration_nimbus_demo_app:
- integration_nimbus_cirrus:
name: Test Demo app with Cirrus
filters:
branches:

84
experimenter/tests/integration/nimbus/conftest.py
Просмотреть файл

@ -27,55 +27,55 @@ from nimbus.pages.experimenter.home import HomePage
from nimbus.utils import helpers
APPLICATION_FEATURE_IDS = {
BaseExperimentApplications.FIREFOX_DESKTOP: helpers.get_feature_id_as_string(
BaseExperimentApplications.FIREFOX_DESKTOP.value: helpers.get_feature_id_as_string(
"no-feature-firefox-desktop", BaseExperimentApplications.FIREFOX_DESKTOP.value
),
BaseExperimentApplications.FENIX: helpers.get_feature_id_as_string(
"no-feature-fenix", BaseExperimentApplications.FENIX.value
BaseExperimentApplications.FIREFOX_FENIX.value: helpers.get_feature_id_as_string(
"no-feature-fenix", BaseExperimentApplications.FIREFOX_FENIX.value
),
BaseExperimentApplications.IOS: helpers.get_feature_id_as_string(
"no-feature-ios", BaseExperimentApplications.IOS.value
BaseExperimentApplications.FIREFOX_IOS.value: helpers.get_feature_id_as_string(
"no-feature-ios", BaseExperimentApplications.FIREFOX_IOS.value
),
BaseExperimentApplications.FOCUS_ANDROID: helpers.get_feature_id_as_string(
BaseExperimentApplications.FOCUS_ANDROID.value: helpers.get_feature_id_as_string(
"no-feature-focus-android", BaseExperimentApplications.FOCUS_ANDROID.value
),
BaseExperimentApplications.FOCUS_IOS: helpers.get_feature_id_as_string(
BaseExperimentApplications.FOCUS_IOS.value: helpers.get_feature_id_as_string(
"no-feature-focus-ios", BaseExperimentApplications.FOCUS_IOS.value
),
BaseExperimentApplications.DEMO_APP: helpers.get_feature_id_as_string(
BaseExperimentApplications.DEMO_APP.value: helpers.get_feature_id_as_string(
"example-feature", BaseExperimentApplications.DEMO_APP.value
),
}
APPLICATION_KINTO_REVIEW_PATH = {
BaseExperimentApplications.FIREFOX_DESKTOP: (
BaseExperimentApplications.FIREFOX_DESKTOP.value: (
"#/buckets/main-workspace/collections/nimbus-desktop-experiments/simple-review"
),
BaseExperimentApplications.FENIX: (
BaseExperimentApplications.FIREFOX_FENIX.value: (
"#/buckets/main-workspace/collections/nimbus-mobile-experiments/simple-review"
),
BaseExperimentApplications.IOS: (
BaseExperimentApplications.FIREFOX_IOS.value: (
"#/buckets/main-workspace/collections/nimbus-mobile-experiments/simple-review"
),
BaseExperimentApplications.FOCUS_ANDROID: (
BaseExperimentApplications.FOCUS_ANDROID.value: (
"#/buckets/main-workspace/collections/nimbus-mobile-experiments/simple-review"
),
BaseExperimentApplications.FOCUS_IOS: (
BaseExperimentApplications.FOCUS_IOS.value: (
"#/buckets/main-workspace/collections/nimbus-mobile-experiments/simple-review"
),
BaseExperimentApplications.DEMO_APP: (
BaseExperimentApplications.DEMO_APP.value: (
"#/buckets/main-workspace/collections/nimbus-web-experiments/simple-review"
),
}
APPLICATION_KINTO_COLLECTION = {
"DESKTOP": KINTO_COLLECTION_DESKTOP,
"FENIX": KINTO_COLLECTION_MOBILE,
"IOS": KINTO_COLLECTION_MOBILE,
"FOCUS_ANDROID": KINTO_COLLECTION_MOBILE,
"FOCUS_IOS": KINTO_COLLECTION_MOBILE,
"DEMO_APP": KINTO_COLLECTION_WEB,
BaseExperimentApplications.FIREFOX_DESKTOP.value: KINTO_COLLECTION_DESKTOP,
BaseExperimentApplications.FIREFOX_FENIX.value: KINTO_COLLECTION_MOBILE,
BaseExperimentApplications.FIREFOX_IOS.value: KINTO_COLLECTION_MOBILE,
BaseExperimentApplications.FOCUS_ANDROID.value: KINTO_COLLECTION_MOBILE,
BaseExperimentApplications.FOCUS_IOS.value: KINTO_COLLECTION_MOBILE,
BaseExperimentApplications.DEMO_APP.value: KINTO_COLLECTION_WEB,
}
@ -128,17 +128,13 @@ def selenium(selenium, experiment_name, kinto_client, base_url, slugify):
@pytest.fixture(
# Use all applications as available parameters in parallel_pytest_args.txt
# Use all applications as available parameters in circle config
params=list(BaseExperimentApplications),
ids=[application.name for application in BaseExperimentApplications],
autouse=True,
)
def application(request):
"""
Returns the current application to use for testing
Will also parametrize the tests
"""
return request.param
return request.param.value
@pytest.fixture(scope="session", autouse=True)
@ -154,7 +150,7 @@ def _verify_url(request, base_url):
@pytest.fixture
def kinto_client(default_data):
return KintoClient(APPLICATION_KINTO_COLLECTION[default_data.application.value])
return KintoClient(APPLICATION_KINTO_COLLECTION[default_data.application])
@pytest.fixture
@ -199,27 +195,18 @@ def default_data(application, experiment_name, load_experiment_outcomes):
feature_config_id = APPLICATION_FEATURE_IDS[application]
outcomes = {
"firefox_desktop": BaseExperimentMetricsDataClass(
BaseExperimentApplications.FIREFOX_DESKTOP.value: BaseExperimentMetricsDataClass(
primary_outcomes=[load_experiment_outcomes["firefox_desktop"][0]],
secondary_outcomes=[load_experiment_outcomes["firefox_desktop"][1]],
),
"fenix": BaseExperimentMetricsDataClass(
BaseExperimentApplications.FIREFOX_FENIX.value: BaseExperimentMetricsDataClass(
primary_outcomes=[load_experiment_outcomes["fenix"][0]],
secondary_outcomes=[load_experiment_outcomes["fenix"][1]],
),
"ios": BaseExperimentMetricsDataClass(
BaseExperimentApplications.FIREFOX_IOS.value: BaseExperimentMetricsDataClass(
primary_outcomes=[load_experiment_outcomes["firefox_ios"][0]],
secondary_outcomes=[load_experiment_outcomes["firefox_ios"][1]],
),
"focus_ios": BaseExperimentMetricsDataClass(
primary_outcomes=[], secondary_outcomes=[]
),
"focus_android": BaseExperimentMetricsDataClass(
primary_outcomes=[], secondary_outcomes=[]
),
"demo_app": BaseExperimentMetricsDataClass(
primary_outcomes=[], secondary_outcomes=[]
),
}
return BaseExperimentDataClass(
@ -238,7 +225,13 @@ def default_data(application, experiment_name, load_experiment_outcomes):
description="treatment description",
),
],
metrics=outcomes[str(application).lower().rsplit(".")[-1]],
metrics=outcomes.get(
application,
BaseExperimentMetricsDataClass(
primary_outcomes=[],
secondary_outcomes=[],
),
),
audience=BaseExperimentAudienceDataClass(
channel=BaseExperimentAudienceChannels.RELEASE,
min_version=106,
@ -265,7 +258,7 @@ def create_experiment(base_url, default_data):
experiment = home.create_new_button()
experiment.public_name = default_data.public_name
experiment.hypothesis = default_data.hypothesis
experiment.application = default_data.application.value
experiment.application = default_data.application
# Fill Overview Page
overview = experiment.save_and_continue()
@ -313,12 +306,15 @@ def create_experiment(base_url, default_data):
audience.targeting = "no_targeting"
audience.percentage = "100"
audience.expected_clients = default_data.audience.expected_clients
if default_data.application.value != "DEMO_APP":
if default_data.application != BaseExperimentApplications.DEMO_APP.value:
audience.min_version = default_data.audience.min_version
audience.percentage = default_data.audience.percentage
audience.targeting = default_data.audience.targeting
audience.countries = ["Canada"]
if default_data.application.value != "DESKTOP":
if (
default_data.application
!= BaseExperimentApplications.FIREFOX_DESKTOP.value
):
audience.languages = ["English"]
else:
audience.locales = ["English (US)"]
@ -353,7 +349,7 @@ def trigger_experiment_loader(selenium):
def fixture_experiment_default_data():
return {
"hypothesis": "Test Hypothesis",
"application": "DESKTOP",
"application": BaseExperimentApplications.FIREFOX_DESKTOP.value,
"changelogMessage": "test updates",
"targetingConfigSlug": "no_targeting",
"publicDescription": "Some sort of Fancy Words",

4
experimenter/tests/integration/nimbus/models/base_dataclass.py
Просмотреть файл

@ -5,8 +5,8 @@ from typing import List, Optional
class BaseExperimentApplications(Enum):
FIREFOX_DESKTOP = "DESKTOP"
FENIX = "FENIX"
IOS = "IOS"
FIREFOX_FENIX = "FENIX"
FIREFOX_IOS = "IOS"
FOCUS_ANDROID = "FOCUS_ANDROID"
FOCUS_IOS = "FOCUS_IOS"
DEMO_APP = "DEMO_APP"

10
experimenter/tests/integration/nimbus/pages/experimenter/audience.py
Просмотреть файл

@ -131,8 +131,8 @@ class AudiencePage(ExperimenterBase):
@languages.setter
def languages(self, text=None):
el = self.wait_for_and_find_element(*self._languages_input_locator)
for _ in text:
el.send_keys(f"{_}")
for char in text:
el.send_keys(char)
el.send_keys(Keys.ENTER)
@property
@ -153,10 +153,8 @@ class AudiencePage(ExperimenterBase):
@proposed_release_date.setter
def proposed_release_date(self, text=None):
el = self.wait_for_and_find_element(*self._languages_input_locator)
for _ in text:
el.send_keys(f"{_}")
el.send_keys(Keys.TAB)
el = self.wait_for_and_find_element(*self._release_date_locator)
el.send_keys(text)
def wait_until_release_date_not_found(self):
self.wait.until_not(

5
experimenter/tests/integration/nimbus/pages/experimenter/new_experiment.py
Просмотреть файл

@ -1,6 +1,9 @@
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import Select
from nimbus.models.base_dataclass import (
BaseExperimentApplications,
)
from nimbus.pages.experimenter.base import ExperimenterBase
from nimbus.pages.experimenter.overview import OverviewPage
@ -41,7 +44,7 @@ class NewExperiment(ExperimenterBase):
return self.wait_for_and_find_element(*self._application_select_locator).text
@application.setter
def application(self, app="DESKTOP"):
def application(self, app=BaseExperimentApplications.FIREFOX_DESKTOP.value):
el = self.wait_for_and_find_element(*self._application_select_locator)
select = Select(el)
select.select_by_value(app)

2
experimenter/tests/integration/nimbus/pages/experimenter/overview.py
Просмотреть файл

@ -49,7 +49,7 @@ class OverviewPage(ExperimenterBase):
@public_description.setter
def public_description(self, text=None):
name = self.wait_for_and_find_element(*self._public_description_locator)
name.send_keys(f"{text}")
name.send_keys(text)
def select_risk_brand_false(self):
el = self.wait_for_and_find_element(*self._risk_brand_locator)

10
experimenter/tests/integration/nimbus/pages/experimenter/summary.py
Просмотреть файл

@ -374,14 +374,14 @@ class SummaryPage(ExperimenterBase):
message="Summary Page: could not find release date on timeline",
)
def wait_until_audience_section_release_date_not_found(self):
def wait_until_release_date_not_found(self):
self.wait.until_not(
EC.presence_of_element_located(self._audience_proposed_release_date_locator),
message="Summary Page: could not find release date in Audience section",
message="Audience Page: could not find release date",
)
def wait_until_timeline_release_date_not_found(self):
def wait_until_first_run_not_found(self):
self.wait.until_not(
EC.presence_of_element_located(self._audience_proposed_release_date_locator),
message="Summary Page: could not find release date on timeline",
EC.presence_of_element_located(self._audience_is_first_run_locator),
message="Audience Page: could not find first run checkbox",
)

8
experimenter/tests/integration/nimbus/test_demo_app_integration.py → experimenter/tests/integration/nimbus/test_cirrus_integration.py
Просмотреть файл

@ -9,8 +9,8 @@ def navigate_to(selenium):
selenium.get(demo_app_url)
@pytest.mark.demo_app
def test_create_new_rollout_approve_remote_settings_demo_app(
@pytest.mark.cirrus_enrollment
def test_create_new_rollout_approve_remote_settings_cirrus(
selenium,
experiment_url,
create_experiment,
@ -70,8 +70,8 @@ def test_create_new_rollout_approve_remote_settings_demo_app(
assert result_text_element.is_displayed()
@pytest.mark.demo_app
def test_create_new_experiment_approve_remote_settings_demo_app(
@pytest.mark.cirrus_enrollment
def test_create_new_experiment_approve_remote_settings_cirrus(
selenium,
experiment_url,
create_experiment,

8
experimenter/tests/integration/nimbus/test_desktop_client_integrations.py
Просмотреть файл

@ -4,6 +4,7 @@ from urllib.parse import urljoin
import pytest
import requests
from nimbus.models.base_dataclass import BaseExperimentApplications
from nimbus.pages.browser import AboutConfig
from nimbus.pages.experimenter.summary import SummaryPage
from nimbus.utils import helpers
@ -76,7 +77,6 @@ def test_check_telemetry_enrollment_unenrollment(
experiment_slug = str(slugify(experiment_name))
data = {
"hypothesis": "Test Hypothesis",
"application": "DESKTOP",
"changelogMessage": "test updates",
"targetingConfigSlug": targeting,
"publicDescription": "Some sort of Fancy Words",
@ -102,7 +102,7 @@ def test_check_telemetry_enrollment_unenrollment(
}
helpers.create_experiment(
experiment_slug,
"desktop",
BaseExperimentApplications.FIREFOX_DESKTOP.value,
targeting,
data,
)
@ -187,7 +187,7 @@ def test_check_telemetry_pref_flip(
experiment_default_data["treatmentBranches"] = []
helpers.create_experiment(
experiment_slug,
"desktop",
BaseExperimentApplications.FIREFOX_DESKTOP.value,
targeting,
experiment_default_data,
)
@ -282,7 +282,7 @@ def test_check_telemetry_sticky_targeting(
experiment_default_data["isSticky"] = True
helpers.create_experiment(
experiment_slug,
"desktop",
BaseExperimentApplications.FIREFOX_DESKTOP.value,
targeting_config_slug,
experiment_default_data,
)

7
experimenter/tests/integration/nimbus/test_desktop_targeting.py
Просмотреть файл

@ -2,6 +2,7 @@ import json
import pytest
from nimbus.models.base_dataclass import BaseExperimentApplications
from nimbus.pages.browser import Browser
from nimbus.utils import helpers
@ -22,7 +23,7 @@ def test_check_advanced_targeting(
experiment_slug = str(slugify(experiment_name))
data = {
"hypothesis": "Test Hypothesis",
"application": "DESKTOP",
"application": BaseExperimentApplications.FIREFOX_DESKTOP.value,
"changelogMessage": "test updates",
"targetingConfigSlug": targeting,
"publicDescription": "Some sort of Fancy Words",
@ -59,7 +60,7 @@ def test_check_advanced_targeting(
}
helpers.create_experiment(
experiment_slug,
"desktop",
BaseExperimentApplications.FIREFOX_DESKTOP.value,
targeting_config_slug,
data,
)
@ -113,7 +114,7 @@ def test_check_audience_targeting(
experiment_default_data.update(audience_field)
helpers.create_experiment(
experiment_slug,
"desktop",
BaseExperimentApplications.FIREFOX_DESKTOP.value,
"no_targeting",
experiment_default_data,
)

229
experimenter/tests/integration/nimbus/test_experimenter_ui.py
Просмотреть файл

@ -3,9 +3,18 @@ from urllib.parse import urljoin
import pytest
from nimbus.models.base_dataclass import (
BaseExperimentApplications,
)
from nimbus.pages.experimenter.home import HomePage
from nimbus.pages.experimenter.summary import SummaryPage
from nimbus.utils import helpers
MOBILE_APPS = [
BaseExperimentApplications.FIREFOX_FENIX.value,
BaseExperimentApplications.FIREFOX_IOS.value,
BaseExperimentApplications.FOCUS_ANDROID.value,
BaseExperimentApplications.FOCUS_IOS.value,
]
@pytest.mark.nimbus_ui
@ -86,16 +95,6 @@ def test_branch_screenshot(
assert summary.branch_screenshot_image is not None
@pytest.mark.nimbus_ui
def test_create_new_experiment_timeout_remote_settings(
selenium,
create_experiment,
):
summary = create_experiment(selenium)
summary.launch_and_approve()
summary.wait_for_timeout_alert()
@pytest.mark.nimbus_ui
def test_every_form_page_can_be_resaved(
selenium,
@ -111,56 +110,32 @@ def test_every_form_page_can_be_resaved(
@pytest.mark.nimbus_ui
@pytest.mark.skipif(
any(app in os.getenv("PYTEST_ARGS") for app in ["FIREFOX_DESKTOP", "DEMO_APP"]),
reason="Only run for mobile applications",
)
def test_first_run_release_date(
def test_first_run_release_date_visible_for_mobile(
base_url,
selenium,
kinto_client,
slugify,
experiment_name,
application,
create_experiment,
):
experiment_slug = str(slugify(experiment_name))
targeting = helpers.load_targeting_configs(app=application)[0]
data = {
"hypothesis": "Test Hypothesis",
"application": application,
"changelogMessage": "test updates",
"targetingConfigSlug": targeting,
"publicDescription": "Some sort of Fancy Words",
"riskRevenue": False,
"riskPartnerRelated": False,
"riskBrand": False,
"featureConfigIds": [2],
"referenceBranch": {
"description": "reference branch",
"name": "Branch 1",
"ratio": 50,
"featureValues": [
{
"featureConfig": "1",
"value": "{}",
},
],
},
"treatmentBranches": [],
"firefoxMinVersion": "FIREFOX_120",
"populationPercent": "100",
"totalEnrolledClients": 55,
"isFirstRun": True,
"proposedReleaseDate": "2023-12-12",
}
helpers.create_experiment(
experiment_slug,
app=application,
targeting=targeting,
data=data,
)
if application not in MOBILE_APPS:
pytest.skip(f"Skipping for {application}")
summary = create_experiment(selenium)
experiment_slug = summary.experiment_slug
audience = summary.navigate_to_audience()
audience.make_first_run()
audience.proposed_release_date = "2023-12-12"
assert audience.is_first_run
assert audience.proposed_release_date == "2023-12-12"
audience.save_and_continue()
summary = SummaryPage(selenium, urljoin(base_url, experiment_slug)).open()
assert summary.proposed_release_date == "2023-12-12"
summary.launch_and_approve()
kinto_client.approve()
@ -173,116 +148,28 @@ def test_first_run_release_date(
@pytest.mark.nimbus_ui
@pytest.mark.skipif(
any(app in os.getenv("PYTEST_ARGS") for app in ["FIREFOX_DESKTOP", "DEMO_APP"]),
reason="Only run for mobile applications",
)
def test_audience_page_release_date(
base_url,
selenium,
slugify,
experiment_name,
):
application = "FENIX"
experiment_slug = str(slugify(experiment_name))
targeting = helpers.load_targeting_configs(app=application)[0]
data = {
"hypothesis": "Test Hypothesis",
"application": application,
"changelogMessage": "test updates",
"targetingConfigSlug": targeting,
"publicDescription": "Some sort of Fancy Words",
"riskRevenue": False,
"riskPartnerRelated": False,
"riskBrand": False,
"featureConfigIds": [2],
"referenceBranch": {
"description": "reference branch",
"name": "Branch 1",
"ratio": 50,
"featureValues": [
{
"featureConfig": "1",
"value": "{}",
},
],
},
"treatmentBranches": [],
"firefoxMinVersion": "FIREFOX_120",
"populationPercent": "100",
"totalEnrolledClients": 55,
"isFirstRun": True,
"proposedReleaseDate": "2023-12-12",
}
helpers.create_experiment(
experiment_slug,
app=application,
targeting=targeting,
data=data,
)
summary = SummaryPage(selenium, urljoin(base_url, experiment_slug)).open()
audience = summary.navigate_to_audience()
assert audience.is_first_run
assert audience.proposed_release_date == "2023-12-12"
audience.make_first_run()
summary = audience.save_and_continue()
assert summary.proposed_release_date == "Not set"
@pytest.mark.nimbus_ui
@pytest.mark.skipif(
any(app in os.getenv("PYTEST_ARGS") for app in ["FIREFOX_DESKTOP", "DEMO_APP"]),
reason="Only run for mobile applications",
)
def test_summary_timeline_release_date(
def test_first_run_release_date_not_visible_for_non_mobile(
base_url,
selenium,
kinto_client,
slugify,
experiment_name,
application,
create_experiment,
):
application = "FENIX"
experiment_slug = str(slugify(experiment_name))
targeting = helpers.load_targeting_configs(app=application)[0]
data = {
"hypothesis": "Test Hypothesis",
"application": application,
"changelogMessage": "test updates",
"targetingConfigSlug": targeting,
"publicDescription": "Some sort of Fancy Words",
"riskRevenue": False,
"riskPartnerRelated": False,
"riskBrand": False,
"featureConfigIds": [2],
"referenceBranch": {
"description": "reference branch",
"name": "Branch 1",
"ratio": 50,
"featureValues": [
{
"featureConfig": "1",
"value": "{}",
},
],
},
"treatmentBranches": [],
"firefoxMinVersion": "FIREFOX_120",
"populationPercent": "100",
"totalEnrolledClients": 55,
"isFirstRun": True,
"proposedReleaseDate": "2023-12-12",
}
helpers.create_experiment(
experiment_slug,
app=application,
targeting=targeting,
data=data,
)
if application in [
*MOBILE_APPS,
BaseExperimentApplications.DEMO_APP.value, # TODO: Remove DEMO_APP with #9817
]:
pytest.skip(f"Skipping for {application}")
summary = SummaryPage(selenium, urljoin(base_url, experiment_slug)).open()
summary = create_experiment(selenium)
experiment_slug = summary.experiment_slug
audience = summary.navigate_to_audience()
audience.wait_until_release_date_not_found()
audience.wait_until_first_run_not_found()
summary = audience.navigate_to_summary()
summary.launch_and_approve()
kinto_client.approve()
@ -291,25 +178,5 @@ def test_summary_timeline_release_date(
summary.wait_for_live_status()
summary.wait_for_timeline_visible()
summary.wait_for_release_date()
@pytest.mark.nimbus_ui
@pytest.mark.skipif(
any(
app in os.getenv("PYTEST_ARGS")
for app in ["FOCUS_IOS", "IOS", "FENIX", "FOCUS_ANDROID", "DEMO_APP"]
),
reason="Only run for non-mobile applications and non desktop",
)
def test_audience_page_release_date_not_visible(
selenium,
create_experiment,
):
summary = create_experiment(selenium)
audience = summary.navigate_to_audience()
release_date = audience.wait_until_release_date_not_found()
assert not release_date
first_run = audience.wait_until_first_run_not_found()
assert not first_run
summary.wait_until_release_date_not_found()
summary.wait_until_first_run_not_found()

5
experimenter/tests/integration/nimbus/test_mobile_targeting.py
Просмотреть файл

@ -5,6 +5,7 @@ import pytest
from nimbus.jexl import collect_exprs
from nimbus.models.base_app_context_dataclass import BaseAppContextDataClass
from nimbus.models.base_dataclass import BaseExperimentApplications
from nimbus.utils import helpers
nimbus = pytest.importorskip("nimbus_rust")
@ -66,7 +67,6 @@ def test_check_mobile_targeting(
experiment_name,
targeting,
):
# The context fixtures can only contain strings or null
context["language"] = context["language"][:2] # strip region
# This context dictionary supports non string values
@ -89,7 +89,7 @@ def test_check_mobile_targeting(
experiment_slug = str(slugify(experiment_name))
helpers.create_basic_experiment(
experiment_slug,
context["app_name"],
BaseExperimentApplications.FIREFOX_FENIX.value,
targeting,
languages=context["language"],
)
@ -97,7 +97,6 @@ def test_check_mobile_targeting(
expression = data["data"]["experimentBySlug"]["jexlTargetingExpression"]
for sub_expr in collect_exprs(expression):
# The evaluator will throw if it detects a syntax error, a comparison type
# mismatch, or an undefined variable
try:

24
experimenter/tests/integration/nimbus/test_remote_settings.py
Просмотреть файл

@ -1,4 +1,3 @@
import os
from urllib.parse import urljoin
import pytest
@ -258,29 +257,10 @@ def test_rollout_live_update_reject_on_experimenter(
@pytest.mark.remote_settings
@pytest.mark.skipif(
any(
app in os.getenv("PYTEST_ARGS")
for app in ["FOCUS_IOS", "IOS", "FENIX", "FOCUS_ANDROID", "DEMO_APP"]
),
reason="Only run for non-mobile applications and non desktop",
)
def test_summary_release_date_not_visible(
def test_create_new_experiment_timeout_remote_settings(
selenium,
kinto_client,
create_experiment,
experiment_url,
):
summary = create_experiment(selenium)
summary.launch_and_approve()
kinto_client.approve()
summary = SummaryPage(selenium, experiment_url).open()
summary.wait_for_live_status()
summary.wait_for_timeline_visible()
timeline_release_date = summary.wait_until_timeline_release_date_not_found()
assert not timeline_release_date
audience_release_date = summary.wait_until_audience_section_release_date_not_found()
assert not audience_release_date
summary.wait_for_timeout_alert()

37
experimenter/tests/integration/nimbus/utils/helpers.py
Просмотреть файл

@ -4,6 +4,10 @@ from functools import lru_cache
import requests
from nimbus.models.base_dataclass import (
BaseExperimentApplications,
)
LOAD_DATA_RETRIES = 10
LOAD_DATA_RETRY_DELAY = 1.0
@ -135,15 +139,16 @@ def load_config_data():
)["data"]["nimbusConfig"]
def load_targeting_configs(app="DESKTOP"):
def load_targeting_configs(app=BaseExperimentApplications.FIREFOX_DESKTOP.value):
config_data = load_config_data()
return [
item["value"]
for item in config_data["targetingConfigs"]
if "DESKTOP" in app
and "DESKTOP" in item["applicationValues"]
or "DESKTOP" not in app
and "DESKTOP" not in item["applicationValues"]
if BaseExperimentApplications.FIREFOX_DESKTOP.value in app
and BaseExperimentApplications.FIREFOX_DESKTOP.value in item["applicationValues"]
or BaseExperimentApplications.FIREFOX_DESKTOP.value not in app
and BaseExperimentApplications.FIREFOX_DESKTOP.value
not in item["applicationValues"]
]
@ -184,7 +189,7 @@ def create_basic_experiment(name, app, targeting, languages=None):
"input": {
"name": name,
"hypothesis": "Test hypothesis",
"application": app.upper(),
"application": app,
"languages": language_ids,
"changelogMessage": "test changelog message",
"targetingConfigSlug": targeting,
@ -203,18 +208,9 @@ def create_basic_experiment(name, app, targeting, languages=None):
)
def create_experiment(slug, app, targeting, data):
# create a basic experiment via graphql so we can get an ID
create_basic_experiment(
slug,
app,
targeting,
)
def update_experiment(slug, data):
experiment_id = load_experiment_data(slug)["data"]["experimentBySlug"]["id"]
data.update({"id": experiment_id})
load_graphql_data(
{
"operationName": "updateExperiment",
@ -230,6 +226,15 @@ def create_experiment(slug, app, targeting, data):
)
def create_experiment(slug, app, targeting, data):
create_basic_experiment(
slug,
app,
targeting,
)
update_experiment(slug, data)
def end_experiment(slug):
experiment_id = load_experiment_data(slug)["data"]["experimentBySlug"]["id"]

2
experimenter/tests/integration/tox.ini
Просмотреть файл

@ -32,4 +32,4 @@ markers =
remote_settings: tests that involve remote settings
nimbus_ui: tests that only involve the UI, nothing else
desktop_enrollment: tests that integrate with nimbus and external services
demo_app : tests that integrate with demo app and cirrus
cirrus_enrollment: tests that integrate with demo app and cirrus