Because:
- we are creating many short-lived containers for commands; and
- we are not cleaning them up except in `make kill`
This commit:
- updates all docker run commands to use the `--rm` flag so that they
are automatically deleted after the container exits.
Fixes#11662
Because:
- make refresh does rebuild the DB but also blows away all containers;
and
- I just want to rebuild the DB
This commit:
- adds a make refresh_db (which refresh depends on) that will just
recreate the database.
Fixes#11661
Because:
- we need the .schema.json files in Desktop;
- they are not currently packaged into either the NPM or PyPI packages;
and
- the generated schemas do not include the custom validation logic
expressed by Pydantic validators
This commit:
- adds JSON schema validation via Pydantic's json_schema_extra hook to
give the JSON schemas validation parity with the Pydantic models and
existing JSON schemas in Desktop;
- generates the JSON schemas and commits them (in schemas/schemas);
- packages the produced schemas into the NPM package;
- packages the produced schemas into the PyPI package (but uncommitted,
since they are duplicates);
- revises the Makefile build steps for schema package generation; and
- bumps the schema package version.
Fixes#11518
Because
- We added the capability to track and store what version of firefox
release is current for use in our integration tests
This commit
- Follows on from that to modify the script so that it will download and
install that specific version
Fixes#11411
Because
- To make it easier to identify failing tests
This commit
- make cirrus pytest tests run in verbose mode
Fixes#9148
Co-authored-by: mosaadm <mohamed.mahmoud@toptal.com>
Because
- We are now able to trigger builds based on changes made to Firefox
Beta on mozilla-central. How ever, we weren't using the firefox images
from those tasks.
This commit
- Allows us to run the firefox beta tests against the recently built
version of Firefox Beta.
Fixes#11311
Because
- the schemas package is used by a lot of data tooling
- the data tooling is starting to complain about schemas requiring
pydantic v1 due to other dependencies
- the pydantic-to-typescript package is outdated and does not support
pydantic v2
This commit
- updates the schemas package to v2
- changes the typescript generation to a script internal to experimenter
Fixes#11214
Because
* Experimenter needs to retrieve the segments from metric-hub as they
are currently not available there
This commit
* allows Experimenter to retrieve segments from metric-hub
Fixes#11242
Because
- We want to run certain jobs based on external dependency changes as
they happen.
This commit
- Changes the fenix integration and enrollment tests to run when a
change has been detected within fenix.
Fixes#11230
Because
- QA should be able to test web experiments on the prod of the
implementing application
This commit
- Supports Cirrus to handle the `nimbus_preview` flag so that it should
return a response based on the preview records
Fixes#10953
---------
Co-authored-by: Jared Lockhart <119884+jaredlockhart@users.noreply.github.com>
Because
* Ruff deprecated the `ruff <path>` interface for `ruff check <path>`
* We updated the check call but not the fix call
This commit
* Updates the ruff fix call in the Makefile to use the `ruff check
--fix` interface
fixes#10990
Because
It would be awesome to extend our coverage to android devices to make
sure those stay healthy.
This commit
* Adds a job to build the fenix test APKs from mozilla central.
* Adds a job to run a small suite of tests on fenix
Fixes#10843
Because
* Now that all of the functionality is finished for the new nimbus ui
list page
* We can setup webpack to bundle all the dependencies
This commit
* Adds webpack to nimbus_ui_new
* Move scripts from being inline to actually stored as static assets
with proper linting
* Setup SASS so we can have some proper styling for bootstrap-select
with dark mode
fixes#10708
Because
* We will now be switching nimbus ui to use django templates
* We need a way to keep the formatting/linting consistent across PRs
This commit
* Adds djlint to the makefile to format/lint django templates locally
and in ci
fixes#10656
Because
- application-services updated uniffi, which resulted in generating
Python with relative imports;
This commit:
- updates our application-services packaging to produce a
`nimbus_megazord` Python package; and
- updates to the latest application-services build.
Fixes#10521
Because
- we want to export metric and image version metadata from jetstream
(https://github.com/mozilla/jetstream/issues/1786)
- schemas_build make command was confusing
This commit
- adds config version to the jetstream metadata schema
- makes schemas_build do the npm and pypi builds
Fixes#10450Fixes#10080
Because:
- we have standardized on using a single version of application-services
across all Experimenter, Cirrus, and integration tests; and
- we do not want to manually manage updating this version
This commit:
- adds a CI task to open a new PR for new application-services
nightlies.
Fixes https://github.com/mozilla/experimenter/issues/9688
Because:
- the cirrus test jobs require the megazords; and
- the cirrus build requires rustc on arm64
This commit:
- adds `build_megazords` as a dependency of `cirrus_build_test`; and
- installs `rustc` inside the `python-builder` phase of the Cirrus
build.
Fixes#10172
feat(nimbus): Use pre-built A-S in Cirrus and Experimenter
Because:
- our integration tests have been updated to use pre-built
application-services artifacts;
- the Cirrus and Experimenter Dockerfiles are currently building
application-services from source;
- Cirrus was using a Debian buster (oldoldstable); and
- Cirrus and Experimenter were using a pinned minor version of Python,
which prevents them from getting security upgrades in minor patches.
This commit:
- updates the Cirrus and Experimenter Dockerfiles to reference the
`experimenter:megazords` image, which contains the pre-built binaries;
- updates Cirrus to use a Debian bullseye (oldstable) image; and
- updates Cirrus and Experimenter to not pin the minor verison of Python
(3.11.2 to 3.11).
Fixes#10150
Because
- The sdk integration test image was stuck on application-services
v97.5.1 due to build system changes, which prevented landing any new
targeting using new targeting attributes from v98 onwards.
- The current system of periodically pushing a integration test image
and pulling that pre-built image in tests lead to headaches and
issues.
- I discovered several oddities in the current
integration-test-nimbus-rust setup (specifying a webdriver driver,
sudo commands that never ran because sudo was not installed).
- The new application services has a very different NimbusClient
interface from the one currently used in tests.
This commit
- Adds a `experimenter:megazords` container, which is responsible for
downloading a tagged nightly of the Cirrus and Nimbus-Experimenter
megazords.
- Adds a new `experimenter:integration-tests` container, which takes the
place of the old "nimbus-rust-image" container. This container is very
barebones, and only includes the application services megazord (copied
from the other container) and tox.
- Removes the old `Dockerfile-rust-image` and its publishing steps from
CI.
- Configures our integration tests to build the image on the fly instead
of pulling a pre-built image from a registry.
- Cleans up a few things in the `test_mobile_targeting.py` suite and
updates the code to match the new interfaces.
- Updates the `integration_test_nimbus_rust` task to run tox with `-vv`,
which includes the output of any commands issued.
Fixes#9820
Because
- we are running into the GitHub API request limit;
- we have an API token in CI; and
- we are not using the API while authenticated because Docker does not
transparently pass environment variables through to containers
This commit
- updates the `feature_manfests` Make target to pass the
`GITHUB_BEARER_TOKEN` through to the docker container.
Fixes#9972
Because
* Building Cirrus can be time consuming
* Cirrus is not necessary for many integration tests or local workflows
* We can add a flag to enable/disable cirrus to speed up the integration
tests or local workflows that don't involve Cirrus
This commit
* Adds a flag to enable/disable Cirrus in the Makefile
* Sets the flag automatically for all Cirrus make commands
* Sets the flag for Cirrus integration tests
fixes#9910
Because
- fetching feature manifests can fail (see #9804 for an example);
- the fetch command does not hard fail when an individual fetch fails;
and
- we are generating a summary of all the fetches
This commit
- updates the fetch command to hard fail if there are fetch failures and
no
successes (i.e., in the case where it would not open or update a PR);
- adds a flag to the fetch command to write the summary to a file; and
- opens and updates PRs with a summary of the fetches in the PR
description
Fixes#9815
Because
- we want to keep track of features on individual product versions
This commit
- fetches features from releases going back 5 versions in FML apps; and
- does not do an initial import for the FML apps because it is a very
large change (see #9751).
Fixes#9685
Because:
- #9694 broke integration tests;
This commit
- reverts #9694; and
- adds an explicit pull of our integration test image in our CI
workflow.
Fixes#9707
Because
- we don't want to manually build application-services megazords
ourselves for each container; and
- application-services publishes megazord artifacts on Taskcluster
This commit
- adds a new as-builder docker container to download cirrus and nimbus
megazords with the appropriate binaries for the platform as Python
packages;
- updates the cirrus and experimenter containers to copy the megazords
from the as-builder container;
- updates cirrus and experimenter to refer to the new megazord packages;
- updates cirrus to be based on Debian Bullseye so it can use the
generated binaries; and
- updates the Dockerfile-rust-image container to be built on top of
as-builder in CI.
Because
* Changes to Application Services can break consuming code in
Cirrus/Experimenter
* We currently consume the main branch of Application Services
* This means that changes can land in main in Application Services and
break main in Cirrus/Experimenter
* We also neglected to explicitly rebuild the Cirrus test container when
running Cirrus tests
This commit
* Explicitly locks Application Services to a specific hash when consumed
in Cirrus/Experimenter
* Explicitly builds the test container for Cirrus when running Cirrus
tests
Because
* As we add a wider variety of elements to the Nimbus project including
binary packages it becomes increasingly important to have a consistent
environment for each of our modules
* Using Docker allows us to control the environment each module runs in
and use a consistent base layer
* Dockerizing the schemas package will allow us to use identical Python,
Node, and Rust environments as the other modules in Nimbus
This commit
* Wraps the schemas module in a Dockerfile that is consistent with the
other modules in Nimbus
Because
- we want to support fetching mozilla-central with manifest-tool
This commit
- adds support for hg.mozilla.org repositories to manifest-tool;
- adds support for legacy experimenter.yaml-based apps to manifest-tool;
- adds Firefox Desktop to apps.yaml; and
- updates make feature_manifests to fetch Firefox Desktop manifests.
Because
* We recently moved our FML parsing into the Experimenter Docker
container
* This requires that the experimenter:dev container is built before
running the make fetch_external_resources command
* This was ommitted and the hourly CircleCI task is broken
This commit
* Adds build_dev as a pre requisite to make fetch_external_resources
Because
- we want to use FML in experimenter
this commit
- adds nimbus-cli to the experimenter:dev docker image;
- adds a wrapper around nimbus-cli (manifest-tool) for fetching all the
FML files and generating experimenter.yaml files;
- updates apps.yaml to include monitor; and
- does an initial import of all the FML files.
Because
- we haven't yet cross compiled libnimbus_experimenter.so for developer
machines
This commit
- Adds an extra git checkout / cargo build to get
libnimbus_experimener.so into experimenter.
N.B. this uses the `--library` form of `uniffi_bindgen`. Once lands, we
should remove the `cdylib_name` entries in the FML and Nimbus
`uniffi.toml` files in application-services.
Because
- As the monitor channels are now staging and production, our cron job
is failing.
This commit
- Modify the make command with the appropriate channel option
`production`
Because
- feature versioning means vendoring several nimbus.fml,
experimenter.yaml, and JSON schema files per application
This commit
- moves apps.yaml into the manifests directory; and
- renames {app}.yaml to {app}/experimenter.yaml because each app will
have multiple manifests.
Because
* We now have access to Docker Hydrobuild
* This allows us to use a shared cloud builder for our docker images so
we can share a build cache between CI and local dev
This commit
* Updates the makefile to explicitly use docker buildx
* Updates the circle config to install the latest docker client and
setup a cloud builder
Because
* We now have the mozilla-nimbus-schemas package for storing all Nimbus
related schemas
* We should add the experiment schema there and update all components to
use it
This commit
* Adds the Nimbus experiment schema to the nimbus schemas package
* Adds all existing test fixtures from shared
Because
- prebuilt versions of nimbus-cli include FML are available for
linux-musl
This commit
- Installs the nimbus-cli as part of deployment
- Uses this to fetch and transform FML files from the Firefox Mobile,
Focus and Monitor apps
This means:
- Faster deploy times
- We can remove the .experimenter.yaml files from the apps' repos and
build scripts.
---------
Co-authored-by: Jared Lockhart <119884+jaredlockhart@users.noreply.github.com>
Because
- monitor changed their channels in
https://github.com/mozilla/blurts-server/pull/3320, which caused
`make convert_feature_manifests` to fail;
- as a result, we have not merged feature manifests since the break
This commit
- updates `make convert_feature_manifests` to use the correct channel;
and
- merges the updated feature manifests from monitor and desktop.
Becuase
* Restarting docker containers can cause images, containers, volumes,
etc to start piling up
* Eventually you run out of disk space and have to docker system prune
-a
* We added some prune commands to remove temporary data on refresh
* This was catching a bunch of things but missing 'anonymous volumes'
This commit
* Adds an additional step to docker_prune to remove anonymous volumes
and prevent them from filling the disk
Because
- We want to test Cirrus end to end with the local remote setting,
including fml and experimenter to launch an experiment for the Demo app
This commit
- Integrate fml, remote settings, experimenter, cirrus to demo app
- Fixes RS pushing to web
PS: currently it uses the static value for client and context, in the
next ticket work, we will add an option in the frontend app to set the
client value and context
https://github.com/mozilla/experimenter/issues/9288fixes#9265
Becuase
* The FML format that lives in an applications repo is not parseable by
Experimenter
* The FML command line tool will convert it into the format expected by
Experimenter
* Currently supported applications do this as part of their build chain
* New applications, particularly web, may not do this so we should do
this conversion on our end as part of the external resource fetch step
This commit
* Adds a step to the fetch_external_resources make command to download
and build the FML tool and convert the FML files into a format
Experimenter can parse
Because
- we want to use the population sizing schemas for graphql and UI
This commit
- sets up generation of typescript schemas from the pydantic schemas
- sets up publishing the typescript schemas to npm
Because
- We want to test cirrus end to end
This commit
- Introduces demo app with frontend and server
- Make command to build and run the demo app- `make build_demo_app`
`make run_demo_app`
- The frontend of the Demo App is accessible at
[http://localhost:8080](http://localhost:8080)
- The backend of the Demo App is accessible at
[http://localhost:3002](http://localhost:3002)
ps: FML and cirrus integration will come in different PR
fixes#9243fixes#9244
Beth Rennie
Beth Rennie
Benjamin Forehand Jr
Mohamed Mahmoud
Mike Williams
Rana Al-Khulaidi
Yashika Khurana
Jared Lockhart
dependabot[bot]
Mathieu Leplatre
jhugman