glean-dictionary/README.md

Glean Dictionary

The Glean dictionary aims to provide a comprehensive index of datasets generated inside Mozilla for applications built using the Glean SDKs.

This project is under active development. For up to date information on project structure and governance, see:

https://wiki.mozilla.org/Data/WorkingGroups/GleanDictionary

The production version of the Glean Dictionary is deployed at:

https://dictionary.telemetry.mozilla.org

Getting Started

You should be able to create your own local copy of the dictionary so long as you have Python (version 3.8+) and node.js (version 18+) installed. You will also need npm v8 or greater: run npm install -g npm@latest if you need to upgrade.

Assuming those requirements are met, follow these instructions:

# Create and activate a python virtual environment.
python3 -m venv venv/
venv/bin/pip install -r requirements.txt

# Build data needed by dashboard
./scripts/gd build-metadata
# Or, on Windows: python3 -m etl build-metadata

# Install npm dependencies and start a local
# instance of the GUI
npm install
npm run dev

If that worked, you should be able to see a local version of Glean at http://localhost:5555

You can speed up the "build data" step by appending the name of a set of application(s) you want to build metadata for. This can speed up the process considerably. For example, to build a metadata index for Fenix (Firefox for Android) only, try:

./scripts/gd build-metadata fenix

Search Service

The Glean Dictionary also includes a search service which enables searching through active metrics. Under the hood, this service is implemented with netlify functions. For example:

https://dictionary.telemetry.mozilla.org/api/v1/metrics_search_burnham?search=techno

You can start it up via the netlify command line interface (assuming you have it installed):

netlify dev

If you have generated metadata as described above, you should then be able to test the search functions locally:

http://localhost:8888/api/v1/metrics_search_burnham?search=techno http://localhost:8888/api/v1/metrics_search_firefox_legacy?search=ms

Storybook

We use Storybook for developing and validating Svelte components used throughout the app. To view the existing list of stories, run:

npm run storybook

Storybook Snapshot Testing

To give us more confidence that changes don't unintentionally break the UI, we run storybook snapshot tests.

You can run them manually as follows:

npm run test:jest

If you intentionally made a change to a component that results in a change to the output of the storybook snapshots, you can re-generate them using the following command:

npm run test:jest -- -u

End-to-End Testing

We use Playwright for our end-to-end tests.

Before testing, download the supported browsers needed for Playwright to execute successfully by running:

npx playwright install

To run the end-to-end tests along with other tests:

npm run test

To run only the Playwright tests:

npx playwright test

ETL Testing

The transforms used by the Glean Dictionary have their own tests. Assuming you've run the set up as described above, you can run these tests by executing:

venv/bin/pytest

Glean Debugging

In order to enable ping logging set the GLEAN_LOG_PINGS environment variable.

GLEAN_LOG_PINGS=true npm run dev

In order to send Glean pings to the debug viewer set the GLEAN_DEBUG_VIEW_TAG environment variable.

GLEAN_DEBUG_VIEW_TAG=my-tag npm run dev

Deployment

A version of the Glean Dictionary running the development branch (main) is accessible at https://glean-dictionary-dev.netlify.app/ .

The production version of the Glean Dictionary (https://dictionary.telemetry.mozilla.org) is deployed from the production branch on this repository, which usually corresponds to the latest GitHub release. To update the Glean Dictionary to the latest version, follow this procedure:

• Do a quick test of https://glean-dictionary-dev.netlify.app to make sure it's working as expected.
• Create a new release off of the main branch:
  • use the auto-generated release notes, omitting dependency updates, unless it's glean.js;
  • use the format vX.Y.Z for the tag, where X.Y.Z is the new version number.
• From a local checkout (assuming origin is the name of the remote):
  • fetch the newly created tags, git fetch --tags origin;
  • switch to the production, git checkout production;
  • make it in sync with the tag you just created, git merge tags/vX.Y.Z (where X.Y.Z is the new version number).
  • push to the production branch, git push origin production.
• Wait for the integration tests to pass by monitoring CircleCi.
• Ensure that https://dictionary.telemetry.mozilla.org is automatically updated to the released version by checking that <HASH> in Built from revision: <HASH> at the bottom of the Glean Dictionary page matches the one reported at the top right of the release page https://github.com/mozilla/glean-dictionary/releases/tag/vX.Y.Z.

Contributing

For more information on contributing, see CONTRIBUTING.md in the root of this repository.