nlp-recipes/tests

Tests

This project uses unit, smoke and integration tests with Python files and notebooks. For more information, see a quick introduction to unit, smoke and integration tests. To manually execute the unit tests in the different environments, first make sure you are in the correct environment as described in the SETUP.md.

Tests are automatically run as part of a DevOps pipeline. The pipelines are defined in .yml files in tests/ci with filenames that align with pipeline names.

Test execution

Click on the following menus to see more details on how to execute the unit, smoke and integration tests:

Unit tests

Unit tests ensure that each class or function behaves as it should. Every time a developer makes a pull request to staging or master branch, a battery of unit tests is executed.

For executing the Python unit tests for the utilities:

pytest tests/unit -m "not notebooks and not gpu"

For executing the Python unit tests for the notebooks:

pytest tests/unit -m "notebooks and not gpu"

For executing the Python GPU unit tests for the utilities:

pytest tests/unit -m "not notebooks and gpu"

For executing the Python GPU unit tests for the notebooks:

pytest tests/unit -m "notebooks and gpu"

Smoke tests

Smoke tests make sure that the system works and are executed just before the integration tests every night.

For executing the Python smoke tests:

pytest tests/smoke -m "smoke and not gpu"

For executing the Python GPU smoke tests:

pytest tests/smoke -m "smoke and gpu"

Integration tests

Integration tests make sure that the program results are acceptable

For executing the Python integration tests:

pytest tests/integration -m "integration and not gpu"

For executing the Python GPU integration tests:

pytest tests/integration -m "integration and gpu"

How to create tests on notebooks with Papermill

In the notebooks of these repo we use Papermill in unit, smoke and integration tests.

In the unit tests we just make sure the notebook runs. In the smoke tests, we run them with a small dataset or a small number of epochs to make sure that, apart from running, they provide reasonable metrics. Finally, in the integration tests, we use a bigger dataset for more epochs and we test that the metrics are what we expect.

For a deep overview on how to integrate papermill on unit, smoke and integration test, please refer to this guide from Microsoft Recommenders repo.

More details on how to integrate Papermill with notebooks can be found in their repo.