зеркало из https://github.com/microsoft/nutter.git
Release 0.1.35 (#74)
* Improved the error message when the call to the parent class constructor is missing in a test fixture * Fixing the Environment variale setting documentation for Windows Powershell * Apply suggestions from code review Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> * Feature: Discover test files with '_test' suffix (#47) * Enable test discovery for test names with suffix 'test' * Combine redundant suffix tests * Remove old suffix tests * Encapsulate test name parsing and have nuttercli call test name validation from api * Have api client results call _is_valid_test_name from api Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Invalid State response is retriable (#49) * fixed import error and refactoring * invalid state is retriable, pull sleep is 5 seconds * Poll wait time as flag (#51) * poll wait time as flag * lint fixes * Parallel runner implementation (#59) * Toc Update (#30) * Release v0.1.34 (#52) * Improved the error message when the call to the parent class constructor is missing in a test fixture * Fixing the Environment variale setting documentation for Windows Powershell * Apply suggestions from code review Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> * Feature: Discover test files with '_test' suffix (#47) * Enable test discovery for test names with suffix 'test' * Combine redundant suffix tests * Remove old suffix tests * Encapsulate test name parsing and have nuttercli call test name validation from api * Have api client results call _is_valid_test_name from api Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Invalid State response is retriable (#49) * fixed import error and refactoring * invalid state is retriable, pull sleep is 5 seconds * Poll wait time as flag (#51) * poll wait time as flag * lint fixes Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Update README.md * Parallel runner * Rename helper class * Fix collect results * Rename execute method * Use new execute method * Introduce add_test_fixture() method Co-authored-by: Jesus Aguilar <3589801+giventocode@users.noreply.github.com> Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Fix Py4JJavaError serialization issue (#60) * Toc Update (#30) * Release v0.1.34 (#52) * Improved the error message when the call to the parent class constructor is missing in a test fixture * Fixing the Environment variale setting documentation for Windows Powershell * Apply suggestions from code review Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> * Feature: Discover test files with '_test' suffix (#47) * Enable test discovery for test names with suffix 'test' * Combine redundant suffix tests * Remove old suffix tests * Encapsulate test name parsing and have nuttercli call test name validation from api * Have api client results call _is_valid_test_name from api Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Invalid State response is retriable (#49) * fixed import error and refactoring * invalid state is retriable, pull sleep is 5 seconds * Poll wait time as flag (#51) * poll wait time as flag * lint fixes Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Update README.md * Fix Py4JJavaError serialization issue * Test Py4JJavaError using mocks * Remove Py4J stuff * Still need to install py4j to run tests Co-authored-by: Jesus Aguilar <3589801+giventocode@users.noreply.github.com> Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Add notebook params and debugger (#68) * Toc Update (#30) * Release v0.1.34 (#52) * Improved the error message when the call to the parent class constructor is missing in a test fixture * Fixing the Environment variale setting documentation for Windows Powershell * Apply suggestions from code review Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> * Feature: Discover test files with '_test' suffix (#47) * Enable test discovery for test names with suffix 'test' * Combine redundant suffix tests * Remove old suffix tests * Encapsulate test name parsing and have nuttercli call test name validation from api * Have api client results call _is_valid_test_name from api Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Invalid State response is retriable (#49) * fixed import error and refactoring * invalid state is retriable, pull sleep is 5 seconds * Poll wait time as flag (#51) * poll wait time as flag * lint fixes Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Update README.md * Add notebook params and debugger * Add example in README.md * Have more explicit notebook_params error message * Revert "Add example in README.md" This reverts commit1aac73c77d. * Add examples for notebook params Co-authored-by: Jesus Aguilar <3589801+giventocode@users.noreply.github.com> Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> Co-authored-by: Neyissa Exilus <68079902+nexilus18@users.noreply.github.com> * upgraded python version * Feature: Discover test files with '_test' suffix (#47) * Enable test discovery for test names with suffix 'test' * Combine redundant suffix tests * Remove old suffix tests * Encapsulate test name parsing and have nuttercli call test name validation from api * Have api client results call _is_valid_test_name from api Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Invalid State response is retriable (#49) * fixed import error and refactoring * invalid state is retriable, pull sleep is 5 seconds * Poll wait time as flag (#51) * poll wait time as flag * lint fixes * Parallel runner implementation (#59) * Toc Update (#30) * Release v0.1.34 (#52) * Improved the error message when the call to the parent class constructor is missing in a test fixture * Fixing the Environment variale setting documentation for Windows Powershell * Apply suggestions from code review Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> * Feature: Discover test files with '_test' suffix (#47) * Enable test discovery for test names with suffix 'test' * Combine redundant suffix tests * Remove old suffix tests * Encapsulate test name parsing and have nuttercli call test name validation from api * Have api client results call _is_valid_test_name from api Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Invalid State response is retriable (#49) * fixed import error and refactoring * invalid state is retriable, pull sleep is 5 seconds * Poll wait time as flag (#51) * poll wait time as flag * lint fixes Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Update README.md * Parallel runner * Rename helper class * Fix collect results * Rename execute method * Use new execute method * Introduce add_test_fixture() method Co-authored-by: Jesus Aguilar <3589801+giventocode@users.noreply.github.com> Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Fix Py4JJavaError serialization issue (#60) * Toc Update (#30) * Release v0.1.34 (#52) * Improved the error message when the call to the parent class constructor is missing in a test fixture * Fixing the Environment variale setting documentation for Windows Powershell * Apply suggestions from code review Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> * Feature: Discover test files with '_test' suffix (#47) * Enable test discovery for test names with suffix 'test' * Combine redundant suffix tests * Remove old suffix tests * Encapsulate test name parsing and have nuttercli call test name validation from api * Have api client results call _is_valid_test_name from api Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Invalid State response is retriable (#49) * fixed import error and refactoring * invalid state is retriable, pull sleep is 5 seconds * Poll wait time as flag (#51) * poll wait time as flag * lint fixes Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Update README.md * Fix Py4JJavaError serialization issue * Test Py4JJavaError using mocks * Remove Py4J stuff * Still need to install py4j to run tests Co-authored-by: Jesus Aguilar <3589801+giventocode@users.noreply.github.com> Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Add notebook params and debugger (#68) * Toc Update (#30) * Release v0.1.34 (#52) * Improved the error message when the call to the parent class constructor is missing in a test fixture * Fixing the Environment variale setting documentation for Windows Powershell * Apply suggestions from code review Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> * Feature: Discover test files with '_test' suffix (#47) * Enable test discovery for test names with suffix 'test' * Combine redundant suffix tests * Remove old suffix tests * Encapsulate test name parsing and have nuttercli call test name validation from api * Have api client results call _is_valid_test_name from api Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Invalid State response is retriable (#49) * fixed import error and refactoring * invalid state is retriable, pull sleep is 5 seconds * Poll wait time as flag (#51) * poll wait time as flag * lint fixes Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> * Update README.md * Add notebook params and debugger * Add example in README.md * Have more explicit notebook_params error message * Revert "Add example in README.md" This reverts commit1aac73c77d. * Add examples for notebook params Co-authored-by: Jesus Aguilar <3589801+giventocode@users.noreply.github.com> Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> Co-authored-by: Neyissa Exilus <68079902+nexilus18@users.noreply.github.com> * upgraded python version * version bump, python 3.7 * added py4j requirement * doc update * doc update Co-authored-by: RobBagby <rob@robbagby.com> Co-authored-by: Prakash Kudkuli Vishnu <prvishnu@microsoft.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: quanuw <quanuw@gmail.com> Co-authored-by: Quan Nguyen <qunguyen@microsoft.com> Co-authored-by: Thomas Conté <199027+tomconte@users.noreply.github.com> Co-authored-by: Andrew Francisque <andrew.francisque@gmail.com> Co-authored-by: Neyissa Exilus <68079902+nexilus18@users.noreply.github.com>
This commit is contained in:
Jesus Aguilar
GitHub
Родитель
e60be0e22b
Коммит
368248bb3c
|
|
@ -9,7 +9,7 @@ jobs:
|
|||
strategy:
|
||||
max-parallel: 4
|
||||
matrix:
|
||||
python-version: [3.5]
|
||||
python-version: [3.7]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v1
|
||||
|
|
@ -34,5 +34,5 @@ jobs:
|
|||
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
|
||||
- name: Test with pytest
|
||||
run: |
|
||||
pip install pytest
|
||||
pip install pytest py4j
|
||||
pytest
|
||||
|
|
|
|||
|
|
@ -0,0 +1,20 @@
|
|||
{
|
||||
"version": "0.2.0",
|
||||
"configurations": [
|
||||
{
|
||||
"name": "Python: Current File",
|
||||
"type": "python",
|
||||
"request": "launch",
|
||||
"console": "integratedTerminal",
|
||||
"python": "python3",
|
||||
"module": "cli.nuttercli",
|
||||
"args": [
|
||||
"run",
|
||||
"<Add test pattern here>",
|
||||
"--cluster_id",
|
||||
"<Add cluster_id here>",
|
||||
"--notebook_params",
|
||||
"{\"example_key_1\": \"example_value_1\", \"example_key_2\": \"example_value_2\"}"
|
||||
]
|
||||
}]
|
||||
}
|
||||
|
|
@ -1,9 +1,10 @@
|
|||
{
|
||||
"python.pythonPath": "/usr/bin/python3",
|
||||
"python.pythonPath": "/usr/bin/python3",
|
||||
"python.testing.pytestArgs": [
|
||||
"tests"
|
||||
],
|
||||
"python.testing.unittestEnabled": false,
|
||||
"python.testing.nosetestsEnabled": false,
|
||||
"python.testing.pytestEnabled": true
|
||||
"python.testing.pytestEnabled": true,
|
||||
"python.envFile": "${workspaceFolder}/.env"
|
||||
}
|
||||
84
README.md
84
README.md
|
|
@ -7,11 +7,12 @@
|
|||
* [Cluster Installation](#cluster-installation)
|
||||
* [Nutter Fixture](#nutter-fixture)
|
||||
* [Test Cases](#test-cases)
|
||||
* [before_all and after_all](#before-all-and-after-all)
|
||||
* [*before_all* and *after_all*](#before-all-and-after-all)
|
||||
* [Running test fixtures in parallel](#running-test-fixtures-in-parallel)
|
||||
- [Nutter CLI](#nutter-cli)
|
||||
* [Getting Started with the Nutter CLI](#getting-started-with-the-nutter-cli)
|
||||
* [Listing Test Notebooks](#listing-test-notebooks)
|
||||
* [Executing Test Notebooks](#executing-test-notebooks)
|
||||
* [Listing test Notebooks](#listing-test-notebooks)
|
||||
* [Executing test Notebooks](#executing-test-notebooks)
|
||||
* [Run single test notebook](#run-single-test-notebook)
|
||||
* [Run multiple tests notebooks](#run-multiple-tests-notebooks)
|
||||
* [Parallel Execution](#parallel-execution)
|
||||
|
|
@ -198,6 +199,64 @@ class TestFixture(NutterFixture):
|
|||
NutterFixture.__init__(self)
|
||||
```
|
||||
|
||||
### Running test fixtures in parallel
|
||||
|
||||
|
||||
Version 0.1.35 includes a parallel runner class ```NutterFixtureParallelRunner``` that facilitates the execution of test fixtures concurrently. This approach could significantly increase the performance of your testing pipeline.
|
||||
|
||||
The following code executes two fixtures, ```CustomerTestFixture``` and ```CountryTestFixture``` in parallel.
|
||||
|
||||
```Python
|
||||
from runtime.runner import NutterFixtureParallelRunner
|
||||
from runtime.nutterfixture import NutterFixture, tag
|
||||
class CustomerTestFixture(NutterFixture):
|
||||
def run_customer_data_is_inserted(self):
|
||||
dbutils.notebook.run('../data/customer_data_import', 600)
|
||||
|
||||
def assertion_customer_data_is_inserted(self):
|
||||
some_tbl = sqlContext.sql('SELECT COUNT(*) AS total FROM customers')
|
||||
first_row = some_tbl.first()
|
||||
assert (first_row[0] == 1)
|
||||
|
||||
class CountryTestFixture(NutterFixture):
|
||||
def run_country_data_is_inserted(self):
|
||||
dbutils.notebook.run('../data/country_data_import', 600)
|
||||
|
||||
def assertion_country_data_is_inserted(self):
|
||||
some_tbl = sqlContext.sql('SELECT COUNT(*) AS total FROM countries')
|
||||
first_row = some_tbl.first()
|
||||
assert (first_row[0] == 1)
|
||||
|
||||
parallel_runner = NutterFixtureParallelRunner(num_of_workers=2)
|
||||
parallel_runner.add_test_fixture(CustomerTestFixture())
|
||||
parallel_runner.add_test_fixture(CountryTestFixture())
|
||||
|
||||
result = parallel_runner.execute()
|
||||
print(result.to_string())
|
||||
# Comment out the next line (result.exit(dbutils)) to see the test result report from within the notebook
|
||||
# result.exit(dbutils)
|
||||
|
||||
```
|
||||
|
||||
The parallel runner combines the test results of both fixtures in a single result.
|
||||
|
||||
``` bash
|
||||
Notebook: N/A - Lifecycle State: N/A, Result: N/A
|
||||
Run Page URL: N/A
|
||||
============================================================
|
||||
PASSING TESTS
|
||||
------------------------------------------------------------
|
||||
country_data_is_inserted (11.446587234000617 seconds)
|
||||
customer_data_is_inserted (11.53276599000128 seconds)
|
||||
|
||||
|
||||
============================================================
|
||||
|
||||
Command took 11.67 seconds -- by foo@bar.com at 12/15/2022, 9:34:24 PM on Foo Cluster
|
||||
```
|
||||
|
||||
|
||||
|
||||
## Nutter CLI
|
||||
|
||||
The Nutter CLI is a command line interface that allows you to execute and list tests via a Command Prompt.
|
||||
|
|
@ -230,7 +289,7 @@ $env:DATABRICKS_TOKEN="TOKEN"
|
|||
|
||||
__Note:__ For more information about personal access tokens review [Databricks API Authentication](https://docs.azuredatabricks.net/dev-tools/api/latest/authentication.html).
|
||||
|
||||
### Listing Test Notebooks
|
||||
### Listing test notebooks
|
||||
|
||||
The following command list all test notebooks in the folder ```/dataload```
|
||||
|
||||
|
|
@ -248,16 +307,16 @@ You can list all test notebooks in the folder structure using the ```--recursive
|
|||
nutter list /dataload --recursive
|
||||
```
|
||||
|
||||
### Executing Test Notebooks
|
||||
### Executing test notebooks
|
||||
|
||||
The ```run``` command schedules the execution of test notebooks and waits for their result.
|
||||
|
||||
### Run single test notebook
|
||||
|
||||
The following command executes the test notebook ```/dataload/test_sourceLoad``` in the cluster ```0123-12334-tonedabc```.
|
||||
The following command executes the test notebook ```/dataload/test_sourceLoad``` in the cluster ```0123-12334-tonedabc``` with the notebook_param key-value pairs of ```{"example_key_1": "example_value_1", "example_key_2": "example_value_2"}``` (Please note the escaping of quotes):
|
||||
|
||||
```bash
|
||||
nutter run dataload/test_sourceLoad --cluster_id 0123-12334-tonedabc
|
||||
nutter run dataload/test_sourceLoad --cluster_id 0123-12334-tonedabc --notebook_params "{\"example_key_1\": \"example_value_1\", \"example_key_2\": \"example_value_2\"}"
|
||||
```
|
||||
|
||||
__Note:__ In Azure Databricks you can get the cluster ID by selecting a cluster name from the Clusters tab and clicking on the JSON view.
|
||||
|
|
@ -267,10 +326,10 @@ __Note:__ In Azure Databricks you can get the cluster ID by selecting a cluster
|
|||
The Nutter CLI supports the execution of multiple notebooks via name pattern matching. The Nutter CLI applies the pattern to the name of test notebook **without** the *test_* prefix. The CLI also expects that you omit the prefix when specifying the pattern.
|
||||
|
||||
|
||||
Say the *dataload* folder has the following test notebooks: *test_srcLoad* and *test_srcValidation*. The following command will result in the execution of both tests.
|
||||
Say the *dataload* folder has the following test notebooks: *test_srcLoad* and *test_srcValidation* with the notebook_param key-value pairs of ```{"example_key_1": "example_value_1", "example_key_2": "example_value_2"}```. The following command will result in the execution of both tests.
|
||||
|
||||
```bash
|
||||
nutter run dataload/src* --cluster_id 0123-12334-tonedabc
|
||||
nutter run dataload/src* --cluster_id 0123-12334-tonedabc --notebook_params "{\"example_key_1\": \"example_value_1\", \"example_key_2\": \"example_value_2\"}"
|
||||
```
|
||||
|
||||
In addition, if you have tests in a hierarchical folder structure, you can recursively execute all tests by setting the ```--recursive``` flag.
|
||||
|
|
@ -316,6 +375,10 @@ FLAGS
|
|||
--max_parallel_tests Sets the level of parallelism for test notebook execution.
|
||||
--recursive Executes all tests in the hierarchical folder structure.
|
||||
--poll_wait_time Polling interval duration for notebook status. Default is 5 (5 seconds).
|
||||
--notebook_params Allows parameters to be passed from the CLI tool to the test notebook. From the
|
||||
notebook, these parameters can then be accessed by the notebook using
|
||||
the 'dbutils.widgets.get('key')' syntax.
|
||||
|
||||
```
|
||||
|
||||
__Note:__ You can also use flags syntax for POSITIONAL ARGUMENTS
|
||||
|
|
@ -435,6 +498,9 @@ steps:
|
|||
condition: succeededOrFailed()
|
||||
```
|
||||
|
||||
### Debugging Locally
|
||||
If using Visual Studio Code, you can use the `example_launch.json` file provided, editing the variables in the `<>` symbols to match your environment. You should be able to use the debugger to see the test run results, much the same as you would in Azure Devops.
|
||||
|
||||
## Contributing
|
||||
|
||||
### Contribution Tips
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ from .resultsvalidator import ExecutionResultsValidator
|
|||
from .reportsman import ReportWriters
|
||||
from . import reportsman as reports
|
||||
|
||||
__version__ = '0.1.34'
|
||||
__version__ = '0.1.35'
|
||||
|
||||
BUILD_NUMBER_ENV_VAR = 'NUTTER_BUILD_NUMBER'
|
||||
|
||||
|
|
@ -53,14 +53,14 @@ class NutterCLI(object):
|
|||
def run(self, test_pattern, cluster_id,
|
||||
timeout=120, junit_report=False,
|
||||
tags_report=False, max_parallel_tests=1,
|
||||
recursive=False, poll_wait_time=DEFAULT_POLL_WAIT_TIME):
|
||||
recursive=False, poll_wait_time=DEFAULT_POLL_WAIT_TIME, notebook_params=None):
|
||||
try:
|
||||
logging.debug(""" Running tests. test_pattern: {} cluster_id: {} timeout: {}
|
||||
logging.debug(""" Running tests. test_pattern: {} cluster_id: {} notebook_params: {} timeout: {}
|
||||
junit_report: {} max_parallel_tests: {}
|
||||
tags_report: {} recursive:{} """
|
||||
.format(test_pattern, cluster_id, timeout,
|
||||
junit_report, max_parallel_tests,
|
||||
tags_report, recursive))
|
||||
tags_report, recursive, notebook_params))
|
||||
|
||||
logging.debug("Executing test(s): {}".format(test_pattern))
|
||||
|
||||
|
|
@ -68,7 +68,7 @@ class NutterCLI(object):
|
|||
logging.debug('Executing pattern')
|
||||
results = self._nutter.run_tests(
|
||||
test_pattern, cluster_id, timeout,
|
||||
max_parallel_tests, recursive, poll_wait_time)
|
||||
max_parallel_tests, recursive, poll_wait_time, notebook_params)
|
||||
self._nutter.events_processor_wait()
|
||||
self._handle_results(results, junit_report, tags_report)
|
||||
return
|
||||
|
|
|
|||
|
|
@ -88,7 +88,7 @@ class Nutter(NutterApi):
|
|||
return tests
|
||||
|
||||
def run_test(self, testpath, cluster_id,
|
||||
timeout=120, pull_wait_time=DEFAULT_POLL_WAIT_TIME):
|
||||
timeout=120, pull_wait_time=DEFAULT_POLL_WAIT_TIME, notebook_params=None):
|
||||
self._add_status_event(NutterStatusEvents.TestExecutionRequest, testpath)
|
||||
test_notebook = TestNotebook.from_path(testpath)
|
||||
if test_notebook is None:
|
||||
|
|
@ -96,13 +96,13 @@ class Nutter(NutterApi):
|
|||
|
||||
result = self.dbclient.execute_notebook(
|
||||
test_notebook.path, cluster_id,
|
||||
timeout=timeout, pull_wait_time=pull_wait_time)
|
||||
timeout=timeout, pull_wait_time=pull_wait_time, notebook_params=notebook_params)
|
||||
|
||||
return result
|
||||
|
||||
def run_tests(self, pattern, cluster_id,
|
||||
timeout=120, max_parallel_tests=1, recursive=False,
|
||||
poll_wait_time=DEFAULT_POLL_WAIT_TIME):
|
||||
poll_wait_time=DEFAULT_POLL_WAIT_TIME, notebook_params=None):
|
||||
|
||||
self._add_status_event(NutterStatusEvents.TestExecutionRequest, pattern)
|
||||
root, pattern_to_match = self._get_root_and_pattern(pattern)
|
||||
|
|
@ -119,7 +119,7 @@ class Nutter(NutterApi):
|
|||
NutterStatusEvents.TestsListingFiltered, len(filtered_notebooks))
|
||||
|
||||
return self._schedule_and_run(
|
||||
filtered_notebooks, cluster_id, max_parallel_tests, timeout, poll_wait_time)
|
||||
filtered_notebooks, cluster_id, max_parallel_tests, timeout, poll_wait_time, notebook_params)
|
||||
|
||||
def events_processor_wait(self):
|
||||
if self._events_processor is None:
|
||||
|
|
@ -168,7 +168,7 @@ class Nutter(NutterApi):
|
|||
return root, valid_pattern
|
||||
|
||||
def _schedule_and_run(self, test_notebooks, cluster_id,
|
||||
max_parallel_tests, timeout, pull_wait_time):
|
||||
max_parallel_tests, timeout, pull_wait_time, notebook_params=None):
|
||||
func_scheduler = scheduler.get_scheduler(max_parallel_tests)
|
||||
for test_notebook in test_notebooks:
|
||||
self._add_status_event(
|
||||
|
|
@ -176,12 +176,12 @@ class Nutter(NutterApi):
|
|||
logging.debug(
|
||||
'Scheduling execution of: {}'.format(test_notebook.path))
|
||||
func_scheduler.add_function(self._execute_notebook,
|
||||
test_notebook.path, cluster_id, timeout, pull_wait_time)
|
||||
test_notebook.path, cluster_id, timeout, pull_wait_time, notebook_params)
|
||||
return self._run_and_await(func_scheduler)
|
||||
|
||||
def _execute_notebook(self, test_notebook_path, cluster_id, timeout, pull_wait_time):
|
||||
def _execute_notebook(self, test_notebook_path, cluster_id, timeout, pull_wait_time, notebook_params=None):
|
||||
result = self.dbclient.execute_notebook(test_notebook_path,
|
||||
cluster_id, None, timeout, pull_wait_time)
|
||||
cluster_id, timeout, pull_wait_time, notebook_params)
|
||||
self._add_status_event(NutterStatusEvents.TestExecuted,
|
||||
ExecutionResultEventData.from_execution_results(result))
|
||||
logging.debug('Executed: {}'.format(test_notebook_path))
|
||||
|
|
|
|||
|
|
@ -56,9 +56,9 @@ class DatabricksAPIClient(object):
|
|||
|
||||
return workspace_path_obj
|
||||
|
||||
def execute_notebook(self, notebook_path, cluster_id,
|
||||
notebook_params=None, timeout=120,
|
||||
pull_wait_time=DEFAULT_POLL_WAIT_TIME):
|
||||
def execute_notebook(self, notebook_path, cluster_id, timeout=120,
|
||||
pull_wait_time=DEFAULT_POLL_WAIT_TIME,
|
||||
notebook_params=None):
|
||||
if not notebook_path:
|
||||
raise ValueError("empty path")
|
||||
if not cluster_id:
|
||||
|
|
@ -68,7 +68,7 @@ class DatabricksAPIClient(object):
|
|||
"Timeout must be greater than {}".format(self.min_timeout))
|
||||
if notebook_params is not None:
|
||||
if not isinstance(notebook_params, dict):
|
||||
raise ValueError("Parameters must be a dictionary")
|
||||
raise ValueError("Parameters must be in the form of a dictionary (See #run-single-test-notebook section in README)")
|
||||
if pull_wait_time <= 1:
|
||||
pull_wait_time = DEFAULT_POLL_WAIT_TIME
|
||||
|
||||
|
|
|
|||
|
|
@ -3,9 +3,13 @@ Copyright (c) Microsoft Corporation.
|
|||
Licensed under the MIT license.
|
||||
"""
|
||||
|
||||
from .pickleserializable import PickleSerializable
|
||||
import pickle
|
||||
import base64
|
||||
import pickle
|
||||
|
||||
from py4j.protocol import Py4JJavaError
|
||||
|
||||
from .pickleserializable import PickleSerializable
|
||||
|
||||
|
||||
def get_test_results():
|
||||
return TestResults()
|
||||
|
|
@ -30,6 +34,9 @@ class TestResults(PickleSerializable):
|
|||
self.total_execution_time = total_execution_time
|
||||
|
||||
def serialize(self):
|
||||
for i in self.results:
|
||||
if isinstance(i.exception, Py4JJavaError):
|
||||
i.exception = Exception(str(i.exception))
|
||||
bin_data = pickle.dumps(self)
|
||||
return str(base64.encodebytes(bin_data), "utf-8")
|
||||
|
||||
|
|
|
|||
|
|
@ -2,4 +2,5 @@ databricks-api
|
|||
requests
|
||||
fire
|
||||
junit_xml
|
||||
py4j
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,56 @@
|
|||
"""
|
||||
Copyright (c) Microsoft Corporation.
|
||||
|
||||
Licensed under the MIT license.
|
||||
"""
|
||||
|
||||
import common.scheduler as scheduler
|
||||
from common.testexecresults import TestExecResults
|
||||
from common.testresult import TestResults
|
||||
|
||||
from runtime.nutterfixture import NutterFixture
|
||||
|
||||
|
||||
class NutterFixtureParallelRunner(object):
|
||||
"""Helper class to execute tests in parallel."""
|
||||
|
||||
def __init__(self, num_of_workers=1):
|
||||
"""Initialize the runner.
|
||||
|
||||
Args:
|
||||
num_of_workers (int): number of parallel workers.
|
||||
"""
|
||||
self.tests = []
|
||||
self.num_of_workers = num_of_workers
|
||||
|
||||
def add_test_fixture(self, fixture):
|
||||
"""Add a test to the list of tests to run.
|
||||
|
||||
Args:
|
||||
fixture (NutterFixture): the test to add.
|
||||
"""
|
||||
if not isinstance(fixture, NutterFixture):
|
||||
raise TypeError("fixture must be of type NutterFixture")
|
||||
self.tests.append(fixture)
|
||||
|
||||
def execute(self):
|
||||
"""Execute the tests."""
|
||||
sched = scheduler.get_scheduler(self.num_of_workers)
|
||||
|
||||
for i in self.tests:
|
||||
sched.add_function(i.execute_tests)
|
||||
|
||||
results = sched.run_and_wait()
|
||||
|
||||
return self._collect_results(results)
|
||||
|
||||
def _collect_results(self, results):
|
||||
"""Collect all results in a single TestExecResults object."""
|
||||
all_results = TestResults()
|
||||
|
||||
for funcres in results:
|
||||
if funcres.func_result is not None:
|
||||
for testres in funcres.func_result.test_results.results:
|
||||
all_results.append(testres)
|
||||
|
||||
return TestExecResults(all_results)
|
||||
2
setup.py
2
setup.py
|
|
@ -32,5 +32,5 @@ setuptools.setup(
|
|||
"License :: OSI Approved :: MIT License",
|
||||
"Operating System :: OS Independent",
|
||||
],
|
||||
python_requires='>=3.5.2',
|
||||
python_requires='>=3.7.0',
|
||||
)
|
||||
|
|
|
|||
|
|
@ -3,11 +3,15 @@ Copyright (c) Microsoft Corporation.
|
|||
Licensed under the MIT license.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import json
|
||||
from common.testresult import TestResults, TestResult
|
||||
import pickle
|
||||
import base64
|
||||
import json
|
||||
import pickle
|
||||
|
||||
import mock
|
||||
import pytest
|
||||
from common.testresult import TestResult, TestResults
|
||||
from py4j.protocol import Py4JError, Py4JJavaError
|
||||
|
||||
|
||||
def test__testresults_append__type_not_testresult__throws_error():
|
||||
# Arrange
|
||||
|
|
@ -74,6 +78,21 @@ def test__deserialize__invalid_pickle_data__throws_Exception():
|
|||
with pytest.raises(Exception):
|
||||
test_results.deserialize(invalid_pickle)
|
||||
|
||||
def test__deserialize__p4jjavaerror__is_serializable_and_deserializable():
|
||||
# Arrange
|
||||
test_results = TestResults()
|
||||
|
||||
py4j_exception = get_mock_py4j_error_exception(get_mock_gateway_client(), mock_target_id="o123")
|
||||
|
||||
test_results.append(TestResult("Test Name", True, 1, [], py4j_exception))
|
||||
|
||||
with mock.patch('py4j.protocol.get_return_value') as mock_get_return_value:
|
||||
mock_get_return_value.return_value = 'foo'
|
||||
serialized_data = test_results.serialize()
|
||||
deserialized_data = TestResults().deserialize(serialized_data)
|
||||
|
||||
assert test_results == deserialized_data
|
||||
|
||||
|
||||
def test__eq__test_results_equal_but_not_same_ref__are_equal():
|
||||
# Arrange
|
||||
|
|
@ -139,3 +158,25 @@ def test__deserialize__data_is_base64_str__can_deserialize():
|
|||
test_results_from_data = TestResults().deserialize(serialized_str)
|
||||
|
||||
assert test_results == test_results_from_data
|
||||
|
||||
|
||||
def get_mock_gateway_client():
|
||||
mock_client = mock.Mock()
|
||||
mock_client.send_command.return_value = "0"
|
||||
mock_client.converters = []
|
||||
mock_client.is_connected.return_value = True
|
||||
mock_client.deque = mock.Mock()
|
||||
return mock_client
|
||||
|
||||
|
||||
def get_mock_java_object(mock_client, mock_target_id):
|
||||
mock_java_object = mock.Mock()
|
||||
mock_java_object._target_id = mock_target_id
|
||||
mock_java_object._gateway_client = mock_client
|
||||
return mock_java_object
|
||||
|
||||
|
||||
def get_mock_py4j_error_exception(mock_client, mock_target_id):
|
||||
mock_java_object = get_mock_java_object(mock_client, mock_target_id)
|
||||
mock_errmsg = "An error occurred while calling {}.load.".format(mock_target_id)
|
||||
return Py4JJavaError(mock_errmsg, java_exception=mock_java_object)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,142 @@
|
|||
"""
|
||||
Copyright (c) Microsoft Corporation.
|
||||
|
||||
Licensed under the MIT license.
|
||||
"""
|
||||
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from runtime.nutterfixture import NutterFixture
|
||||
from runtime.runner import NutterFixtureParallelRunner
|
||||
|
||||
test_cases = [
|
||||
(1, 1),
|
||||
(1, 2),
|
||||
(2, 1),
|
||||
(2, 2),
|
||||
(3, 1),
|
||||
(3, 2),
|
||||
(3, 3)
|
||||
]
|
||||
|
||||
@pytest.mark.parametrize('num_of_tests, num_of_workers', test_cases)
|
||||
def test__execute_tests__x_tests_x_workers__results_ok(num_of_tests, num_of_workers):
|
||||
# Assemble list of tests
|
||||
runner = NutterFixtureParallelRunner(num_of_workers)
|
||||
for i in range(num_of_tests):
|
||||
test_case = RunnerTestFixture()
|
||||
runner.add_test_fixture(test_case)
|
||||
|
||||
# Execute tests
|
||||
results = runner.execute()
|
||||
|
||||
# Assert results
|
||||
assert len(results.test_results.results) == num_of_tests
|
||||
assert results.test_results.passed() == True
|
||||
|
||||
def test__execute_tests__3_tests_in_sequence_with_failed_assertion__results_ok():
|
||||
# Arrange
|
||||
tests = [
|
||||
RunnerTestFixture(),
|
||||
RunnerTestFixtureFailAssert(),
|
||||
RunnerTestFixture()
|
||||
]
|
||||
|
||||
runner = NutterFixtureParallelRunner()
|
||||
for i in tests:
|
||||
runner.add_test_fixture(i)
|
||||
|
||||
# Act
|
||||
results = runner.execute()
|
||||
|
||||
# Assert
|
||||
assert len(results.test_results.results) == len(tests)
|
||||
assert results.test_results.results[0].passed == True
|
||||
assert results.test_results.results[1].passed == False
|
||||
assert results.test_results.results[2].passed == True
|
||||
|
||||
def test__execute_tests__3_tests_in_sequence_with_run_exception__results_ok():
|
||||
# Arrange
|
||||
tests = [
|
||||
RunnerTestFixture(),
|
||||
RunnerTestFixtureRunException(),
|
||||
RunnerTestFixture()
|
||||
]
|
||||
|
||||
runner = NutterFixtureParallelRunner()
|
||||
for i in tests:
|
||||
runner.add_test_fixture(i)
|
||||
|
||||
# Act
|
||||
results = runner.execute()
|
||||
|
||||
# Assert
|
||||
assert len(results.test_results.results) == len(tests)
|
||||
assert results.test_results.results[0].passed == True
|
||||
assert results.test_results.results[1].passed == False
|
||||
assert results.test_results.results[2].passed == True
|
||||
|
||||
def test__execute_tests__3_tests_in_sequence_with_exec_exception__results_ok():
|
||||
# Arrange
|
||||
tests = [
|
||||
RunnerTestFixture(),
|
||||
RunnerTestFixtureExecuteException(),
|
||||
RunnerTestFixture()
|
||||
]
|
||||
|
||||
runner = NutterFixtureParallelRunner()
|
||||
for i in tests:
|
||||
runner.add_test_fixture(i)
|
||||
|
||||
# Act
|
||||
results = runner.execute()
|
||||
|
||||
# Assert
|
||||
assert len(results.test_results.results) == len(tests) - 1
|
||||
assert results.test_results.results[0].passed == True
|
||||
assert results.test_results.results[1].passed == True
|
||||
|
||||
class RunnerTestFixture(NutterFixture):
|
||||
def before_test(self):
|
||||
pass
|
||||
|
||||
def run_test(self):
|
||||
pass
|
||||
|
||||
def assertion_test(self):
|
||||
assert 1 == 1
|
||||
|
||||
def after_test(self):
|
||||
pass
|
||||
|
||||
class RunnerTestFixtureFailAssert(NutterFixture):
|
||||
def before_test(self):
|
||||
pass
|
||||
|
||||
def run_test(self):
|
||||
pass
|
||||
|
||||
def assertion_test(self):
|
||||
assert 1 != 1
|
||||
|
||||
def after_test(self):
|
||||
pass
|
||||
|
||||
class RunnerTestFixtureRunException(NutterFixture):
|
||||
def before_test(self):
|
||||
pass
|
||||
|
||||
def run_test(self):
|
||||
raise(Exception())
|
||||
|
||||
def assertion_test(self):
|
||||
assert 1 == 1
|
||||
|
||||
def after_test(self):
|
||||
pass
|
||||
|
||||
class RunnerTestFixtureExecuteException(NutterFixture):
|
||||
def execute_tests(self):
|
||||
raise(Exception())
|
||||
Загрузка…
Ссылка в новой задаче