зеркало из https://github.com/microsoft/lisa.git
convert md files to rst files
This commit is contained in:
WenyiQ
Chi Song
Родитель
4bebeb17d5
Коммит
0523bd8faa
63
README.md
63
README.md
|
|
@ -1,63 +0,0 @@
|
|||
# Linux Integration Services Automation (LISA)
|
||||
|
||||
[](https://github.com/microsoft/lisa/actions?query=workflow%3A%22CI+Workflow+for+LISAv3%22+event%3Apush+branch%3Amain)
|
||||
[](https://github.com/microsoft/lisa/blob/main/LICENSE)
|
||||
|
||||
**Linux Integration Services Automation (LISA)** is a Linux quality validation system,
|
||||
which consists of two parts：
|
||||
|
||||
* A test framework to drive test execution.
|
||||
* A set of test suites to verify Linux kernel/distribution quality.
|
||||
|
||||
`LISA` was originally designed and implemented for Microsoft Azure and
|
||||
Windows HyperV platforms; now it can be used to validate Linux quality on
|
||||
any platforms if the proper orchestrator module implemented.
|
||||
|
||||
## Why LISA
|
||||
|
||||
* **Scalable**：Benefit from the appropriate abstractions, `LISA` can be used
|
||||
to test the quality of numerous Linux distributions without duplication of code
|
||||
implementation.
|
||||
|
||||
* **Customizable**: The test suites created on top of `LISA` can be customized
|
||||
to support different quality validation needs.
|
||||
|
||||
* **Support multiple platforms**: `LISA` is created with modular design, to
|
||||
support various of Linux platforms including Microsoft Azure, Windows HyperV,
|
||||
Linux bare metal, and other cloud based platforms.
|
||||
|
||||
* **End-to-end**: `LISA` supports platform specific orchestrator to create and
|
||||
delete test environment automatically; it also provides flexibility to preserve
|
||||
environment for troubleshooting if test failed.
|
||||
|
||||
## Documents
|
||||
|
||||
* [Quick start](docs/quick_start.md)
|
||||
* [Run tests](docs/run_test/run.md)
|
||||
* [Microsoft tests](docs/run_test/microsoft_tests.md)
|
||||
* [Write test cases in LISA](docs/write_test/write_case.md)
|
||||
* [Command line reference](docs/run_test/command_line.md)
|
||||
* [Runbook reference](docs/run_test/runbook.md)
|
||||
* [Extend and customize LISA](docs/write_test/extension.md)
|
||||
* [Run previous version LISA (aka LISAv2)](docs/run_test/run_legacy.md)
|
||||
|
||||
## Contribute
|
||||
|
||||
You are very welcome to contribute. Please follow [the contribution
|
||||
document](docs/contributing.md) for details.
|
||||
|
||||
## History and road map
|
||||
|
||||
The previous LISA called LISAv2, which is in [master
|
||||
branch](https://github.com/microsoft/lisa/tree/master). The previous LISA can be
|
||||
used standalone or called from the current LISA. Learn more from [how to run
|
||||
LISAv2 test cases](docs/run_test/run_legacy.md).
|
||||
|
||||
LISA is in active developing, and a lot of exciting features are implementing.
|
||||
We're listening your [feedback](https://github.com/microsoft/lisa/issues/new).
|
||||
|
||||
## License
|
||||
|
||||
The entire codebase is under [MIT license](LICENSE).
|
||||
|
|
@ -0,0 +1,75 @@
|
|||
Linux Integration Services Automation (LISA)
|
||||
============================================
|
||||
|
||||
|CI Workflow| |GitHub license|
|
||||
|
||||
**Linux Integration Services Automation (LISA)** is a Linux quality
|
||||
validation system, which consists of two parts：
|
||||
|
||||
- A test framework to drive test execution.
|
||||
- A set of test suites to verify Linux kernel/distribution quality.
|
||||
|
||||
``LISA`` was originally designed and implemented for Microsoft Azure and
|
||||
Windows HyperV platforms; now it can be used to validate Linux quality
|
||||
on any platforms if the proper orchestrator module implemented.
|
||||
|
||||
Why LISA
|
||||
--------
|
||||
|
||||
- **Scalable**\ ：Benefit from the appropriate abstractions, ``LISA``
|
||||
can be used to test the quality of numerous Linux distributions
|
||||
without duplication of code implementation.
|
||||
|
||||
- **Customizable**: The test suites created on top of ``LISA`` can be
|
||||
customized to support different quality validation needs.
|
||||
|
||||
- **Support multiple platforms**: ``LISA`` is created with modular
|
||||
design, to support various of Linux platforms including Microsoft
|
||||
Azure, Windows HyperV, Linux bare metal, and other cloud based
|
||||
platforms.
|
||||
|
||||
- **End-to-end**: ``LISA`` supports platform specific orchestrator to
|
||||
create and delete test environment automatically; it also provides
|
||||
flexibility to preserve environment for troubleshooting if test
|
||||
failed.
|
||||
|
||||
Documents
|
||||
---------
|
||||
|
||||
- `Quick start <docs/quick_start.html>`__
|
||||
- `Run tests <docs/run_test/run.html>`__
|
||||
- `Microsoft tests <docs/run_test/microsoft_tests.html>`__
|
||||
- `Write test cases in LISA <docs/write_test/write_case.html>`__
|
||||
- `Command line reference <docs/run_test/command_line.html>`__
|
||||
- `Runbook reference <docs/run_test/runbook.html>`__
|
||||
- `Extend and customize LISA <docs/write_test/extension.html>`__
|
||||
- `Run previous version LISA (aka
|
||||
LISAv2) <docs/run_test/run_legacy.html>`__
|
||||
|
||||
Contribute
|
||||
----------
|
||||
|
||||
You are very welcome to contribute. Please follow `the contribution
|
||||
document <docs/contributing.html>`__ for details.
|
||||
|
||||
History and road map
|
||||
--------------------
|
||||
|
||||
The previous LISA called LISAv2, which is in `master
|
||||
branch <https://github.com/microsoft/lisa/tree/master>`__. The previous
|
||||
LISA can be used standalone or called from the current LISA. Learn more
|
||||
from `how to run LISAv2 test cases <docs/run_test/run_legacy.html>`__.
|
||||
|
||||
LISA is in active developing, and a lot of exciting features are
|
||||
implementing. We’re listening your
|
||||
`feedback <https://github.com/microsoft/lisa/issues/new>`__.
|
||||
|
||||
License
|
||||
-------
|
||||
|
||||
The entire codebase is under `MIT license <LICENSE>`__.
|
||||
|
||||
.. |CI Workflow| image:: https://github.com/microsoft/lisa/workflows/CI%20Workflow/badge.svg?branch=main
|
||||
:target: https://github.com/microsoft/lisa/actions?query=workflow%3A%22CI+Workflow+for+LISAv3%22+event%3Apush+branch%3Amain
|
||||
.. |GitHub license| image:: https://img.shields.io/github/license/microsoft/lisa
|
||||
:target: https://github.com/microsoft/lisa/blob/main/LICENSE
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
# Contributing
|
||||
|
||||
We welcome any form of contribution.
|
||||
|
||||
Please read and follow the guides below to help not only us but also you to get
|
||||
more efficient.
|
||||
|
||||
- [Provide feedback or ask a question](#provide-feedback-or-ask-a-question)
|
||||
- [Propose or implement a new feature](#propose-or-implement-a-new-feature)
|
||||
- [Report a bug](#report-a-bug)
|
||||
- [Contributing fixes or extensions](#contributing-fixes-or-extensions)
|
||||
- [Code of Conduct](#code-of-conduct)
|
||||
|
||||
Most contributions require you to agree to a Contributor License Agreement (CLA)
|
||||
declaring that you have the right to, and do, grant us the rights to use your
|
||||
contribution. For details, visit https://cla.opensource.microsoft.com.
|
||||
|
||||
When you submit a pull request (PR), a CLA bot will automatically determine
|
||||
whether you need to provide a CLA and decorate the PR appropriately (e.g.,
|
||||
status check, comment). Simply follow the instructions provided by the bot. You
|
||||
will only need to do this once across all repos using our CLA.
|
||||
|
||||
## Provide feedback or ask a question
|
||||
|
||||
You are welcome to provide any feedback by [filing an
|
||||
issue](https://github.com/microsoft/lisa/issues/new).
|
||||
|
||||
When there is any question, please check [FAQ and
|
||||
troubleshooting](troubleshooting.md) first. If there is no satisfying answer,
|
||||
don't hesitate to [ask us](https://github.com/microsoft/lisa/issues/new).
|
||||
|
||||
## Propose or implement a new feature
|
||||
|
||||
If you would like to add any additonal feature, please [file an
|
||||
issue](https://github.com/microsoft/lisa/issues/new) to describe why a feature
|
||||
is needed. Furthermore, if you have a good idea how to implement it, please tell
|
||||
us as well. Once we review and agree on that, follow [Contributing fixes or
|
||||
extensions](#contributing-fixes-or-extensions) to implement.
|
||||
|
||||
## Report a bug
|
||||
|
||||
Before you report a bug, please search in [existing
|
||||
issues](https://github.com/microsoft/lisa/issues) first in case similar bugs are
|
||||
in presence. If you do not find a similar bug, feel free to [file a new
|
||||
one](https://github.com/microsoft/lisa/issues/new).
|
||||
|
||||
## Contributing fixes or extensions
|
||||
|
||||
If there is a bug and you want to fix it, feel free to do so by submitting a
|
||||
pull request. We will review it proactively. Once it's approved, we will merge
|
||||
it.
|
||||
|
||||
If you'd like to suggest any extension, please submit a pull request.
|
||||
|
||||
Learn more from [development guidelines](write_test/guidelines.md) and [how to
|
||||
write extensions](write_test/extension.md).
|
||||
|
||||
## Code of Conduct
|
||||
|
||||
This project has adopted the [Microsoft Open Source Code of
|
||||
Conduct](https://opensource.microsoft.com/codeofconduct/). For more information
|
||||
see the [Code of Conduct
|
||||
FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or contact
|
||||
[opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional
|
||||
questions or comments.
|
||||
|
||||
---
|
||||
**Thank you for taking the time to contribute!**
|
||||
|
|
@ -0,0 +1,82 @@
|
|||
Contributing
|
||||
============
|
||||
|
||||
We welcome any form of contribution.
|
||||
|
||||
Please read and follow the guides below to help not only us but also you
|
||||
to get more efficient.
|
||||
|
||||
- `Provide feedback or ask a
|
||||
question <#provide-feedback-or-ask-a-question>`__
|
||||
- `Propose or implement a new
|
||||
feature <#propose-or-implement-a-new-feature>`__
|
||||
- `Report a bug <#report-a-bug>`__
|
||||
- `Contributing fixes or
|
||||
extensions <#contributing-fixes-or-extensions>`__
|
||||
- `Code of Conduct <#code-of-conduct>`__
|
||||
|
||||
Most contributions require you to agree to a Contributor License
|
||||
Agreement (CLA) declaring that you have the right to, and do, grant us
|
||||
the rights to use your contribution. For details, visit
|
||||
https://cla.opensource.microsoft.com.
|
||||
|
||||
When you submit a pull request (PR), a CLA bot will automatically
|
||||
determine whether you need to provide a CLA and decorate the PR
|
||||
appropriately (e.g., status check, comment). Simply follow the
|
||||
instructions provided by the bot. You will only need to do this once
|
||||
across all repos using our CLA.
|
||||
|
||||
Provide feedback or ask a question
|
||||
----------------------------------
|
||||
|
||||
You are welcome to provide any feedback by `filing an
|
||||
issue <https://github.com/microsoft/lisa/issues/new>`__.
|
||||
|
||||
When there is any question, please check `FAQ and
|
||||
troubleshooting <troubleshooting.html>`__ first. If there is no
|
||||
satisfying answer, don’t hesitate to `ask
|
||||
us <https://github.com/microsoft/lisa/issues/new>`__.
|
||||
|
||||
Propose or implement a new feature
|
||||
----------------------------------
|
||||
|
||||
If you would like to add any additonal feature, please `file an
|
||||
issue <https://github.com/microsoft/lisa/issues/new>`__ to describe why
|
||||
a feature is needed. Furthermore, if you have a good idea how to
|
||||
implement it, please tell us as well. Once we review and agree on that,
|
||||
follow `Contributing fixes or
|
||||
extensions <#contributing-fixes-or-extensions>`__ to implement.
|
||||
|
||||
Report a bug
|
||||
------------
|
||||
|
||||
Before you report a bug, please search in `existing
|
||||
issues <https://github.com/microsoft/lisa/issues>`__ first in case
|
||||
similar bugs are in presence. If you do not find a similar bug, feel
|
||||
free to `file a new
|
||||
one <https://github.com/microsoft/lisa/issues/new>`__.
|
||||
|
||||
Contributing fixes or extensions
|
||||
--------------------------------
|
||||
|
||||
If there is a bug and you want to fix it, feel free to do so by
|
||||
submitting a pull request. We will review it proactively. Once it’s
|
||||
approved, we will merge it.
|
||||
|
||||
If you’d like to suggest any extension, please submit a pull request.
|
||||
|
||||
Learn more from `development guidelines <write_test/guidelines.html>`__
|
||||
and `how to write extensions <write_test/extension.html>`__.
|
||||
|
||||
Code of Conduct
|
||||
---------------
|
||||
|
||||
This project has adopted the `Microsoft Open Source Code of
|
||||
Conduct <https://opensource.microsoft.com/codeofconduct/>`__. For more
|
||||
information see the `Code of Conduct
|
||||
FAQ <https://opensource.microsoft.com/codeofconduct/faq/>`__ or contact
|
||||
opencode@microsoft.com with any additional questions or comments.
|
||||
|
||||
--------------
|
||||
|
||||
**Thank you for taking the time to contribute!**
|
||||
|
|
@ -1,87 +0,0 @@
|
|||
# Install LISA on Linux
|
||||
|
||||
- [Install Python on Linux](#install-python-on-linux)
|
||||
- [Install dependencies on Linux](#install-dependencies-on-linux)
|
||||
- [Clone code](#clone-code)
|
||||
- [Install Poetry on Linux](#install-poetry-on-linux)
|
||||
- [Verify installation](#verify-installation)
|
||||
- [FAQ and Troubleshooting](#faq-and-troubleshooting)
|
||||
|
||||
## Install Python on Linux
|
||||
|
||||
LISA has been tested on [Python 3.8 64 bits](https://www.python.org/). The
|
||||
latest version of Python 3 is recommended. If you found LISA is not compatible
|
||||
with higher version Python, [please file an
|
||||
issue](https://github.com/microsoft/lisa/issues/new).
|
||||
|
||||
Run following commands to install Python 3.8 in Ubuntu 20.04.
|
||||
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install software-properties-common -y
|
||||
sudo add-apt-repository ppa:deadsnakes/ppa -y
|
||||
sudo apt install python3.8 python3.8-dev -y
|
||||
```
|
||||
|
||||
## Install dependencies on Linux
|
||||
|
||||
In Linux, for example, on Ubuntu 20.04, please use the command below to install
|
||||
the dependencies:
|
||||
|
||||
```bash
|
||||
sudo apt install git gcc libgirepository1.0-dev libcairo2-dev virtualenv python3-pip -y
|
||||
pip3 install virtualenv
|
||||
```
|
||||
|
||||
## Clone code
|
||||
|
||||
```sh
|
||||
git clone https://github.com/microsoft/lisa.git
|
||||
cd lisa
|
||||
```
|
||||
|
||||
## Install Poetry on Linux
|
||||
|
||||
Poetry is used to manage Python dependencies of LISA.
|
||||
|
||||
:warning: Please enter the root folder of LISA source code to run following
|
||||
commands to install poetry, since Poetry manages dependencies by the working
|
||||
folder.
|
||||
|
||||
```bash
|
||||
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python3 -
|
||||
```
|
||||
|
||||
After running this, you should see `Add export
|
||||
PATH="/home/YOURUSERNAME/.local/bin:$PATH" to your shell configuration file` on
|
||||
the console. Then do
|
||||
|
||||
```bash
|
||||
source $HOME/.profile
|
||||
poetry install
|
||||
```
|
||||
|
||||
## Verify installation
|
||||
|
||||
`lisa.sh` is provided in Linux to wrap `Poetry` for you to run LISA test.
|
||||
|
||||
In Linux, you could create an alias for this simple script. For example, add
|
||||
below line to add to `.bashrc`:
|
||||
|
||||
```bash
|
||||
alias lisa="./lisa.sh"
|
||||
```
|
||||
|
||||
With no argument specified, LISA will run some sample test cases with the
|
||||
default runbook (`examples/runbook/hello_world.yml`) on your local computer. In
|
||||
the root folder of LISA, you can run this command to verify your local LISA
|
||||
environment setup. This test will not modify your computer.
|
||||
|
||||
```bash
|
||||
lisa
|
||||
```
|
||||
|
||||
## FAQ and Troubleshooting
|
||||
|
||||
If there's any problem during the installation, please refer to [FAQ and
|
||||
troubleshooting](troubleshooting.md).
|
||||
|
|
@ -0,0 +1,97 @@
|
|||
Install LISA on Linux
|
||||
=====================
|
||||
|
||||
- `Install Python on Linux <#install-python-on-linux>`__
|
||||
- `Install dependencies on Linux <#install-dependencies-on-linux>`__
|
||||
- `Clone code <#clone-code>`__
|
||||
- `Install Poetry on Linux <#install-poetry-on-linux>`__
|
||||
- `Verify installation <#verify-installation>`__
|
||||
- `FAQ and Troubleshooting <#faq-and-troubleshooting>`__
|
||||
|
||||
Install Python on Linux
|
||||
-----------------------
|
||||
|
||||
LISA has been tested on `Python 3.8 64
|
||||
bits <https://www.python.org/>`__. The latest version of Python 3 is
|
||||
recommended. If you found LISA is not compatible with higher version
|
||||
Python, `please file an
|
||||
issue <https://github.com/microsoft/lisa/issues/new>`__.
|
||||
|
||||
Run following commands to install Python 3.8 in Ubuntu 20.04.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo apt update
|
||||
sudo apt install software-properties-common -y
|
||||
sudo add-apt-repository ppa:deadsnakes/ppa -y
|
||||
sudo apt install python3.8 python3.8-dev -y
|
||||
|
||||
Install dependencies on Linux
|
||||
-----------------------------
|
||||
|
||||
In Linux, for example, on Ubuntu 20.04, please use the command below to
|
||||
install the dependencies:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
sudo apt install git gcc libgirepository1.0-dev libcairo2-dev virtualenv python3-pip -y
|
||||
pip3 install virtualenv
|
||||
|
||||
Clone code
|
||||
----------
|
||||
|
||||
.. code:: sh
|
||||
|
||||
git clone https://github.com/microsoft/lisa.git
|
||||
cd lisa
|
||||
|
||||
Install Poetry on Linux
|
||||
-----------------------
|
||||
|
||||
Poetry is used to manage Python dependencies of LISA.
|
||||
|
||||
:warning: Please enter the root folder of LISA source code to run
|
||||
following commands to install poetry, since Poetry manages dependencies
|
||||
by the working folder.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python3 -
|
||||
|
||||
After running this, you should see
|
||||
``Add export PATH="/home/YOURUSERNAME/.local/bin:$PATH" to your shell configuration file``
|
||||
on the console. Then do
|
||||
|
||||
.. code:: bash
|
||||
|
||||
source $HOME/.profile
|
||||
poetry install
|
||||
|
||||
Verify installation
|
||||
-------------------
|
||||
|
||||
``lisa.sh`` is provided in Linux to wrap ``Poetry`` for you to run LISA
|
||||
test.
|
||||
|
||||
In Linux, you could create an alias for this simple script. For example,
|
||||
add below line to add to ``.bashrc``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
alias lisa="./lisa.sh"
|
||||
|
||||
With no argument specified, LISA will run some sample test cases with
|
||||
the default runbook (``examples/runbook/hello_world.yml``) on your local
|
||||
computer. In the root folder of LISA, you can run this command to verify
|
||||
your local LISA environment setup. This test will not modify your
|
||||
computer.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
lisa
|
||||
|
||||
FAQ and Troubleshooting
|
||||
-----------------------
|
||||
|
||||
If there’s any problem during the installation, please refer to `FAQ and
|
||||
troubleshooting <troubleshooting.html>`__.
|
||||
|
|
@ -1,91 +0,0 @@
|
|||
# Install LISA on Windows
|
||||
|
||||
- [Install Python on Windows](#install-python-on-windows)
|
||||
- [Install dependencies on Windows](#install-dependencies-on-windows)
|
||||
- [Clone code](#clone-code)
|
||||
- [Install Poetry on Windows](#install-poetry-on-windows)
|
||||
- [Verify installation](#verify-installation)
|
||||
- [FAQ and Troubleshooting](#faq-and-troubleshooting)
|
||||
|
||||
:blue_book: On Windows, after you finished an installation, or made an
|
||||
environment variable change, you might need to restart your shell before moving
|
||||
to next step, to make sure your changes take effect.
|
||||
|
||||
:blue_book: Please run your command prompt or shell with elevated privilege
|
||||
(such as `'Run as Administrator'` on Windows) when you see access denied message
|
||||
when install tools.
|
||||
|
||||
## Install Python on Windows
|
||||
|
||||
LISA has been tested on [Python 3.8 64 bits](https://www.python.org/). The
|
||||
latest version of Python 3 is recommended. If you found LISA is not compatible
|
||||
with higher version Python, [please file an
|
||||
issue](https://github.com/microsoft/lisa/issues/new).
|
||||
|
||||
Navigate to [Python releases for
|
||||
Windows](https://www.python.org/downloads/windows/). Download and install
|
||||
*Windows installer (64-bit)* for Python 3.8 64-bits or higher version.
|
||||
|
||||
:warning: Please make sure the `Python` directory and its `Scripts` directory
|
||||
are added to your `PATH` environment variable. To check, type in console
|
||||
|
||||
```powershell
|
||||
echo $env:path
|
||||
```
|
||||
and if you see such two paths in the output, you are good. Otherwise please
|
||||
manually add these two paths.
|
||||
```powercell
|
||||
...;C:\Users\username\AppData\Local\Programs\Python\Python39;C:\Users\username\AppData\Local\Programs\Python\Python39\Scripts;...
|
||||
```
|
||||
|
||||
If this is your first time installing Python, simply check "add Python to PATH"
|
||||
option in installation.
|
||||
|
||||
## Install dependencies on Windows
|
||||
|
||||
Please install `git` on your computer to clone LISA source code from this repo
|
||||
and `pip` for later installation of Poetry.
|
||||
|
||||
In Windows, you need to install [git](https://git-scm.com/downloads),
|
||||
`virtualenv`(by running ```pip install virtualenv```) and [Visual C++
|
||||
redistributable package](https://aka.ms/vs/16/release/vc_redist.x64.exe)
|
||||
|
||||
## Clone code
|
||||
|
||||
```sh
|
||||
git clone https://github.com/microsoft/lisa.git
|
||||
cd lisa
|
||||
```
|
||||
|
||||
## Install Poetry on Windows
|
||||
|
||||
Poetry is used to manage Python dependencies of LISA.
|
||||
|
||||
Enter the `PowerShell` command prompt and then execute below commands:
|
||||
|
||||
```powershell
|
||||
(Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py -UseBasicParsing).Content | python -
|
||||
|
||||
# Add poetry.exe's path to your `PATH` environment variable.
|
||||
$env:PATH += ";$env:APPDATA\Python\Scripts"
|
||||
|
||||
poetry install
|
||||
```
|
||||
|
||||
## Verify installation
|
||||
|
||||
`lisa.cmd` is provided in Windows to wrap `Poetry` for you to run LISA test.
|
||||
|
||||
With no argument specified, LISA will run some sample test cases with the
|
||||
default runbook (`examples/runbook/hello_world.yml`) on your local computer. In
|
||||
the root folder of LISA, you can run this command to verify your local LISA
|
||||
environment setup. This test will not modify your computer.
|
||||
|
||||
```bash
|
||||
.\lisa
|
||||
```
|
||||
|
||||
## FAQ and Troubleshooting
|
||||
|
||||
If there's any problem during the installation, please refer to [FAQ and
|
||||
troubleshooting](troubleshooting.md).
|
||||
|
|
@ -0,0 +1,107 @@
|
|||
Install LISA on Windows
|
||||
=======================
|
||||
|
||||
- `Install Python on Windows <#install-python-on-windows>`__
|
||||
- `Install dependencies on
|
||||
Windows <#install-dependencies-on-windows>`__
|
||||
- `Clone code <#clone-code>`__
|
||||
- `Install Poetry on Windows <#install-poetry-on-windows>`__
|
||||
- `Verify installation <#verify-installation>`__
|
||||
- `FAQ and Troubleshooting <#faq-and-troubleshooting>`__
|
||||
|
||||
:blue_book: On Windows, after you finished an installation, or made an
|
||||
environment variable change, you might need to restart your shell before
|
||||
moving to next step, to make sure your changes take effect.
|
||||
|
||||
:blue_book: Please run your command prompt or shell with elevated
|
||||
privilege (such as ``'Run as Administrator'`` on Windows) when you see
|
||||
access denied message when install tools.
|
||||
|
||||
Install Python on Windows
|
||||
-------------------------
|
||||
|
||||
LISA has been tested on `Python 3.8 64
|
||||
bits <https://www.python.org/>`__. The latest version of Python 3 is
|
||||
recommended. If you found LISA is not compatible with higher version
|
||||
Python, `please file an
|
||||
issue <https://github.com/microsoft/lisa/issues/new>`__.
|
||||
|
||||
Navigate to `Python releases for
|
||||
Windows <https://www.python.org/downloads/windows/>`__. Download and
|
||||
install *Windows installer (64-bit)* for Python 3.8 64-bits or higher
|
||||
version.
|
||||
|
||||
:warning: Please make sure the ``Python`` directory and its ``Scripts``
|
||||
directory are added to your ``PATH`` environment variable. To check,
|
||||
type in console
|
||||
|
||||
.. code:: powershell
|
||||
|
||||
echo $env:path
|
||||
|
||||
and if you see such two paths in the output, you are good. Otherwise
|
||||
please manually add these two paths.
|
||||
|
||||
.. code:: powercell
|
||||
|
||||
...;C:\Users\username\AppData\Local\Programs\Python\Python39;C:\Users\username\AppData\Local\Programs\Python\Python39\Scripts;...
|
||||
|
||||
If this is your first time installing Python, simply check “add Python
|
||||
to PATH” option in installation.
|
||||
|
||||
Install dependencies on Windows
|
||||
-------------------------------
|
||||
|
||||
Please install ``git`` on your computer to clone LISA source code from
|
||||
this repo and ``pip`` for later installation of Poetry.
|
||||
|
||||
In Windows, you need to install `git <https://git-scm.com/downloads>`__,
|
||||
``virtualenv``\ (by running ``pip install virtualenv``) and `Visual C++
|
||||
redistributable
|
||||
package <https://aka.ms/vs/16/release/vc_redist.x64.exe>`__
|
||||
|
||||
Clone code
|
||||
----------
|
||||
|
||||
.. code:: sh
|
||||
|
||||
git clone https://github.com/microsoft/lisa.git
|
||||
cd lisa
|
||||
|
||||
Install Poetry on Windows
|
||||
-------------------------
|
||||
|
||||
Poetry is used to manage Python dependencies of LISA.
|
||||
|
||||
Enter the ``PowerShell`` command prompt and then execute below commands:
|
||||
|
||||
.. code:: powershell
|
||||
|
||||
(Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py -UseBasicParsing).Content | python -
|
||||
|
||||
# Add poetry.exe's path to your `PATH` environment variable.
|
||||
$env:PATH += ";$env:APPDATA\Python\Scripts"
|
||||
|
||||
poetry install
|
||||
|
||||
Verify installation
|
||||
-------------------
|
||||
|
||||
``lisa.cmd`` is provided in Windows to wrap ``Poetry`` for you to run
|
||||
LISA test.
|
||||
|
||||
With no argument specified, LISA will run some sample test cases with
|
||||
the default runbook (``examples/runbook/hello_world.yml``) on your local
|
||||
computer. In the root folder of LISA, you can run this command to verify
|
||||
your local LISA environment setup. This test will not modify your
|
||||
computer.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
.\lisa
|
||||
|
||||
FAQ and Troubleshooting
|
||||
-----------------------
|
||||
|
||||
If there’s any problem during the installation, please refer to `FAQ and
|
||||
troubleshooting <troubleshooting.html>`__.
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
# Quick Start
|
||||
|
||||
- [Prerequisites](#prerequisites)
|
||||
- [Installation](#installation)
|
||||
- [Run tests](#run-tests)
|
||||
- [Write tests](#write-tests)
|
||||
- [FAQ and Troubleshooting](#faq-and-troubleshooting)
|
||||
|
||||
LISA can be used to run test against the local node, or a remote node; if it is
|
||||
used to run against a remote node, you don't need to configure anything on the
|
||||
remote node.
|
||||
|
||||
|
||||
|
||||
## Prerequisites
|
||||
|
||||
LISA can be launched on a Windows or a Linux OS. It needs to be installed on a
|
||||
computer which has network access to the platform and the node to be tested.
|
||||
|
||||
- It is recommended that this computer at least has 2 CPU cores and 4GB memory.
|
||||
|
||||
## Installation
|
||||
|
||||
Follow the installation guide below to install LISA on your OS.
|
||||
|
||||
- [Installation on Linux](installation_linux.md)
|
||||
- [Installation on Windows](installation_windows.md)
|
||||
|
||||
## Run tests
|
||||
|
||||
Follow [Getting started with Azure](run_test/quick_run.md) to quickly run your
|
||||
first test. See [Run LISA](run_test/run.md) for more detailed information.
|
||||
|
||||
## Write tests
|
||||
|
||||
See [Info for
|
||||
developers](write_test/write.md) for more detailed information.
|
||||
|
||||
## FAQ and Troubleshooting
|
||||
|
||||
If there is any problem, please refer to the [FAQ and
|
||||
troubleshooting](troubleshooting.md).
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
Quick Start
|
||||
===========
|
||||
|
||||
- `Prerequisites <#prerequisites>`__
|
||||
- `Installation <#installation>`__
|
||||
- `Run tests <#run-tests>`__
|
||||
- `Write tests <#write-tests>`__
|
||||
- `FAQ and Troubleshooting <#faq-and-troubleshooting>`__
|
||||
|
||||
LISA can be used to run test against the local node, or a remote node;
|
||||
if it is used to run against a remote node, you don’t need to configure
|
||||
anything on the remote node.
|
||||
|
||||
.. figure:: img/deploy.svg
|
||||
:alt: deploy
|
||||
|
||||
deploy
|
||||
|
||||
Prerequisites
|
||||
-------------
|
||||
|
||||
LISA can be launched on a Windows or a Linux OS. It needs to be
|
||||
installed on a computer which has network access to the platform and the
|
||||
node to be tested.
|
||||
|
||||
- It is recommended that this computer at least has 2 CPU cores and 4GB
|
||||
memory.
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
Follow the installation guide below to install LISA on your OS.
|
||||
|
||||
- `Installation on Linux <installation_linux.html>`__
|
||||
- `Installation on Windows <installation_windows.html>`__
|
||||
|
||||
Run tests
|
||||
---------
|
||||
|
||||
Follow `Getting started with Azure <run_test/quick_run.html>`__ to
|
||||
quickly run your first test. See `Run LISA <run_test/run.html>`__ for
|
||||
more detailed information.
|
||||
|
||||
Write tests
|
||||
-----------
|
||||
|
||||
See `Info for developers <write_test/write.html>`__ for more detailed
|
||||
information.
|
||||
|
||||
FAQ and Troubleshooting
|
||||
-----------------------
|
||||
|
||||
If there is any problem, please refer to the `FAQ and
|
||||
troubleshooting <troubleshooting.html>`__.
|
||||
|
|
@ -1,88 +0,0 @@
|
|||
# Command line reference
|
||||
|
||||
- [Common arguments](#common-arguments)
|
||||
- [-r, --runbook](#-r---runbook)
|
||||
- [-d, --debug](#-d---debug)
|
||||
- [-h, --help](#-h---help)
|
||||
- [-v, --variable](#-v---variable)
|
||||
- [run](#run)
|
||||
- [check](#check)
|
||||
- [list](#list)
|
||||
|
||||
## Common arguments
|
||||
|
||||
### -r, --runbook
|
||||
|
||||
Specify the path of [runbook](runbook.md). It can be an absolute path or a
|
||||
relative path. This parameter is required in every run other than run with -h.
|
||||
|
||||
```sh
|
||||
lisa -r ./microsoft/runbook/azure.yml
|
||||
```
|
||||
|
||||
### -d, --debug
|
||||
|
||||
By default, the console will display INFO or higher level logs, but will not
|
||||
display DEBUG level logs. This option enables the console to output DEBUG level
|
||||
logs. Note the log file will not be affected by this setting and will always
|
||||
contain the DEBUG level messages.
|
||||
|
||||
```sh
|
||||
lisa -d
|
||||
```
|
||||
|
||||
### -h, --help
|
||||
|
||||
Show help messages.
|
||||
|
||||
```sh
|
||||
lisa -h
|
||||
```
|
||||
|
||||
### -v, --variable
|
||||
|
||||
Define one or more variables in the format of `name:value`, which will overwrite
|
||||
the value in the YAML file. It can support secret values in the format of
|
||||
`s:name:value`.
|
||||
|
||||
```sh
|
||||
lisa -r ./microsoft/runbook/azure.yml -v location:westus2 -v "gallery_image:Canonical UbuntuServer 18.04-LTS Latest"
|
||||
```
|
||||
|
||||
## run
|
||||
|
||||
An optional command since it is the default operation. The following two lines
|
||||
perform the same operation.
|
||||
|
||||
```sh
|
||||
lisa run -r ./microsoft/runbook/azure.yml
|
||||
```
|
||||
|
||||
```sh
|
||||
lisa -r ./microsoft/runbook/azure.yml
|
||||
```
|
||||
|
||||
## check
|
||||
|
||||
Check whether the specified YAML file and variables are valid.
|
||||
|
||||
```sh
|
||||
lisa check -r ./microsoft/runbook/azure.yml
|
||||
```
|
||||
|
||||
## list
|
||||
|
||||
Output information of this run.
|
||||
|
||||
- `-t` or `--type` specifies the information type. It supports `case`.
|
||||
|
||||
```sh
|
||||
lisa list -r ./microsoft/runbook/local.yml -v tier:0 -t case
|
||||
```
|
||||
|
||||
- With `-a` or `--all`, it will ignore test case selection, and display all test
|
||||
cases.
|
||||
|
||||
```sh
|
||||
lisa list -r ./microsoft/runbook/local.yml -v tier:0 -t case -a
|
||||
```
|
||||
|
|
@ -0,0 +1,101 @@
|
|||
Command line reference
|
||||
======================
|
||||
|
||||
- `Common arguments <#common-arguments>`__
|
||||
|
||||
- `-r, –runbook <#-r---runbook>`__
|
||||
- `-d, –debug <#-d---debug>`__
|
||||
- `-h, –help <#-h---help>`__
|
||||
- `-v, –variable <#-v---variable>`__
|
||||
|
||||
- `run <#run>`__
|
||||
- `check <#check>`__
|
||||
- `list <#list>`__
|
||||
|
||||
Common arguments
|
||||
----------------
|
||||
|
||||
-r, –runbook
|
||||
~~~~~~~~~~~~
|
||||
|
||||
Specify the path of `runbook <runbook.html>`__. It can be an absolute
|
||||
path or a relative path. This parameter is required in every run other
|
||||
than run with -h.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
lisa -r ./microsoft/runbook/azure.yml
|
||||
|
||||
-d, –debug
|
||||
~~~~~~~~~~
|
||||
|
||||
By default, the console will display INFO or higher level logs, but will
|
||||
not display DEBUG level logs. This option enables the console to output
|
||||
DEBUG level logs. Note the log file will not be affected by this setting
|
||||
and will always contain the DEBUG level messages.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
lisa -d
|
||||
|
||||
-h, –help
|
||||
~~~~~~~~~
|
||||
|
||||
Show help messages.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
lisa -h
|
||||
|
||||
-v, –variable
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Define one or more variables in the format of ``name:value``, which will
|
||||
overwrite the value in the YAML file. It can support secret values in
|
||||
the format of ``s:name:value``.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
lisa -r ./microsoft/runbook/azure.yml -v location:westus2 -v "gallery_image:Canonical UbuntuServer 18.04-LTS Latest"
|
||||
|
||||
run
|
||||
---
|
||||
|
||||
An optional command since it is the default operation. The following two
|
||||
lines perform the same operation.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
lisa run -r ./microsoft/runbook/azure.yml
|
||||
|
||||
.. code:: sh
|
||||
|
||||
lisa -r ./microsoft/runbook/azure.yml
|
||||
|
||||
check
|
||||
-----
|
||||
|
||||
Check whether the specified YAML file and variables are valid.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
lisa check -r ./microsoft/runbook/azure.yml
|
||||
|
||||
list
|
||||
----
|
||||
|
||||
Output information of this run.
|
||||
|
||||
- ``-t`` or ``--type`` specifies the information type. It supports
|
||||
``case``.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
lisa list -r ./microsoft/runbook/local.yml -v tier:0 -t case
|
||||
|
||||
- With ``-a`` or ``--all``, it will ignore test case selection, and
|
||||
display all test cases.
|
||||
|
||||
.. code:: sh
|
||||
|
||||
lisa list -r ./microsoft/runbook/local.yml -v tier:0 -t case -a
|
||||
|
|
@ -1,98 +0,0 @@
|
|||
# Microsoft tests
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Some terms](#some-terms)
|
||||
- [Test priority](#test-priority)
|
||||
- [Test tier](#test-tier)
|
||||
- [How to run Microsoft tests](#how-to-run-microsoft-tests)
|
||||
- [Advanced](#advanced)
|
||||
- [Test cases specification](#test-cases-specification)
|
||||
|
||||
## Overview
|
||||
|
||||
The test suite in LISA is called Microsoft tests, which are provided by
|
||||
Microsoft Linux System Group. This document introduces how Microsoft tests were
|
||||
defined, categorized, and how to have the appropriate coverage.
|
||||
|
||||
## Some terms
|
||||
|
||||
### Test priority
|
||||
|
||||
The priority of each test case is determined by the impact if it's failed. The
|
||||
smaller number means the higher priority. For example, if a high-priority test
|
||||
case fails, it means the operating system cannot start. If a lower-priority test
|
||||
case fails, it may mean that a function does not work.
|
||||
|
||||
Note that when multiple test cases fail, we should first check the failure of
|
||||
high-priority test cases to speed up the analysis.
|
||||
|
||||
- **P0**. The system fails/hangs on start/restart using default settings.
|
||||
- **P1**. The system fails/hangs on start/restart using popular configurations,
|
||||
for example, add firewall rules, install some popular packages. There is data
|
||||
loss with popular configurations. The system cannot be connected via network
|
||||
with default settings. The system performance drops significantly, like SRIOV
|
||||
doesn't work as expected; only one CPU core works on multiple core machine; an
|
||||
important feature doesn't work with default settings; or the system can be
|
||||
used with limited functionality.
|
||||
- **P2**. The system fails/hangs on start/restart using unpopular
|
||||
configurations. Data loss with unpopular configurations. The system cannot be
|
||||
connected with popular configurations. The system performance drops obviously.
|
||||
An important feature doesn't work with popular configurations.
|
||||
- **P3**. A feature doesn't work with unpopular configurations with low impact.
|
||||
- **P4**. The system has obvious but not serious problems on long-haul, stress
|
||||
or performance test scenarios.
|
||||
|
||||
Please Note that the above examples do not cover all situations and are for
|
||||
reference. For example, in a cloud environment, one host version may cause
|
||||
problems of some Linux virtual machines. The impact is affected by the
|
||||
percentage the problematic version also.
|
||||
|
||||
### Test tier
|
||||
|
||||
Ideally, all tests should be run to maximize the coverage. But the time and
|
||||
resource are limited, and the risks need to be minimized based on the
|
||||
limitations. In LISA, Microsoft tests are organized into several tiers to have
|
||||
the appropriate coverage using limited resource.
|
||||
|
||||
Test tiers can be T0, T1, T2, T3, T4. It maps to priorities of test cases. For
|
||||
example, T0 means all P0 test cases are selected in a test run. T2 means all P0,
|
||||
P1, P2 test cases are selected in a test run.
|
||||
|
||||
| name | test priorities | time restriction | resource restriction | automation requirement |
|
||||
| ---- | ------------------ | ---------------- | --------------------------------------- | ------------------------------------------------------------ |
|
||||
| T0 | P0 | 5 minutes | single VM | 100% automation, and no need for manual analysis of results. |
|
||||
| T1 | P0, P1 | 2 hours | 2 environments, and two VMs in each one | 100% automation, and no need for manual analysis of results. |
|
||||
| T2 | P0, P1, P2 | 8 hours | 2 environments | 100% automation |
|
||||
| T3 | P0, P1, P2, P3 | 16 hours | 2 environments | 100% automation |
|
||||
| T4 | P0, P1, P2, P3, P4 | no limitation | no limitation | 100% automation |
|
||||
|
||||
## How to run Microsoft tests
|
||||
|
||||
Microsoft tests are organized under the folder `microsoft/runbook`. The root
|
||||
folder contains runbooks for azure, ready, and local. Learn more from [how to
|
||||
run LISA tests](run.md) to run different tiers on an image or existing
|
||||
environment.
|
||||
|
||||
LISA comes with a set of test suites to verify Linux distro/kernel quality on
|
||||
Microsoft's platforms (including Azure, and HyperV). The test cases in those
|
||||
test suites are organized with multiple test `Tiers` (`T0`, `T1`, `T2`, `T3`,
|
||||
`T4`).
|
||||
|
||||
You can specify the test cases by the test tier, with `-v tier:<tier id>`:
|
||||
|
||||
```bash
|
||||
lisa -r ./microsoft/runbook/azure.yml -v subscription_id:<subscription id> -v "admin_private_key_file:<private key file>" -v tier:<tier id>
|
||||
```
|
||||
|
||||
### Advanced
|
||||
|
||||
If you want to verify on specified conditions, like to select some VM size in
|
||||
azure, or select test cases by names, learn more from [runbook
|
||||
reference](runbook.md).
|
||||
|
||||
## Test cases specification
|
||||
|
||||
TODO: add spec of LISA test cases.
|
||||
|
||||
Learn more on not migrated [legacy LISAv2
|
||||
tests](https://github.com/microsoft/lisa/blob/master/Documents/LISAv2-TestCase-Statistics.md).
|
||||
|
|
@ -0,0 +1,136 @@
|
|||
Microsoft tests
|
||||
===============
|
||||
|
||||
- `Overview <#overview>`__
|
||||
- `Some terms <#some-terms>`__
|
||||
|
||||
- `Test priority <#test-priority>`__
|
||||
- `Test tier <#test-tier>`__
|
||||
|
||||
- `How to run Microsoft tests <#how-to-run-microsoft-tests>`__
|
||||
|
||||
- `Advanced <#advanced>`__
|
||||
|
||||
- `Test cases specification <#test-cases-specification>`__
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
The test suite in LISA is called Microsoft tests, which are provided by
|
||||
Microsoft Linux System Group. This document introduces how Microsoft
|
||||
tests were defined, categorized, and how to have the appropriate
|
||||
coverage.
|
||||
|
||||
Some terms
|
||||
----------
|
||||
|
||||
Test priority
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
The priority of each test case is determined by the impact if it’s
|
||||
failed. The smaller number means the higher priority. For example, if a
|
||||
high-priority test case fails, it means the operating system cannot
|
||||
start. If a lower-priority test case fails, it may mean that a function
|
||||
does not work.
|
||||
|
||||
Note that when multiple test cases fail, we should first check the
|
||||
failure of high-priority test cases to speed up the analysis.
|
||||
|
||||
- **P0**. The system fails/hangs on start/restart using default
|
||||
settings.
|
||||
- **P1**. The system fails/hangs on start/restart using popular
|
||||
configurations, for example, add firewall rules, install some popular
|
||||
packages. There is data loss with popular configurations. The system
|
||||
cannot be connected via network with default settings. The system
|
||||
performance drops significantly, like SRIOV doesn’t work as expected;
|
||||
only one CPU core works on multiple core machine; an important
|
||||
feature doesn’t work with default settings; or the system can be used
|
||||
with limited functionality.
|
||||
- **P2**. The system fails/hangs on start/restart using unpopular
|
||||
configurations. Data loss with unpopular configurations. The system
|
||||
cannot be connected with popular configurations. The system
|
||||
performance drops obviously. An important feature doesn’t work with
|
||||
popular configurations.
|
||||
- **P3**. A feature doesn’t work with unpopular configurations with low
|
||||
impact.
|
||||
- **P4**. The system has obvious but not serious problems on long-haul,
|
||||
stress or performance test scenarios.
|
||||
|
||||
Please Note that the above examples do not cover all situations and are
|
||||
for reference. For example, in a cloud environment, one host version may
|
||||
cause problems of some Linux virtual machines. The impact is affected by
|
||||
the percentage the problematic version also.
|
||||
|
||||
Test tier
|
||||
~~~~~~~~~
|
||||
|
||||
Ideally, all tests should be run to maximize the coverage. But the time
|
||||
and resource are limited, and the risks need to be minimized based on
|
||||
the limitations. In LISA, Microsoft tests are organized into several
|
||||
tiers to have the appropriate coverage using limited resource.
|
||||
|
||||
Test tiers can be T0, T1, T2, T3, T4. It maps to priorities of test
|
||||
cases. For example, T0 means all P0 test cases are selected in a test
|
||||
run. T2 means all P0, P1, P2 test cases are selected in a test run.
|
||||
|
||||
+---+--------+-------+-------------------+------------------------------+
|
||||
| n | test | time | resource | automation requirement |
|
||||
| a | prio | r | restriction | |
|
||||
| m | rities | estri | | |
|
||||
| e | | ction | | |
|
||||
+===+========+=======+===================+==============================+
|
||||
| T | P0 | 5 | single VM | 100% automation, and no need |
|
||||
| 0 | | mi | | for manual analysis of |
|
||||
| | | nutes | | results. |
|
||||
+---+--------+-------+-------------------+------------------------------+
|
||||
| T | P0, P1 | 2 | 2 environments, | 100% automation, and no need |
|
||||
| 1 | | hours | and two VMs in | for manual analysis of |
|
||||
| | | | each one | results. |
|
||||
+---+--------+-------+-------------------+------------------------------+
|
||||
| T | P0, | 8 | 2 environments | 100% automation |
|
||||
| 2 | P1, P2 | hours | | |
|
||||
+---+--------+-------+-------------------+------------------------------+
|
||||
| T | P0, | 16 | 2 environments | 100% automation |
|
||||
| 3 | P1, | hours | | |
|
||||
| | P2, P3 | | | |
|
||||
+---+--------+-------+-------------------+------------------------------+
|
||||
| T | P0, | no | no limitation | 100% automation |
|
||||
| 4 | P1, | limit | | |
|
||||
| | P2, | ation | | |
|
||||
| | P3, P4 | | | |
|
||||
+---+--------+-------+-------------------+------------------------------+
|
||||
|
||||
How to run Microsoft tests
|
||||
--------------------------
|
||||
|
||||
Microsoft tests are organized under the folder ``microsoft/runbook``.
|
||||
The root folder contains runbooks for azure, ready, and local. Learn
|
||||
more from `how to run LISA tests <run.html>`__ to run different tiers on
|
||||
an image or existing environment.
|
||||
|
||||
LISA comes with a set of test suites to verify Linux distro/kernel
|
||||
quality on Microsoft’s platforms (including Azure, and HyperV). The test
|
||||
cases in those test suites are organized with multiple test ``Tiers``
|
||||
(``T0``, ``T1``, ``T2``, ``T3``, ``T4``).
|
||||
|
||||
You can specify the test cases by the test tier, with
|
||||
``-v tier:<tier id>``:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
lisa -r ./microsoft/runbook/azure.yml -v subscription_id:<subscription id> -v "admin_private_key_file:<private key file>" -v tier:<tier id>
|
||||
|
||||
Advanced
|
||||
~~~~~~~~
|
||||
|
||||
If you want to verify on specified conditions, like to select some VM
|
||||
size in azure, or select test cases by names, learn more from `runbook
|
||||
reference <runbook.html>`__.
|
||||
|
||||
Test cases specification
|
||||
------------------------
|
||||
|
||||
TODO: add spec of LISA test cases.
|
||||
|
||||
Learn more on not migrated `legacy LISAv2
|
||||
tests <https://github.com/microsoft/lisa/blob/master/Documents/LISAv2-TestCase-Statistics.html>`__.
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
# Run tests on different platforms
|
||||
|
||||
- [Run on Azure](#run-on-azure)
|
||||
- [Run on Azure without deployment](#run-on-azure-without-deployment)
|
||||
- [Run on Ready computers](#run-on-ready-computers)
|
||||
|
||||
## Run on Azure
|
||||
|
||||
Follow the same procedure in [Getting started with Azure](quick_run.md).
|
||||
|
||||
## Run on Azure without deployment
|
||||
|
||||
In addition to deploying a new Azure server and running tests, you can skip the
|
||||
deployment phase and use existing resource group.
|
||||
|
||||
The advantage is that it can run all test cases of Azure. The shortage is that
|
||||
the VM name is fixed, and it should be node-0, so each resource group can put
|
||||
only one VM.
|
||||
|
||||
## Run on Ready computers
|
||||
|
||||
If you have prepared a Linux computer for testing, please run LISA with `ready`
|
||||
runbook:
|
||||
|
||||
1. Get the IP address of your computer for testing.
|
||||
|
||||
2. Get the SSH public/private key pair which can access this computer.
|
||||
|
||||
3. Run LISA with parameters below:
|
||||
|
||||
```bash
|
||||
lisa -r ./microsoft/runbook/ready.yml -v public_address:<public address> -v "user_name:<user name>" -v "admin_private_key_file:<private key file>"
|
||||
```
|
||||
|
||||
The advantage is it's not related to any infra. The shortage is that, some test
|
||||
cases won't run in Ready platform, for example, test cases cannot get serial log
|
||||
from a VM directly.
|
||||
|
||||
`ready` runbook also supports tests which require multiple computers (for
|
||||
example, networking testing); and, it supports password authentication too.
|
||||
Learn more from [runbook reference](runbook.md).
|
||||
|
||||
For a comprehensive introduction to LISA supported test parameters and runbook
|
||||
schema, please read [command-line reference](command_line.md) and [runbook
|
||||
reference](runbook.md).
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
Run tests on different platforms
|
||||
================================
|
||||
|
||||
- `Run on Azure <#run-on-azure>`__
|
||||
- `Run on Azure without
|
||||
deployment <#run-on-azure-without-deployment>`__
|
||||
- `Run on Ready computers <#run-on-ready-computers>`__
|
||||
|
||||
Run on Azure
|
||||
------------
|
||||
|
||||
Follow the same procedure in `Getting started with
|
||||
Azure <quick_run.html>`__.
|
||||
|
||||
Run on Azure without deployment
|
||||
-------------------------------
|
||||
|
||||
In addition to deploying a new Azure server and running tests, you can
|
||||
skip the deployment phase and use existing resource group.
|
||||
|
||||
The advantage is that it can run all test cases of Azure. The shortage
|
||||
is that the VM name is fixed, and it should be node-0, so each resource
|
||||
group can put only one VM.
|
||||
|
||||
Run on Ready computers
|
||||
----------------------
|
||||
|
||||
If you have prepared a Linux computer for testing, please run LISA with
|
||||
``ready`` runbook:
|
||||
|
||||
1. Get the IP address of your computer for testing.
|
||||
|
||||
2. Get the SSH public/private key pair which can access this computer.
|
||||
|
||||
3. Run LISA with parameters below:
|
||||
|
||||
.. code:: bash
|
||||
|
||||
lisa -r ./microsoft/runbook/ready.yml -v public_address:<public address> -v "user_name:<user name>" -v "admin_private_key_file:<private key file>"
|
||||
|
||||
The advantage is it’s not related to any infra. The shortage is that,
|
||||
some test cases won’t run in Ready platform, for example, test cases
|
||||
cannot get serial log from a VM directly.
|
||||
|
||||
``ready`` runbook also supports tests which require multiple computers
|
||||
(for example, networking testing); and, it supports password
|
||||
authentication too. Learn more from `runbook
|
||||
reference <runbook.html>`__.
|
||||
|
||||
For a comprehensive introduction to LISA supported test parameters and
|
||||
runbook schema, please read `command-line
|
||||
reference <command_line.html>`__ and `runbook
|
||||
reference <runbook.html>`__.
|
||||
|
|
@ -1,93 +0,0 @@
|
|||
# Getting started with Azure
|
||||
|
||||
In this document you will find the test procedure using a powerful cloud
|
||||
computing service [Azure](https://azure.microsoft.com/). Follow the steps below
|
||||
to configure your local computer and run LISA test against Linux VM on Azure.
|
||||
|
||||
1. Sign in to Azure
|
||||
|
||||
Make sure either [Azure
|
||||
CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli) or [Azure
|
||||
PowerShell](https://docs.microsoft.com/en-us/powershell/azure/install-az-ps)
|
||||
has been installed on your local computer. Then log in to your Azure
|
||||
subscription to authenticate your current session. LISA also supports other
|
||||
Azure authentications, for more information, please refer to [runbook
|
||||
reference](runbook.md).
|
||||
|
||||
Here, let's choose `Azure CLI` for the setup. You should see a page pop up
|
||||
and all your Azure subscriptions shown in console after running the
|
||||
following command.
|
||||
|
||||
```bash
|
||||
az login
|
||||
```
|
||||
|
||||
2. Get the subscription ID
|
||||
|
||||
A subscription ID is a unique identifier for your server. LISA needs to know
|
||||
the Azure subscription ID for your testing. Run below command to retrieve
|
||||
subscription information.
|
||||
|
||||
```bash
|
||||
az account show --subscription "<your subscription Name>"
|
||||
```
|
||||
|
||||
You should get something in the following format. For now you only need the
|
||||
`<subscription id>` for future use.
|
||||
|
||||
```json
|
||||
{
|
||||
"environmentName": "AzureCloud",
|
||||
"homeTenantId": "<tenant id>",
|
||||
"id": "<subscription id>",
|
||||
"isDefault": true,
|
||||
"managedByTenants": [],
|
||||
"name": "<subscription name>",
|
||||
"state": "Enabled",
|
||||
"tenantId": "<tenant id>",
|
||||
"user": {
|
||||
"name": "<user account>",
|
||||
"type": "user"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Note although the example subscription named "AzureCloud" has the attribute
|
||||
`isDefault` as true, it's not necessary to do so as long as you provide the
|
||||
correct `<subscription id>`.
|
||||
|
||||
3. Prepare SSH key pair
|
||||
|
||||
LISA connects to the Azure test VM by SSH with key authentication; please
|
||||
have your key pair (public key and private key) ready before running the
|
||||
test.
|
||||
|
||||
You can skip this step if you already have a key pair. However, if you don't
|
||||
have a key pair, run below command to create a new one.
|
||||
|
||||
```bash
|
||||
ssh-keygen
|
||||
```
|
||||
|
||||
:warning: Don't use passphrase to protect your key. LISA doesn't support
|
||||
that.
|
||||
|
||||
4. Run LISA
|
||||
|
||||
Use above `<subscription id>` and `<private key file>` to run LISA with the
|
||||
default [runbook](runbook.md). It might take several minutes to complete.
|
||||
|
||||
```bash
|
||||
lisa -r ./microsoft/runbook/azure.yml -v subscription_id:<subscription id> -v "admin_private_key_file:<private key file>"
|
||||
```
|
||||
|
||||
5. Verify test result
|
||||
|
||||
After the test is completed, you can check the LISA console log, or the html
|
||||
report file for the test results. Refer to [Understand test
|
||||
results](understand_results.md) for more detailed explanation of the logs
|
||||
and report. See an example html report as below:
|
||||
|
||||
|
||||
|
||||
## See [Run LISA](run.md) for more advanced usages.
|
||||
|
|
@ -0,0 +1,101 @@
|
|||
Getting started with Azure
|
||||
==========================
|
||||
|
||||
In this document you will find the test procedure using a powerful cloud
|
||||
computing service `Azure <https://azure.microsoft.com/>`__. Follow the
|
||||
steps below to configure your local computer and run LISA test against
|
||||
Linux VM on Azure.
|
||||
|
||||
1. Sign in to Azure
|
||||
|
||||
Make sure either `Azure
|
||||
CLI <https://docs.microsoft.com/en-us/cli/azure/install-azure-cli>`__
|
||||
or `Azure
|
||||
PowerShell <https://docs.microsoft.com/en-us/powershell/azure/install-az-ps>`__
|
||||
has been installed on your local computer. Then log in to your Azure
|
||||
subscription to authenticate your current session. LISA also supports
|
||||
other Azure authentications, for more information, please refer to
|
||||
`runbook reference <runbook.html>`__.
|
||||
|
||||
Here, let’s choose ``Azure CLI`` for the setup. You should see a page
|
||||
pop up and all your Azure subscriptions shown in console after
|
||||
running the following command.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
az login
|
||||
|
||||
2. Get the subscription ID
|
||||
|
||||
A subscription ID is a unique identifier for your server. LISA needs
|
||||
to know the Azure subscription ID for your testing. Run below command
|
||||
to retrieve subscription information.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
az account show --subscription "<your subscription Name>"
|
||||
|
||||
You should get something in the following format. For now you only
|
||||
need the ``<subscription id>`` for future use.
|
||||
|
||||
.. code:: json
|
||||
|
||||
{
|
||||
"environmentName": "AzureCloud",
|
||||
"homeTenantId": "<tenant id>",
|
||||
"id": "<subscription id>",
|
||||
"isDefault": true,
|
||||
"managedByTenants": [],
|
||||
"name": "<subscription name>",
|
||||
"state": "Enabled",
|
||||
"tenantId": "<tenant id>",
|
||||
"user": {
|
||||
"name": "<user account>",
|
||||
"type": "user"
|
||||
}
|
||||
}
|
||||
|
||||
Note although the example subscription named “AzureCloud” has the
|
||||
attribute ``isDefault`` as true, it’s not necessary to do so as long
|
||||
as you provide the correct ``<subscription id>``.
|
||||
|
||||
3. Prepare SSH key pair
|
||||
|
||||
LISA connects to the Azure test VM by SSH with key authentication;
|
||||
please have your key pair (public key and private key) ready before
|
||||
running the test.
|
||||
|
||||
You can skip this step if you already have a key pair. However, if
|
||||
you don’t have a key pair, run below command to create a new one.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
ssh-keygen
|
||||
|
||||
:warning: Don’t use passphrase to protect your key. LISA doesn’t
|
||||
support that.
|
||||
|
||||
4. Run LISA
|
||||
|
||||
Use above ``<subscription id>`` and ``<private key file>`` to run
|
||||
LISA with the default `runbook <runbook.html>`__. It might take
|
||||
several minutes to complete.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
lisa -r ./microsoft/runbook/azure.yml -v subscription_id:<subscription id> -v "admin_private_key_file:<private key file>"
|
||||
|
||||
5. Verify test result
|
||||
|
||||
After the test is completed, you can check the LISA console log, or
|
||||
the html report file for the test results. Refer to `Understand test
|
||||
results <understand_results.html>`__ for more detailed explanation of
|
||||
the logs and report. See an example html report as below:
|
||||
|
||||
.. figure:: ../img/smoke_test_result.png
|
||||
:alt: image
|
||||
|
||||
image
|
||||
|
||||
See `Run LISA <run.html>`__ for more advanced usages.
|
||||
-----------------------------------------------------
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
# Run LISA
|
||||
|
||||
- [Quick start](quick_run.md)
|
||||
- [Getting started with Azure](quick_run.md)
|
||||
- [Learn Microsoft tests](microsoft_tests.md)
|
||||
- [Overview](microsoft_tests.md)
|
||||
- [How to run](microsoft_tests.md#how-to)
|
||||
- [Test cases specification](microsoft_tests.md#test-cases-specification)
|
||||
- [LISA legacy tests](run_legacy.md)
|
||||
- [Preparation](run_legacy.md#preparation)
|
||||
- [Limitations](run_legacy.md#limitation)
|
||||
- [Roadmap](run_legacy.md#roadmap)
|
||||
- [Table of test cases](test_table.md)
|
||||
- [Run tests on different platforms](platform.md)
|
||||
- [Azure](platform.md#run-on-azure)
|
||||
- [Use runbook](runbook.md)
|
||||
- [How to configure](runbook.md#how-to)
|
||||
- [Reference](runbook.md#reference)
|
||||
- [Analyze test results](understand_results.md)
|
||||
- [Understand log messages](understand_results.md#understand-log-messages)
|
||||
- [Troubleshoot failure](understand_results.md#troubleshoot-failure)
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
Run LISA
|
||||
========
|
||||
|
||||
- `Quick start <quick_run.html>`__
|
||||
|
||||
- `Getting started with Azure <quick_run.html>`__
|
||||
|
||||
- `Learn Microsoft tests <microsoft_tests.html>`__
|
||||
|
||||
- `Overview <microsoft_tests.html>`__
|
||||
- `How to run <microsoft_tests.html#how-to>`__
|
||||
- `Test cases
|
||||
specification <microsoft_tests.html#test-cases-specification>`__
|
||||
- `LISA legacy tests <run_legacy.html>`__
|
||||
|
||||
- `Preparation <run_legacy.html#preparation>`__
|
||||
- `Limitations <run_legacy.html#limitation>`__
|
||||
- `Roadmap <run_legacy.html#roadmap>`__
|
||||
|
||||
- `Table of test cases <test_table.html>`__
|
||||
|
||||
- `Run tests on different platforms <platform.html>`__
|
||||
|
||||
- `Azure <platform.html#run-on-azure>`__
|
||||
|
||||
- `Use runbook <runbook.html>`__
|
||||
|
||||
- `How to configure <runbook.html#how-to>`__
|
||||
- `Reference <runbook.html#reference>`__
|
||||
|
||||
- `Analyze test results <understand_results.html>`__
|
||||
|
||||
- `Understand log
|
||||
messages <understand_results.html#understand-log-messages>`__
|
||||
- `Troubleshoot
|
||||
failure <understand_results.html#troubleshoot-failure>`__
|
||||
|
|
@ -1,52 +0,0 @@
|
|||
# How to run Legacy LISAv2 tests
|
||||
|
||||
[LISAv2](https://github.com/microsoft/lisa) brings incredible value to Linux
|
||||
quality on Microsoft virtualization platforms. To increase the value of LISA, we
|
||||
decide to continue to invest and have more innovation on LISA. So, we started
|
||||
the next version.
|
||||
|
||||
During the transition time, we keep validating Linux kernels and distributions.
|
||||
We cannot and don't want to stop to wait for the exciting current LISA. The two
|
||||
versions will be co-existing for a while. So, the current LISA version supports
|
||||
to invoke previous LISAv2.
|
||||
|
||||
With this document, you will know how to run LISAv2 tests in the current LISA.
|
||||
The current LISA clones the LISAv2 repo, start LISAv2, and parse LISAv2 log to
|
||||
generate test results with new format.
|
||||
|
||||
## :construction: WIP :construction:
|
||||
|
||||
Attention: Currently we are migrating previous LISAv2 test cases to this LISA
|
||||
framework. Before we complete the test case migration, only T0 test cases can be
|
||||
launched on LISA. Other test cases can be executed in LISA with "Compatibility
|
||||
mode", which will invoke a shim layer to call LISAv2; so you need to run LISA on
|
||||
a Windows computer and provide the secret file.
|
||||
|
||||
## Preparation
|
||||
|
||||
LISAv2 should run in the latest Windows 10 client 64 bits, or Windows Server
|
||||
2019 editions.
|
||||
|
||||
Follow [LISAv2
|
||||
document](https://github.com/microsoft/lisa/blob/master/README.md) to understand
|
||||
prerequisites and prepare secret files.
|
||||
|
||||
Note, you don't need to clone the LISAv2 yourself, the current LISA will clone
|
||||
it when running.
|
||||
|
||||
## Limitations
|
||||
|
||||
1. The Test in Parallel of LISAv2 doesn't support with the current LISA
|
||||
together. The current LISA implements test matrix to replace current test in
|
||||
parallel in LISAv2. There is no plan for compatibility with LISAv2 on this.
|
||||
2. The LISAv2 results is possible to be missed in a rare race condition. There
|
||||
may be accessing conflict of LISAv2 test log. If it happens on key logs which
|
||||
relates to test result parsing, it may cause the status of results are not
|
||||
shown correctly.
|
||||
|
||||
## Road map
|
||||
|
||||
We're migrating LISAv2 test cases to current LISA by test case priority. We will
|
||||
keep t0 to tx runbooks update to date. When test cases migrated, they will be
|
||||
included in current LISA, and remove from LISAv2. It's transparent when running
|
||||
LISA tests.
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
How to run Legacy LISAv2 tests
|
||||
==============================
|
||||
|
||||
`LISAv2 <https://github.com/microsoft/lisa>`__ brings incredible value
|
||||
to Linux quality on Microsoft virtualization platforms. To increase the
|
||||
value of LISA, we decide to continue to invest and have more innovation
|
||||
on LISA. So, we started the next version.
|
||||
|
||||
During the transition time, we keep validating Linux kernels and
|
||||
distributions. We cannot and don’t want to stop to wait for the exciting
|
||||
current LISA. The two versions will be co-existing for a while. So, the
|
||||
current LISA version supports to invoke previous LISAv2.
|
||||
|
||||
With this document, you will know how to run LISAv2 tests in the current
|
||||
LISA. The current LISA clones the LISAv2 repo, start LISAv2, and parse
|
||||
LISAv2 log to generate test results with new format.
|
||||
|
||||
:construction: WIP :construction:
|
||||
---------------------------------
|
||||
|
||||
Attention: Currently we are migrating previous LISAv2 test cases to this
|
||||
LISA framework. Before we complete the test case migration, only T0 test
|
||||
cases can be launched on LISA. Other test cases can be executed in LISA
|
||||
with “Compatibility mode”, which will invoke a shim layer to call
|
||||
LISAv2; so you need to run LISA on a Windows computer and provide the
|
||||
secret file.
|
||||
|
||||
Preparation
|
||||
-----------
|
||||
|
||||
LISAv2 should run in the latest Windows 10 client 64 bits, or Windows
|
||||
Server 2019 editions.
|
||||
|
||||
Follow `LISAv2
|
||||
document <https://github.com/microsoft/lisa/blob/master/README.html>`__
|
||||
to understand prerequisites and prepare secret files.
|
||||
|
||||
Note, you don’t need to clone the LISAv2 yourself, the current LISA will
|
||||
clone it when running.
|
||||
|
||||
Limitations
|
||||
-----------
|
||||
|
||||
1. The Test in Parallel of LISAv2 doesn’t support with the current LISA
|
||||
together. The current LISA implements test matrix to replace current
|
||||
test in parallel in LISAv2. There is no plan for compatibility with
|
||||
LISAv2 on this.
|
||||
2. The LISAv2 results is possible to be missed in a rare race condition.
|
||||
There may be accessing conflict of LISAv2 test log. If it happens on
|
||||
key logs which relates to test result parsing, it may cause the
|
||||
status of results are not shown correctly.
|
||||
|
||||
Road map
|
||||
--------
|
||||
|
||||
We’re migrating LISAv2 test cases to current LISA by test case priority.
|
||||
We will keep t0 to tx runbooks update to date. When test cases migrated,
|
||||
they will be included in current LISA, and remove from LISAv2. It’s
|
||||
transparent when running LISA tests.
|
||||
|
|
@ -1,565 +0,0 @@
|
|||
# Runbook Reference
|
||||
|
||||
- [What is a runbook](#what-is-a-runbook)
|
||||
- [How-to](#how-to)
|
||||
- [Configure Azure deployment](#configure-azure-deployment)
|
||||
- [Select and set test cases](#select-and-set-test-cases)
|
||||
- [Use variable and secrets](#use-variable-and-secrets)
|
||||
- [Use partial runbook](#use-partial-runbook)
|
||||
- [Use extensions](#use-extensions)
|
||||
- [Reference](#reference)
|
||||
- [name](#name)
|
||||
- [test_project](#test_project)
|
||||
- [test_pass](#test_pass)
|
||||
- [tags](#tags)
|
||||
- [concurrency](#concurrency)
|
||||
- [include](#include)
|
||||
- [path](#path)
|
||||
- [extension](#extension)
|
||||
- [name](#name-1)
|
||||
- [path](#path-1)
|
||||
- [variable](#variable)
|
||||
- [is_secret](#is_secret)
|
||||
- [file](#file)
|
||||
- [name](#name-2)
|
||||
- [value](#value)
|
||||
- [transformer](#transformer)
|
||||
- [type](#type)
|
||||
- [name](#name-3)
|
||||
- [prefix](#prefix)
|
||||
- [depends_on](#depends_on)
|
||||
- [rename](#rename)
|
||||
- [combinator](#combinator)
|
||||
- [grid combinator](#grid-combinator)
|
||||
- [items](#items)
|
||||
- [batch combinator](#batch-combinator)
|
||||
- [items](#items-1)
|
||||
- [notifier](#notifier)
|
||||
- [console](#console)
|
||||
- [log_level](#log_level)
|
||||
- [html](#html)
|
||||
- [path](#path-2)
|
||||
- [auto_open](#auto_open)
|
||||
- [environment](#environment)
|
||||
- [environments](#environments)
|
||||
- [name](#name-4)
|
||||
- [topology](#topology)
|
||||
- [nodes](#nodes)
|
||||
- [nodes_requirement](#nodes_requirement)
|
||||
- [type](#type-1)
|
||||
- [platform](#platform)
|
||||
- [testcase](#testcase)
|
||||
- [criteria](#criteria)
|
||||
|
||||
## What is a runbook
|
||||
|
||||
In simple terms,
|
||||
|
||||
> The **runbook** contains all the configurations of LISA operation. It keeps
|
||||
you from lengthy command-line commands and makes it easy to adjust
|
||||
configurations.
|
||||
|
||||
See [runbook](../write_test/concepts.md#runbook) for further knowledge.
|
||||
|
||||
## How-to
|
||||
|
||||
### Configure Azure deployment
|
||||
|
||||
Below section is for running cases on Azure platform, it specifies:
|
||||
|
||||
- admin_private_key_file: the private key file to access the Azure VM.
|
||||
- subscription_id: Azure VM is created under this subscription.
|
||||
|
||||
```yaml
|
||||
platform:
|
||||
- type: azure
|
||||
admin_private_key_file: $(admin_private_key_file)
|
||||
azure:
|
||||
subscription_id: $(subscription_id)
|
||||
```
|
||||
|
||||
### Select and set test cases
|
||||
|
||||
Below section is to specify P0 and P1 test cases excluding case with name
|
||||
`hello`.
|
||||
|
||||
```yaml
|
||||
testcase:
|
||||
- criteria:
|
||||
priority: [0, 1]
|
||||
- criteria:
|
||||
name: hello
|
||||
select_action: exclude
|
||||
```
|
||||
|
||||
### Use variable and secrets
|
||||
|
||||
Below section is to specify the variable in name/value format. We can use this
|
||||
variable in other field in this format `$(location)`.
|
||||
|
||||
```yaml
|
||||
variable:
|
||||
- name: location
|
||||
value: westus2
|
||||
```
|
||||
|
||||
The value of variable passed from command line will override the value in
|
||||
runbook yaml file.
|
||||
|
||||
```bash
|
||||
lisa -r sample.yml -v "location:eastus2"
|
||||
```
|
||||
|
||||
Below section is to specify the path of yaml file which stores the secret
|
||||
values.
|
||||
|
||||
```yaml
|
||||
variable:
|
||||
- file: secret.yml
|
||||
```
|
||||
|
||||
Content of secret.yml.
|
||||
|
||||
```yaml
|
||||
subscription_id:
|
||||
value: replace_your_subscription_id_here
|
||||
is_secret: true
|
||||
mask: guid
|
||||
```
|
||||
|
||||
### Use partial runbook
|
||||
|
||||
Below three yaml files will be loaded in this sequence.
|
||||
|
||||
```bash
|
||||
loading runbook sample.yml
|
||||
|-- loading include tier.yml
|
||||
| |-- loading include t0.yml
|
||||
```
|
||||
|
||||
The variable values in the included yaml file(s) will be overridden by
|
||||
the including yaml file(s). The relative path is always relative to
|
||||
the including yaml file.
|
||||
|
||||
Part of sample.yml
|
||||
|
||||
```yaml
|
||||
include:
|
||||
- path: ./tier.yml
|
||||
```
|
||||
|
||||
Part of tier.yml.
|
||||
|
||||
```yaml
|
||||
include:
|
||||
- path: ./t$(tier).yml
|
||||
variable:
|
||||
- name: tier
|
||||
value: 0
|
||||
```
|
||||
|
||||
Part of t0.yml.
|
||||
|
||||
```yaml
|
||||
testcase:
|
||||
- criteria:
|
||||
priority: 0
|
||||
```
|
||||
|
||||
### Use extensions
|
||||
|
||||
Below section is to specify path of extensions, the extensions are modules for
|
||||
test cases or extended features.
|
||||
|
||||
```yaml
|
||||
extension:
|
||||
- name: extended_features
|
||||
path: ../../extensions
|
||||
- ../../lisa/microsoft/testsuites/core
|
||||
```
|
||||
|
||||
## Reference
|
||||
|
||||
### name
|
||||
|
||||
type: str, optional, default is "not_named"
|
||||
|
||||
Part of the test run name. This name will be used to group results and put it in
|
||||
title of the html report, also the created resources's name contains this
|
||||
specified str.
|
||||
|
||||
```yaml
|
||||
name: Azure Default
|
||||
```
|
||||
|
||||
### test_project
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
The project name of this test run. This name will be used to group test results
|
||||
in html, it also shows up in notifier message.
|
||||
|
||||
```yaml
|
||||
test_project: Azure Image Weekly Testing
|
||||
```
|
||||
|
||||
### test_pass
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
The test pass name of this test run. This name combined with test project name
|
||||
will be used to group test results in html report, it also shows up in notifier
|
||||
message.
|
||||
|
||||
```yaml
|
||||
test_pass: bvt testing
|
||||
```
|
||||
|
||||
### tags
|
||||
|
||||
type: list of str, optional, default is empty
|
||||
|
||||
The tags of the test run. This name combined with test project name and test
|
||||
pass name will be used to group test results in html report, it also shows up in
|
||||
notifier message.
|
||||
|
||||
```yaml
|
||||
tags:
|
||||
- test
|
||||
- bvt
|
||||
```
|
||||
|
||||
### concurrency
|
||||
|
||||
type: int, optional, default is 1.
|
||||
|
||||
The number of concurrent running environments.
|
||||
|
||||
|
||||
### include
|
||||
|
||||
type: list of path, optional, default is empty
|
||||
|
||||
Share runbook parts for similar runs, including the shared content via
|
||||
that yaml primitive.
|
||||
|
||||
#### path
|
||||
|
||||
It can be absolute or relative path of current runbook.
|
||||
|
||||
### extension
|
||||
|
||||
type: list of path str or name/path pairs, optional, default: empty
|
||||
|
||||
The path and the name of the modules, we can also just specify the extension
|
||||
path directly.
|
||||
|
||||
```yaml
|
||||
extension:
|
||||
- name: ms
|
||||
path: ../../extensions
|
||||
```
|
||||
|
||||
#### name
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
Each extension can be specified a name. With the name, one extension can
|
||||
reference another one, using above example extension, in code we can reference
|
||||
it like this way ms.submodule.
|
||||
|
||||
#### path
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
Path of extension, it can be absolute or relative path of current runbook file.
|
||||
|
||||
### variable
|
||||
|
||||
type: list of path str or name/value pairs, optional, default: empty
|
||||
|
||||
Used to support variables in other fields.
|
||||
|
||||
The values pass from command line has the highest priority, with below example,
|
||||
any places use `${subscription_id}` will be replaced with value `subscription id
|
||||
B`.
|
||||
|
||||
```bash
|
||||
lisa -r ./microsoft/runbook/azure.yml -v "subscription_id:<subscription id A>"
|
||||
```
|
||||
|
||||
```yaml
|
||||
variable:
|
||||
- name: subscription_id
|
||||
value: subscription id B
|
||||
```
|
||||
|
||||
The variable values in the runbook have higher priority than the same
|
||||
variables defined in any included runbook file. Thus, `${location}`
|
||||
will be replaced with value `northeurope` in the following example.
|
||||
|
||||
```yaml
|
||||
include:
|
||||
- path: tier.yml
|
||||
variable:
|
||||
- name: location
|
||||
value: northeurope
|
||||
```
|
||||
|
||||
tier.yml
|
||||
|
||||
```yaml
|
||||
variable:
|
||||
- name: location
|
||||
value: westus2
|
||||
```
|
||||
|
||||
The later defined variables values in runbook have higher priority than the same
|
||||
variables previous defined. `${location}` will be replaced with value
|
||||
`northeurope`.
|
||||
|
||||
```yaml
|
||||
variable:
|
||||
- name: location
|
||||
value: westus2
|
||||
- name: location
|
||||
value: northeurope
|
||||
```
|
||||
|
||||
#### is_secret
|
||||
|
||||
type: bool, optional, default is False.
|
||||
|
||||
When set to True, the value of this variable will be masked in log and other
|
||||
output information.
|
||||
|
||||
Recommend to use secret file or env variable. It's not recommended to specify
|
||||
secret value in runbook directly.
|
||||
|
||||
#### file
|
||||
|
||||
type: list of str, optional, default: empty
|
||||
|
||||
Specify path of other yml files which define variables.
|
||||
|
||||
#### name
|
||||
|
||||
type: str, optional, default is empty.
|
||||
|
||||
Variable name.
|
||||
|
||||
#### value
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
Value of the paired variable.
|
||||
|
||||
### transformer
|
||||
|
||||
type: list of Transformer, default is empty
|
||||
|
||||
#### type
|
||||
|
||||
type: str, required, the type of transformer. See
|
||||
[transformers](../../lisa/transformers) for all transformers.
|
||||
|
||||
#### name
|
||||
|
||||
type: str, optional, default is the `type`.
|
||||
|
||||
Unique name of the transformer. It's depended by other transformers.
|
||||
If it's not specified, it will use the `type` field. But if there are two
|
||||
transformers with the same type, one of them should have name at least.
|
||||
|
||||
#### prefix
|
||||
|
||||
type: str, optional, default is the `name`.
|
||||
|
||||
The prefix of generated variables from this transformer. If it's not specified,
|
||||
it will use the `name` field.
|
||||
|
||||
#### depends_on
|
||||
|
||||
type: list of str, optional, default is None.
|
||||
|
||||
The depended transformers. The depended transformers will run before this one.
|
||||
|
||||
#### rename
|
||||
|
||||
type: Dict[str, str], optional, default is None.
|
||||
|
||||
The variables, which need to be renamed. If the variable exists already, its
|
||||
value will be overwritten by the transformer. For example, `["to_list_image",
|
||||
"image"]` means change the variable name `to_list_image` to `image`. The
|
||||
original variable name must exist in the output variables of the transformer.
|
||||
|
||||
### combinator
|
||||
|
||||
type: str, required.
|
||||
|
||||
The type of combinator, for example, `grid` or `batch`.
|
||||
|
||||
#### grid combinator
|
||||
|
||||
##### items
|
||||
|
||||
type: List[Variable], required.
|
||||
|
||||
The variables which are in the matrix. Each variable must be a list.
|
||||
|
||||
For example,
|
||||
|
||||
```yaml
|
||||
- type: grid
|
||||
items:
|
||||
- name: image
|
||||
value:
|
||||
- Ubuntu
|
||||
- CentOs
|
||||
- name: vm_size
|
||||
value:
|
||||
- Standard_DS2_v2
|
||||
- Standard_DS3_v2
|
||||
- Standard_DS4_v2
|
||||
```
|
||||
|
||||
#### batch combinator
|
||||
|
||||
##### items
|
||||
|
||||
type: List[Dict[str, Any]], required.
|
||||
|
||||
Specify batches of variables. Each batch will run once.
|
||||
|
||||
For example,
|
||||
|
||||
```yaml
|
||||
- type: batch
|
||||
items:
|
||||
- image: Ubuntu
|
||||
vm_size: Standard_DS2_v2
|
||||
- image: Ubuntu
|
||||
vm_size: Standard_DS3_v2
|
||||
- image: CentOS
|
||||
vm_size: Standard_DS3_v2
|
||||
```
|
||||
|
||||
### notifier
|
||||
|
||||
Receive messages during the test run and output them somewhere.
|
||||
|
||||
#### console
|
||||
|
||||
One of notifier type. It outputs messages to the console and file log and
|
||||
demonstrates how to implement notification procedures.
|
||||
|
||||
Example of console notifier:
|
||||
|
||||
```yaml
|
||||
notifier:
|
||||
- type: console
|
||||
log_level: INFO
|
||||
```
|
||||
|
||||
##### log_level
|
||||
|
||||
type: str, optional, default: DEBUG, values: DEBUG, INFO, WARNING...
|
||||
|
||||
Set log level of notification messages.
|
||||
|
||||
#### html
|
||||
|
||||
Output test results in html format. It can be used for local development or as
|
||||
the body of an email.
|
||||
|
||||
##### path
|
||||
|
||||
type: str, optional, default: lisa.html
|
||||
|
||||
Specify the output file name and path.
|
||||
|
||||
##### auto_open
|
||||
|
||||
type: bool, optional, default: False
|
||||
|
||||
When set to True, the html will be opened in the browser after completion.
|
||||
Useful in local run.
|
||||
|
||||
Example of html notifier:
|
||||
|
||||
```yaml
|
||||
notifier:
|
||||
- type: html
|
||||
path: ./lisa.html
|
||||
auto_open: true
|
||||
```
|
||||
|
||||
### environment
|
||||
|
||||
List of environments. For more information, refer [node and
|
||||
environment](https://github.com/microsoft/lisa/blob/main/docs/concepts.md#node-and-environment).
|
||||
|
||||
#### environments
|
||||
|
||||
List of test run environment.
|
||||
|
||||
##### name
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
The name of the environment.
|
||||
|
||||
##### topology
|
||||
|
||||
type: str, optional, default is "subnet"
|
||||
|
||||
The topology of the environment, current only support value "subnet".
|
||||
|
||||
##### nodes
|
||||
|
||||
List of node, it can be a virtual machine on Azure or Hyper-V, bare metal or
|
||||
others. For more information, refer [node and
|
||||
environment](https://github.com/microsoft/lisa/blob/main/docs/concepts.md#node-and-environment).
|
||||
|
||||
##### nodes_requirement
|
||||
|
||||
List of testing required environments, by default node_count (default is 1),
|
||||
core_count (default is 1), memory_mb (default is 512 MB), disk_count (default is
|
||||
1), nic_count (default is 1), gpu_count (default is 0). The node can be created
|
||||
once the node requirement is met.
|
||||
|
||||
###### type
|
||||
|
||||
type: str, optional, default value is "requirement", supported values are
|
||||
"requirement", "remote", "local".
|
||||
|
||||
### platform
|
||||
|
||||
List of platform, default value is "ready", current support values are "ready",
|
||||
"azure".
|
||||
|
||||
### testcase
|
||||
|
||||
type: list of str, optional, default: lisa
|
||||
|
||||
Criteria to select cases.
|
||||
|
||||
#### criteria
|
||||
|
||||
type: list of dictionary, optional, default is empty
|
||||
|
||||
Select test cases by area, category, name, priority or tags combined with select
|
||||
action.
|
||||
|
||||
select_action can be "none", "include", "exclude", "forceInclude" and
|
||||
"forceExclude", default value is "none".
|
||||
|
||||
```yaml
|
||||
testcase:
|
||||
- criteria:
|
||||
priority: 0
|
||||
select_action: include
|
||||
- criteria:
|
||||
priority: 1
|
||||
select_action: exclude
|
||||
```
|
||||
|
|
@ -0,0 +1,661 @@
|
|||
Runbook Reference
|
||||
=================
|
||||
|
||||
- `What is a runbook <#what-is-a-runbook>`__
|
||||
- `How-to <#how-to>`__
|
||||
|
||||
- `Configure Azure deployment <#configure-azure-deployment>`__
|
||||
- `Select and set test cases <#select-and-set-test-cases>`__
|
||||
- `Use variable and secrets <#use-variable-and-secrets>`__
|
||||
- `Use partial runbook <#use-partial-runbook>`__
|
||||
- `Use extensions <#use-extensions>`__
|
||||
|
||||
- `Reference <#reference>`__
|
||||
|
||||
- `name <#name>`__
|
||||
- `test_project <#test_project>`__
|
||||
- `test_pass <#test_pass>`__
|
||||
- `tags <#tags>`__
|
||||
- `concurrency <#concurrency>`__
|
||||
- `include <#include>`__
|
||||
|
||||
- `path <#path>`__
|
||||
|
||||
- `extension <#extension>`__
|
||||
|
||||
- `name <#name-1>`__
|
||||
- `path <#path-1>`__
|
||||
|
||||
- `variable <#variable>`__
|
||||
|
||||
- `is_secret <#is_secret>`__
|
||||
- `file <#file>`__
|
||||
- `name <#name-2>`__
|
||||
- `value <#value>`__
|
||||
|
||||
- `transformer <#transformer>`__
|
||||
|
||||
- `type <#type>`__
|
||||
- `name <#name-3>`__
|
||||
- `prefix <#prefix>`__
|
||||
- `depends_on <#depends_on>`__
|
||||
- `rename <#rename>`__
|
||||
|
||||
- `combinator <#combinator>`__
|
||||
|
||||
- `grid combinator <#grid-combinator>`__
|
||||
|
||||
- `items <#items>`__
|
||||
|
||||
- `batch combinator <#batch-combinator>`__
|
||||
|
||||
- `items <#items-1>`__
|
||||
|
||||
- `notifier <#notifier>`__
|
||||
|
||||
- `console <#console>`__
|
||||
|
||||
- `log_level <#log_level>`__
|
||||
|
||||
- `html <#html>`__
|
||||
|
||||
- `path <#path-2>`__
|
||||
- `auto_open <#auto_open>`__
|
||||
|
||||
- `environment <#environment>`__
|
||||
|
||||
- `environments <#environments>`__
|
||||
|
||||
- `name <#name-4>`__
|
||||
- `topology <#topology>`__
|
||||
- `nodes <#nodes>`__
|
||||
- `nodes_requirement <#nodes_requirement>`__
|
||||
|
||||
- `type <#type-1>`__
|
||||
|
||||
- `platform <#platform>`__
|
||||
- `testcase <#testcase>`__
|
||||
|
||||
- `criteria <#criteria>`__
|
||||
|
||||
What is a runbook
|
||||
-----------------
|
||||
|
||||
In simple terms,
|
||||
|
||||
The **runbook** contains all the configurations of LISA operation. It
|
||||
keeps you from lengthy command-line commands and makes it easy to
|
||||
adjust configurations.
|
||||
|
||||
See `runbook <../write_test/concepts.html#runbook>`__ for further
|
||||
knowledge.
|
||||
|
||||
How-to
|
||||
------
|
||||
|
||||
Configure Azure deployment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Below section is for running cases on Azure platform, it specifies:
|
||||
|
||||
- admin_private_key_file: the private key file to access the Azure VM.
|
||||
- subscription_id: Azure VM is created under this subscription.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
platform:
|
||||
- type: azure
|
||||
admin_private_key_file: $(admin_private_key_file)
|
||||
azure:
|
||||
subscription_id: $(subscription_id)
|
||||
|
||||
Select and set test cases
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Below section is to specify P0 and P1 test cases excluding case with
|
||||
name ``hello``.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
testcase:
|
||||
- criteria:
|
||||
priority: [0, 1]
|
||||
- criteria:
|
||||
name: hello
|
||||
select_action: exclude
|
||||
|
||||
Use variable and secrets
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Below section is to specify the variable in name/value format. We can
|
||||
use this variable in other field in this format ``$(location)``.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
variable:
|
||||
- name: location
|
||||
value: westus2
|
||||
|
||||
The value of variable passed from command line will override the value
|
||||
in runbook yaml file.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
lisa -r sample.yml -v "location:eastus2"
|
||||
|
||||
Below section is to specify the path of yaml file which stores the
|
||||
secret values.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
variable:
|
||||
- file: secret.yml
|
||||
|
||||
Content of secret.yml.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
subscription_id:
|
||||
value: replace_your_subscription_id_here
|
||||
is_secret: true
|
||||
mask: guid
|
||||
|
||||
Use partial runbook
|
||||
~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Below three yaml files will be loaded in this sequence.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
loading runbook sample.yml
|
||||
|-- loading include tier.yml
|
||||
| |-- loading include t0.yml
|
||||
|
||||
The variable values in the included yaml file(s) will be overridden by
|
||||
the including yaml file(s). The relative path is always relative to
|
||||
the including yaml file.
|
||||
|
||||
Part of sample.yml
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
include:
|
||||
- path: ./tier.yml
|
||||
|
||||
Part of tier.yml.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
include:
|
||||
- path: ./t$(tier).yml
|
||||
variable:
|
||||
- name: tier
|
||||
value: 0
|
||||
|
||||
Part of t0.yml.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
testcase:
|
||||
- criteria:
|
||||
priority: 0
|
||||
|
||||
Use extensions
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Below section is to specify path of extensions, the extensions are
|
||||
modules for test cases or extended features.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
extension:
|
||||
- name: extended_features
|
||||
path: ../../extensions
|
||||
- ../../lisa/microsoft/testsuites/core
|
||||
|
||||
Reference
|
||||
---------
|
||||
|
||||
name
|
||||
~~~~
|
||||
|
||||
type: str, optional, default is “not_named”
|
||||
|
||||
Part of the test run name. This name will be used to group results and
|
||||
put it in title of the html report, also the created resources’s name
|
||||
contains this specified str.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
name: Azure Default
|
||||
|
||||
test_project
|
||||
~~~~~~~~~~~~
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
The project name of this test run. This name will be used to group test
|
||||
results in html, it also shows up in notifier message.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
test_project: Azure Image Weekly Testing
|
||||
|
||||
test_pass
|
||||
~~~~~~~~~
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
The test pass name of this test run. This name combined with test
|
||||
project name will be used to group test results in html report, it also
|
||||
shows up in notifier message.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
test_pass: bvt testing
|
||||
|
||||
tags
|
||||
~~~~
|
||||
|
||||
type: list of str, optional, default is empty
|
||||
|
||||
The tags of the test run. This name combined with test project name and
|
||||
test pass name will be used to group test results in html report, it
|
||||
also shows up in notifier message.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
tags:
|
||||
- test
|
||||
- bvt
|
||||
|
||||
concurrency
|
||||
~~~~~~~~~~~
|
||||
|
||||
type: int, optional, default is 1.
|
||||
|
||||
The number of concurrent running environments.
|
||||
|
||||
include
|
||||
~~~~~~
|
||||
|
||||
type: list of path, optional, default is empty
|
||||
|
||||
Share runbook parts for similar runs, including the shared content via
|
||||
that yaml primitive.
|
||||
|
||||
path
|
||||
^^^^
|
||||
|
||||
It can be absolute or relative path of current runbook.
|
||||
|
||||
extension
|
||||
~~~~~~~~~
|
||||
|
||||
type: list of path str or name/path pairs, optional, default: empty
|
||||
|
||||
The path and the name of the modules, we can also just specify the
|
||||
extension path directly.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
extension:
|
||||
- name: ms
|
||||
path: ../../extensions
|
||||
|
||||
.. _name-1:
|
||||
|
||||
name
|
||||
^^^^
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
Each extension can be specified a name. With the name, one extension can
|
||||
reference another one, using above example extension, in code we can
|
||||
reference it like this way ms.submodule.
|
||||
|
||||
.. _path-1:
|
||||
|
||||
path
|
||||
^^^^
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
Path of extension, it can be absolute or relative path of current
|
||||
runbook file.
|
||||
|
||||
variable
|
||||
~~~~~~~~
|
||||
|
||||
type: list of path str or name/value pairs, optional, default: empty
|
||||
|
||||
Used to support variables in other fields.
|
||||
|
||||
The values pass from command line has the highest priority, with below
|
||||
example, any places use ``${subscription_id}`` will be replaced with
|
||||
value ``subscription id B``.
|
||||
|
||||
.. code:: bash
|
||||
|
||||
lisa -r ./microsoft/runbook/azure.yml -v "subscription_id:<subscription id A>"
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
variable:
|
||||
- name: subscription_id
|
||||
value: subscription id B
|
||||
|
||||
The variable values in the runbook have higher priority than the same
|
||||
variables defined in any included runbook file. Thus, `${location}`
|
||||
will be replaced with value `northeurope` in the following example.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
include:
|
||||
- path: tier.yml
|
||||
variable:
|
||||
- name: location
|
||||
value: northeurope
|
||||
|
||||
tier.yml
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
variable:
|
||||
- name: location
|
||||
value: westus2
|
||||
|
||||
The later defined variables values in runbook have higher priority than
|
||||
the same variables previous defined. ``${location}`` will be replaced
|
||||
with value ``northeurope``.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
variable:
|
||||
- name: location
|
||||
value: westus2
|
||||
- name: location
|
||||
value: northeurope
|
||||
|
||||
is_secret
|
||||
^^^^^^^^^
|
||||
|
||||
type: bool, optional, default is False.
|
||||
|
||||
When set to True, the value of this variable will be masked in log and
|
||||
other output information.
|
||||
|
||||
Recommend to use secret file or env variable. It’s not recommended to
|
||||
specify secret value in runbook directly.
|
||||
|
||||
file
|
||||
^^^^
|
||||
|
||||
type: list of str, optional, default: empty
|
||||
|
||||
Specify path of other yml files which define variables.
|
||||
|
||||
.. _name-2:
|
||||
|
||||
name
|
||||
^^^^
|
||||
|
||||
type: str, optional, default is empty.
|
||||
|
||||
Variable name.
|
||||
|
||||
value
|
||||
^^^^^
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
Value of the paired variable.
|
||||
|
||||
transformer
|
||||
~~~~~~~~~~~
|
||||
|
||||
type: list of Transformer, default is empty
|
||||
|
||||
type
|
||||
^^^^
|
||||
|
||||
type: str, required, the type of transformer. See
|
||||
`transformers <../../lisa/transformers>`__ for all transformers.
|
||||
|
||||
.. _name-3:
|
||||
|
||||
name
|
||||
^^^^
|
||||
|
||||
type: str, optional, default is the ``type``.
|
||||
|
||||
Unique name of the transformer. It’s depended by other transformers. If
|
||||
it’s not specified, it will use the ``type`` field. But if there are two
|
||||
transformers with the same type, one of them should have name at least.
|
||||
|
||||
prefix
|
||||
^^^^^^
|
||||
|
||||
type: str, optional, default is the ``name``.
|
||||
|
||||
The prefix of generated variables from this transformer. If it’s not
|
||||
specified, it will use the ``name`` field.
|
||||
|
||||
depends_on
|
||||
^^^^^^^^^^
|
||||
|
||||
type: list of str, optional, default is None.
|
||||
|
||||
The depended transformers. The depended transformers will run before
|
||||
this one.
|
||||
|
||||
rename
|
||||
^^^^^^
|
||||
|
||||
type: Dict[str, str], optional, default is None.
|
||||
|
||||
The variables, which need to be renamed. If the variable exists already,
|
||||
its value will be overwritten by the transformer. For example,
|
||||
``["to_list_image", "image"]`` means change the variable name
|
||||
``to_list_image`` to ``image``. The original variable name must exist in
|
||||
the output variables of the transformer.
|
||||
|
||||
combinator
|
||||
~~~~~~~~~~
|
||||
|
||||
type: str, required.
|
||||
|
||||
The type of combinator, for example, ``grid`` or ``batch``.
|
||||
|
||||
grid combinator
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
items
|
||||
'''''
|
||||
|
||||
type: List\ `Variable <#variable>`__, required.
|
||||
|
||||
The variables which are in the matrix. Each variable must be a list.
|
||||
|
||||
For example,
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- type: grid
|
||||
items:
|
||||
- name: image
|
||||
value:
|
||||
- Ubuntu
|
||||
- CentOs
|
||||
- name: vm_size
|
||||
value:
|
||||
- Standard_DS2_v2
|
||||
- Standard_DS3_v2
|
||||
- Standard_DS4_v2
|
||||
|
||||
batch combinator
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
.. _items-1:
|
||||
|
||||
items
|
||||
'''''
|
||||
|
||||
type: List[Dict[str, Any]], required.
|
||||
|
||||
Specify batches of variables. Each batch will run once.
|
||||
|
||||
For example,
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
- type: batch
|
||||
items:
|
||||
- image: Ubuntu
|
||||
vm_size: Standard_DS2_v2
|
||||
- image: Ubuntu
|
||||
vm_size: Standard_DS3_v2
|
||||
- image: CentOS
|
||||
vm_size: Standard_DS3_v2
|
||||
|
||||
notifier
|
||||
~~~~~~~~
|
||||
|
||||
Receive messages during the test run and output them somewhere.
|
||||
|
||||
console
|
||||
^^^^^^^
|
||||
|
||||
One of notifier type. It outputs messages to the console and file log
|
||||
and demonstrates how to implement notification procedures.
|
||||
|
||||
Example of console notifier:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
notifier:
|
||||
- type: console
|
||||
log_level: INFO
|
||||
|
||||
log_level
|
||||
'''''''''
|
||||
|
||||
type: str, optional, default: DEBUG, values: DEBUG, INFO, WARNING…
|
||||
|
||||
Set log level of notification messages.
|
||||
|
||||
html
|
||||
^^^^
|
||||
|
||||
Output test results in html format. It can be used for local development
|
||||
or as the body of an email.
|
||||
|
||||
.. _path-2:
|
||||
|
||||
path
|
||||
''''
|
||||
|
||||
type: str, optional, default: lisa.html
|
||||
|
||||
Specify the output file name and path.
|
||||
|
||||
auto_open
|
||||
'''''''''
|
||||
|
||||
type: bool, optional, default: False
|
||||
|
||||
When set to True, the html will be opened in the browser after
|
||||
completion. Useful in local run.
|
||||
|
||||
Example of html notifier:
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
notifier:
|
||||
- type: html
|
||||
path: ./lisa.html
|
||||
auto_open: true
|
||||
|
||||
environment
|
||||
~~~~~~~~~~~
|
||||
|
||||
List of environments. For more information, refer `node and
|
||||
environment <https://github.com/microsoft/lisa/blob/main/docs/concepts.html#node-and-environment>`__.
|
||||
|
||||
environments
|
||||
^^^^^^^^^^^^
|
||||
|
||||
List of test run environment.
|
||||
|
||||
.. _name-4:
|
||||
|
||||
name
|
||||
''''
|
||||
|
||||
type: str, optional, default is empty
|
||||
|
||||
The name of the environment.
|
||||
|
||||
topology
|
||||
''''''''
|
||||
|
||||
type: str, optional, default is “subnet”
|
||||
|
||||
The topology of the environment, current only support value “subnet”.
|
||||
|
||||
nodes
|
||||
'''''
|
||||
|
||||
List of node, it can be a virtual machine on Azure or Hyper-V, bare
|
||||
metal or others. For more information, refer `node and
|
||||
environment <https://github.com/microsoft/lisa/blob/main/docs/concepts.html#node-and-environment>`__.
|
||||
|
||||
nodes_requirement
|
||||
'''''''''''''''''
|
||||
|
||||
List of testing required environments, by default node_count (default is
|
||||
1), core_count (default is 1), memory_mb (default is 512 MB), disk_count
|
||||
(default is 1), nic_count (default is 1), gpu_count (default is 0). The
|
||||
node can be created once the node requirement is met.
|
||||
|
||||
.. _type-1:
|
||||
|
||||
type
|
||||
|
||||
|
||||
type: str, optional, default value is “requirement”, supported values
|
||||
are “requirement”, “remote”, “local”.
|
||||
|
||||
platform
|
||||
~~~~~~~~
|
||||
|
||||
List of platform, default value is “ready”, current support values are
|
||||
“ready”, “azure”.
|
||||
|
||||
testcase
|
||||
~~~~~~~~
|
||||
|
||||
type: list of str, optional, default: lisa
|
||||
|
||||
Criteria to select cases.
|
||||
|
||||
criteria
|
||||
^^^^^^^^
|
||||
|
||||
type: list of dictionary, optional, default is empty
|
||||
|
||||
Select test cases by area, category, name, priority or tags combined
|
||||
with select action.
|
||||
|
||||
select_action can be “none”, “include”, “exclude”, “forceInclude” and
|
||||
“forceExclude”, default value is “none”.
|
||||
|
||||
.. code:: yaml
|
||||
|
||||
testcase:
|
||||
- criteria:
|
||||
priority: 0
|
||||
select_action: include
|
||||
- criteria:
|
||||
priority: 1
|
||||
select_action: exclude
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
# Table of Test Cases
|
||||
|
||||
See also [LISAv2
|
||||
Tests](https://github.com/microsoft/lisa/blob/master/Documents/LISAv2-TestCase-Statistics.md)
|
||||
|
||||
| Index | LISA TestName | Priority | Platform | Category | Area |
|
||||
| ----- | :------------ | -------: | :------: | :------: | :---: |
|
||||
|
||||
[//]: <In the format of LISAv2 if applicable>
|
||||
|
|
@ -0,0 +1,10 @@
|
|||
Table of Test Cases
|
||||
===================
|
||||
|
||||
See also `LISAv2
|
||||
Tests <https://github.com/microsoft/lisa/blob/master/Documents/LISAv2-TestCase-Statistics.html>`__
|
||||
|
||||
===== ============= ======== ======== ======== ====
|
||||
Index LISA TestName Priority Platform Category Area
|
||||
===== ============= ======== ======== ======== ====
|
||||
===== ============= ======== ======== ======== ====
|
||||
|
|
@ -1,78 +0,0 @@
|
|||
# Understand test results
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Intermediate results](#intermediate-results)
|
||||
- [Final results](#final-results)
|
||||
|
||||
## Overview
|
||||
|
||||
It's essential to understand the results after running tests. LISA has 7 kinds
|
||||
of test results in total: 3 of which are intermediate results, and 4 of which
|
||||
are final results, as explained in sections below. Each test case can and will
|
||||
be moved from one result to another but can never have two or more results at
|
||||
the same time.
|
||||
|
||||
|
||||
|
||||
## Intermediate results
|
||||
|
||||
An intermediate result shows information of an unfinished test. It will show up
|
||||
when a test changes its state. If a test run terminates because of error or
|
||||
exception prior to running a test case, only the intermediate result will be
|
||||
provided.
|
||||
|
||||
- **QUEUED**
|
||||
|
||||
QUEUED tests are tests that are created, and planned to run (but have not
|
||||
started yet). They are pre-selected by extension/runbook criteria. You can
|
||||
check log to see which test cases are included by such criteria. They suggest
|
||||
that there are some tests waiting to be performed.
|
||||
|
||||
QUEUED tests will try to match every created environment. They will move
|
||||
forward to ASSIGNED if they match any, and to SKIPPED if they match none of
|
||||
the environments.
|
||||
|
||||
- **ASSIGNED**
|
||||
|
||||
ASSIGNED tests are tests that are assigned to an environment, and will start
|
||||
to run, if applicable, once the environment is deployed/initialized. They
|
||||
suggest some environmental setting up is going on.
|
||||
|
||||
ASSIGNED tests will end with FAILED if the environment fails to deploy.
|
||||
Otherwise, they move forward to RUNNING. They will also move backward to
|
||||
QUEUED if the environment is deployed and initialized successfully.
|
||||
|
||||
- **RUNNING**
|
||||
|
||||
RUNNING tests are tests that are in test procedure.
|
||||
|
||||
RUNNING tests will end with one of the following final results.
|
||||
|
||||
## Final results
|
||||
|
||||
A final result shows information of a terminated test. It provides more valuable
|
||||
information than the intermediate result. It only appears in the end of a
|
||||
successful test run.
|
||||
|
||||
- **FAILED**
|
||||
|
||||
FAILED tests are tests that did not finish successfully and terminated because
|
||||
of failures like `LISA exceptions` or `Assertion failure`. You can use them to
|
||||
trace where the problem was and why the problem happened.
|
||||
|
||||
- **PASSED**
|
||||
|
||||
PASSED tests are tests that passed, or at least partially passed, with a
|
||||
special `PASSException` that warns there are minor errors in the run but they
|
||||
do not affect the test result.
|
||||
|
||||
- **SKIPPED**
|
||||
|
||||
SKIPPPED tests are tests that did not start and would no longer run. They
|
||||
suggest failure to meet some requirements in the environments involved with
|
||||
the test.
|
||||
|
||||
- **ATTEMPTED**
|
||||
|
||||
ATTEMPTED tests are a special category of FAILED tests because of known
|
||||
issues, which are not likely to be fixed soon.
|
||||
|
|
@ -0,0 +1,89 @@
|
|||
Understand test results
|
||||
=======================
|
||||
|
||||
- `Overview <#overview>`__
|
||||
- `Intermediate results <#intermediate-results>`__
|
||||
- `Final results <#final-results>`__
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
It’s essential to understand the results after running tests. LISA has 7
|
||||
kinds of test results in total: 3 of which are intermediate results, and
|
||||
4 of which are final results, as explained in sections below. Each test
|
||||
case can and will be moved from one result to another but can never have
|
||||
two or more results at the same time.
|
||||
|
||||
.. figure:: ../img/test_results.png
|
||||
:alt: test_results
|
||||
|
||||
test_results
|
||||
|
||||
Intermediate results
|
||||
--------------------
|
||||
|
||||
An intermediate result shows information of an unfinished test. It will
|
||||
show up when a test changes its state. If a test run terminates because
|
||||
of error or exception prior to running a test case, only the
|
||||
intermediate result will be provided.
|
||||
|
||||
- **QUEUED**
|
||||
|
||||
QUEUED tests are tests that are created, and planned to run (but have
|
||||
not started yet). They are pre-selected by extension/runbook
|
||||
criteria. You can check log to see which test cases are included by
|
||||
such criteria. They suggest that there are some tests waiting to be
|
||||
performed.
|
||||
|
||||
QUEUED tests will try to match every created environment. They will
|
||||
move forward to ASSIGNED if they match any, and to SKIPPED if they
|
||||
match none of the environments.
|
||||
|
||||
- **ASSIGNED**
|
||||
|
||||
ASSIGNED tests are tests that are assigned to an environment, and
|
||||
will start to run, if applicable, once the environment is
|
||||
deployed/initialized. They suggest some environmental setting up is
|
||||
going on.
|
||||
|
||||
ASSIGNED tests will end with FAILED if the environment fails to
|
||||
deploy. Otherwise, they move forward to RUNNING. They will also move
|
||||
backward to QUEUED if the environment is deployed and initialized
|
||||
successfully.
|
||||
|
||||
- **RUNNING**
|
||||
|
||||
RUNNING tests are tests that are in test procedure.
|
||||
|
||||
RUNNING tests will end with one of the following final results.
|
||||
|
||||
Final results
|
||||
-------------
|
||||
|
||||
A final result shows information of a terminated test. It provides more
|
||||
valuable information than the intermediate result. It only appears in
|
||||
the end of a successful test run.
|
||||
|
||||
- **FAILED**
|
||||
|
||||
FAILED tests are tests that did not finish successfully and
|
||||
terminated because of failures like ``LISA exceptions`` or
|
||||
``Assertion failure``. You can use them to trace where the problem
|
||||
was and why the problem happened.
|
||||
|
||||
- **PASSED**
|
||||
|
||||
PASSED tests are tests that passed, or at least partially passed,
|
||||
with a special ``PASSException`` that warns there are minor errors in
|
||||
the run but they do not affect the test result.
|
||||
|
||||
- **SKIPPED**
|
||||
|
||||
SKIPPPED tests are tests that did not start and would no longer run.
|
||||
They suggest failure to meet some requirements in the environments
|
||||
involved with the test.
|
||||
|
||||
- **ATTEMPTED**
|
||||
|
||||
ATTEMPTED tests are a special category of FAILED tests because of
|
||||
known issues, which are not likely to be fixed soon.
|
||||
|
|
@ -1,80 +0,0 @@
|
|||
# Troubleshooting
|
||||
|
||||
- [Installation](#installation)
|
||||
- [How to use LISA in WSL](#how-to-use-lisa-in-wsl)
|
||||
- [Cannot find package after run \`poetry
|
||||
install\`](#cannot-find-package-after-run-poetry-install)
|
||||
- [Error: Poetry could not find a pyproject.toml
|
||||
file](#error-poetry-could-not-find-a-pyprojecttoml-file)
|
||||
- [Using VSCode](#using-vscode)
|
||||
- [Cannot find Python Interpreter by
|
||||
Poetry](#cannot-find-python-interpreter-by-poetry)
|
||||
- [VSCode Python extension no longer supports "python.pythonPath" in
|
||||
"setting.json"](#vscode-python-extension-no-longer-supports-pythonpythonpath-in-settingjson)
|
||||
- [Other issues](#other-issues)
|
||||
- [Poetry related questions](#poetry-related-questions)
|
||||
|
||||
## Installation
|
||||
|
||||
### How to use LISA in WSL
|
||||
|
||||
If you are using WSL, installing Poetry on both Windows and WSL may cause both
|
||||
platforms' versions of Poetry to be on your path, as Windows binaries are mapped
|
||||
into `PATH` of WSL. This means that the WSL `poetry` binary _must_ appear in
|
||||
your `PATH` before the Windows version, otherwise this error will appear:
|
||||
|
||||
> `/usr/bin/env: ‘python\r’: No such file or directory`
|
||||
|
||||
### Cannot find package after run \`poetry install\`
|
||||
|
||||
Poetry is case sensitive, which means it differentiates directories like
|
||||
`C:\abc` and `C:\ABC` in Windows, although Windows in fact does not allow this
|
||||
(as a case insensitive system). When reading the path, please make sure there's
|
||||
no case mismatch in the path.
|
||||
|
||||
### Error: Poetry could not find a pyproject.toml file
|
||||
|
||||
Poetry provides different packages according to the folder, and depends on the
|
||||
`pyproject.toml` file in the current folder. Make sure to run `poetry` in the
|
||||
root folder of LISA.
|
||||
|
||||
## Using VSCode
|
||||
|
||||
### Cannot find Python Interpreter by Poetry
|
||||
|
||||
In the root folder of LISA, run the command below. It will return the path of
|
||||
the virtual environment that Poetry set up. Use that path to find the Python
|
||||
interpreter accordingly (in most cases open the path and look for
|
||||
`\Scripts\python.exe`).
|
||||
|
||||
```powershell
|
||||
poetry env info -p
|
||||
```
|
||||
|
||||
### VSCode Python extension no longer supports "python.pythonPath" in "setting.json"
|
||||
|
||||
> We removed the "python.pythonPath" setting from your settings.json file as the
|
||||
> setting is no longer used by the Python extension. You can get the path of
|
||||
> your selected interpreter in the Python output channel.
|
||||
|
||||
Refer to
|
||||
[DeprecatePythonPath](https://github.com/microsoft/vscode-python/wiki/AB-Experiments)
|
||||
for more information.
|
||||
|
||||
An alternative way is to simply select the Poetry Python interpreter as the
|
||||
default interpreter in the workspace, as in [Cannot find Python Interpreter by
|
||||
Poetry](#cannot-find-python-interpreter-by-poetry)
|
||||
|
||||
## Other issues
|
||||
|
||||
Please check [known issues](https://github.com/microsoft/lisa/issues) or [file a
|
||||
new issue](https://github.com/microsoft/lisa/issues/new) if it doesn't exist.
|
||||
|
||||
### Poetry related questions
|
||||
|
||||
Poetry is very useful to manage dependencies of Python. It's a virtual
|
||||
environment, not a complete interpreter like Conda. So make sure the right and
|
||||
effective version of Python interpreter is installed. You can learn more about
|
||||
Poetry in the official documentation like
|
||||
[installation](https://python-poetry.org/docs/#installation) or
|
||||
[commands](https://python-poetry.org/docs/cli/).
|
||||
|
|
@ -0,0 +1,98 @@
|
|||
Troubleshooting
|
||||
===============
|
||||
|
||||
- `Installation <#installation>`__
|
||||
|
||||
- `How to use LISA in WSL <#how-to-use-lisa-in-wsl>`__
|
||||
- `Cannot find package after run \`poetry
|
||||
install\` <#cannot-find-package-after-run-poetry-install>`__
|
||||
- `Error: Poetry could not find a pyproject.toml
|
||||
file <#error-poetry-could-not-find-a-pyprojecttoml-file>`__
|
||||
|
||||
- `Using VSCode <#using-vscode>`__
|
||||
|
||||
- `Cannot find Python Interpreter by
|
||||
Poetry <#cannot-find-python-interpreter-by-poetry>`__
|
||||
- `VSCode Python extension no longer supports “python.pythonPath” in
|
||||
“setting.json” <#vscode-python-extension-no-longer-supports-pythonpythonpath-in-settingjson>`__
|
||||
|
||||
- `Other issues <#other-issues>`__
|
||||
|
||||
- `Poetry related questions <#poetry-related-questions>`__
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
How to use LISA in WSL
|
||||
~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If you are using WSL, installing Poetry on both Windows and WSL may
|
||||
cause both platforms’ versions of Poetry to be on your path, as Windows
|
||||
binaries are mapped into ``PATH`` of WSL. This means that the WSL
|
||||
``poetry`` binary *must* appear in your ``PATH`` before the Windows
|
||||
version, otherwise this error will appear:
|
||||
|
||||
``/usr/bin/env: ‘python\r’: No such file or directory``
|
||||
|
||||
Cannot find package after run \`poetry install\`
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Poetry is case sensitive, which means it differentiates directories like
|
||||
``C:\abc`` and ``C:\ABC`` in Windows, although Windows in fact does not
|
||||
allow this (as a case insensitive system). When reading the path, please
|
||||
make sure there’s no case mismatch in the path.
|
||||
|
||||
Error: Poetry could not find a pyproject.toml file
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Poetry provides different packages according to the folder, and depends
|
||||
on the ``pyproject.toml`` file in the current folder. Make sure to run
|
||||
``poetry`` in the root folder of LISA.
|
||||
|
||||
Using VSCode
|
||||
------------
|
||||
|
||||
Cannot find Python Interpreter by Poetry
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In the root folder of LISA, run the command below. It will return the
|
||||
path of the virtual environment that Poetry set up. Use that path to
|
||||
find the Python interpreter accordingly (in most cases open the path and
|
||||
look for ``\Scripts\python.exe``).
|
||||
|
||||
.. code:: powershell
|
||||
|
||||
poetry env info -p
|
||||
|
||||
VSCode Python extension no longer supports “python.pythonPath” in “setting.json”
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
We removed the “python.pythonPath” setting from your settings.json
|
||||
file as the setting is no longer used by the Python extension. You
|
||||
can get the path of your selected interpreter in the Python output
|
||||
channel.
|
||||
|
||||
Refer to
|
||||
`DeprecatePythonPath <https://github.com/microsoft/vscode-python/wiki/AB-Experiments>`__
|
||||
for more information.
|
||||
|
||||
An alternative way is to simply select the Poetry Python interpreter as
|
||||
the default interpreter in the workspace, as in `Cannot find Python
|
||||
Interpreter by Poetry <#cannot-find-python-interpreter-by-poetry>`__
|
||||
|
||||
Other issues
|
||||
------------
|
||||
|
||||
Please check `known issues <https://github.com/microsoft/lisa/issues>`__
|
||||
or `file a new issue <https://github.com/microsoft/lisa/issues/new>`__
|
||||
if it doesn’t exist.
|
||||
|
||||
Poetry related questions
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Poetry is very useful to manage dependencies of Python. It’s a virtual
|
||||
environment, not a complete interpreter like Conda. So make sure the
|
||||
right and effective version of Python interpreter is installed. You can
|
||||
learn more about Poetry in the official documentation like
|
||||
`installation <https://python-poetry.org/docs/#installation>`__ or
|
||||
`commands <https://python-poetry.org/docs/cli/>`__.
|
||||
|
|
@ -1,152 +0,0 @@
|
|||
# Basic concepts
|
||||
|
||||
- [Test case and Test suite](#test-case-and-test-suite)
|
||||
- [Node and Environment](#node-and-environment)
|
||||
- [Extension](#extension)
|
||||
- [Platform](#platform)
|
||||
- [Tool and Script](#tool-and-script)
|
||||
- [Feature](#feature)
|
||||
- [Combinator](#combinator)
|
||||
- [Grid combinator](#grid-combinator)
|
||||
- [Batch combinator](#batch-combinator)
|
||||
- [Transformer](#transformer)
|
||||
- [Runbook](#runbook)
|
||||
- [Requirement and Capability](#requirement-and-capability)
|
||||
|
||||
## Test case and Test suite
|
||||
|
||||
A **test case** is the smallest unit that runs and generates test results.
|
||||
|
||||
A **test suite** contains one or more test cases that focus on certain functions
|
||||
or tested domains.
|
||||
|
||||
Test cases in the same test suite serve for the same or similar purpose and
|
||||
share the same setup/clean-up method. Each test case describes its own steps and
|
||||
sets the appropriate priority.
|
||||
|
||||
Both test cases and test suites have metadata, including an overall description,
|
||||
priority, test requirement, and other information.
|
||||
|
||||
## Node and Environment
|
||||
|
||||
A **node** is a system under test. Usually a node is a virtual machine (VM) in a
|
||||
virtualization platform, but it can also be a physical computer. LISA supports
|
||||
further customization of other types of nodes as well.
|
||||
|
||||
An **environment** contains one or more nodes and how they are connected. For
|
||||
example, certain Hyper-V tests need to be run on two hosts. This kind of
|
||||
information is not within the scope of the node and is thus described at the
|
||||
environmental level.
|
||||
|
||||
## Extension
|
||||
|
||||
LISA wraps the shared code logic in different kinds of extensions as follows.
|
||||
|
||||
### Platform
|
||||
|
||||
A **platform** provides a test environment, such as Azure, Hyper-V or WSL. In
|
||||
sequence, a platform will call its API to measure test requirements, deploy the
|
||||
environment and delete the used environment.
|
||||
|
||||
In most cases, after the environment is deployed, it has nothing to do with the
|
||||
source platform. In this way, tests from different platforms can be run in a
|
||||
consistent manner.
|
||||
|
||||
### Tool and Script
|
||||
|
||||
A **tool** includes runnable commands in a node, it needs to be installed and
|
||||
can be installed in many ways.
|
||||
|
||||
A **script** is also considered a tool, except it can only be uploaded to a
|
||||
node.
|
||||
|
||||
In different Linux distributions, tools may have different installation methods,
|
||||
commands, or command-line parameters. LISA tools, however, provide a simple test
|
||||
interface with consistency to focus on verifying logic and do not need to deal
|
||||
with the diversity of distributions.
|
||||
|
||||
A collection of tools are provided on each node. After one tool is initialized,
|
||||
it will be added to the collection, and will be available during the lifetime of
|
||||
the node.
|
||||
|
||||
### Feature
|
||||
|
||||
A **feature** is like a tool, except it supports operations outside the node,
|
||||
for example, getting serial logs of nodes, adding disks, etc.
|
||||
|
||||
Note that there are test cases that require certain features to run. To run such
|
||||
test cases, the platform must support the required features.
|
||||
|
||||
### Combinator
|
||||
|
||||
The **combinator** helps to run large-scale test cases with different variables,
|
||||
such as running multiple images with different VM sizes or other variables.
|
||||
There are two kinds of combinators provided: grid combinator and batch
|
||||
combinator. You can also write your own combinator.
|
||||
|
||||
#### Grid combinator
|
||||
|
||||
A **grid combinator** selects all the combination of provided images and
|
||||
variables. For example, if you are to run 10 images with 3 different VM sizes
|
||||
([`img1`, `vm1`], [`img1`, `vm2`], [`img1`, `vm3`], [`img2`, `vm1`], etc.), the
|
||||
grid combinator saves you from expanding that combination out manually in
|
||||
complete form and would automatically give all $10\times3=30$ combinations.
|
||||
|
||||
#### Batch combinator
|
||||
|
||||
A **batch combinator** instead runs a batch of specified combinations. For
|
||||
example, maybe we only want `img1` to run with `vm1` but we want `img2` to run
|
||||
with `vm1`, `vm2` and `vm3`, you can give the batch combinator such
|
||||
specification and all tests would run as you expect.
|
||||
|
||||
### Transformer
|
||||
|
||||
The **transformers** are used to prepare test environments and others before
|
||||
test runs. A transformer generates variables from other variables, and multiple
|
||||
transformers can run one by one to achieve complex transformation. For example,
|
||||
the first transformer can build Linux kernel and another one can save the VM to
|
||||
a VHD. The two transformers can be reused in other workflows.
|
||||
|
||||
## Runbook
|
||||
|
||||
The **runbook** contains all the configurations of LISA operation. It keeps you
|
||||
from lengthy command-line commands and makes it easy to adjust configurations.
|
||||
|
||||
The previous version of LISA is powerful and supports many scenarios. However,
|
||||
the trade-off there is increased complexity of the command-line parameters, and
|
||||
it relies on a secret file to work. The current version of LISA provides a
|
||||
consistent way to manage configuration and maximize customization capabilities
|
||||
through a runbook.
|
||||
|
||||
One runbook can refer to other runbooks so that test case selection, platform
|
||||
configuration and other options can be separated. It helps reduce redundancy.
|
||||
|
||||
The runbook supports custom variables, and these variables can be provided in
|
||||
the runbook, command line or environment variables. Variables are only used in
|
||||
the runbook and are resolved to actual values before the start of the test.
|
||||
|
||||
The configuration of the runbook can be extended to certain components. For
|
||||
example, when defining a notifier, you can define its configuration structure
|
||||
together. The corresponding configuration will be loaded into the notifier with
|
||||
the same schema. The extended configuration also supports variables.
|
||||
|
||||
## Requirement and Capability
|
||||
|
||||
A test case may have certain **requirements** for the test environment. For
|
||||
example, it may need two nodes, four CPU cores, serial logging, a certain Linux
|
||||
distribution or other requirements. When writing test cases, other requirements
|
||||
of the test cases in addition to the default ones should be defined explicitly.
|
||||
|
||||
The platform provides environments with different **capability**. The
|
||||
capabilities of environments are loaded from the platform's API. Only if the
|
||||
capability of an environment meets the requirements of a test case, the test
|
||||
case can be run on this environment.
|
||||
|
||||
The figure below shows a test case that requires at least one core, 2G memory
|
||||
and three network interface cards (NIC). The Azure VM Standard_DS2_v2 in the
|
||||
middle meets the first two requirements, as it has two cores and 7G memory; but
|
||||
because it can only have two NICs at most, it fails the test requirements and
|
||||
thus the test case will be performed on another VM. Standard_DS3_v2 supports up
|
||||
to four NICs and it meets all three requirements, so it will run the test cases.
|
||||
|
||||
|
||||
|
|
@ -0,0 +1,184 @@
|
|||
Basic concepts
|
||||
==============
|
||||
|
||||
- `Test case and Test suite <#test-case-and-test-suite>`__
|
||||
- `Node and Environment <#node-and-environment>`__
|
||||
- `Extension <#extension>`__
|
||||
|
||||
- `Platform <#platform>`__
|
||||
- `Tool and Script <#tool-and-script>`__
|
||||
- `Feature <#feature>`__
|
||||
- `Combinator <#combinator>`__
|
||||
|
||||
- `Grid combinator <#grid-combinator>`__
|
||||
- `Batch combinator <#batch-combinator>`__
|
||||
|
||||
- `Transformer <#transformer>`__
|
||||
|
||||
- `Runbook <#runbook>`__
|
||||
- `Requirement and Capability <#requirement-and-capability>`__
|
||||
|
||||
Test case and Test suite
|
||||
------------------------
|
||||
|
||||
A **test case** is the smallest unit that runs and generates test
|
||||
results.
|
||||
|
||||
A **test suite** contains one or more test cases that focus on certain
|
||||
functions or tested domains.
|
||||
|
||||
Test cases in the same test suite serve for the same or similar purpose
|
||||
and share the same setup/clean-up method. Each test case describes its
|
||||
own steps and sets the appropriate priority.
|
||||
|
||||
Both test cases and test suites have metadata, including an overall
|
||||
description, priority, test requirement, and other information.
|
||||
|
||||
Node and Environment
|
||||
--------------------
|
||||
|
||||
A **node** is a system under test. Usually a node is a virtual machine
|
||||
(VM) in a virtualization platform, but it can also be a physical
|
||||
computer. LISA supports further customization of other types of nodes as
|
||||
well.
|
||||
|
||||
An **environment** contains one or more nodes and how they are
|
||||
connected. For example, certain Hyper-V tests need to be run on two
|
||||
hosts. This kind of information is not within the scope of the node and
|
||||
is thus described at the environmental level.
|
||||
|
||||
Extension
|
||||
---------
|
||||
|
||||
LISA wraps the shared code logic in different kinds of extensions as
|
||||
follows.
|
||||
|
||||
Platform
|
||||
~~~~~~~~
|
||||
|
||||
A **platform** provides a test environment, such as Azure, Hyper-V or
|
||||
WSL. In sequence, a platform will call its API to measure test
|
||||
requirements, deploy the environment and delete the used environment.
|
||||
|
||||
In most cases, after the environment is deployed, it has nothing to do
|
||||
with the source platform. In this way, tests from different platforms
|
||||
can be run in a consistent manner.
|
||||
|
||||
Tool and Script
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
A **tool** includes runnable commands in a node, it needs to be
|
||||
installed and can be installed in many ways.
|
||||
|
||||
A **script** is also considered a tool, except it can only be uploaded
|
||||
to a node.
|
||||
|
||||
In different Linux distributions, tools may have different installation
|
||||
methods, commands, or command-line parameters. LISA tools, however,
|
||||
provide a simple test interface with consistency to focus on verifying
|
||||
logic and do not need to deal with the diversity of distributions.
|
||||
|
||||
A collection of tools are provided on each node. After one tool is
|
||||
initialized, it will be added to the collection, and will be available
|
||||
during the lifetime of the node.
|
||||
|
||||
Feature
|
||||
~~~~~~~
|
||||
|
||||
A **feature** is like a tool, except it supports operations outside the
|
||||
node, for example, getting serial logs of nodes, adding disks, etc.
|
||||
|
||||
Note that there are test cases that require certain features to run. To
|
||||
run such test cases, the platform must support the required features.
|
||||
|
||||
Combinator
|
||||
~~~~~~~~~~
|
||||
|
||||
The **combinator** helps to run large-scale test cases with different
|
||||
variables, such as running multiple images with different VM sizes or
|
||||
other variables. There are two kinds of combinators provided: grid
|
||||
combinator and batch combinator. You can also write your own combinator.
|
||||
|
||||
Grid combinator
|
||||
^^^^^^^^^^^^^^^
|
||||
|
||||
A **grid combinator** selects all the combination of provided images and
|
||||
variables. For example, if you are to run 10 images with 3 different VM
|
||||
sizes ([``img1``, ``vm1``], [``img1``, ``vm2``], [``img1``, ``vm3``],
|
||||
[``img2``, ``vm1``], etc.), the grid combinator saves you from expanding
|
||||
that combination out manually in complete form and would automatically
|
||||
give all :math:`10\times3=30` combinations.
|
||||
|
||||
Batch combinator
|
||||
^^^^^^^^^^^^^^^^
|
||||
|
||||
A **batch combinator** instead runs a batch of specified combinations.
|
||||
For example, maybe we only want ``img1`` to run with ``vm1`` but we want
|
||||
``img2`` to run with ``vm1``, ``vm2`` and ``vm3``, you can give the
|
||||
batch combinator such specification and all tests would run as you
|
||||
expect.
|
||||
|
||||
Transformer
|
||||
~~~~~~~~~~~
|
||||
|
||||
The **transformers** are used to prepare test environments and others before
|
||||
test runs. A transformer generates variables from other variables, and multiple
|
||||
transformers can run one by one to achieve complex transformation. For
|
||||
example, the first transformer can build Linux kernel and another one
|
||||
can save the VM to a VHD. The two transformers can be reused in other
|
||||
workflows.
|
||||
|
||||
Runbook
|
||||
-------
|
||||
|
||||
The **runbook** contains all the configurations of LISA operation. It
|
||||
keeps you from lengthy command-line commands and makes it easy to adjust
|
||||
configurations.
|
||||
|
||||
The previous version of LISA is powerful and supports many scenarios.
|
||||
However, the trade-off there is increased complexity of the command-line
|
||||
parameters, and it relies on a secret file to work. The current version
|
||||
of LISA provides a consistent way to manage configuration and maximize
|
||||
customization capabilities through a runbook.
|
||||
|
||||
One runbook can refer to other runbooks so that test case selection,
|
||||
platform configuration and other options can be separated. It helps
|
||||
reduce redundancy.
|
||||
|
||||
The runbook supports custom variables, and these variables can be
|
||||
provided in the runbook, command line or environment variables.
|
||||
Variables are only used in the runbook and are resolved to actual values
|
||||
before the start of the test.
|
||||
|
||||
The configuration of the runbook can be extended to certain components.
|
||||
For example, when defining a notifier, you can define its configuration
|
||||
structure together. The corresponding configuration will be loaded into
|
||||
the notifier with the same schema. The extended configuration also
|
||||
supports variables.
|
||||
|
||||
Requirement and Capability
|
||||
--------------------------
|
||||
|
||||
A test case may have certain **requirements** for the test environment.
|
||||
For example, it may need two nodes, four CPU cores, serial logging, a
|
||||
certain Linux distribution or other requirements. When writing test
|
||||
cases, other requirements of the test cases in addition to the default
|
||||
ones should be defined explicitly.
|
||||
|
||||
The platform provides environments with different **capability**. The
|
||||
capabilities of environments are loaded from the platform’s API. Only if
|
||||
the capability of an environment meets the requirements of a test case,
|
||||
the test case can be run on this environment.
|
||||
|
||||
The figure below shows a test case that requires at least one core, 2G
|
||||
memory and three network interface cards (NIC). The Azure VM
|
||||
Standard_DS2_v2 in the middle meets the first two requirements, as it
|
||||
has two cores and 7G memory; but because it can only have two NICs at
|
||||
most, it fails the test requirements and thus the test case will be
|
||||
performed on another VM. Standard_DS3_v2 supports up to four NICs and it
|
||||
meets all three requirements, so it will run the test cases.
|
||||
|
||||
.. figure:: ../img/req_cap.png
|
||||
:alt: requirements to capability
|
||||
|
||||
requirements to capability
|
||||
|
|
@ -1,280 +0,0 @@
|
|||
# How to write extensions in LISA
|
||||
|
||||
- [Notifier](#notifier)
|
||||
- [Tool](#tool)
|
||||
- [CustomScript](#customscript)
|
||||
- [Feature](#feature)
|
||||
- [Support an existing feature in a platform](#support-an-existing-feature-in-a-platform)
|
||||
- [Create a new feature](#create-a-new-feature)
|
||||
- [Use a feature](#use-a-feature)
|
||||
- [Combinator](#combinator)
|
||||
- [Transformer](#transformer)
|
||||
- [Platform](#platform)
|
||||
- [Hooks](#hooks)
|
||||
- [Implement a hook](#implement-a-hook)
|
||||
- [Some notes](#some-notes)
|
||||
- [Extend schema](#extend-schema)
|
||||
- [Which method must be implemented](#which-method-must-be-implemented)
|
||||
|
||||
LISA uses extensions to share code in test cases and makes it flexibly
|
||||
applicable to various situations. Before starting to extend, please make sure
|
||||
you understand the [concepts](concepts.md) of each extension.
|
||||
|
||||
The following content links to the code, which will be constructed using
|
||||
docstrings in the future.
|
||||
|
||||
## Notifier
|
||||
|
||||
The base class is the `Notifier` in [notifier.py](../../lisa/notifier.py). All
|
||||
examples are in [notifier](../../lisa/notifiers).
|
||||
|
||||
- [console.py](../../lisa/notifiers/console.py) is the simplest example.
|
||||
- [html.py](../../lisa/notifiers/html.py) is a complete example.
|
||||
|
||||
If the notifier needs to be set up from the runbook, implement `TypedSchema`.
|
||||
Learn more from `ConsoleSchema` in
|
||||
[console.py](../../lisa/notifiers/console.py).
|
||||
|
||||
Note that the current implementation does not process messages in isolated
|
||||
threads, so if the implementation is slow, it may slow down the overall
|
||||
operation speed.
|
||||
|
||||
## Tool
|
||||
|
||||
The base class is the `Tool` in [executable.py](../../lisa/executable.py). All
|
||||
examples are in [tools](../../lisa/tools).
|
||||
|
||||
- [cat.py](../../lisa/tools/cat.py) is the simplest example.
|
||||
- [gcc.py](../../lisa/tools/gcc.py) supports installation.
|
||||
- [echo.py](../../lisa/tools/echo.py) supports Windows.
|
||||
- [ntttcp.py](../../lisa/tools/ntttcp.py) shows how to specify dependencies
|
||||
between tools through the `dependencies` property.
|
||||
- [lsvmbus.py](../../lisa/tools/lsvmbus.py) is a complex example, that handles
|
||||
different behaviors of Linux distributions and returns structured results to
|
||||
test cases.
|
||||
|
||||
In simple terms, the tool runs the command, returns the output, and parses it
|
||||
into a structure. When implementing tools, try to avoid returning original
|
||||
results to test cases, instead, parse the result and return a structured object,
|
||||
such as in [lsvmbus.py](../../lisa/tools/lsvmbus.py). This code logic is
|
||||
preferred because it allows more coherence.
|
||||
|
||||
> Note, although in [using extensions](write_case.md#extensions) we told you
|
||||
that installation is automatically checked and done, yet you must implement the
|
||||
`_install` method with the correct dependency as a prerequisite. See
|
||||
[gcc.py](../../lisa/tools/gcc.py).
|
||||
|
||||
Learn more about how to use the tool from
|
||||
[helloworld.py](../../examples/testsuites/helloworld.py).
|
||||
|
||||
```python
|
||||
echo = node.tools[Echo]
|
||||
...
|
||||
result = echo.run(hello_world)
|
||||
assert_that(result.stdout).is_equal_to(hello_world)
|
||||
```
|
||||
|
||||
## CustomScript
|
||||
|
||||
The `CustomScript` is like a lightweight tool. However, **please avoid using
|
||||
it** unless there are serious performance issues or other reasons, because it
|
||||
will return the original results to the test case. You can also package custom
|
||||
scripts as tools.
|
||||
|
||||
The base class is the `CustomScript` in
|
||||
[executable.py](../../lisa/executable.py).
|
||||
|
||||
To use the scripts,
|
||||
|
||||
1. Define the scripts using `CustomScriptBuilder`.
|
||||
|
||||
```python
|
||||
self._echo_script = CustomScriptBuilder(
|
||||
Path(__file__).parent.joinpath("scripts"), ["echo.sh"]
|
||||
)
|
||||
```
|
||||
|
||||
2. Use it like a tool.
|
||||
|
||||
```python
|
||||
script: CustomScript = node.tools[self._echo_script]
|
||||
result1 = script.run()
|
||||
```
|
||||
|
||||
3. Learn more from [withscript.py](../../examples/testsuites/withscript.py).
|
||||
|
||||
## Feature
|
||||
|
||||
The base class is [feature.py](../../lisa/feature.py). All examples are in
|
||||
[features](../../lisa/features) and Azure's
|
||||
[features.py](../../lisa/sut_orchestrator/azure/features.py).
|
||||
|
||||
The following content takes `SerialConsole` as an example to introduce the
|
||||
feature.
|
||||
|
||||
### Support an existing feature in a platform
|
||||
|
||||
1. Implement the feature, so that it can work normally. Learn more from the
|
||||
`SerialConsole` implementation in Azure's
|
||||
[features.py](../../lisa/sut_orchestrator/azure/features.py).
|
||||
|
||||
2. The platform should declare which features it supports, and where the
|
||||
implementations of features are.
|
||||
|
||||
```python
|
||||
@classmethod
|
||||
def supported_features(cls) -> List[Type[Feature]]:
|
||||
return [features.StartStop, features.SerialConsole]
|
||||
```
|
||||
|
||||
3. When preparing an environment, the platform should set the supported features
|
||||
on nodes.
|
||||
|
||||
```python
|
||||
node_space.features = search_space.SetSpace[str](is_allow_set=True)
|
||||
node_space.features.update(
|
||||
[features.StartStop.name(), features.SerialConsole.name()]
|
||||
)
|
||||
```
|
||||
|
||||
4. Learn more from Azure's
|
||||
[platform_.py](../../lisa/sut_orchestrator/azure/platform_.py).
|
||||
|
||||
### Create a new feature
|
||||
|
||||
To create a new feature, you need to implement a base class that is called by
|
||||
the test cases, as to keep a common and shareable code logic. Learn more from
|
||||
`SerialConsole` in [serial_console.py](../../lisa/features/serial_console.py).
|
||||
|
||||
### Use a feature
|
||||
|
||||
1. Declare in the metadata which features are required. If the environment does
|
||||
not support this feature, the test case will be skipped.
|
||||
|
||||
```python
|
||||
requirement=simple_requirement(
|
||||
supported_features=[SerialConsole],
|
||||
```
|
||||
|
||||
2. Using features is like using tools.
|
||||
|
||||
```python
|
||||
serial_console = node.features[SerialConsole]
|
||||
# if there is any panic, fail before partial pass
|
||||
serial_console.check_panic(saved_path=case_path, stage="reboot")
|
||||
```
|
||||
|
||||
3. Learn more from
|
||||
[provisioning.py](../../microsoft/testsuites/core/provisioning.py).
|
||||
|
||||
## Combinator
|
||||
|
||||
The base class is [combinator.py](../../lisa/combinator.py). All examples are in
|
||||
[combinators](../../lisa/combinators).
|
||||
|
||||
- [grid_combinator.py](../../lisa/combinators/grid_combinator.py) supports a
|
||||
full matrix combination.
|
||||
- [batch_combinator.py](../../lisa/combinators/batch_combinator.py) supports a
|
||||
batch combination.
|
||||
|
||||
## Transformer
|
||||
|
||||
The base class is [transformer.py](../../lisa/transformer.py). All examples are
|
||||
in [transformers](../../lisa/transformers).
|
||||
|
||||
- [to_list.py](../../lisa/transfomers/to_list.py) is the simplest example.
|
||||
|
||||
## Platform
|
||||
|
||||
The base class is [platform_.py](../../lisa/platform_.py).
|
||||
|
||||
- [ready.py](../../lisa/sut_orchestrator/ready.py) is the simplest example.
|
||||
- [platform_.py](../../lisa/sut_orchestrator/azure/platform_.py) is a complete
|
||||
example of Azure.
|
||||
|
||||
If a platform needs to specify settings in runbook, it can be implemented in two
|
||||
places.
|
||||
|
||||
1. Platform schema. Learn more from `AzurePlatformSchema` in Azure's
|
||||
[platform_.py](../../lisa/sut_orchestrator/azure/platform_.py).
|
||||
|
||||
1. Node schema. Learn more from `AzureNodeSchema` in Azure's
|
||||
[common.py](../../lisa/sut_orchestrator/azure/common.py).
|
||||
|
||||
1. Use them in the platform code. Learn more from Azure's
|
||||
[platform_.py](../../lisa/sut_orchestrator/azure/platform_.py).
|
||||
|
||||
```python
|
||||
azure_runbook: AzurePlatformSchema = self._runbook.get_extended_runbook(
|
||||
AzurePlatformSchema
|
||||
)
|
||||
azure_node_runbook = node_space.get_extended_runbook(
|
||||
AzureNodeSchema, type_name=AZURE
|
||||
)
|
||||
```
|
||||
|
||||
## Hooks
|
||||
|
||||
Hooks are imported by [pluggy](https://pluggy.readthedocs.io/en/latest/). The current list of hooks
|
||||
will expand due to new requirements. Take a look at [A definitive
|
||||
example](https://github.com/pytest-dev/pluggy/blob/master/README.rst) to quickly
|
||||
get started with [pluggy](https://pluggy.readthedocs.io/en/latest/).
|
||||
|
||||
### Implement a hook
|
||||
|
||||
1. Create a hook specification namespace.
|
||||
|
||||
```python
|
||||
class AzureHookSpec:
|
||||
|
||||
@hookspec
|
||||
def azure_deploy_failed(self, error_message: str) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
2. Define a hook and add some functions.
|
||||
|
||||
```python
|
||||
class Platform(...):
|
||||
|
||||
@hookimpl # type: ignore
|
||||
def get_environment_information(self, environment: Environment) -> Dict[str, str]:
|
||||
...
|
||||
```
|
||||
|
||||
3. Add the spec to the manager and register the hook in place.
|
||||
|
||||
```python
|
||||
plugin_manager.add_hookspecs(AzureHookSpec)
|
||||
plugin_manager.register(AzureHookSpecDefaultImpl())
|
||||
```
|
||||
|
||||
4. Learn more from hooks in [platform_.py](../../lisa/platform_.py).
|
||||
|
||||
## Some notes
|
||||
|
||||
### Extend schema
|
||||
|
||||
Extensions such as platforms and notifications support extended schema in
|
||||
runbook.
|
||||
|
||||
The runbook uses [dataclass](https://docs.python.org/3/library/dataclasses.html)
|
||||
for definition, [dataclass-json](https://github.com/lidatong/dataclasses-json/)
|
||||
for deserialization, and
|
||||
[marshmallow](https://marshmallow.readthedocs.io/en/3.0/api_reference.html) to
|
||||
validate the schema.
|
||||
|
||||
See more examples in [schema.py](../../lisa/schema.py), if you need to extend
|
||||
runbook schema.
|
||||
|
||||
### Which method must be implemented
|
||||
|
||||
If a method in a parent class needs to be implemented in child class, it may
|
||||
raise a `NotImplementedError` inside the method body in the parent class and be
|
||||
annotated with `@abstractmethod`. Be careful with `@abstractmethod` to use use
|
||||
it only with `NotImplementedError` and nowhere else, because it is not support
|
||||
as a type in `typing`.
|
||||
|
||||
---
|
||||
|
||||
Back to [how to write tests](write_case.md).
|
||||
|
|
@ -0,0 +1,318 @@
|
|||
How to write extensions in LISA
|
||||
===============================
|
||||
|
||||
- `Notifier <#notifier>`__
|
||||
- `Tool <#tool>`__
|
||||
- `CustomScript <#customscript>`__
|
||||
- `Feature <#feature>`__
|
||||
|
||||
- `Support an existing feature in a
|
||||
platform <#support-an-existing-feature-in-a-platform>`__
|
||||
- `Create a new feature <#create-a-new-feature>`__
|
||||
- `Use a feature <#use-a-feature>`__
|
||||
|
||||
- `Combinator <#combinator>`__
|
||||
- `Transformer <#transformer>`__
|
||||
- `Platform <#platform>`__
|
||||
- `Hooks <#hooks>`__
|
||||
|
||||
- `Implement a hook <#implement-a-hook>`__
|
||||
|
||||
- `Some notes <#some-notes>`__
|
||||
|
||||
- `Extend schema <#extend-schema>`__
|
||||
- `Which method must be
|
||||
implemented <#which-method-must-be-implemented>`__
|
||||
|
||||
LISA uses extensions to share code in test cases and makes it flexibly
|
||||
applicable to various situations. Before starting to extend, please make
|
||||
sure you understand the `concepts <concepts.html>`__ of each extension.
|
||||
|
||||
The following content links to the code, which will be constructed using
|
||||
docstrings in the future.
|
||||
|
||||
Notifier
|
||||
--------
|
||||
|
||||
The base class is the ``Notifier`` in
|
||||
`notifier.py <../../lisa/notifier.py>`__. All examples are in
|
||||
`notifier <../../lisa/notifiers>`__.
|
||||
|
||||
- `console.py <../../lisa/notifiers/console.py>`__ is the simplest
|
||||
example.
|
||||
- `html.py <../../lisa/notifiers/html.py>`__ is a complete example.
|
||||
|
||||
If the notifier needs to be set up from the runbook, implement
|
||||
``TypedSchema``. Learn more from ``ConsoleSchema`` in
|
||||
`console.py <../../lisa/notifiers/console.py>`__.
|
||||
|
||||
Note that the current implementation does not process messages in
|
||||
isolated threads, so if the implementation is slow, it may slow down the
|
||||
overall operation speed.
|
||||
|
||||
Tool
|
||||
----
|
||||
|
||||
The base class is the ``Tool`` in
|
||||
`executable.py <../../lisa/executable.py>`__. All examples are in
|
||||
`tools <../../lisa/tools>`__.
|
||||
|
||||
- `cat.py <../../lisa/tools/cat.py>`__ is the simplest example.
|
||||
- `gcc.py <../../lisa/tools/gcc.py>`__ supports installation.
|
||||
- `echo.py <../../lisa/tools/echo.py>`__ supports Windows.
|
||||
- `ntttcp.py <../../lisa/tools/ntttcp.py>`__ shows how to specify
|
||||
dependencies between tools through the ``dependencies`` property.
|
||||
- `lsvmbus.py <../../lisa/tools/lsvmbus.py>`__ is a complex example,
|
||||
that handles different behaviors of Linux distributions and returns
|
||||
structured results to test cases.
|
||||
|
||||
In simple terms, the tool runs the command, returns the output, and
|
||||
parses it into a structure. When implementing tools, try to avoid
|
||||
returning original results to test cases, instead, parse the result and
|
||||
return a structured object, such as in
|
||||
`lsvmbus.py <../../lisa/tools/lsvmbus.py>`__. This code logic is
|
||||
preferred because it allows more coherence.
|
||||
|
||||
Note, although in `using extensions <write_case.html#extensions>`__
|
||||
we told you that installation is automatically checked and done, yet
|
||||
you must implement the ``_install`` method with the correct
|
||||
dependency as a prerequisite. See
|
||||
`gcc.py <../../lisa/tools/gcc.py>`__.
|
||||
|
||||
Learn more about how to use the tool from
|
||||
`helloworld.py <../../examples/testsuites/helloworld.py>`__.
|
||||
|
||||
.. code:: python
|
||||
|
||||
echo = node.tools[Echo]
|
||||
...
|
||||
result = echo.run(hello_world)
|
||||
assert_that(result.stdout).is_equal_to(hello_world)
|
||||
|
||||
CustomScript
|
||||
------------
|
||||
|
||||
The ``CustomScript`` is like a lightweight tool. However, **please avoid
|
||||
using it** unless there are serious performance issues or other reasons,
|
||||
because it will return the original results to the test case. You can
|
||||
also package custom scripts as tools.
|
||||
|
||||
The base class is the ``CustomScript`` in
|
||||
`executable.py <../../lisa/executable.py>`__.
|
||||
|
||||
To use the scripts,
|
||||
|
||||
1. Define the scripts using ``CustomScriptBuilder``.
|
||||
|
||||
.. code:: python
|
||||
|
||||
self._echo_script = CustomScriptBuilder(
|
||||
Path(__file__).parent.joinpath("scripts"), ["echo.sh"]
|
||||
)
|
||||
|
||||
2. Use it like a tool.
|
||||
|
||||
.. code:: python
|
||||
|
||||
script: CustomScript = node.tools[self._echo_script]
|
||||
result1 = script.run()
|
||||
|
||||
3. Learn more from
|
||||
`withscript.py <../../examples/testsuites/withscript.py>`__.
|
||||
|
||||
Feature
|
||||
-------
|
||||
|
||||
The base class is `feature.py <../../lisa/feature.py>`__. All examples
|
||||
are in `features <../../lisa/features>`__ and Azure’s
|
||||
`features.py <../../lisa/sut_orchestrator/azure/features.py>`__.
|
||||
|
||||
The following content takes ``SerialConsole`` as an example to introduce
|
||||
the feature.
|
||||
|
||||
Support an existing feature in a platform
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
1. Implement the feature, so that it can work normally. Learn more from
|
||||
the ``SerialConsole`` implementation in Azure’s
|
||||
`features.py <../../lisa/sut_orchestrator/azure/features.py>`__.
|
||||
|
||||
2. The platform should declare which features it supports, and where the
|
||||
implementations of features are.
|
||||
|
||||
.. code:: python
|
||||
|
||||
@classmethod
|
||||
def supported_features(cls) -> List[Type[Feature]]:
|
||||
return [features.StartStop, features.SerialConsole]
|
||||
|
||||
3. When preparing an environment, the platform should set the supported
|
||||
features on nodes.
|
||||
|
||||
.. code:: python
|
||||
|
||||
node_space.features = search_space.SetSpace[str](is_allow_set=True)
|
||||
node_space.features.update(
|
||||
[features.StartStop.name(), features.SerialConsole.name()]
|
||||
)
|
||||
|
||||
4. Learn more from Azure’s
|
||||
`platform_.py <../../lisa/sut_orchestrator/azure/platform_.py>`__.
|
||||
|
||||
Create a new feature
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
To create a new feature, you need to implement a base class that is
|
||||
called by the test cases, as to keep a common and shareable code logic.
|
||||
Learn more from ``SerialConsole`` in
|
||||
`serial_console.py <../../lisa/features/serial_console.py>`__.
|
||||
|
||||
Use a feature
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
1. Declare in the metadata which features are required. If the
|
||||
environment does not support this feature, the test case will be
|
||||
skipped.
|
||||
|
||||
.. code:: python
|
||||
|
||||
requirement=simple_requirement(
|
||||
supported_features=[SerialConsole],
|
||||
|
||||
2. Using features is like using tools.
|
||||
|
||||
.. code:: python
|
||||
|
||||
serial_console = node.features[SerialConsole]
|
||||
# if there is any panic, fail before partial pass
|
||||
serial_console.check_panic(saved_path=case_path, stage="reboot")
|
||||
|
||||
3. Learn more from
|
||||
`provisioning.py <../../microsoft/testsuites/core/provisioning.py>`__.
|
||||
|
||||
Combinator
|
||||
----------
|
||||
|
||||
The base class is `combinator.py <../../lisa/combinator.py>`__. All
|
||||
examples are in `combinators <../../lisa/combinators>`__.
|
||||
|
||||
- `grid_combinator.py <../../lisa/combinators/grid_combinator.py>`__
|
||||
supports a full matrix combination.
|
||||
- `batch_combinator.py <../../lisa/combinators/batch_combinator.py>`__
|
||||
supports a batch combination.
|
||||
|
||||
Transformer
|
||||
-----------
|
||||
|
||||
The base class is `transformer.py <../../lisa/transformer.py>`__. All
|
||||
examples are in `transformers <../../lisa/transformers>`__.
|
||||
|
||||
- `to_list.py <../../lisa/transfomers/to_list.py>`__ is the simplest
|
||||
example.
|
||||
|
||||
Platform
|
||||
--------
|
||||
|
||||
The base class is `platform_.py <../../lisa/platform_.py>`__.
|
||||
|
||||
- `ready.py <../../lisa/sut_orchestrator/ready.py>`__ is the simplest
|
||||
example.
|
||||
- `platform_.py <../../lisa/sut_orchestrator/azure/platform_.py>`__ is
|
||||
a complete example of Azure.
|
||||
|
||||
If a platform needs to specify settings in runbook, it can be
|
||||
implemented in two places.
|
||||
|
||||
1. Platform schema. Learn more from ``AzurePlatformSchema`` in Azure’s
|
||||
`platform_.py <../../lisa/sut_orchestrator/azure/platform_.py>`__.
|
||||
|
||||
2. Node schema. Learn more from ``AzureNodeSchema`` in Azure’s
|
||||
`common.py <../../lisa/sut_orchestrator/azure/common.py>`__.
|
||||
|
||||
3. Use them in the platform code. Learn more from Azure’s
|
||||
`platform_.py <../../lisa/sut_orchestrator/azure/platform_.py>`__.
|
||||
|
||||
.. code:: python
|
||||
|
||||
azure_runbook: AzurePlatformSchema = self._runbook.get_extended_runbook(
|
||||
AzurePlatformSchema
|
||||
)
|
||||
azure_node_runbook = node_space.get_extended_runbook(
|
||||
AzureNodeSchema, type_name=AZURE
|
||||
)
|
||||
|
||||
Hooks
|
||||
-----
|
||||
|
||||
Hooks are imported by
|
||||
`pluggy <https://pluggy.readthedocs.io/en/latest/>`__. The current list
|
||||
of hooks will expand due to new requirements. Take a look at `A
|
||||
definitive
|
||||
example <https://github.com/pytest-dev/pluggy/blob/master/README.rst>`__
|
||||
to quickly get started with
|
||||
`pluggy <https://pluggy.readthedocs.io/en/latest/>`__.
|
||||
|
||||
Implement a hook
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
1. Create a hook specification namespace.
|
||||
|
||||
.. code:: python
|
||||
|
||||
class AzureHookSpec:
|
||||
|
||||
@hookspec
|
||||
def azure_deploy_failed(self, error_message: str) -> None:
|
||||
...
|
||||
|
||||
2. Define a hook and add some functions.
|
||||
|
||||
.. code:: python
|
||||
|
||||
class Platform(...):
|
||||
|
||||
@hookimpl # type: ignore
|
||||
def get_environment_information(self, environment: Environment) -> Dict[str, str]:
|
||||
...
|
||||
|
||||
3. Add the spec to the manager and register the hook in place.
|
||||
|
||||
.. code:: python
|
||||
|
||||
plugin_manager.add_hookspecs(AzureHookSpec)
|
||||
plugin_manager.register(AzureHookSpecDefaultImpl())
|
||||
|
||||
4. Learn more from hooks in `platform_.py <../../lisa/platform_.py>`__.
|
||||
|
||||
Some notes
|
||||
----------
|
||||
|
||||
Extend schema
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Extensions such as platforms and notifications support extended schema
|
||||
in runbook.
|
||||
|
||||
The runbook uses
|
||||
`dataclass <https://docs.python.org/3/library/dataclasses.html>`__ for
|
||||
definition,
|
||||
`dataclass-json <https://github.com/lidatong/dataclasses-json/>`__ for
|
||||
deserialization, and
|
||||
`marshmallow <https://marshmallow.readthedocs.io/en/3.0/api_reference.html>`__
|
||||
to validate the schema.
|
||||
|
||||
See more examples in `schema.py <../../lisa/schema.py>`__, if you need
|
||||
to extend runbook schema.
|
||||
|
||||
Which method must be implemented
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If a method in a parent class needs to be implemented in child class, it
|
||||
may raise a ``NotImplementedError`` inside the method body in the parent
|
||||
class and be annotated with ``@abstractmethod``. Be careful with
|
||||
``@abstractmethod`` to use use it only with ``NotImplementedError`` and
|
||||
nowhere else, because it is not support as a type in ``typing``.
|
||||
|
||||
--------------
|
||||
|
||||
Back to `how to write tests <write_case.html>`__.
|
||||
|
|
@ -1,264 +0,0 @@
|
|||
# Coding guidelines
|
||||
|
||||
- [Naming Conventions](#naming-conventions)
|
||||
- [Test code excellence](#test-code-excellence)
|
||||
- [Code comments](#code-comments)
|
||||
- [Commit messages](#commit-messages)
|
||||
- [Logging](#logging)
|
||||
- [Error message](#error-message)
|
||||
- [Assertion](#assertion)
|
||||
- [Troubleshooting excellence](#troubleshooting-excellence)
|
||||
- [Document excellence](#document-excellence)
|
||||
- [Tips for non-native English speakers by non-native English speakers](#tips-for-non-native-english-speakers-by-non-native-english-speakers)
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
Please read the [naming
|
||||
conventions](https://www.python.org/dev/peps/pep-0008/#naming-conventions)
|
||||
section of PEP 8, which explains the meaning of each of the styles. A brief
|
||||
overview of the most important parts:
|
||||
|
||||
- Modules (and files) should use lowercase short names.
|
||||
- Class (and exception) names should use the `CapWords` convention (also known
|
||||
as `CamelCase`)
|
||||
- Function and variable names should use lowercase letters, and words should be
|
||||
separated by underscores to improve readability (also called `snake_case`).
|
||||
- To avoid conflicts with the standard library, you can add an underscore, such
|
||||
as `id_`.
|
||||
- Leading lines such as `_data` apply to non-public methods and instance
|
||||
variables. Subclasses can use it. If you don't use it in a subclass, use it
|
||||
like `__data` in a superclass.
|
||||
- If there is a pair of `get_x` and `set_x` methods without additional
|
||||
parameters, please use the built-in `@property` decorator to convert them to
|
||||
properties.
|
||||
- Constants should be similar to `CAPITALIZED_SNAKE_CASE`.
|
||||
- When importing a function, try to avoid renaming it with `import as` because
|
||||
it introduces cognitive overhead to keep track of another name. If the name
|
||||
conflicts, please use the package name as the namespace, such as `import
|
||||
schema`, and use it as `schema.Node`.
|
||||
|
||||
If in doubt, follow existing conventions or check the style guide.
|
||||
|
||||
## Test code excellence
|
||||
|
||||
Your code would be an example for others, and they might follow your approach.
|
||||
Therefore, both good and bad practices will be amplified.
|
||||
|
||||
In LISA, test code should be organized according to business logic, which means
|
||||
that the code should perform the purpose of the test like a test specification.
|
||||
The underlying logic should be implemented elsewhere, such as tools, functions,
|
||||
or private methods in test suites.
|
||||
|
||||
An example: Be careful when using `sleep`! The only way to use sleep is in
|
||||
polling mode. This means that you must wait for something with regular
|
||||
inspections. In the inspection cycle, you can wait for a reasonable period.
|
||||
Don't wait for 10 seconds of sleep. This causes two problems, 1) if it is too
|
||||
short, the case may fail; 2) if it is long enough, it will slow down the running
|
||||
speed.
|
||||
|
||||
Please keep in mind that your code may be referred to by others.
|
||||
|
||||
## Code comments
|
||||
|
||||
How to write good code comments is a hot topic, and many best practices are also
|
||||
valuable. Here are some highlights.
|
||||
|
||||
- Do not repeat the code logic. Code comments are always in the same place as
|
||||
the code, which is different from metadata. Do not repeat `if/else` statement
|
||||
like "if ... else ...", do not repeat the content that already exists in the
|
||||
log string and exception message, do not repeat what can be clearly seen from
|
||||
the variable name.
|
||||
- Record business logic. Code logic is more detailed than business logic. Some
|
||||
complex code logic may not be intuitive for understanding business logic. Code
|
||||
comments can help summarize complex code logic.
|
||||
- Record trick things. We cannot avoid writing tricky code. For example, magic
|
||||
numbers, special handling of the Linux version, or other content.
|
||||
- Provide regular expression examples. LISA uses many regular expressions to
|
||||
parse command output. It is simple and useful, but it may not match. When you
|
||||
need to create or update a regular expression, it needs to check the sample
|
||||
for regression. These examples also help to understand what the expression
|
||||
does.
|
||||
|
||||
## Commit messages
|
||||
|
||||
The commit message is used to explain why this change was made. The code
|
||||
comments describe the current state. The commit message describes the reason for
|
||||
the change. If you think the content is also suitable for writing in the code,
|
||||
please write it as a code comment.
|
||||
|
||||
## Logging
|
||||
|
||||
The log has two purposes, 1) display progress, and 2) troubleshoot.
|
||||
|
||||
To show progress, the log should be simple and logical. To troubleshoot, it
|
||||
requires more detailed information. These two goals sound contradictory, but
|
||||
they can be achieved through different INFO and DEBUG levels. LISA always
|
||||
enables the DEBUG level in the log file, while the INFO level is the default
|
||||
setting on the console.
|
||||
|
||||
In LISA, when writing log lines in the code, it's recommended to consider what
|
||||
the test runner needs to know, instead of what the developer needs to know,
|
||||
which should be done in code comments.
|
||||
|
||||
- **DEBUG** level log should provide the *correct level* detail. The only way to
|
||||
write at the "correct level" is to use it from the beginning.
|
||||
|
||||
When writing code, please keep using and improving the log. If you need to
|
||||
debug step by step, it means you need to improve the log. If you don’t
|
||||
understand the meaning of the log, others may not as well, so please optimize
|
||||
the log at DEBUG level. In addition, if you find duplicate information, please
|
||||
merge it.
|
||||
|
||||
- **INFO** level log should be *like a story*, to illustrate what happened.
|
||||
|
||||
Even if the whole process goes smoothly, this is what you want to know every
|
||||
time. It should be friendly so that new users can understand what is going on.
|
||||
It should be as little as possible. It should tell the user to wait before
|
||||
performing a long operation.
|
||||
|
||||
- **WARNING** level logs should be avoided.
|
||||
|
||||
The warning message indicates that it is important, but there is no need to
|
||||
stop. But in most cases, you will find that it is either not as important as
|
||||
the information level, or it is so important to stop running.
|
||||
|
||||
At the time of writing, there are 3 warning messages in LISA. After review, I
|
||||
converted them all into information or error level. There is only one left,
|
||||
and it is up to the user to suppress errors.
|
||||
|
||||
- **ERROR** level log should be reviewed carefully.
|
||||
|
||||
Error level logs can help identify potential problems. If there are too many
|
||||
error level logs, it will hide the actual problem. When it goes smoothly,
|
||||
there should be no error level logs. According to experience, 95% of
|
||||
successful runs should not contain any error level logs.
|
||||
|
||||
Some tips:
|
||||
|
||||
- By reading the log, you should be able to understand the progress without
|
||||
having to look at the code. And logs describe business logic, not code logic.
|
||||
A bad example, "4 items found: [a , b , c]", should be "found 4 channels,
|
||||
unique names: [a, b, c]".
|
||||
- Make each log line unique in the code. If you must check where the log is
|
||||
printed in the code. We can quickly find the code by searching. A bad example,
|
||||
`log.info("received stop signal")`, should be `log.info("received stop signal
|
||||
in lisa_runner")`.
|
||||
- Do not repeat similar lines in succession. It is worth adding logic and
|
||||
variables to reduce redundant logs.
|
||||
- Reduce log lines. If two lines of logs always appear together, merge them into
|
||||
one line. The impact of log lines on readability is much greater than the
|
||||
length of the log.
|
||||
- Associate related logs through shared context. In the case of concurrency,
|
||||
this is very important. A bad example, "cmd: echo hello world", "cmd: hello
|
||||
world" can be "cmd[666]: echo hello world", "cmd[666]: hello world".
|
||||
|
||||
## Error message
|
||||
|
||||
There are two kinds of error messages in LISA. The first is an error message,
|
||||
and it does not fail. It will be printed as stderr and will be more obvious when
|
||||
the test case fails. The second is a one-line message in the failed test case.
|
||||
This section applies to two of them, but the second one is more important
|
||||
because we want it to be the only information that helps understand the failed
|
||||
test case.
|
||||
|
||||
In LISA, failed, skipped, and some passed test cases have a message. It
|
||||
specifies the reason the test case failed or skipped. Through this message, the
|
||||
user can understand what will happen and can act. Therefore, this message should
|
||||
be as helpful as possible.
|
||||
|
||||
The error message should include what happened and how to resolve it. It may not
|
||||
be easy to provide all the information for the first time, but guesswork is also
|
||||
helpful. At the same time, the original error message is also useful, please
|
||||
don't hide it.
|
||||
|
||||
For examples,
|
||||
|
||||
- "The subscription ID [aaa] could not be found, please make sure it exists and
|
||||
is accessible by the current account". A bad example, "The subscription ID
|
||||
[aaa] could not be found". This bad example illustrates what happened, but
|
||||
there is no suggestion.
|
||||
- "The vm size [aaa] could not be found on the location [bbb]. This may be
|
||||
because the virtual machine size is not available in this location". A bad
|
||||
example, "The vm size [aaa] could not be found on the location [bbb]". It
|
||||
explains what happened, but it does not provide a guess at the root cause.
|
||||
|
||||
## Assertion
|
||||
|
||||
Assertions are heavily used in test code. Assertions are a simple pattern of "if
|
||||
some checks fail, raise an exception".
|
||||
|
||||
The assertion library includes commonly used patterns and detailed error
|
||||
messages. LISA uses `assertpy` as a standard assertion library, which provides
|
||||
Pythonic and test-friendly assertions.
|
||||
|
||||
When writing the assertion,
|
||||
|
||||
- Put the actual value in `assert_that` to keep the style consistent, and you
|
||||
can compare it with multiple expected values continuously.
|
||||
- Assertions should be as comprehensive as possible, but do not repeat existing
|
||||
checks. For example, `assert_that(str1).is_equal_to('hello')` is enough, no
|
||||
need like `assert_that(str1).is_instance_of(str).is_equal_to('hello')`.
|
||||
- Add a description to explain the business logic. If a malfunction occurs,
|
||||
these instructions will be displayed. For example,
|
||||
`assert_that(str1).described_as('echo back result is
|
||||
unexpected').is_equal_to('hello')` is better than
|
||||
`assert_that(str1).is_equal_to('hello')`.
|
||||
- Try to use native assertions instead of manipulating the data yourself.
|
||||
`assert_that(vmbuses).is_length(6)` is better than
|
||||
`assert_that(len(vmbuses)).is_equal_to(6)`. It is simpler and the error
|
||||
message is clearer.
|
||||
- Don't forget to use powerful collection assertions. They can compare ordered
|
||||
list by `contains` (actual value is superset), `is_subset_of` (actual value is
|
||||
subset), and others.
|
||||
|
||||
Learn more from [examples](../../examples/testsuites) and [assertpy
|
||||
document](https://github.com/assertpy/assertpy#readme).
|
||||
|
||||
## Troubleshooting excellence
|
||||
|
||||
Test failure is a common phenomenon. Therefore, perform troubleshooting
|
||||
frequently. There are some useful ways to troubleshoot failures. In the list
|
||||
below, the higher items are better than the lower items because of its lower
|
||||
cost of analysis.
|
||||
|
||||
1. Single line message. A one-line message is sent with the test result status.
|
||||
If this message clearly describes the root cause, no other digging is
|
||||
necessary. You can even perform some automated actions to match messages and
|
||||
act.
|
||||
2. Test case log. LISA provides a complete log for each run, which includes the
|
||||
output of all test cases, all threads, and all nodes. This file can be
|
||||
regarded as the default log, which is easy to search.
|
||||
3. Other log files. Some original logs may be divided into test cases. After
|
||||
finding out the cause, it is easier to find out. But it needs to download and
|
||||
browse the test result files.
|
||||
4. Reproduce in the environment. It is costly but contains most of the original
|
||||
information. But sometimes, the problem cannot be reproduced.
|
||||
|
||||
In LISA, test cases fail due to exceptions, and exception messages are treated
|
||||
as single-line messages. When writing test cases, it's time to adjust the
|
||||
exception message. Therefore, after completing the test case, many errors will
|
||||
be explained well.
|
||||
|
||||
## Document excellence
|
||||
|
||||
The documentation is the opportunity to make things clear and easy to maintain.
|
||||
A longer document is not always a better document. Each kind of documentation
|
||||
has its own purpose. Good technical documentation should be *useful and
|
||||
accurate*.
|
||||
|
||||
## Tips for non-native English speakers by non-native English speakers
|
||||
|
||||
Today, there are a lot of great tools to help you create high-quality English
|
||||
documents. If writing in English is challenging, please try the following steps:
|
||||
|
||||
1. Read our documentations.
|
||||
2. Write in your language first.
|
||||
3. Use machine translation such as [Microsoft
|
||||
Translator](https://www.bing.com/translator/) and [Google
|
||||
translate](https://translate.google.com/) to convert it to English.
|
||||
4. Convert the English version back to your language and check. If it doesn't
|
||||
make sense after translating back, it means the sentence is too complicated.
|
||||
Make it simpler, and then start from step 1 again.
|
||||
5. Once satisfied, you can use [Microsoft
|
||||
Editor](https://www.microsoft.com/en-us/microsoft-365/microsoft-editor) to
|
||||
further refine the grammar and wordings.
|
||||
|
|
@ -0,0 +1,289 @@
|
|||
Coding guidelines
|
||||
=================
|
||||
|
||||
- `Naming Conventions <#naming-conventions>`__
|
||||
- `Test code excellence <#test-code-excellence>`__
|
||||
- `Code comments <#code-comments>`__
|
||||
- `Commit messages <#commit-messages>`__
|
||||
- `Logging <#logging>`__
|
||||
- `Error message <#error-message>`__
|
||||
- `Assertion <#assertion>`__
|
||||
- `Troubleshooting excellence <#troubleshooting-excellence>`__
|
||||
- `Document excellence <#document-excellence>`__
|
||||
- `Tips for non-native English speakers by non-native English
|
||||
speakers <#tips-for-non-native-english-speakers-by-non-native-english-speakers>`__
|
||||
|
||||
Naming Conventions
|
||||
------------------
|
||||
|
||||
Please read the `naming
|
||||
conventions <https://www.python.org/dev/peps/pep-0008/#naming-conventions>`__
|
||||
section of PEP 8, which explains the meaning of each of the styles. A
|
||||
brief overview of the most important parts:
|
||||
|
||||
- Modules (and files) should use lowercase short names.
|
||||
- Class (and exception) names should use the ``CapWords`` convention
|
||||
(also known as ``CamelCase``)
|
||||
- Function and variable names should use lowercase letters, and words
|
||||
should be separated by underscores to improve readability (also
|
||||
called ``snake_case``).
|
||||
- To avoid conflicts with the standard library, you can add an
|
||||
underscore, such as ``id_``.
|
||||
- Leading lines such as ``_data`` apply to non-public methods and
|
||||
instance variables. Subclasses can use it. If you don’t use it in a
|
||||
subclass, use it like ``__data`` in a superclass.
|
||||
- If there is a pair of ``get_x`` and ``set_x`` methods without
|
||||
additional parameters, please use the built-in ``@property``
|
||||
decorator to convert them to properties.
|
||||
- Constants should be similar to ``CAPITALIZED_SNAKE_CASE``.
|
||||
- When importing a function, try to avoid renaming it with
|
||||
``import as`` because it introduces cognitive overhead to keep track
|
||||
of another name. If the name conflicts, please use the package name
|
||||
as the namespace, such as ``import schema``, and use it as
|
||||
``schema.Node``.
|
||||
|
||||
If in doubt, follow existing conventions or check the style guide.
|
||||
|
||||
Test code excellence
|
||||
--------------------
|
||||
|
||||
Your code would be an example for others, and they might follow your
|
||||
approach. Therefore, both good and bad practices will be amplified.
|
||||
|
||||
In LISA, test code should be organized according to business logic,
|
||||
which means that the code should perform the purpose of the test like a
|
||||
test specification. The underlying logic should be implemented
|
||||
elsewhere, such as tools, functions, or private methods in test suites.
|
||||
|
||||
An example: Be careful when using ``sleep``! The only way to use sleep
|
||||
is in polling mode. This means that you must wait for something with
|
||||
regular inspections. In the inspection cycle, you can wait for a
|
||||
reasonable period. Don’t wait for 10 seconds of sleep. This causes two
|
||||
problems, 1) if it is too short, the case may fail; 2) if it is long
|
||||
enough, it will slow down the running speed.
|
||||
|
||||
Please keep in mind that your code may be referred to by others.
|
||||
|
||||
Code comments
|
||||
-------------
|
||||
|
||||
How to write good code comments is a hot topic, and many best practices
|
||||
are also valuable. Here are some highlights.
|
||||
|
||||
- Do not repeat the code logic. Code comments are always in the same
|
||||
place as the code, which is different from metadata. Do not repeat
|
||||
``if/else`` statement like “if … else …”, do not repeat the content
|
||||
that already exists in the log string and exception message, do not
|
||||
repeat what can be clearly seen from the variable name.
|
||||
- Record business logic. Code logic is more detailed than business
|
||||
logic. Some complex code logic may not be intuitive for understanding
|
||||
business logic. Code comments can help summarize complex code logic.
|
||||
- Record trick things. We cannot avoid writing tricky code. For
|
||||
example, magic numbers, special handling of the Linux version, or
|
||||
other content.
|
||||
- Provide regular expression examples. LISA uses many regular
|
||||
expressions to parse command output. It is simple and useful, but it
|
||||
may not match. When you need to create or update a regular
|
||||
expression, it needs to check the sample for regression. These
|
||||
examples also help to understand what the expression does.
|
||||
|
||||
Commit messages
|
||||
---------------
|
||||
|
||||
The commit message is used to explain why this change was made. The code
|
||||
comments describe the current state. The commit message describes the
|
||||
reason for the change. If you think the content is also suitable for
|
||||
writing in the code, please write it as a code comment.
|
||||
|
||||
Logging
|
||||
-------
|
||||
|
||||
The log has two purposes, 1) display progress, and 2) troubleshoot.
|
||||
|
||||
To show progress, the log should be simple and logical. To troubleshoot,
|
||||
it requires more detailed information. These two goals sound
|
||||
contradictory, but they can be achieved through different INFO and DEBUG
|
||||
levels. LISA always enables the DEBUG level in the log file, while the
|
||||
INFO level is the default setting on the console.
|
||||
|
||||
In LISA, when writing log lines in the code, it’s recommended to
|
||||
consider what the test runner needs to know, instead of what the
|
||||
developer needs to know, which should be done in code comments.
|
||||
|
||||
- **DEBUG** level log should provide the *correct level* detail. The
|
||||
only way to write at the “correct level” is to use it from the
|
||||
beginning.
|
||||
|
||||
When writing code, please keep using and improving the log. If you
|
||||
need to debug step by step, it means you need to improve the log. If
|
||||
you don’t understand the meaning of the log, others may not as well,
|
||||
so please optimize the log at DEBUG level. In addition, if you find
|
||||
duplicate information, please merge it.
|
||||
|
||||
- **INFO** level log should be *like a story*, to illustrate what
|
||||
happened.
|
||||
|
||||
Even if the whole process goes smoothly, this is what you want to
|
||||
know every time. It should be friendly so that new users can
|
||||
understand what is going on. It should be as little as possible. It
|
||||
should tell the user to wait before performing a long operation.
|
||||
|
||||
- **WARNING** level logs should be avoided.
|
||||
|
||||
The warning message indicates that it is important, but there is no
|
||||
need to stop. But in most cases, you will find that it is either not
|
||||
as important as the information level, or it is so important to stop
|
||||
running.
|
||||
|
||||
At the time of writing, there are 3 warning messages in LISA. After
|
||||
review, I converted them all into information or error level. There
|
||||
is only one left, and it is up to the user to suppress errors.
|
||||
|
||||
- **ERROR** level log should be reviewed carefully.
|
||||
|
||||
Error level logs can help identify potential problems. If there are
|
||||
too many error level logs, it will hide the actual problem. When it
|
||||
goes smoothly, there should be no error level logs. According to
|
||||
experience, 95% of successful runs should not contain any error level
|
||||
logs.
|
||||
|
||||
Some tips:
|
||||
|
||||
- By reading the log, you should be able to understand the progress
|
||||
without having to look at the code. And logs describe business logic,
|
||||
not code logic. A bad example, “4 items found: [a , b , c]”, should
|
||||
be “found 4 channels, unique names: [a, b, c]”.
|
||||
- Make each log line unique in the code. If you must check where the
|
||||
log is printed in the code. We can quickly find the code by
|
||||
searching. A bad example, ``log.info("received stop signal")``,
|
||||
should be ``log.info("received stop signal in lisa_runner")``.
|
||||
- Do not repeat similar lines in succession. It is worth adding logic
|
||||
and variables to reduce redundant logs.
|
||||
- Reduce log lines. If two lines of logs always appear together, merge
|
||||
them into one line. The impact of log lines on readability is much
|
||||
greater than the length of the log.
|
||||
- Associate related logs through shared context. In the case of
|
||||
concurrency, this is very important. A bad example, “cmd: echo hello
|
||||
world”, “cmd: hello world” can be “cmd[666]: echo hello world”,
|
||||
“cmd[666]: hello world”.
|
||||
|
||||
Error message
|
||||
-------------
|
||||
|
||||
There are two kinds of error messages in LISA. The first is an error
|
||||
message, and it does not fail. It will be printed as stderr and will be
|
||||
more obvious when the test case fails. The second is a one-line message
|
||||
in the failed test case. This section applies to two of them, but the
|
||||
second one is more important because we want it to be the only
|
||||
information that helps understand the failed test case.
|
||||
|
||||
In LISA, failed, skipped, and some passed test cases have a message. It
|
||||
specifies the reason the test case failed or skipped. Through this
|
||||
message, the user can understand what will happen and can act.
|
||||
Therefore, this message should be as helpful as possible.
|
||||
|
||||
The error message should include what happened and how to resolve it. It
|
||||
may not be easy to provide all the information for the first time, but
|
||||
guesswork is also helpful. At the same time, the original error message
|
||||
is also useful, please don’t hide it.
|
||||
|
||||
For examples,
|
||||
|
||||
- “The subscription ID [aaa] could not be found, please make sure it
|
||||
exists and is accessible by the current account”. A bad example, “The
|
||||
subscription ID [aaa] could not be found”. This bad example
|
||||
illustrates what happened, but there is no suggestion.
|
||||
- “The vm size [aaa] could not be found on the location [bbb]. This may
|
||||
be because the virtual machine size is not available in this
|
||||
location”. A bad example, “The vm size [aaa] could not be found on
|
||||
the location [bbb]”. It explains what happened, but it does not
|
||||
provide a guess at the root cause.
|
||||
|
||||
Assertion
|
||||
---------
|
||||
|
||||
Assertions are heavily used in test code. Assertions are a simple
|
||||
pattern of “if some checks fail, raise an exception”.
|
||||
|
||||
The assertion library includes commonly used patterns and detailed error
|
||||
messages. LISA uses ``assertpy`` as a standard assertion library, which
|
||||
provides Pythonic and test-friendly assertions.
|
||||
|
||||
When writing the assertion,
|
||||
|
||||
- Put the actual value in ``assert_that`` to keep the style consistent,
|
||||
and you can compare it with multiple expected values continuously.
|
||||
- Assertions should be as comprehensive as possible, but do not repeat
|
||||
existing checks. For example,
|
||||
``assert_that(str1).is_equal_to('hello')`` is enough, no need like
|
||||
``assert_that(str1).is_instance_of(str).is_equal_to('hello')``.
|
||||
- Add a description to explain the business logic. If a malfunction
|
||||
occurs, these instructions will be displayed. For example,
|
||||
``assert_that(str1).described_as('echo back result is unexpected').is_equal_to('hello')``
|
||||
is better than ``assert_that(str1).is_equal_to('hello')``.
|
||||
- Try to use native assertions instead of manipulating the data
|
||||
yourself. ``assert_that(vmbuses).is_length(6)`` is better than
|
||||
``assert_that(len(vmbuses)).is_equal_to(6)``. It is simpler and the
|
||||
error message is clearer.
|
||||
- Don’t forget to use powerful collection assertions. They can compare
|
||||
ordered list by ``contains`` (actual value is superset),
|
||||
``is_subset_of`` (actual value is subset), and others.
|
||||
|
||||
Learn more from `examples <../../examples/testsuites>`__ and `assertpy
|
||||
document <https://github.com/assertpy/assertpy#readme>`__.
|
||||
|
||||
Troubleshooting excellence
|
||||
--------------------------
|
||||
|
||||
Test failure is a common phenomenon. Therefore, perform troubleshooting
|
||||
frequently. There are some useful ways to troubleshoot failures. In the
|
||||
list below, the higher items are better than the lower items because of
|
||||
its lower cost of analysis.
|
||||
|
||||
1. Single line message. A one-line message is sent with the test result
|
||||
status. If this message clearly describes the root cause, no other
|
||||
digging is necessary. You can even perform some automated actions to
|
||||
match messages and act.
|
||||
2. Test case log. LISA provides a complete log for each run, which
|
||||
includes the output of all test cases, all threads, and all nodes.
|
||||
This file can be regarded as the default log, which is easy to
|
||||
search.
|
||||
3. Other log files. Some original logs may be divided into test cases.
|
||||
After finding out the cause, it is easier to find out. But it needs
|
||||
to download and browse the test result files.
|
||||
4. Reproduce in the environment. It is costly but contains most of the
|
||||
original information. But sometimes, the problem cannot be
|
||||
reproduced.
|
||||
|
||||
In LISA, test cases fail due to exceptions, and exception messages are
|
||||
treated as single-line messages. When writing test cases, it’s time to
|
||||
adjust the exception message. Therefore, after completing the test case,
|
||||
many errors will be explained well.
|
||||
|
||||
Document excellence
|
||||
-------------------
|
||||
|
||||
The documentation is the opportunity to make things clear and easy to
|
||||
maintain. A longer document is not always a better document. Each kind
|
||||
of documentation has its own purpose. Good technical documentation
|
||||
should be *useful and accurate*.
|
||||
|
||||
Tips for non-native English speakers by non-native English speakers
|
||||
-------------------------------------------------------------------
|
||||
|
||||
Today, there are a lot of great tools to help you create high-quality
|
||||
English documents. If writing in English is challenging, please try the
|
||||
following steps:
|
||||
|
||||
1. Read our documentations.
|
||||
2. Write in your language first.
|
||||
3. Use machine translation such as `Microsoft
|
||||
Translator <https://www.bing.com/translator/>`__ and `Google
|
||||
translate <https://translate.google.com/>`__ to convert it to
|
||||
English.
|
||||
4. Convert the English version back to your language and check. If it
|
||||
doesn’t make sense after translating back, it means the sentence is
|
||||
too complicated. Make it simpler, and then start from step 1 again.
|
||||
5. Once satisfied, you can use `Microsoft
|
||||
Editor <https://www.microsoft.com/en-us/microsoft-365/microsoft-editor>`__
|
||||
to further refine the grammar and wordings.
|
||||
|
|
@ -1,144 +0,0 @@
|
|||
# Development setup
|
||||
|
||||
This document describes the existing developer tooling we have in place (and
|
||||
what to expect of it).
|
||||
|
||||
- [Environment Setup](#environment-setup)
|
||||
- [Visual Studio Code](#visual-studio-code)
|
||||
- [Emacs](#emacs)
|
||||
- [Other setups](#other-setups)
|
||||
- [Code checks](#code-checks)
|
||||
- [Extended reading](#extended-reading)
|
||||
|
||||
## Environment Setup
|
||||
|
||||
Follow the [installation](../quick_start#installation) steps to prepare the
|
||||
source code. Then follow the steps below to set up the corresponding development
|
||||
environment.
|
||||
|
||||
### Visual Studio Code
|
||||
|
||||
1. Click on the Python version at the bottom left of the editor's window and
|
||||
select the Python interpreter which Poetry just created. If you do not find
|
||||
it, check [FAQ and troubleshooting](../troubleshooting.md) for extra
|
||||
instructions. This step is important because it ensures that the current
|
||||
workspace uses the correct Poetry virtual environment which provides all
|
||||
dependencies required.
|
||||
|
||||
1. You can copy the settings below into `.vscode/settings.json`.
|
||||
|
||||
```json
|
||||
{
|
||||
"markdown.extension.toc.levels": "2..6",
|
||||
"python.analysis.typeCheckingMode": "strict",
|
||||
"python.formatting.provider": "black",
|
||||
"python.linting.enabled": true,
|
||||
"python.linting.flake8Enabled": true,
|
||||
"python.linting.mypyEnabled": true,
|
||||
"python.linting.pylintEnabled": false,
|
||||
"editor.formatOnSave": true,
|
||||
"python.linting.mypyArgs": [
|
||||
"--strict",
|
||||
"--namespace-packages",
|
||||
"--implicit-reexport",
|
||||
"--show-column-numbers"
|
||||
],
|
||||
"python.sortImports.path": "isort",
|
||||
"python.analysis.useLibraryCodeForTypes": false,
|
||||
"python.analysis.autoImportCompletions": false,
|
||||
"files.eol": "\n",
|
||||
"terminal.integrated.env.windows": {
|
||||
"mypypath": "${workspaceFolder}\\typings"
|
||||
},
|
||||
"python.analysis.diagnosticSeverityOverrides": {
|
||||
"reportUntypedClassDecorator": "none",
|
||||
"reportUnknownMemberType": "none",
|
||||
"reportGeneralTypeIssues": "none",
|
||||
"reportUnknownVariableType": "none",
|
||||
"reportUnknownArgumentType": "none",
|
||||
"reportUnknownParameterType": "none",
|
||||
"reportUnboundVariable": "none",
|
||||
"reportPrivateUsage": "none",
|
||||
"reportImportCycles": "none",
|
||||
"reportUnnecessaryIsInstance": "none",
|
||||
},
|
||||
"python.languageServer": "Pylance",
|
||||
"markdown.extension.toc.levels": "2..6",
|
||||
}
|
||||
```
|
||||
|
||||
1. Install extensions.
|
||||
|
||||
- Install
|
||||
[Pylance](https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance)
|
||||
to get best code intelligence experience.
|
||||
- Install
|
||||
[Rewrap](https://marketplace.visualstudio.com/items?itemName=stkb.rewrap)
|
||||
to automatically wrap.
|
||||
- If there is need to update the documentation, it is recommended to install
|
||||
[Markdown All in
|
||||
One](https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one).
|
||||
It helps to maintain the table of contents in the documentation.
|
||||
|
||||
### Emacs
|
||||
|
||||
Use the [pyvenv](https://github.com/jorgenschaefer/pyvenv) package:
|
||||
|
||||
```emacs-lisp
|
||||
(use-package pyvenv
|
||||
:ensure t
|
||||
:hook (python-mode . pyvenv-tracking-mode))
|
||||
```
|
||||
|
||||
Then run `M-x add-dir-local-variable RET python-mode RET pyvenv-activate RET
|
||||
<path/to/virtualenv>` where the value is the path given by the command above.
|
||||
This will create a `.dir-locals.el` file as follows:
|
||||
|
||||
```emacs-lisp
|
||||
;;; Directory Local Variables
|
||||
;;; For more information see (info "(emacs) Directory Variables")
|
||||
|
||||
((python-mode . ((pyvenv-activate . "~/.cache/pypoetry/virtualenvs/lisa-s7Q404Ij-py3.8"))))
|
||||
```
|
||||
|
||||
### Other setups
|
||||
|
||||
- Install and enable [ShellCheck](https://github.com/koalaman/shellcheck) to
|
||||
find bash errors locally.
|
||||
|
||||
## Code checks
|
||||
|
||||
If the development environment is set up correctly, the following tools will
|
||||
automatically check the code. If there is any problem with the development
|
||||
environment settings, please feel free to submit an issue to us or create a pull
|
||||
request for repair. You can also run the check manually.
|
||||
|
||||
- [Black](https://github.com/psf/black), the opinionated code formatter resolves
|
||||
all disputes about how to format our Python files. This will become clearer
|
||||
after following [PEP 8](https://www.python.org/dev/peps/pep-0008/) (official
|
||||
Python style guide).
|
||||
- [Flake8](https://flake8.pycqa.org/en/latest/) (and integrations), the semantic
|
||||
analyzer, used to coordinate most other tools.
|
||||
- [isort](https://timothycrosley.github.io/isort/), the `import` sorter, it will
|
||||
automatically divide the import into the expected alphabetical order.
|
||||
- [mypy](http://mypy-lang.org/), the static type checker, which allows us to
|
||||
find potential errors by annotating and checking types.
|
||||
- [rope](https://github.com/python-rope/rope), provides completion and renaming
|
||||
support for pyls.
|
||||
|
||||
## Extended reading
|
||||
|
||||
- [Python Design Patterns](https://python-patterns.guide/). A fantastic
|
||||
collection of material for using Python's design patterns.
|
||||
- [The Hitchhiker’s Guide to Python](https://docs.python-guide.org/). This
|
||||
handcrafted guide exists to provide both novice and expert Python developers a
|
||||
best practice handbook for the installation, configuration, and usage of
|
||||
Python on a daily basis.
|
||||
- LISA performs static type checking to help finding bugs. Learn more from [mypy
|
||||
cheat sheet](https://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html) and
|
||||
[typing lib](https://docs.python.org/3/library/typing.html). You can also
|
||||
learn from LISA code.
|
||||
- [How to write best commit
|
||||
messages](https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html)
|
||||
and [Git best
|
||||
practice](http://sethrobertson.github.io/GitBestPractices/#sausage).
|
||||
|
|
@ -0,0 +1,161 @@
|
|||
Development setup
|
||||
=================
|
||||
|
||||
This document describes the existing developer tooling we have in place
|
||||
(and what to expect of it).
|
||||
|
||||
- `Environment Setup <#environment-setup>`__
|
||||
|
||||
- `Visual Studio Code <#visual-studio-code>`__
|
||||
- `Emacs <#emacs>`__
|
||||
- `Other setups <#other-setups>`__
|
||||
|
||||
- `Code checks <#code-checks>`__
|
||||
- `Extended reading <#extended-reading>`__
|
||||
|
||||
Environment Setup
|
||||
-----------------
|
||||
|
||||
Follow the `installation <../quick_start#installation>`__ steps to
|
||||
prepare the source code. Then follow the steps below to set up the
|
||||
corresponding development environment.
|
||||
|
||||
Visual Studio Code
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
1. Click on the Python version at the bottom left of the editor’s window
|
||||
and select the Python interpreter which Poetry just created. If you
|
||||
do not find it, check `FAQ and
|
||||
troubleshooting <../troubleshooting.html>`__ for extra instructions.
|
||||
This step is important because it ensures that the current workspace
|
||||
uses the correct Poetry virtual environment which provides all
|
||||
dependencies required.
|
||||
|
||||
2. You can copy the settings below into ``.vscode/settings.json``.
|
||||
|
||||
.. code:: json
|
||||
|
||||
{
|
||||
"markdown.extension.toc.levels": "2..6",
|
||||
"python.analysis.typeCheckingMode": "strict",
|
||||
"python.formatting.provider": "black",
|
||||
"python.linting.enabled": true,
|
||||
"python.linting.flake8Enabled": true,
|
||||
"python.linting.mypyEnabled": true,
|
||||
"python.linting.pylintEnabled": false,
|
||||
"editor.formatOnSave": true,
|
||||
"python.linting.mypyArgs": [
|
||||
"--strict",
|
||||
"--namespace-packages",
|
||||
"--implicit-reexport",
|
||||
"--show-column-numbers"
|
||||
],
|
||||
"python.sortImports.path": "isort",
|
||||
"python.analysis.useLibraryCodeForTypes": false,
|
||||
"python.analysis.autoImportCompletions": false,
|
||||
"files.eol": "\n",
|
||||
"terminal.integrated.env.windows": {
|
||||
"mypypath": "${workspaceFolder}\\typings"
|
||||
},
|
||||
"python.analysis.diagnosticSeverityOverrides": {
|
||||
"reportUntypedClassDecorator": "none",
|
||||
"reportUnknownMemberType": "none",
|
||||
"reportGeneralTypeIssues": "none",
|
||||
"reportUnknownVariableType": "none",
|
||||
"reportUnknownArgumentType": "none",
|
||||
"reportUnknownParameterType": "none",
|
||||
"reportUnboundVariable": "none",
|
||||
"reportPrivateUsage": "none",
|
||||
"reportImportCycles": "none",
|
||||
"reportUnnecessaryIsInstance": "none",
|
||||
},
|
||||
"python.languageServer": "Pylance",
|
||||
"markdown.extension.toc.levels": "2..6",
|
||||
}
|
||||
|
||||
3. Install extensions.
|
||||
|
||||
- Install
|
||||
`Pylance <https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance>`__
|
||||
to get best code intelligence experience.
|
||||
- Install
|
||||
`Rewrap <https://marketplace.visualstudio.com/items?itemName=stkb.rewrap>`__
|
||||
to automatically wrap.
|
||||
- If there is need to update the documentation, it is recommended to
|
||||
install `Markdown All in
|
||||
One <https://marketplace.visualstudio.com/items?itemName=yzhang.markdown-all-in-one>`__.
|
||||
It helps to maintain the table of contents in the documentation.
|
||||
|
||||
Emacs
|
||||
~~~~~
|
||||
|
||||
Use the `pyvenv <https://github.com/jorgenschaefer/pyvenv>`__ package:
|
||||
|
||||
.. code:: emacs-lisp
|
||||
|
||||
(use-package pyvenv
|
||||
:ensure t
|
||||
:hook (python-mode . pyvenv-tracking-mode))
|
||||
|
||||
Then run
|
||||
``M-x add-dir-local-variable RET python-mode RET pyvenv-activate RET <path/to/virtualenv>``
|
||||
where the value is the path given by the command above. This will create
|
||||
a ``.dir-locals.el`` file as follows:
|
||||
|
||||
.. code:: emacs-lisp
|
||||
|
||||
;;; Directory Local Variables
|
||||
;;; For more information see (info "(emacs) Directory Variables")
|
||||
|
||||
((python-mode . ((pyvenv-activate . "~/.cache/pypoetry/virtualenvs/lisa-s7Q404Ij-py3.8"))))
|
||||
|
||||
Other setups
|
||||
~~~~~~~~~~~~
|
||||
|
||||
- Install and enable
|
||||
`ShellCheck <https://github.com/koalaman/shellcheck>`__ to find bash
|
||||
errors locally.
|
||||
|
||||
Code checks
|
||||
-----------
|
||||
|
||||
If the development environment is set up correctly, the following tools
|
||||
will automatically check the code. If there is any problem with the
|
||||
development environment settings, please feel free to submit an issue to
|
||||
us or create a pull request for repair. You can also run the check
|
||||
manually.
|
||||
|
||||
- `Black <https://github.com/psf/black>`__, the opinionated code
|
||||
formatter resolves all disputes about how to format our Python files.
|
||||
This will become clearer after following `PEP
|
||||
8 <https://www.python.org/dev/peps/pep-0008/>`__ (official Python
|
||||
style guide).
|
||||
- `Flake8 <https://flake8.pycqa.org/en/latest/>`__ (and integrations),
|
||||
the semantic analyzer, used to coordinate most other tools.
|
||||
- `isort <https://timothycrosley.github.io/isort/>`__, the ``import``
|
||||
sorter, it will automatically divide the import into the expected
|
||||
alphabetical order.
|
||||
- `mypy <http://mypy-lang.org/>`__, the static type checker, which
|
||||
allows us to find potential errors by annotating and checking types.
|
||||
- `rope <https://github.com/python-rope/rope>`__, provides completion
|
||||
and renaming support for pyls.
|
||||
|
||||
Extended reading
|
||||
----------------
|
||||
|
||||
- `Python Design Patterns <https://python-patterns.guide/>`__. A
|
||||
fantastic collection of material for using Python’s design patterns.
|
||||
- `The Hitchhiker’s Guide to
|
||||
Python <https://docs.python-guide.org/>`__. This handcrafted guide
|
||||
exists to provide both novice and expert Python developers a best
|
||||
practice handbook for the installation, configuration, and usage of
|
||||
Python on a daily basis.
|
||||
- LISA performs static type checking to help finding bugs. Learn more
|
||||
from `mypy cheat
|
||||
sheet <https://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html>`__
|
||||
and `typing lib <https://docs.python.org/3/library/typing.html>`__.
|
||||
You can also learn from LISA code.
|
||||
- `How to write best commit
|
||||
messages <https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html>`__
|
||||
and `Git best
|
||||
practice <http://sethrobertson.github.io/GitBestPractices/#sausage>`__.
|
||||
|
|
@ -1,46 +0,0 @@
|
|||
# Info for developers
|
||||
|
||||
- [Basic concepts](concepts.md)
|
||||
- [Test case and Test suite](concepts.md#test-case-and-test-suite)
|
||||
- [Node and Environment](concepts.md#node-and-environment)
|
||||
- [Extension](concepts.md#extension)
|
||||
- [Platform](concepts.md#platform)
|
||||
- [Tool and Script](concepts.md#tool-and-script)
|
||||
- [Feature](concepts.md#feature)
|
||||
- [Combinator](concepts.md#combinator)
|
||||
- [Grid combinator](concepts.md#grid-combinator)
|
||||
- [Batch combinator](concepts.md#batch-combinator)
|
||||
- [Transformer](concepts.md#transformer)
|
||||
- [Runbook](concepts.md#runbook)
|
||||
- [Requirement and Capability](concepts.md#requirement-and-capability)
|
||||
- [Development setup](setup.md)
|
||||
- [How to write a test case](write_case.md)
|
||||
- [Preparation](write_case.md#preparation)
|
||||
- [Test composition](write_case.md#test-composition)
|
||||
- [Metadata](write_case.md#metadata)
|
||||
- [Test case body](write_case.md#test-case-body)
|
||||
- [Setup and clean-up](write_case.md#setup-and-clean-up)
|
||||
- [Using extensions](write_case.md#using-extensions)
|
||||
- [Best practices](write_case.md#best-practices)
|
||||
- [How to write extensions](extension.md)
|
||||
- [Notifier](extension.md#notifier)
|
||||
- [Tool](extension.md#tool)
|
||||
- [CustomScript](extension.md#customscript)
|
||||
- [Feature](extension.md#feature)
|
||||
- [Combinator](extension.md#combinator)
|
||||
- [Transformer](extension.md#transformer)
|
||||
- [Platform](extension.md#platform)
|
||||
- [Hooks](extension.md#hooks)
|
||||
- [Some notes](extension.md#some-notes)
|
||||
- [Code quality](guidelines.md)
|
||||
- [Naming Conventions](guidelines.md#naming-conventions)
|
||||
- [Test code excellence](guidelines.md#test-code-excellence)
|
||||
- [Code comments](guidelines.md#code-comments)
|
||||
- [Commit messages](guidelines.md#commit-messages)
|
||||
- [Logging](guidelines.md#logging)
|
||||
- [Error message](guidelines.md#error-message)
|
||||
- [Assertion](guidelines.md#assertion)
|
||||
- [Troubleshooting excellence](guidelines.md#troubleshooting-excellence)
|
||||
- [Document excellence](guidelines.md#document-excellence)
|
||||
- [Tips for non-native English speakers by non-native English
|
||||
speakers](guidelines.md#tips-for-non-native-english-speakers-by-non-native-english-speakers)
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
Info for developers
|
||||
===================
|
||||
|
||||
- `Basic concepts <concepts.html>`__
|
||||
|
||||
- `Test case and Test
|
||||
suite <concepts.html#test-case-and-test-suite>`__
|
||||
- `Node and Environment <concepts.html#node-and-environment>`__
|
||||
- `Extension <concepts.html#extension>`__
|
||||
|
||||
- `Platform <concepts.html#platform>`__
|
||||
- `Tool and Script <concepts.html#tool-and-script>`__
|
||||
- `Feature <concepts.html#feature>`__
|
||||
- `Combinator <concepts.html#combinator>`__
|
||||
|
||||
- `Grid combinator <concepts.html#grid-combinator>`__
|
||||
- `Batch combinator <concepts.html#batch-combinator>`__
|
||||
|
||||
- `Transformer <concepts.html#transformer>`__
|
||||
|
||||
- `Runbook <concepts.html#runbook>`__
|
||||
- `Requirement and
|
||||
Capability <concepts.html#requirement-and-capability>`__
|
||||
|
||||
- `Development setup <setup.html>`__
|
||||
- `How to write a test case <write_case.html>`__
|
||||
|
||||
- `Preparation <write_case.html#preparation>`__
|
||||
- `Test composition <write_case.html#test-composition>`__
|
||||
|
||||
- `Metadata <write_case.html#metadata>`__
|
||||
- `Test case body <write_case.html#test-case-body>`__
|
||||
- `Setup and clean-up <write_case.html#setup-and-clean-up>`__
|
||||
|
||||
- `Using extensions <write_case.html#using-extensions>`__
|
||||
- `Best practices <write_case.html#best-practices>`__
|
||||
|
||||
- `How to write extensions <extension.html>`__
|
||||
|
||||
- `Notifier <extension.html#notifier>`__
|
||||
- `Tool <extension.html#tool>`__
|
||||
- `CustomScript <extension.html#customscript>`__
|
||||
- `Feature <extension.html#feature>`__
|
||||
- `Combinator <extension.html#combinator>`__
|
||||
- `Transformer <extension.html#transformer>`__
|
||||
- `Platform <extension.html#platform>`__
|
||||
- `Hooks <extension.html#hooks>`__
|
||||
- `Some notes <extension.html#some-notes>`__
|
||||
|
||||
- `Code quality <guidelines.html>`__
|
||||
|
||||
- `Naming Conventions <guidelines.html#naming-conventions>`__
|
||||
- `Test code excellence <guidelines.html#test-code-excellence>`__
|
||||
- `Code comments <guidelines.html#code-comments>`__
|
||||
- `Commit messages <guidelines.html#commit-messages>`__
|
||||
- `Logging <guidelines.html#logging>`__
|
||||
- `Error message <guidelines.html#error-message>`__
|
||||
- `Assertion <guidelines.html#assertion>`__
|
||||
- `Troubleshooting
|
||||
excellence <guidelines.html#troubleshooting-excellence>`__
|
||||
- `Document excellence <guidelines.html#document-excellence>`__
|
||||
- `Tips for non-native English speakers by non-native English
|
||||
speakers <guidelines.html#tips-for-non-native-english-speakers-by-non-native-english-speakers>`__
|
||||
|
|
@ -1,298 +0,0 @@
|
|||
# How to write test suites/cases
|
||||
|
||||
- [Preparation](#preparation)
|
||||
- [Test composition](#test-composition)
|
||||
- [Metadata](#metadata)
|
||||
- [Metadata in test suite](#metadata-in-test-suite)
|
||||
- [Metadata in test case](#metadata-in-test-case)
|
||||
- [Test case body](#test-case-body)
|
||||
- [Setup and clean-up](#setup-and-clean-up)
|
||||
- [Using extensions](#using-extensions)
|
||||
- [Environment and node](#environment-and-node)
|
||||
- [Tool](#tool)
|
||||
- [Scripts](#scripts)
|
||||
- [Features](#features)
|
||||
- [Hooks](#hooks)
|
||||
- [`get_environment_information`](#get_environment_information)
|
||||
- [`azure_deploy_failed`](#azure_deploy_failed)
|
||||
- [`azure_update_arm_template`](#azure_update_arm_template)
|
||||
- [Best practices](#best-practices)
|
||||
- [Debug in ready environment](#debug-in-ready-environment)
|
||||
|
||||
## Preparation
|
||||
|
||||
Before getting down to do some exciting coding, we recommend that you read the
|
||||
following documents to ensure a better LISA development experience. We believe
|
||||
that the engineering excellence is equally important in addition to new test
|
||||
cases, since any test case will be run thousands of times, and many people will
|
||||
read and troubleshoot it. Therefore, a good test case following the guidelines
|
||||
can save everyone's time.
|
||||
|
||||
- [Basic concepts](concepts.md) introduces design considerations and how
|
||||
components work together in LISA. We recommend every LISA developer go through
|
||||
this before coding.
|
||||
- [Coding guidelines](guidelines.md) covers our coding guidelines such as
|
||||
naming, code, comment conventions, etc.
|
||||
- [Development setup](setup.md) introduces how to setup environment and code
|
||||
checks.
|
||||
- [Extensions](extension.md) introduces how to develop extensions for LISA. In
|
||||
some cases, you may need to improve or implement extensions for new test
|
||||
cases.
|
||||
|
||||
## Test composition
|
||||
|
||||
The LISA test is composed of [metadata](#metadata), [test body](#test-case-body)
|
||||
and [setup/clean-up](#setup-and-clean-up).
|
||||
|
||||
### Metadata
|
||||
|
||||
Metadata provides documentations and settings for test cases and test suites,
|
||||
illustrates the main test logic, and is used to generate specifications. Both of
|
||||
the following examples are taken from
|
||||
[provision.py](../../microsoft/testsuites/core/provisioning.py). See [example
|
||||
tests](../../examples/testsuites) for more examples.
|
||||
|
||||
#### Metadata in test suite
|
||||
|
||||
A test suite is a set of test cases with similar test purposes or shared steps.
|
||||
|
||||
```python
|
||||
@TestSuiteMetadata(
|
||||
area="provisioning",
|
||||
category="functional",
|
||||
description="""
|
||||
This test suite is to verify if an environment can be provisioned correct or not.
|
||||
|
||||
- The basic smoke test can run on all images to determine if a image can boot and
|
||||
reboot.
|
||||
- Other provisioning tests verify if an environment can be provisioned with special
|
||||
hardware configurations.
|
||||
""",
|
||||
)
|
||||
class Provisioning(TestSuite):
|
||||
...
|
||||
```
|
||||
|
||||
- **area** classifies test suites by their task field. When it needs to have a
|
||||
special validation on some area, it can be used to filter test cases. It can
|
||||
be provisioning, CPU, memory, storage, network, etc.
|
||||
- **category** categorizes test cases by test type. It includes functional,
|
||||
performance, stress, and community. Performance and stress test cases take
|
||||
longer time to run, which are not included in regular operations. Community
|
||||
test cases are wrappers that help provide results comparable to the community.
|
||||
- **description** introduces purpose, coverage, why these test cases are bundled
|
||||
together and other content of the test suite, which makes clarity the test
|
||||
suite.
|
||||
- **name** is optional. The default name is the class name and will be
|
||||
overridden by this field if provided. It is part of the test name, just like
|
||||
the namespace in a programming language.
|
||||
- **requirement** is optional. A test case without this field means it does not
|
||||
have any requirement. It defines the default requirement for this test suite
|
||||
and can be overwritten at the test case level. Learn more from
|
||||
[concepts](concepts.md#requirement-and-capability).
|
||||
|
||||
#### Metadata in test case
|
||||
|
||||
A test case is a test that has its own test purpose and steps.
|
||||
|
||||
```python
|
||||
@TestCaseMetadata(
|
||||
description="""
|
||||
This case verifies whether a node is operating normally.
|
||||
|
||||
Steps,
|
||||
1. Connect to TCP port 22. If it's not connectable, failed and check whether
|
||||
there is kernel panic.
|
||||
2. Connect to SSH port 22, and reboot the node. If there is an error and kernel
|
||||
panic, fail the case. If it's not connectable, also fail the case.
|
||||
3. If there is another error, but not kernel panic or TCP connection, pass with
|
||||
warning.
|
||||
4. Otherwise, fully passed.
|
||||
""",
|
||||
priority=0,
|
||||
requirement=simple_requirement(
|
||||
environment_status=EnvironmentStatus.Deployed,
|
||||
supported_features=[SerialConsole],
|
||||
),
|
||||
)
|
||||
def smoke_test(self, case_name: str) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
- **description** explains the purpose and procedures of the test. As said
|
||||
before, it is also used to generate test specification documents.
|
||||
- **priority** depends on the impact of the test case and is used to determine
|
||||
how often to run the case. A lower priority means a test case of more
|
||||
importance, and thus it will be run more often. The lowest value (most
|
||||
prioritized) is `0`.
|
||||
- **requirement** defines the requirements in this case. If no requirement
|
||||
specified, the test suite's or the default global requirements will apply.
|
||||
|
||||
Note for a regression test case, which deals with further issues that the fixed
|
||||
bug might cause, the related bugs should be presented. It is also helpful to
|
||||
include impact of failure in metadata.
|
||||
|
||||
### Test case body
|
||||
|
||||
The test case body contains the actual implementations of the test. You can
|
||||
import existing `tools` to verify certain purposes. If existing `tools` cannot
|
||||
realize your test purpose, it is recommended that you wrap your test codes into
|
||||
functions, integrate them into new `tools`, and then only call functions like
|
||||
`assert_that` in test case body to verify. The section below explains how to do
|
||||
this.
|
||||
|
||||
The method accepts `environment`, `node` and other arguments as follows. An
|
||||
example from [helloworld.py](../../examples/testsuites/helloworld.py):
|
||||
|
||||
```python
|
||||
def hello(self, case_name: str, node: Node, environment: Environment) -> None:
|
||||
...
|
||||
assert_that(result.stdout).is_equal_to(hello_world)
|
||||
assert_that(result.stderr).is_equal_to("")
|
||||
assert_that(result.exit_code).is_equal_to(0)
|
||||
```
|
||||
|
||||
Find more examples in [example tests](../../examples/testsuites) and [Microsoft
|
||||
tests](../../microsoft/testsuites).
|
||||
|
||||
### Setup and clean-up
|
||||
|
||||
There are four methods in two pairs:
|
||||
|
||||
1. `before_suite` & `after_suite`
|
||||
|
||||
2. `before_case` & `after_case`
|
||||
|
||||
They are used to share common logic or variables among test cases. They will be
|
||||
called in the corresponding step.
|
||||
|
||||
The kwargs supports variables similar to those in test methods.
|
||||
|
||||
```python
|
||||
def before_suite(self, **kwargs: Any) -> None:
|
||||
...
|
||||
|
||||
def after_suite(self, **kwargs: Any) -> None:
|
||||
...
|
||||
|
||||
def before_case(self, **kwargs: Any) -> None:
|
||||
...
|
||||
|
||||
def after_case(self, **kwargs: Any) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
## Using extensions
|
||||
|
||||
When implementing test cases, you may need to use some existing extensions, or
|
||||
you are welcome to create your own. This section focuses on how to use them in
|
||||
the test code.
|
||||
|
||||
Read
|
||||
- [concepts](concepts.md) to understand which extension does what and
|
||||
- [how to write extensions](extension.md) to develop new extensions
|
||||
|
||||
### Environment and node
|
||||
|
||||
The `environment` and `node` variables are obtained from the method arguments
|
||||
`def hello(self, node: Node, environment: Environment)`. If there are multiple
|
||||
nodes in the environment, you can use `environment.nodes` to get them. The node
|
||||
per se can run any command, but it is recommended to implement the logic in
|
||||
`tools` and obtain the tool by `node.tools[ToolName]`.
|
||||
|
||||
### Tool
|
||||
|
||||
As said, call `node.tools[ToolName]` to obtain the tool. When called, LISA will
|
||||
first check if the tool is installed. If not, LISA will install it, and after
|
||||
that, an instance of the tool will be returned. The instance is available until
|
||||
the node is recycled, which means the same tool is already ready to use when
|
||||
`node.tools[ToolName]` is called again, as to avoid the redundant installation.
|
||||
|
||||
### Scripts
|
||||
|
||||
The `script`, like the `tool`, needs to be uploaded to the node before use. In
|
||||
addition, you need to define the following script builder before using the
|
||||
script.
|
||||
|
||||
```python
|
||||
self._echo_script = CustomScriptBuilder(
|
||||
Path(__file__).parent.joinpath("scripts"), ["echo.sh"]
|
||||
)
|
||||
```
|
||||
|
||||
Once defined, the script can be used like `script: CustomScript =
|
||||
node.tools[self._echo_script]`.
|
||||
|
||||
Please note that it is recommended that you use the tools in LISA instead of
|
||||
writing scripts. Bash scripts are not as flexible as Python, so we prefer to
|
||||
write logic in Python.
|
||||
|
||||
### Features
|
||||
|
||||
The `feature` needs to be declared in the requirements of the test suite or test
|
||||
case, as shown below. It means that the test case requires the feature, and if
|
||||
the feature is not available in the environment, the test case will be skipped.
|
||||
|
||||
```python
|
||||
@TestCaseMetadata(
|
||||
requirement=simple_requirement(
|
||||
supported_features=[SerialConsole],
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
After the declaration, you can use the feature just like the tool, by calling
|
||||
`node.features[SerialConsole]`.
|
||||
|
||||
### Hooks
|
||||
|
||||
Hooks are used to insert extension logic in the platform.
|
||||
|
||||
#### `get_environment_information`
|
||||
|
||||
It returns the information of an environment. It's called when a test case is
|
||||
completed.
|
||||
|
||||
Please note that to avoid the mutual influence of hooks, there is no upper
|
||||
`try...except...`. If a hook fails, it will fail the entire run. If you find
|
||||
such a problem, please solve it first.
|
||||
|
||||
```python
|
||||
@hookimpl # type: ignore
|
||||
def get_environment_information(self, environment: Environment) -> Dict[str, str]:
|
||||
information: Dict[str, str] = {}
|
||||
```
|
||||
|
||||
#### `azure_deploy_failed`
|
||||
|
||||
Called when Azure deployment fails. This is an opportunity to return a better
|
||||
error message. Learn from example in
|
||||
[hooks.py](../../lisa/sut_orchestrator/azure/hooks.py).
|
||||
|
||||
```python
|
||||
@hookimpl # type: ignore
|
||||
def azure_deploy_failed(self, error_message: str) -> None:
|
||||
for message, pattern, exception_type in self.__error_maps:
|
||||
if pattern.findall(error_message):
|
||||
raise exception_type(f"{message}. {error_message}")
|
||||
```
|
||||
|
||||
#### `azure_update_arm_template`
|
||||
|
||||
Called when it needs to update ARM template before deploying to Azure.
|
||||
|
||||
```python
|
||||
@hookimpl
|
||||
def azure_update_arm_template(
|
||||
self, template: Any, environment: Environment
|
||||
) -> None:
|
||||
...
|
||||
```
|
||||
|
||||
## Best practices
|
||||
|
||||
### Debug in ready environment
|
||||
|
||||
Debugging test cases or tools can be done on a local computer, in the ready
|
||||
environment, or in the deployed Azure environment. We recommend the latter two
|
||||
methods as they can save a lot of deployment time.
|
||||
|
|
@ -0,0 +1,334 @@
|
|||
How to write test suites/cases
|
||||
==============================
|
||||
|
||||
- `Preparation <#preparation>`__
|
||||
- `Test composition <#test-composition>`__
|
||||
|
||||
- `Metadata <#metadata>`__
|
||||
|
||||
- `Metadata in test suite <#metadata-in-test-suite>`__
|
||||
- `Metadata in test case <#metadata-in-test-case>`__
|
||||
|
||||
- `Test case body <#test-case-body>`__
|
||||
- `Setup and clean-up <#setup-and-clean-up>`__
|
||||
|
||||
- `Using extensions <#using-extensions>`__
|
||||
|
||||
- `Environment and node <#environment-and-node>`__
|
||||
- `Tool <#tool>`__
|
||||
- `Scripts <#scripts>`__
|
||||
- `Features <#features>`__
|
||||
- `Hooks <#hooks>`__
|
||||
|
||||
- ```get_environment_information`` <#get_environment_information>`__
|
||||
- ```azure_deploy_failed`` <#azure_deploy_failed>`__
|
||||
- ```azure_update_arm_template`` <#azure_update_arm_template>`__
|
||||
|
||||
- `Best practices <#best-practices>`__
|
||||
|
||||
- `Debug in ready environment <#debug-in-ready-environment>`__
|
||||
|
||||
Preparation
|
||||
-----------
|
||||
|
||||
Before getting down to do some exciting coding, we recommend that you
|
||||
read the following documents to ensure a better LISA development
|
||||
experience. We believe that the engineering excellence is equally
|
||||
important in addition to new test cases, since any test case will be run
|
||||
thousands of times, and many people will read and troubleshoot it.
|
||||
Therefore, a good test case following the guidelines can save everyone’s
|
||||
time.
|
||||
|
||||
- `Basic concepts <concepts.html>`__ introduces design considerations
|
||||
and how components work together in LISA. We recommend every LISA
|
||||
developer go through this before coding.
|
||||
- `Coding guidelines <guidelines.html>`__ covers our coding guidelines
|
||||
such as naming, code, comment conventions, etc.
|
||||
- `Development setup <setup.html>`__ introduces how to setup
|
||||
environment and code checks.
|
||||
- `Extensions <extension.html>`__ introduces how to develop extensions
|
||||
for LISA. In some cases, you may need to improve or implement
|
||||
extensions for new test cases.
|
||||
|
||||
Test composition
|
||||
----------------
|
||||
|
||||
The LISA test is composed of `metadata <#metadata>`__, `test
|
||||
body <#test-case-body>`__ and `setup/clean-up <#setup-and-clean-up>`__.
|
||||
|
||||
Metadata
|
||||
~~~~~~~~
|
||||
|
||||
Metadata provides documentations and settings for test cases and test
|
||||
suites, illustrates the main test logic, and is used to generate
|
||||
specifications. Both of the following examples are taken from
|
||||
`provision.py <../../microsoft/testsuites/core/provisioning.py>`__. See
|
||||
`example tests <../../examples/testsuites>`__ for more examples.
|
||||
|
||||
Metadata in test suite
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
A test suite is a set of test cases with similar test purposes or shared
|
||||
steps.
|
||||
|
||||
.. code:: python
|
||||
|
||||
@TestSuiteMetadata(
|
||||
area="provisioning",
|
||||
category="functional",
|
||||
description="""
|
||||
This test suite is to verify if an environment can be provisioned correct or not.
|
||||
|
||||
- The basic smoke test can run on all images to determine if a image can boot and
|
||||
reboot.
|
||||
- Other provisioning tests verify if an environment can be provisioned with special
|
||||
hardware configurations.
|
||||
""",
|
||||
)
|
||||
class Provisioning(TestSuite):
|
||||
...
|
||||
|
||||
- **area** classifies test suites by their task field. When it needs to
|
||||
have a special validation on some area, it can be used to filter test
|
||||
cases. It can be provisioning, CPU, memory, storage, network, etc.
|
||||
- **category** categorizes test cases by test type. It includes
|
||||
functional, performance, stress, and community. Performance and
|
||||
stress test cases take longer time to run, which are not included in
|
||||
regular operations. Community test cases are wrappers that help
|
||||
provide results comparable to the community.
|
||||
- **description** introduces purpose, coverage, why these test cases
|
||||
are bundled together and other content of the test suite, which makes
|
||||
clarity the test suite.
|
||||
- **name** is optional. The default name is the class name and will be
|
||||
overridden by this field if provided. It is part of the test name,
|
||||
just like the namespace in a programming language.
|
||||
- **requirement** is optional. A test case without this field means it
|
||||
does not have any requirement. It defines the default requirement for
|
||||
this test suite and can be overwritten at the test case level. Learn
|
||||
more from `concepts <concepts.html#requirement-and-capability>`__.
|
||||
|
||||
Metadata in test case
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
A test case is a test that has its own test purpose and steps.
|
||||
|
||||
.. code:: python
|
||||
|
||||
@TestCaseMetadata(
|
||||
description="""
|
||||
This case verifies whether a node is operating normally.
|
||||
|
||||
Steps,
|
||||
1. Connect to TCP port 22. If it's not connectable, failed and check whether
|
||||
there is kernel panic.
|
||||
2. Connect to SSH port 22, and reboot the node. If there is an error and kernel
|
||||
panic, fail the case. If it's not connectable, also fail the case.
|
||||
3. If there is another error, but not kernel panic or TCP connection, pass with
|
||||
warning.
|
||||
4. Otherwise, fully passed.
|
||||
""",
|
||||
priority=0,
|
||||
requirement=simple_requirement(
|
||||
environment_status=EnvironmentStatus.Deployed,
|
||||
supported_features=[SerialConsole],
|
||||
),
|
||||
)
|
||||
def smoke_test(self, case_name: str) -> None:
|
||||
...
|
||||
|
||||
- **description** explains the purpose and procedures of the test. As
|
||||
said before, it is also used to generate test specification
|
||||
documents.
|
||||
- **priority** depends on the impact of the test case and is used to
|
||||
determine how often to run the case. A lower priority means a test
|
||||
case of more importance, and thus it will be run more often. The
|
||||
lowest value (most prioritized) is ``0``.
|
||||
- **requirement** defines the requirements in this case. If no
|
||||
requirement specified, the test suite’s or the default global
|
||||
requirements will apply.
|
||||
|
||||
Note for a regression test case, which deals with further issues that
|
||||
the fixed bug might cause, the related bugs should be presented. It is
|
||||
also helpful to include impact of failure in metadata.
|
||||
|
||||
Test case body
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
The test case body contains the actual implementations of the test. You
|
||||
can import existing ``tools`` to verify certain purposes. If existing
|
||||
``tools`` cannot realize your test purpose, it is recommended that you
|
||||
wrap your test codes into functions, integrate them into new ``tools``,
|
||||
and then only call functions like ``assert_that`` in test case body to
|
||||
verify. The section below explains how to do this.
|
||||
|
||||
The method accepts ``environment``, ``node`` and other arguments as
|
||||
follows. An example from
|
||||
`helloworld.py <../../examples/testsuites/helloworld.py>`__:
|
||||
|
||||
.. code:: python
|
||||
|
||||
def hello(self, case_name: str, node: Node, environment: Environment) -> None:
|
||||
...
|
||||
assert_that(result.stdout).is_equal_to(hello_world)
|
||||
assert_that(result.stderr).is_equal_to("")
|
||||
assert_that(result.exit_code).is_equal_to(0)
|
||||
|
||||
Find more examples in `example tests <../../examples/testsuites>`__ and
|
||||
`Microsoft tests <../../microsoft/testsuites>`__.
|
||||
|
||||
Setup and clean-up
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
There are four methods in two pairs:
|
||||
|
||||
1. ``before_suite`` & ``after_suite``
|
||||
|
||||
2. ``before_case`` & ``after_case``
|
||||
|
||||
They are used to share common logic or variables among test cases. They
|
||||
will be called in the corresponding step.
|
||||
|
||||
The kwargs supports variables similar to those in test methods.
|
||||
|
||||
.. code:: python
|
||||
|
||||
def before_suite(self, **kwargs: Any) -> None:
|
||||
...
|
||||
|
||||
def after_suite(self, **kwargs: Any) -> None:
|
||||
...
|
||||
|
||||
def before_case(self, **kwargs: Any) -> None:
|
||||
...
|
||||
|
||||
def after_case(self, **kwargs: Any) -> None:
|
||||
...
|
||||
|
||||
Using extensions
|
||||
----------------
|
||||
|
||||
When implementing test cases, you may need to use some existing
|
||||
extensions, or you are welcome to create your own. This section focuses
|
||||
on how to use them in the test code.
|
||||
|
||||
Read - `concepts <concepts.html>`__ to understand which extension does
|
||||
what and - `how to write extensions <extension.html>`__ to develop new
|
||||
extensions
|
||||
|
||||
Environment and node
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The ``environment`` and ``node`` variables are obtained from the method
|
||||
arguments ``def hello(self, node: Node, environment: Environment)``. If
|
||||
there are multiple nodes in the environment, you can use
|
||||
``environment.nodes`` to get them. The node per se can run any command,
|
||||
but it is recommended to implement the logic in ``tools`` and obtain the
|
||||
tool by ``node.tools[ToolName]``.
|
||||
|
||||
Tool
|
||||
~~~~
|
||||
|
||||
As said, call ``node.tools[ToolName]`` to obtain the tool. When called,
|
||||
LISA will first check if the tool is installed. If not, LISA will
|
||||
install it, and after that, an instance of the tool will be returned.
|
||||
The instance is available until the node is recycled, which means the
|
||||
same tool is already ready to use when ``node.tools[ToolName]`` is
|
||||
called again, as to avoid the redundant installation.
|
||||
|
||||
Scripts
|
||||
~~~~~~~
|
||||
|
||||
The ``script``, like the ``tool``, needs to be uploaded to the node
|
||||
before use. In addition, you need to define the following script builder
|
||||
before using the script.
|
||||
|
||||
.. code:: python
|
||||
|
||||
self._echo_script = CustomScriptBuilder(
|
||||
Path(__file__).parent.joinpath("scripts"), ["echo.sh"]
|
||||
)
|
||||
|
||||
Once defined, the script can be used like
|
||||
``script: CustomScript = node.tools[self._echo_script]``.
|
||||
|
||||
Please note that it is recommended that you use the tools in LISA
|
||||
instead of writing scripts. Bash scripts are not as flexible as Python,
|
||||
so we prefer to write logic in Python.
|
||||
|
||||
Features
|
||||
~~~~~~~~
|
||||
|
||||
The ``feature`` needs to be declared in the requirements of the test
|
||||
suite or test case, as shown below. It means that the test case requires
|
||||
the feature, and if the feature is not available in the environment, the
|
||||
test case will be skipped.
|
||||
|
||||
.. code:: python
|
||||
|
||||
@TestCaseMetadata(
|
||||
requirement=simple_requirement(
|
||||
supported_features=[SerialConsole],
|
||||
),
|
||||
)
|
||||
|
||||
After the declaration, you can use the feature just like the tool, by
|
||||
calling ``node.features[SerialConsole]``.
|
||||
|
||||
Hooks
|
||||
~~~~~
|
||||
|
||||
Hooks are used to insert extension logic in the platform.
|
||||
|
||||
``get_environment_information``
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
It returns the information of an environment. It’s called when a test
|
||||
case is completed.
|
||||
|
||||
Please note that to avoid the mutual influence of hooks, there is no
|
||||
upper ``try...except...``. If a hook fails, it will fail the entire run.
|
||||
If you find such a problem, please solve it first.
|
||||
|
||||
.. code:: python
|
||||
|
||||
@hookimpl # type: ignore
|
||||
def get_environment_information(self, environment: Environment) -> Dict[str, str]:
|
||||
information: Dict[str, str] = {}
|
||||
|
||||
``azure_deploy_failed``
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Called when Azure deployment fails. This is an opportunity to return a
|
||||
better error message. Learn from example in
|
||||
`hooks.py <../../lisa/sut_orchestrator/azure/hooks.py>`__.
|
||||
|
||||
.. code:: python
|
||||
|
||||
@hookimpl # type: ignore
|
||||
def azure_deploy_failed(self, error_message: str) -> None:
|
||||
for message, pattern, exception_type in self.__error_maps:
|
||||
if pattern.findall(error_message):
|
||||
raise exception_type(f"{message}. {error_message}")
|
||||
|
||||
``azure_update_arm_template``
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Called when it needs to update ARM template before deploying to Azure.
|
||||
|
||||
.. code:: python
|
||||
|
||||
@hookimpl
|
||||
def azure_update_arm_template(
|
||||
self, template: Any, environment: Environment
|
||||
) -> None:
|
||||
...
|
||||
|
||||
Best practices
|
||||
--------------
|
||||
|
||||
Debug in ready environment
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Debugging test cases or tools can be done on a local computer, in the
|
||||
ready environment, or in the deployed Azure environment. We recommend
|
||||
the latter two methods as they can save a lot of deployment time.
|
||||
Загрузка…
Ссылка в новой задаче