lisa/docs/run_test/runbook.rst

Runbook Reference
=================

-  `What is a runbook <#what-is-a-runbook>`__
-  `How-to <#how-to>`__

   -  `Configure Azure deployment <#configure-azure-deployment>`__
   -  `Select and set test cases <#select-and-set-test-cases>`__
   -  `Use variable and secrets <#use-variable-and-secrets>`__
   -  `Use partial runbook <#use-partial-runbook>`__
   -  `Use extensions <#use-extensions>`__

-  `Reference <#reference>`__

   -  `name <#name>`__
   -  `test_project <#test-project>`__
   -  `test_pass <#test-pass>`__
   -  `tags <#tags>`__
   -  `concurrency <#concurrency>`__
   -  `include <#include>`__

      -  `path <#path>`__

   -  `extension <#extension>`__

      -  `name <#name-1>`__
      -  `path <#path-1>`__

   -  `variable <#variable>`__

      -  `is_case_visible <#is-case-visible>`__
      -  `is_secret <#is-secret>`__
      -  `file <#file>`__
      -  `name <#name-2>`__
      -  `value <#value>`__

   -  `transformer <#transformer>`__

      -  `type <#type>`__
      -  `name <#name-3>`__
      -  `prefix <#prefix>`__
      -  `depends_on <#depends-on>`__
      -  `rename <#rename>`__

   -  `combinator <#combinator>`__

      -  `grid combinator <#grid-combinator>`__

         -  `items <#items>`__

      -  `batch combinator <#batch-combinator>`__

         -  `items <#items-1>`__
      -   `bisect combinator <#bisect-combinator>`__

   -  `notifier <#notifier>`__

      -  `console <#console>`__

         -  `log_level <#log-level>`__

      -  `html <#html>`__

         -  `path <#path-2>`__
         -  `auto_open <#auto-open>`__

   -  `environment <#environment>`__

      -  `retry <#retry>`__

      -  `environments <#environments>`__

         -  `name <#name-4>`__
         -  `topology <#topology>`__
         -  `nodes <#nodes>`__
         -  `nodes_requirement <#nodes-requirement>`__

            -  `type <#type-1>`__

   -  `platform <#platform>`__
   -  `testcase <#testcase>`__

      -  `criteria <#criteria>`__

What is a runbook
-----------------

In simple terms:
   `The **runbook** contains all the configurations of LISA operation. It keeps
   you from lengthy command-line commands and makes it easy to adjust
   configurations.`

See :ref:`write_test/concepts:runbook` for further knowledge.

How-to
------

Configure Azure deployment
~~~~~~~~~~~~~~~~~~~~~~~~~~

Below section is for running cases on Azure platform, it specifies:

-  admin_private_key_file: the private key file to access the Azure VM.
-  subscription_id: Azure VM is created under this subscription.

.. code:: yaml

   platform:
     - type: azure
       admin_private_key_file: $(admin_private_key_file)
       azure:
         subscription_id: $(subscription_id)

Select and set test cases
~~~~~~~~~~~~~~~~~~~~~~~~~

Below section is to specify P0 and P1 test cases excluding case with
name ``hello``.

.. code:: yaml

   testcase:
     - criteria:
         priority: [0, 1]
     - criteria:
         name: hello
       select_action: exclude

Use variable and secrets
~~~~~~~~~~~~~~~~~~~~~~~~

Below section is to specify the variable in name/value format. We can
use this variable in other field in this format ``$(location)``.

.. code:: yaml

   variable:
     - name: location
       value: westus3

The value of variable passed from command line will override the value
in runbook yaml file.

.. code:: bash

   lisa -r sample.yml -v "location:westus3"

Below section is to specify the path of yaml file which stores the
secret values.

.. code:: yaml

   variable:
     - file: secret.yml

Content of secret.yml.

.. code:: yaml

   subscription_id:
     value: replace_your_subscription_id_here
     is_secret: true
     mask: guid

Use partial runbook
~~~~~~~~~~~~~~~~~~~

Below three yaml files will be loaded in this sequence.

.. code:: bash

   loading runbook sample.yml
   |-- loading include tier.yml
   |   |-- loading include t0.yml

The variable values in the included yaml file(s) will be overridden by
the including yaml file(s). The relative path is always relative to
the including yaml file.

Part of sample.yml

.. code:: yaml

   include:
     - path: ./tier.yml

Part of tier.yml.

.. code:: yaml

   include:
     - path: ./t$(tier).yml
   variable:
     - name: tier
       value: 0

Part of t0.yml.

.. code:: yaml

   testcase:
     - criteria:
         priority: 0

Use extensions
~~~~~~~~~~~~~~

Below section is to specify path of extensions, the extensions are
modules for test cases or extended features.

.. code:: yaml

   extension:
     - name: extended_features
       path: ../../extensions
     - ../../lisa/microsoft/testsuites/core

Use transformers
~~~~~~~~~~~~~~~~

Transformers are executed one by one. The order is decided by their
dependencies. If there is no dependencies, their order in runbook affects the
execution order.

Below transformer shows how to deploy a VM in Azure, and export it to a VHD.
Before the exporting, other transformers can be added, like install kernel.

.. code:: yaml

   transformer:
   - type: azure_deploy
     requirement:
       azure:
         marketplace: redhat rhel 7_9 7.9.2021051701
   - type: azure_vhd
     resource_group_name: $(azure_deploy_resource_group_name)
     rename:
       azure_vhd_url: vhd
   - type: azure_delete
     resource_group_name: $(azure_deploy_resource_group_name)

Below is the transformer to build kernel from source code and patches.

.. code:: yaml

   transformer:
   - type: azure_deploy
     requirement:
       azure:
         marketplace: $(marketplace_image)
       core_count: 16
     enabled: true
   - type: kernel_installer
     connection:
       address: $(azure_deploy_address)
       private_key_file: $(admin_private_key_file)
     installer:
       type: source
       location:
         type: repo
         path: /mnt/code
         ref: tags/v4.9.184
       modifier:
         - type: patch
           repo: https://github.com/microsoft/azure-linux-kernel.git
           file_pattern: Patches_Following_Mainline_History/4.9.184/*.patch

Reference
---------

name
~~~~

type: str, optional, default is “not_named”

Part of the test run name. This name will be used to group results and
put it in title of the html report, also the created resources' name
contains this specified str.

.. code:: yaml

   name: Azure Default

test_project
~~~~~~~~~~~~

type: str, optional, default is empty

The project name of this test run. This name will be used to group test
results in html, it also shows up in notifier message.

.. code:: yaml

   test_project: Azure Image Weekly Testing

test_pass
~~~~~~~~~

type: str, optional, default is empty

The test pass name of this test run. This name combined with test
project name will be used to group test results in html report, it also
shows up in notifier message.

.. code:: yaml

   test_pass: bvt testing

tags
~~~~

type: list of str, optional, default is empty

The tags of the test run. This name combined with test project name and
test pass name will be used to group test results in html report, it
also shows up in notifier message.

.. code:: yaml

   tags:
     - test
     - bvt

concurrency
~~~~~~~~~~~

type: int, optional, default is 1.

The number of concurrent running environments.

include
~~~~~~~

type: list of path, optional, default is empty

Share runbook parts for similar runs, including the shared content via
that yaml primitive.

path
^^^^

It can be absolute or relative path of current runbook.

extension
~~~~~~~~~

type: list of path str or name/path pairs, optional, default: empty

The path and the name of the modules, we can also just specify the
extension path directly.

.. code:: yaml

   extension:
     - name: ms
       path: ../../extensions

.. _name-1:

name
^^^^

type: str, optional, default is empty

Each extension can be specified a name. With the name, one extension can
reference another one, using above example extension, in code we can
reference it like this way ms.submodule.

.. _path-1:

path
^^^^

type: str, optional, default is empty

Path of extension, it can be absolute or relative path of current
runbook file.

variable
~~~~~~~~

type: list of path str or name/value pairs, optional, default: empty

Used to support variables in other fields.

The values pass from command line has the highest priority, with below
example, any places use ``${subscription_id}`` will be replaced with
value ``subscription id B``.

.. code:: bash

   lisa -r ./microsoft/runbook/azure.yml -v "subscription_id:<subscription id A>"

.. code:: yaml

   variable:
     - name: subscription_id
       value: subscription id B

The variable values in the runbook have higher priority than the same variables
defined in any included runbook file. Thus, ``${location}`` will be replaced with
value ``northeurope`` in the following example.

.. code:: yaml

   include:
     - path: tier.yml
   variable:
     - name: location
       value: northeurope

tier.yml

.. code:: yaml

   variable:
     - name: location
       value: westus3

The later defined variables values in runbook have higher priority than
the same variables previous defined. ``${location}`` will be replaced
with value ``northeurope``.

.. code:: yaml

   variable:
     - name: location
       value: westus3
     - name: location
       value: northeurope

is_case_visible
^^^^^^^^^^^^^^^

type: bool, optional, default is False.

When set to True, the value of this variable will be passed to the testcases,
such as ``perf_nested_kvm_storage_singledisk`` which requires information
about nested image.

is_secret
^^^^^^^^^

type: bool, optional, default is False.

When set to True, the value of this variable will be masked in log and
other output information.

Recommend to use secret file or env variable. It's not recommended to
specify secret value in runbook directly.

file
^^^^

type: list of str, optional, default: empty

Specify path of other yml files which define variables.

.. _name-2:

name
^^^^

type: str, optional, default is empty.

Variable name.

value
^^^^^

type: str, optional, default is empty

Value of the paired variable.

transformer
~~~~~~~~~~~

type: list of Transformer, default is empty

type
^^^^

type: str, required, the type of transformer. See `transformers
<https://github.com/microsoft/lisa/tree/main/lisa/transformers>`__ for all
transformers.

See :doc:`documentation for transformers<transformers>`.

.. _name-3:

name
^^^^

type: str, optional, default is the ``type``.

Unique name of the transformer. It's depended by other transformers. If
it's not specified, it will use the ``type`` field. But if there are two
transformers with the same type, one of them should have name at least.

prefix
^^^^^^

type: str, optional, default is the ``name``.

The prefix of generated variables from this transformer. If it's not
specified, it will use the ``name`` field.

depends_on
^^^^^^^^^^

type: list of str, optional, default is None.

The depended transformers. The depended transformers will run before
this one.

rename
^^^^^^

type: Dict[str, str], optional, default is None.

The variables, which need to be renamed. If the variable exists already,
its value will be overwritten by the transformer. For example,
``["to_list_image", "image"]`` means change the variable name
``to_list_image`` to ``image``. The original variable name must exist in
the output variables of the transformer.

.. _combinator:

combinator
~~~~~~~~~~

type: str, required.

The type of combinator, for example, ``grid`` or ``batch``.

grid combinator
^^^^^^^^^^^^^^^

items
'''''

type: List[Variable], required.

The variables which are in the matrix. Each variable must be a list.

For example,

.. code:: yaml

   - type: grid
     items:
     - name: image
       value:
         - Ubuntu
         - CentOs
     - name: vm_size
       value:
         - Standard_DS2_v2
         - Standard_DS3_v2
         - Standard_DS4_v2

batch combinator
^^^^^^^^^^^^^^^^

.. _items-1:

items
'''''

type: List[Dict[str, Any]], required.

Specify batches of variables. Each batch will run once.

For example,

.. code:: yaml

   - type: batch
     items:
     - image: Ubuntu
       vm_size: Standard_DS2_v2
     - image: Ubuntu
       vm_size: Standard_DS3_v2
     - image: CentOS
       vm_size: Standard_DS3_v2


bisect combinator
^^^^^^^^^^^^^^^^^

Specify a git repo url, the good commit and bad commit. The combinator
performs bisect operations on VM specified under 'connection'.

The runbook will be iterated until the bisect operations completes.

For example,

.. code:: yaml

  combinator:
    type: git_bisect
    repo: $(repo_url)
    bad_commit: $(bad_commit)
    good_commit: $(good_commit)
    connection:
      address: $(bisect_vm_address)
      private_key_file: $(admin_private_key_file)

Refer `Sample runbook <https://github.com/microsoft/lisa/blob/main/examples/runbook/git_bisect.yml>`__

notifier
~~~~~~~~

Receive messages during the test run and output them somewhere.

console
^^^^^^^

One of notifier type. It outputs messages to the console and file log
and demonstrates how to implement notification procedures.

Example of console notifier:

.. code:: yaml

   notifier:
     - type: console
       log_level: INFO

log_level
'''''''''

type: str, optional, default: DEBUG, values: DEBUG, INFO, WARNING…

Set log level of notification messages.

html
^^^^

Output test results in html format. It can be used for local development
or as the body of an email.

.. _path-2:

path
''''

type: str, optional, default: lisa.html

Specify the output file name and path.

auto_open
'''''''''

type: bool, optional, default: False

When set to True, the html will be opened in the browser after
completion. Useful in local run.

Example of html notifier:

.. code:: yaml

   notifier:
     - type: html
       path: ./lisa.html
       auto_open: true

environment
~~~~~~~~~~~

List of environments. For more information, refer to
:ref:`write_test/concepts:node and environment`.

retry
^^^^^^^^^^^^

Number of retry attempts for failed deployments, default value is 0.

environments
^^^^^^^^^^^^

List of test run environment.

.. _name-4:

name
''''

type: str, optional, default is empty

The name of the environment.

topology
''''''''

type: str, optional, default is “subnet”

The topology of the environment, current only support value “subnet”.

nodes
'''''

List of node, it can be a virtual machine on Azure or Hyper-V, bare metal or
others. For more information, refer to :ref:`write_test/concepts:node and
environment`.

nodes_requirement
'''''''''''''''''

List of testing required environments, by default node_count (default is
1), core_count (default is 1), memory_mb (default is 512 MB), data_disk_count
(default is 0), nic_count (default is 1), gpu_count (default is 0). The
node can be created once the node requirement is met.

.. _type-1:

type
    

type: str, optional, default value is “requirement”, supported values
are “requirement”, “remote”, “local”.

platform
~~~~~~~~

List of platform, default value is “ready”, current support values are
“ready”, “azure”.

testcase
~~~~~~~~

type: list of str, optional, default: lisa

Criteria to select cases.

criteria
^^^^^^^^

type: list of dictionary, optional, default is empty

Select test cases by area, category, name, priority or tags combined
with select action.

select_action can be “none”, “include”, “exclude”, “forceInclude” and
“forceExclude”, default value is “none”.

.. code:: yaml

   testcase:
     - criteria:
         priority: 0
       select_action: include
     - criteria:
         priority: 1
       select_action: exclude