Update docs for nox / setuptools changes

This commit is contained in:
Avram Lubkin 2022-11-03 13:35:54 -04:00 коммит произвёл LiliDeng
Родитель ecd95f6815
Коммит ec74d3d60f
7 изменённых файлов: 396 добавлений и 182 удалений

35
docs/extras.rst Normal file
Просмотреть файл

@ -0,0 +1,35 @@
Extra Dependencies
==================
What are extras?
----------------
Python has a concept of `Extras`, which are groups of dependencies that can be
with a package to provide additional functionality.
To `install extras`_, simply include the name(s) in square brackets after the
package name.
.. code:: bash
pip install lisa[azure,libvirt]
LISA's extras
-------------
LISA has several supported extra dependency groups.
aws
Provides dependencies for running LISA on Amazon Web Services
azure
Provides dependencies for running LISA on Microsoft Azure
legacy
Provides dependencies for running LISA v2 tests
libvirt
Provides dependencies for running LISA on libvirt-managed hypervisors
.. _install extras: https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-extras

Просмотреть файл

@ -9,3 +9,4 @@ Currently we support installation on Linux & Windows.
Installation on Linux <installation_linux>
Installation on Windows <installation_windows>
Updating LISA <update>
extras

Просмотреть файл

@ -1,41 +1,79 @@
Install LISA on Linux
=====================
Minimum System Requirement
--------------------------
1. Your favourite Linux distro supporting Python 3.8+
Minimum System Requirements
---------------------------
1. Your favorite Linux distribution supporting Python 3.8 - 3.10
2. Dual core processor
3. 4 GB system memory
We will guide you through the installation of LISA on Linux. The following commands assume Ubuntu 18.04 as the underlying Linux distro.
We will guide you through the installation of LISA on Linux.
The following commands assume Ubuntu is being used.
Install Python on Linux
-----------------------
LISA has been tested to work with `Python 3.8 64-bit
<https://www.python.org/>`__. The latest version of Python 3 is
recommended. If you find that LISA is not compatible with higher version
Python, `please file an
issue <https://github.com/microsoft/lisa/issues/new>`__.
LISA has been tested to work with `Python 3.8 - 3.10 64-bit <https://www.python.org/>`__.
Python 3.10 is recommended. Support for 3.11+ is under development.
If you find that LISA is not compatible with a supported version,
`please file an issue <https://github.com/microsoft/lisa/issues/new>`__.
Run following commands to install Python 3.8:
To check which version of Python is used on your system, run the following:
.. code:: bash
python3 --version
If you need to install a different Python package, there are likely packaged versions for
your distro.
Here is an example to install Python 3.10 on Ubuntu 20.04
.. code:: bash
sudo apt update
sudo apt install software-properties-common -y
sudo add-apt-repository ppa:deadsnakes/ppa -y
sudo apt install python3.8 python3.8-dev -y
sudo apt install python3.10 python3.10-dev -y
Install dependencies on Linux
-----------------------------
Run the command below to install the dependencies:
Install system dependencies
---------------------------
Run the command below to install the dependencies on Ubuntu:
.. code:: bash
sudo apt install git gcc libgirepository1.0-dev libcairo2-dev qemu-utils libvirt-dev python3-pip python3-venv -y
If you're using a different distribution or python version, adjust the command as needed
Check PATH
----------
When installing Python packages via ``pip``, they will be installed as a local user unless invoked
as root. Some Python packages contain entry point scripts which act as user-facing commands
for the Python package. When installed as a user, these scripts are placed in ``$HOME/.local/bin``.
To ensure you're able to run these commands, make sure ``$HOME/.local/bin`` is at the beginning
of your ``$PATH``. The following command will highlight this section of the ``$PATH`` variable
if it exists.
.. code:: bash
echo $PATH | grep --color=always "$HOME/\.local/bin\|$
Ideally, this section is at the beginning of your ``$PATH``. If not, you can add the following to
the bottom of your ``~/.profile`` or ``~.bash_profile`` files.
.. code:: bash
export PATH="$HOME/.local/bin:$PATH
Clone code
----------
@ -44,54 +82,36 @@ Clone code
git clone https://github.com/microsoft/lisa.git
cd lisa
Install Poetry on Linux
-----------------------
Poetry is used to manage Python dependencies of LISA.
Directly install LISA (Option 1)
--------------------------------
.. warning::
Please enter the root folder of LISA source code to run
following commands to install poetry, since Poetry manages dependencies
by the working folder.
This will install LISA directly for the invoking user.
To install system-wide, preface the command with ``sudo``.
.. code:: bash
curl -sSL https://install.python-poetry.org | python3 -
python3 -m pip install .[azure, libvirt]
After running this, you should see
``Add export PATH="/home/YOURUSERNAME/.local/bin:$PATH" to your shell configuration file``
message on the console. Follow the message and add the necessary exports to ``$HOME/.profile`` file.
.. code:: bash
source $HOME/.profile
Install LISA in a virtual environment (Option 2)
------------------------------------------------
[Optional] Create poetry virtual environment in the same folder as LISA for VS Code to automatically
pick up the python environment. Run the following commands to update poetry configuration:
If you wish to keep LISA and it's dependencies separate, you can install it
into a virtual environment. This `guide`_ can be used if you wish to do this manually.
Or, to use a development virtual environment, follow the instructions in :ref:`DevVirtEnv`.
.. code:: bash
.. _guide: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html
poetry config virtualenvs.in-project true
Install python dependencies
.. code:: bash
make setup
Verify installation
-------------------
``lisa.sh`` is provided in Linux to wrap ``Poetry`` for you to run LISA
test.
Ensure LISA is installed or a virtual environment is activated.
In Linux, you could create an alias for this simple script. For example,
add below line to add to ``.bashrc``:
.. code:: bash
alias lisa="./lisa.sh"
Run LISA with the ``lisa`` command
With no argument specified, LISA will run some sample test cases with
the default runbook (``examples/runbook/hello_world.yml``) on your local

Просмотреть файл

@ -5,28 +5,45 @@ We will guide you through the installation of LISA on Windows.
.. note::
On Windows, after you finished an installation, or made an
On Windows, after you finished an installation, or make an
environment variable change, you might need to restart your shell before moving
to next step, to make sure your changes take effect.
to next step to make sure your changes take effect.
.. note::
Please run your command prompt or shell with elevated privilege
(such as ``'Run as Administrator'`` on Windows) when you see access denied
message when install tools.
message when installing tools.
Install Python on Windows
-------------------------
LISA has been tested on `Python 3.8 64
bits <https://www.python.org/>`__. The latest version of Python 3 is
recommended. If you found LISA is not compatible with higher version
Python, `please file an
issue <https://github.com/microsoft/lisa/issues/new>`__.
LISA has been tested to work with `Python 3.8 64-bit <https://www.python.org/>`__ and above.
The latest version of Python 3 is recommended. If you find that LISA is not compatible
with higher version Python, `please file an issue <https://github.com/microsoft/lisa/issues/new>`__.
Navigate to `Python releases for
Windows <https://www.python.org/downloads/windows/>`__. Download and
install *Windows installer (64-bit)* for Python 3.8 64-bits or higher
version.
Install from Microsoft Store (recommended)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It is recommended to install Python from the Microsoft Store. Packages are regularly
published by the Python Software Foundation and will set up paths as needed.
To install from the Microsoft Store, search for Python in the store interface or,
if no other Python version is installed, running `python3` from the command line
will bring up the latest version.
More details can be found `here<https://docs.python.org/3/using/windows.html#windows-store>`.
Install using full installer (alternative)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The full installer allows greater customization and doesn't have the security restriction
of the Microsoft Store packages, so may be preferred in some situations.
Navigate to `Python releases for Windows <https://www.python.org/downloads/windows/>`__.
Download and install *Windows installer (64-bit)* for Python 3.8 - 3.10 64-bit.
More information on the full installer, including installation without a GUI,
can be found `here <https://docs.python.org/3/using/windows.html#the-full-installer>`_.
.. warning::
@ -48,15 +65,13 @@ please manually add these two paths.
If this is your first time installing Python, simply check “add Python
to PATH” option in installation.
Install dependencies on Windows
-------------------------------
Please install ``git`` on your computer to clone LISA source code from
this repo and ``pip`` for later installation of Poetry.
Install system dependencies on Windows
--------------------------------------
In Windows, you need to install `git <https://git-scm.com/downloads>`__,
``virtualenv``\ (by running ``pip install virtualenv``) and `Visual C++
redistributive package <https://aka.ms/vs/16/release/vc_redist.x64.exe>`__
and `Visual C++ redistributive package <https://aka.ms/vs/16/release/vc_redist.x64.exe>`__
Clone code
----------
@ -66,38 +81,34 @@ Clone code
git clone https://github.com/microsoft/lisa.git
cd lisa
Install Poetry on Windows
-------------------------
Poetry is used to manage Python dependencies of LISA.
Directly install LISA (Option 1)
--------------------------------
Enter the ``PowerShell`` command prompt and then execute below commands:
This will install LISA directly for the invoking user.
To install system-wide, run from and Administrator console.
.. code:: powershell
.. code:: bash
(Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -UseBasicParsing).Content | python -
pip3 install .[azure]
# Add poetry.exe's path to your `PATH` environment variable.
$env:PATH += ";$env:APPDATA\Python\Scripts"
[Optional] Create poetry virtual environment in the same folder as LISA for VS Code to automatically
pick up the python environment. Run the following commands to update poetry configuration:
.. code:: powershell
Install LISA in a virtual environment (Option 2)
------------------------------------------------
poetry config virtualenvs.in-project true
If you wish to keep LISA and it's dependencies separate, you can install it
into a virtual environment. This `guide`_ can be used if you wish to do this manually.
Or, to use a development virtual environment, follow the instructions in :ref:`DevVirtEnv`.
Install python dependencies
.. code:: powershell
poetry install -E "azure libvirt"
.. _guide: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html
Verify installation
-------------------
``lisa.cmd`` is provided in Windows to wrap ``Poetry`` for you to run
LISA test.
Ensure LISA is installed or a virtual environment is activated.
Run LISA with the ``lisa`` command
With no argument specified, LISA will run some sample test cases with
the default runbook (``examples/runbook/hello_world.yml``) on your local
@ -107,7 +118,7 @@ computer.
.. code:: bash
.\lisa
lisa
FAQ and Troubleshooting
-----------------------

Просмотреть файл

@ -1,55 +1,14 @@
Troubleshooting
===============
- `Installation <#installation>`__
.. contents::
:local:
:depth: 2
- `How to use LISA in WSL <#how-to-use-lisa-in-wsl>`__
- `Many missing packages <#many-missing-packages>`__
- `Error: Poetry could not find a pyproject.toml
file <#error-poetry-could-not-find-a-pyproject-toml-file>`__
- `Error: Poetry \"The virtual environment seems to be broken\"
<#error-poetry-the-virtual-environment-seems-to-be-broken>`__
- `Using VSCode <#using-vscode>`__
- `Cannot find Python Interpreter by Poetry <#cannot-find-python-interpreter-by-poetry>`__
- `VSCode Python extension no longer supports “python.pythonPath” in “setting.json” <#vscode-python-extension-no-longer-supports-python-pythonpath-in-setting-json>`__
- `Other issues <#other-issues>`__
- `Poetry related questions <#poetry-related-questions>`__
Installation
------------
How to use LISA in WSL
~~~~~~~~~~~~~~~~~~~~~~
If you are using WSL, installing Poetry on both Windows and WSL may
cause both platforms versions of Poetry to be on your path, as Windows
binaries are mapped into ``PATH`` of WSL. This means that the WSL
``poetry`` binary *must* appear in your ``PATH`` before the Windows
version, otherwise this error will appear:
``/usr/bin/env: python\r: No such file or directory``
Many missing packages
~~~~~~~~~~~~~~~~~~~~~
Poetry is case sensitive, which means it differentiates directories like
``C:\abc`` and ``C:\ABC`` in Windows, although Windows in fact does not allow
this (as a case insensitive system). When reading the path, please make sure
theres no case mismatch in the path. Then run ``poetry install`` again at the
root folder (where your LISA source code is) to make sure all packages are
correctly installed.
Error: Poetry could not find a pyproject.toml file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Poetry provides different packages according to the folder, and depends
on the ``pyproject.toml`` file in the current folder. Make sure to run
``poetry`` in the root folder of LISA.
Error: (For Windows installation) Cannot open Scripts folder
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -58,32 +17,15 @@ The Shell interface tries to locate the Scripts folder inside python.exe.
Error line: Cannot open <Your local AppData dir>\\Microsoft\\WindowsApps\
\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\python.exe\\Scripts'
Reason: This could happen when you install the Python version via Microsoft
store.
Reason: This can happen when you install the Python version via Microsoft
store or use a user installation.
Fix: uninstall the Microsoft Store version and install the standalone
version from https://www.python.org/downloads/windows/
Error: Poetry "The virtual environment seems to be broken"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Ensure that ``python3 --version`` returns python3.8 before trying to install poetry. If the command points to an older version of python3, you must uninstall then reinstall poetry after ensuring that virtualenv is installed with pip3 using python3.8.
Using VSCode
------------
Cannot find Python Interpreter by Poetry
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the root folder of LISA, run the command below. It will return the
path of the virtual environment that Poetry set up. Use that path to
find the Python interpreter accordingly (in most cases open the path and
look for ``\Scripts\python.exe``).
.. code:: powershell
poetry env info -p
VSCode Python extension no longer supports “python.pythonPath” in “setting.json”
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@ -98,9 +40,8 @@ for more information.
can get the path of your selected interpreter in the Python output
channel."
An alternative way is to simply select the Poetry Python interpreter as
the default interpreter in the workspace, as in `Cannot find Python
Interpreter by Poetry <#cannot-find-python-interpreter-by-poetry>`__
An alternative way is to simply select the virtual environment ``.venv`` as
the default interpreter in the workspace.
Other issues
------------
@ -108,13 +49,3 @@ Other issues
Please check `known issues <https://github.com/microsoft/lisa/issues>`__
or `file a new issue <https://github.com/microsoft/lisa/issues/new>`__
if it doesnt exist.
Poetry related questions
~~~~~~~~~~~~~~~~~~~~~~~~
Poetry is very useful to manage dependencies of Python. Its a virtual
environment, not a complete interpreter like Conda. So make sure the
right and effective version of Python interpreter is installed. You can
learn more about Poetry in the official documentation like
`installation <https://python-poetry.org/docs/#installation>`__ or
`commands <https://python-poetry.org/docs/cli/>`__.

Просмотреть файл

@ -1,20 +1,27 @@
Updating LISA
=============
Once LISA has been successfully installed on your computer, simply follow below
steps to update.
Once LISA has been successfully installed on your computer,
the steps below can be used to keep it updated.
To update the the local repo, ensure you are in the main branch and run
In the root folder of LISA ``(...\lisa\)``, run
.. code:: bash
git pull
to keep your local source code in sync with the latest code in repo, and then
run
If you installed LISA, reinstall to get the latest version.
.. code:: bash
poetry install -E "azure libvirt"
python3 -m pip install .[azure, libvirt]
If you're using a virtual environment, recreate the virtual environment.
.. code:: bash
nox -vs dev
to keep all packages up-to-date as well.

Просмотреть файл

@ -4,15 +4,10 @@ Development setup
This document describes the existing developer tooling we have in place (and
what to expect of it).
- `Environment Setup <#environment-setup>`__
.. contents::
:local:
:depth: 2
- `Visual Studio Code <#visual-studio-code>`__
- `Emacs <#emacs>`__
- `Other setups <#other-setups>`__
- `Code checks <#code-checks>`__
- `Local Documentation <#local-documentation>`__
- `Extended reading <#extended-reading>`__
Environment Setup
-----------------
@ -21,6 +16,77 @@ Follow the :ref:`quick_start:installation` steps to
prepare the source code. Then follow the steps below to set up the corresponding
development environment.
.. _DevVirtEnv:
Creating a LISA development virtual environment
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Nox is used to manage virtual environments for LISA.
See `Using Nox`_ for more information.
Nox can be installed with `pip`.
.. code:: bash
pip3 install nox toml
The following creates a virtual environment in ``.venv`` with an editable install
of LISA. An editable install allows code changes in the repo to immediately affect
the virtual environment. This is useful for iterative development.
From the root of the LISA repo, run the following to create or update virtual environment.
.. code:: bash
nox -vs dev
By default, the extra dependencies for Azure will be installed.
On Linux, the extra dependencies for libvirt will also be installed.
This behavior can be overridden by passing dependency groups as additional arguments.
For example, the following will only install extra dependencies for Azure.
.. code:: bash
nox -vs dev -- azure
Information on what extra dependency groups are supported can be found in
:ref:`extras:LISA's extras`.
VSCode should automatically detect and activate the virtual environment on startup
and you can manually activate it in a shell with the following commands.
Activate virtual environment (Bash)
.. code:: bash
source .venv/bin/activate
Activate virtual environment (Powershell)
.. code:: powershell
.venv\Scripts\activate.ps1
Activate virtual environment (cmd)
.. code:: batch
.venv\Scripts\activate.bat
When the virtual environment is active, your command prompt will be prefixed with `(lisa)`.
If you wish to deactivate the virtual environment, use the ``deactivate`` command.
.. code:: bash
deactivate
Visual Studio Code
~~~~~~~~~~~~~~~~~~
@ -117,7 +183,7 @@ a ``.dir-locals.el`` file as follows:
;;; Directory Local Variables
;;; For more information see (info "(emacs) Directory Variables")
((python-mode . ((pyvenv-activate . "~/.cache/pypoetry/virtualenvs/lisa-s7Q404Ij-py3.8"))))
((python-mode . ((pyvenv-activate . ".venv"))))
Other setups
~~~~~~~~~~~~
@ -150,24 +216,167 @@ manually.
- `rope <https://github.com/python-rope/rope>`__, provides completion
and renaming support for pyls.
Using Nox
---------
Nox is test automation utility that allows running tests and utilities in
virtual environments. This allows isolation and consistency for these actions.
Sessions
~~~~~~~~
Nox tasks are called sessions. A number of Nox sessions have been configured
for LISA. They can be displayed by running ``nox --list``.
.. code:: console
$ nox --list
Nox configuration file
See https://nox.thea.codes/en/stable/config.html
Sessions defined in /srv/Development/lisa/noxfile.py:
* test -> Run tests
* example -> Run example
* coverage -> Check test coverage
* black -> Run black
* isort -> Run isort
* flake8 -> Run flake8
* mypy -> Run mypy
* docs -> Build docs
* dev -> Create virtual environment for development
sessions marked with * are selected, sessions marked with - are skipped.
An individual session can be run with ``nox -vs <session>``.
.. code:: console
$ nox -vs flake8
nox > Running session flake8
nox > Creating virtual environment (virtualenv) using python3 in .nox/flake8
...
nox > flake8
nox > Session flake8 was successful.
Tags
~~~~
Another way to call Nox sessions is with tags. Tags can not currently be
listed on the command line, but the following have been define:
all
Runs various checks and tests to do before pushing a commit
format
Run formatting tools such as isort and black
linting
Run linting tools such as flake8
test
Run unit tests and test scenarios
typing
Run typing tools such as mypy
To execute all sessions with a given tag, use the ``-t`` option.
.. code:: console
$ nox -vt format
nox > Running session black
...
nox > Running session isort
...
nox > Ran multiple sessions:
nox > * black: success
nox > * isort: success
To determine which sessions will be called for a tag without running them,
use the ``--list`` option.
.. code:: console
$ nox -t format --list
Nox configuration file
See https://nox.thea.codes/en/stable/config.html
Sessions defined in /srv/Development/lisa/noxfile.py:
- test -> Run tests
- example -> Run example
- coverage -> Check test coverage
* black -> Run black
* isort -> Run isort
- flake8 -> Run flake8
- mypy -> Run mypy
- docs -> Build docs
- dev -> Create virtual environment for development
Running with a different Python interpreter
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Nox sessions for LISA are configured to run with the same Python interpreter
used to run Nox. To use a different interpreter, use the `--force-python` option.
You can either specify a Python version or the path to an executable.
.. code:: console
$ nox -vs test --force-python 3.11
.. code:: console
$ nox -vs test --force-python /usr/bin/python3.11
Speeding up Nox
~~~~~~~~~~~~~~~
By default, Nox will recreate a virtual environment every time it runs.
This ensures there are no stale dependencies, but is not always necessary.
To reuse a virtual environment, use the ``-r`` option. To reuse the virtual
environment without reinstalling any dependencies, use the ``-R`` option.
This will have a greater impact for sessions with a large number of
dependencies.
.. code:: console
$ time nox -vs flake8
...
real 0m9.827s
$ time nox -vrs flake8
...
real 0m6.433s
$ time nox -vRs flake8
...
real 0m5.638s
Additional information
~~~~~~~~~~~~~~~~~~~~~~
More information on Nox can be found `here <https://nox.thea.codes>`_.
Local Documentation
-------------------
It's recommended to build the documentation locally using ``Sphinx`` for preview.
To do so, in ``./lisa/docs``, run
To do so, run
.. code:: bash
poetry run make html
nox -vs docs
You can find all generated documents in ``./lisa/docs/_build/html`` folder. Open
them with a browser to view.
.. note::
If there are already generated documents in ``./lisa/docs/_build/html``, run
``poetry run make clean`` to ensure the documentation is clean and not
affected by the previous build.
Extended reading
----------------