diff --git a/docs/extras.rst b/docs/extras.rst new file mode 100644 index 000000000..89a9e0ea2 --- /dev/null +++ b/docs/extras.rst @@ -0,0 +1,35 @@ +Extra Dependencies +================== + +What are extras? +---------------- + +Python has a concept of `Extras`, which are groups of dependencies that can be +with a package to provide additional functionality. + +To `install extras`_, simply include the name(s) in square brackets after the +package name. + +.. code:: bash + + pip install lisa[azure,libvirt] + +LISA's extras +------------- + +LISA has several supported extra dependency groups. + +aws + Provides dependencies for running LISA on Amazon Web Services + +azure + Provides dependencies for running LISA on Microsoft Azure + +legacy + Provides dependencies for running LISA v2 tests + +libvirt + Provides dependencies for running LISA on libvirt-managed hypervisors + + +.. _install extras: https://packaging.python.org/en/latest/tutorials/installing-packages/#installing-extras \ No newline at end of file diff --git a/docs/install.rst b/docs/install.rst index bb3e4ab03..ed8534ecd 100644 --- a/docs/install.rst +++ b/docs/install.rst @@ -5,7 +5,8 @@ Currently we support installation on Linux & Windows. .. toctree:: :maxdepth: 2 - + Installation on Linux Installation on Windows Updating LISA + extras diff --git a/docs/installation_linux.rst b/docs/installation_linux.rst index ee0d1adfd..c03823f22 100644 --- a/docs/installation_linux.rst +++ b/docs/installation_linux.rst @@ -1,41 +1,79 @@ Install LISA on Linux ===================== -Minimum System Requirement --------------------------- -1. Your favourite Linux distro supporting Python 3.8+ +Minimum System Requirements +--------------------------- + +1. Your favorite Linux distribution supporting Python 3.8 - 3.10 2. Dual core processor 3. 4 GB system memory -We will guide you through the installation of LISA on Linux. The following commands assume Ubuntu 18.04 as the underlying Linux distro. +We will guide you through the installation of LISA on Linux. +The following commands assume Ubuntu is being used. + Install Python on Linux ----------------------- -LISA has been tested to work with `Python 3.8 64-bit -`__. The latest version of Python 3 is -recommended. If you find that LISA is not compatible with higher version -Python, `please file an -issue `__. +LISA has been tested to work with `Python 3.8 - 3.10 64-bit `__. +Python 3.10 is recommended. Support for 3.11+ is under development. +If you find that LISA is not compatible with a supported version, +`please file an issue `__. -Run following commands to install Python 3.8: +To check which version of Python is used on your system, run the following: + +.. code:: bash + + python3 --version + +If you need to install a different Python package, there are likely packaged versions for +your distro. + +Here is an example to install Python 3.10 on Ubuntu 20.04 .. code:: bash sudo apt update sudo apt install software-properties-common -y sudo add-apt-repository ppa:deadsnakes/ppa -y - sudo apt install python3.8 python3.8-dev -y + sudo apt install python3.10 python3.10-dev -y -Install dependencies on Linux ------------------------------ -Run the command below to install the dependencies: +Install system dependencies +--------------------------- + +Run the command below to install the dependencies on Ubuntu: .. code:: bash sudo apt install git gcc libgirepository1.0-dev libcairo2-dev qemu-utils libvirt-dev python3-pip python3-venv -y +If you're using a different distribution or python version, adjust the command as needed + + +Check PATH +---------- + +When installing Python packages via ``pip``, they will be installed as a local user unless invoked +as root. Some Python packages contain entry point scripts which act as user-facing commands +for the Python package. When installed as a user, these scripts are placed in ``$HOME/.local/bin``. + +To ensure you're able to run these commands, make sure ``$HOME/.local/bin`` is at the beginning +of your ``$PATH``. The following command will highlight this section of the ``$PATH`` variable +if it exists. + +.. code:: bash + + echo $PATH | grep --color=always "$HOME/\.local/bin\|$ + +Ideally, this section is at the beginning of your ``$PATH``. If not, you can add the following to +the bottom of your ``~/.profile`` or ``~.bash_profile`` files. + +.. code:: bash + + export PATH="$HOME/.local/bin:$PATH + + Clone code ---------- @@ -44,54 +82,36 @@ Clone code git clone https://github.com/microsoft/lisa.git cd lisa -Install Poetry on Linux ------------------------ -Poetry is used to manage Python dependencies of LISA. +Directly install LISA (Option 1) +-------------------------------- -.. warning:: - - Please enter the root folder of LISA source code to run - following commands to install poetry, since Poetry manages dependencies - by the working folder. +This will install LISA directly for the invoking user. +To install system-wide, preface the command with ``sudo``. .. code:: bash - curl -sSL https://install.python-poetry.org | python3 - + python3 -m pip install .[azure, libvirt] -After running this, you should see -``Add export PATH="/home/YOURUSERNAME/.local/bin:$PATH" to your shell configuration file`` -message on the console. Follow the message and add the necessary exports to ``$HOME/.profile`` file. -.. code:: bash - source $HOME/.profile +Install LISA in a virtual environment (Option 2) +------------------------------------------------ -[Optional] Create poetry virtual environment in the same folder as LISA for VS Code to automatically -pick up the python environment. Run the following commands to update poetry configuration: +If you wish to keep LISA and it's dependencies separate, you can install it +into a virtual environment. This `guide`_ can be used if you wish to do this manually. +Or, to use a development virtual environment, follow the instructions in :ref:`DevVirtEnv`. -.. code:: bash +.. _guide: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html - poetry config virtualenvs.in-project true -Install python dependencies - -.. code:: bash - - make setup Verify installation ------------------- -``lisa.sh`` is provided in Linux to wrap ``Poetry`` for you to run LISA -test. +Ensure LISA is installed or a virtual environment is activated. -In Linux, you could create an alias for this simple script. For example, -add below line to add to ``.bashrc``: - -.. code:: bash - - alias lisa="./lisa.sh" +Run LISA with the ``lisa`` command With no argument specified, LISA will run some sample test cases with the default runbook (``examples/runbook/hello_world.yml``) on your local diff --git a/docs/installation_windows.rst b/docs/installation_windows.rst index 2848bed13..2b8acf259 100644 --- a/docs/installation_windows.rst +++ b/docs/installation_windows.rst @@ -5,28 +5,45 @@ We will guide you through the installation of LISA on Windows. .. note:: - On Windows, after you finished an installation, or made an + On Windows, after you finished an installation, or make an environment variable change, you might need to restart your shell before moving - to next step, to make sure your changes take effect. + to next step to make sure your changes take effect. .. note:: Please run your command prompt or shell with elevated privilege (such as ``'Run as Administrator'`` on Windows) when you see access denied - message when install tools. + message when installing tools. + Install Python on Windows ------------------------- -LISA has been tested on `Python 3.8 64 -bits `__. The latest version of Python 3 is -recommended. If you found LISA is not compatible with higher version -Python, `please file an -issue `__. +LISA has been tested to work with `Python 3.8 64-bit `__ and above. +The latest version of Python 3 is recommended. If you find that LISA is not compatible +with higher version Python, `please file an issue `__. -Navigate to `Python releases for -Windows `__. Download and -install *Windows installer (64-bit)* for Python 3.8 64-bits or higher -version. +Install from Microsoft Store (recommended) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +It is recommended to install Python from the Microsoft Store. Packages are regularly +published by the Python Software Foundation and will set up paths as needed. + +To install from the Microsoft Store, search for Python in the store interface or, +if no other Python version is installed, running `python3` from the command line +will bring up the latest version. +More details can be found `here`. + +Install using full installer (alternative) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The full installer allows greater customization and doesn't have the security restriction +of the Microsoft Store packages, so may be preferred in some situations. + +Navigate to `Python releases for Windows `__. +Download and install *Windows installer (64-bit)* for Python 3.8 - 3.10 64-bit. + +More information on the full installer, including installation without a GUI, +can be found `here `_. .. warning:: @@ -48,15 +65,13 @@ please manually add these two paths. If this is your first time installing Python, simply check “add Python to PATH” option in installation. -Install dependencies on Windows -------------------------------- -Please install ``git`` on your computer to clone LISA source code from -this repo and ``pip`` for later installation of Poetry. +Install system dependencies on Windows +-------------------------------------- In Windows, you need to install `git `__, -``virtualenv``\ (by running ``pip install virtualenv``) and `Visual C++ -redistributive package `__ +and `Visual C++ redistributive package `__ + Clone code ---------- @@ -66,38 +81,34 @@ Clone code git clone https://github.com/microsoft/lisa.git cd lisa -Install Poetry on Windows -------------------------- -Poetry is used to manage Python dependencies of LISA. +Directly install LISA (Option 1) +-------------------------------- -Enter the ``PowerShell`` command prompt and then execute below commands: +This will install LISA directly for the invoking user. +To install system-wide, run from and Administrator console. -.. code:: powershell +.. code:: bash - (Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py -UseBasicParsing).Content | python - + pip3 install .[azure] - # Add poetry.exe's path to your `PATH` environment variable. - $env:PATH += ";$env:APPDATA\Python\Scripts" -[Optional] Create poetry virtual environment in the same folder as LISA for VS Code to automatically -pick up the python environment. Run the following commands to update poetry configuration: -.. code:: powershell +Install LISA in a virtual environment (Option 2) +------------------------------------------------ - poetry config virtualenvs.in-project true +If you wish to keep LISA and it's dependencies separate, you can install it +into a virtual environment. This `guide`_ can be used if you wish to do this manually. +Or, to use a development virtual environment, follow the instructions in :ref:`DevVirtEnv`. -Install python dependencies - -.. code:: powershell - - poetry install -E "azure libvirt" +.. _guide: https://sublime-and-sphinx-guide.readthedocs.io/en/latest/references.html Verify installation ------------------- -``lisa.cmd`` is provided in Windows to wrap ``Poetry`` for you to run -LISA test. +Ensure LISA is installed or a virtual environment is activated. + +Run LISA with the ``lisa`` command With no argument specified, LISA will run some sample test cases with the default runbook (``examples/runbook/hello_world.yml``) on your local @@ -107,7 +118,7 @@ computer. .. code:: bash - .\lisa + lisa FAQ and Troubleshooting ----------------------- diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst index 183466bb3..d90565758 100644 --- a/docs/troubleshooting.rst +++ b/docs/troubleshooting.rst @@ -1,55 +1,14 @@ Troubleshooting =============== -- `Installation <#installation>`__ +.. contents:: + :local: + :depth: 2 - - `How to use LISA in WSL <#how-to-use-lisa-in-wsl>`__ - - `Many missing packages <#many-missing-packages>`__ - - `Error: Poetry could not find a pyproject.toml - file <#error-poetry-could-not-find-a-pyproject-toml-file>`__ - - `Error: Poetry \"The virtual environment seems to be broken\" - <#error-poetry-the-virtual-environment-seems-to-be-broken>`__ - -- `Using VSCode <#using-vscode>`__ - - - `Cannot find Python Interpreter by Poetry <#cannot-find-python-interpreter-by-poetry>`__ - - `VSCode Python extension no longer supports “python.pythonPath” in “setting.json” <#vscode-python-extension-no-longer-supports-python-pythonpath-in-setting-json>`__ - -- `Other issues <#other-issues>`__ - - - `Poetry related questions <#poetry-related-questions>`__ Installation ------------ -How to use LISA in WSL -~~~~~~~~~~~~~~~~~~~~~~ - -If you are using WSL, installing Poetry on both Windows and WSL may -cause both platforms’ versions of Poetry to be on your path, as Windows -binaries are mapped into ``PATH`` of WSL. This means that the WSL -``poetry`` binary *must* appear in your ``PATH`` before the Windows -version, otherwise this error will appear: - -``/usr/bin/env: ‘python\r’: No such file or directory`` - -Many missing packages -~~~~~~~~~~~~~~~~~~~~~ - -Poetry is case sensitive, which means it differentiates directories like -``C:\abc`` and ``C:\ABC`` in Windows, although Windows in fact does not allow -this (as a case insensitive system). When reading the path, please make sure -there’s no case mismatch in the path. Then run ``poetry install`` again at the -root folder (where your LISA source code is) to make sure all packages are -correctly installed. - -Error: Poetry could not find a pyproject.toml file -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Poetry provides different packages according to the folder, and depends -on the ``pyproject.toml`` file in the current folder. Make sure to run -``poetry`` in the root folder of LISA. - Error: (For Windows installation) Cannot open Scripts folder ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -58,32 +17,15 @@ The Shell interface tries to locate the Scripts folder inside python.exe. Error line: Cannot open \\Microsoft\\WindowsApps\ \PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\\python.exe\\Scripts' -Reason: This could happen when you install the Python version via Microsoft -store. +Reason: This can happen when you install the Python version via Microsoft +store or use a user installation. Fix: uninstall the Microsoft Store version and install the standalone version from https://www.python.org/downloads/windows/ -Error: Poetry "The virtual environment seems to be broken" -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Ensure that ``python3 --version`` returns python3.8 before trying to install poetry. If the command points to an older version of python3, you must uninstall then reinstall poetry after ensuring that virtualenv is installed with pip3 using python3.8. - Using VSCode ------------ -Cannot find Python Interpreter by Poetry -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In the root folder of LISA, run the command below. It will return the -path of the virtual environment that Poetry set up. Use that path to -find the Python interpreter accordingly (in most cases open the path and -look for ``\Scripts\python.exe``). - -.. code:: powershell - - poetry env info -p - VSCode Python extension no longer supports “python.pythonPath” in “setting.json” ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -98,9 +40,8 @@ for more information. can get the path of your selected interpreter in the Python output channel." -An alternative way is to simply select the Poetry Python interpreter as -the default interpreter in the workspace, as in `Cannot find Python -Interpreter by Poetry <#cannot-find-python-interpreter-by-poetry>`__ +An alternative way is to simply select the virtual environment ``.venv`` as +the default interpreter in the workspace. Other issues ------------ @@ -108,13 +49,3 @@ Other issues Please check `known issues `__ or `file a new issue `__ if it doesn’t exist. - -Poetry related questions -~~~~~~~~~~~~~~~~~~~~~~~~ - -Poetry is very useful to manage dependencies of Python. It’s a virtual -environment, not a complete interpreter like Conda. So make sure the -right and effective version of Python interpreter is installed. You can -learn more about Poetry in the official documentation like -`installation `__ or -`commands `__. diff --git a/docs/update.rst b/docs/update.rst index d7f9d9253..a0f6bedea 100644 --- a/docs/update.rst +++ b/docs/update.rst @@ -1,20 +1,27 @@ Updating LISA ============= -Once LISA has been successfully installed on your computer, simply follow below -steps to update. +Once LISA has been successfully installed on your computer, +the steps below can be used to keep it updated. + +To update the the local repo, ensure you are in the main branch and run -In the root folder of LISA ``(...\lisa\)``, run .. code:: bash git pull -to keep your local source code in sync with the latest code in repo, and then -run +If you installed LISA, reinstall to get the latest version. .. code:: bash - poetry install -E "azure libvirt" + python3 -m pip install .[azure, libvirt] + + +If you're using a virtual environment, recreate the virtual environment. + +.. code:: bash + + nox -vs dev + -to keep all packages up-to-date as well. diff --git a/docs/write_test/dev_setup.rst b/docs/write_test/dev_setup.rst index 336e7527f..abbd76ed4 100644 --- a/docs/write_test/dev_setup.rst +++ b/docs/write_test/dev_setup.rst @@ -4,15 +4,10 @@ Development setup This document describes the existing developer tooling we have in place (and what to expect of it). -- `Environment Setup <#environment-setup>`__ +.. contents:: + :local: + :depth: 2 - - `Visual Studio Code <#visual-studio-code>`__ - - `Emacs <#emacs>`__ - - `Other setups <#other-setups>`__ - -- `Code checks <#code-checks>`__ -- `Local Documentation <#local-documentation>`__ -- `Extended reading <#extended-reading>`__ Environment Setup ----------------- @@ -21,6 +16,77 @@ Follow the :ref:`quick_start:installation` steps to prepare the source code. Then follow the steps below to set up the corresponding development environment. +.. _DevVirtEnv: + +Creating a LISA development virtual environment +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Nox is used to manage virtual environments for LISA. +See `Using Nox`_ for more information. + +Nox can be installed with `pip`. + +.. code:: bash + + pip3 install nox toml + + +The following creates a virtual environment in ``.venv`` with an editable install +of LISA. An editable install allows code changes in the repo to immediately affect +the virtual environment. This is useful for iterative development. + +From the root of the LISA repo, run the following to create or update virtual environment. + +.. code:: bash + + nox -vs dev + +By default, the extra dependencies for Azure will be installed. +On Linux, the extra dependencies for libvirt will also be installed. +This behavior can be overridden by passing dependency groups as additional arguments. + +For example, the following will only install extra dependencies for Azure. + +.. code:: bash + + nox -vs dev -- azure + +Information on what extra dependency groups are supported can be found in +:ref:`extras:LISA's extras`. + + +VSCode should automatically detect and activate the virtual environment on startup +and you can manually activate it in a shell with the following commands. + + +Activate virtual environment (Bash) + +.. code:: bash + + source .venv/bin/activate + +Activate virtual environment (Powershell) + +.. code:: powershell + + .venv\Scripts\activate.ps1 + +Activate virtual environment (cmd) + +.. code:: batch + + .venv\Scripts\activate.bat + + +When the virtual environment is active, your command prompt will be prefixed with `(lisa)`. + +If you wish to deactivate the virtual environment, use the ``deactivate`` command. + +.. code:: bash + + deactivate + + Visual Studio Code ~~~~~~~~~~~~~~~~~~ @@ -117,7 +183,7 @@ a ``.dir-locals.el`` file as follows: ;;; Directory Local Variables ;;; For more information see (info "(emacs) Directory Variables") - ((python-mode . ((pyvenv-activate . "~/.cache/pypoetry/virtualenvs/lisa-s7Q404Ij-py3.8")))) + ((python-mode . ((pyvenv-activate . ".venv")))) Other setups ~~~~~~~~~~~~ @@ -150,24 +216,167 @@ manually. - `rope `__, provides completion and renaming support for pyls. +Using Nox +--------- + +Nox is test automation utility that allows running tests and utilities in +virtual environments. This allows isolation and consistency for these actions. + +Sessions +~~~~~~~~ + +Nox tasks are called sessions. A number of Nox sessions have been configured +for LISA. They can be displayed by running ``nox --list``. + +.. code:: console + + $ nox --list + Nox configuration file + See https://nox.thea.codes/en/stable/config.html + + Sessions defined in /srv/Development/lisa/noxfile.py: + + * test -> Run tests + * example -> Run example + * coverage -> Check test coverage + * black -> Run black + * isort -> Run isort + * flake8 -> Run flake8 + * mypy -> Run mypy + * docs -> Build docs + * dev -> Create virtual environment for development + + sessions marked with * are selected, sessions marked with - are skipped. + +An individual session can be run with ``nox -vs ``. + +.. code:: console + + $ nox -vs flake8 + nox > Running session flake8 + nox > Creating virtual environment (virtualenv) using python3 in .nox/flake8 + ... + nox > flake8 + nox > Session flake8 was successful. + +Tags +~~~~ + +Another way to call Nox sessions is with tags. Tags can not currently be +listed on the command line, but the following have been define: + +all + Runs various checks and tests to do before pushing a commit + +format + Run formatting tools such as isort and black + +linting + Run linting tools such as flake8 + +test + Run unit tests and test scenarios + +typing + Run typing tools such as mypy + + +To execute all sessions with a given tag, use the ``-t`` option. + +.. code:: console + + $ nox -vt format + nox > Running session black + ... + nox > Running session isort + ... + nox > Ran multiple sessions: + nox > * black: success + nox > * isort: success + + +To determine which sessions will be called for a tag without running them, +use the ``--list`` option. + +.. code:: console + + $ nox -t format --list + Nox configuration file + See https://nox.thea.codes/en/stable/config.html + + Sessions defined in /srv/Development/lisa/noxfile.py: + + - test -> Run tests + - example -> Run example + - coverage -> Check test coverage + * black -> Run black + * isort -> Run isort + - flake8 -> Run flake8 + - mypy -> Run mypy + - docs -> Build docs + - dev -> Create virtual environment for development + +Running with a different Python interpreter +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Nox sessions for LISA are configured to run with the same Python interpreter +used to run Nox. To use a different interpreter, use the `--force-python` option. +You can either specify a Python version or the path to an executable. + +.. code:: console + + $ nox -vs test --force-python 3.11 + +.. code:: console + + $ nox -vs test --force-python /usr/bin/python3.11 + +Speeding up Nox +~~~~~~~~~~~~~~~ + +By default, Nox will recreate a virtual environment every time it runs. +This ensures there are no stale dependencies, but is not always necessary. +To reuse a virtual environment, use the ``-r`` option. To reuse the virtual +environment without reinstalling any dependencies, use the ``-R`` option. +This will have a greater impact for sessions with a large number of +dependencies. + + +.. code:: console + + $ time nox -vs flake8 + ... + real 0m9.827s + + $ time nox -vrs flake8 + ... + real 0m6.433s + + $ time nox -vRs flake8 + ... + real 0m5.638s + + + +Additional information +~~~~~~~~~~~~~~~~~~~~~~ + +More information on Nox can be found `here `_. + Local Documentation ------------------- It's recommended to build the documentation locally using ``Sphinx`` for preview. -To do so, in ``./lisa/docs``, run +To do so, run .. code:: bash - poetry run make html + nox -vs docs You can find all generated documents in ``./lisa/docs/_build/html`` folder. Open them with a browser to view. -.. note:: - If there are already generated documents in ``./lisa/docs/_build/html``, run - ``poetry run make clean`` to ensure the documentation is clean and not - affected by the previous build. Extended reading ----------------