incubator-airflow/LOCAL_VIRTUALENV.rst

 .. Licensed to the Apache Software Foundation (ASF) under one
    or more contributor license agreements.  See the NOTICE file
    distributed with this work for additional information
    regarding copyright ownership.  The ASF licenses this file
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at

 ..   http://www.apache.org/licenses/LICENSE-2.0

 .. Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    KIND, either express or implied.  See the License for the
    specific language governing permissions and limitations
    under the License.

.. contents:: :local:


Local virtualenv environment
============================

Installation
------------

Install Python (3.5 or 3.6), MySQL, and libxml by using system-level
package managers like yum, apt-get for Linux, or Homebrew for Mac OS at
first. Refer to the `Dockerfile <Dockerfile>`__ for a comprehensive list
of required packages.

In order to use your IDE you need you can use the virtual environment.
Ideally you should setup virtualenv for all python versions that Airflow
supports (3.5, 3.6). An easy way to create the virtualenv is to use
`virtualenvwrapper <https://virtualenvwrapper.readthedocs.io/en/latest/>`__
- it allows you to easily switch between virtualenvs using ``workon``
command and mange your virtual environments more easily. Typically
creating the environment can be done by:

.. code:: bash

    mkvirtualenv <ENV_NAME> --python=python<VERSION>

Then you need to install python PIP requirements. Typically it can be
done with: ``pip install -e ".[devel]"``.

After creating the virtualenv, run this command to create the Airflow
sqlite database:

.. code:: bash

    airflow db init


Creating virtualenv can be automated with `Breeze environment <BREEZE.rst#configuring-local-virtualenv>`_

Once initialization is done, you should select the virtualenv you
initialized as the project's default virtualenv in your IDE.

After setting it up - you can use the usual "Run Test" option of the IDE
and have the autocomplete and documentation support from IDE as well as
you can debug and view the sources of Airflow - which is very helpful
during development.

Installing other extras
-----------------------

You can also other extras (like ``[mysql]``, ``[gcp]`` etc. via
``pip install -e [EXTRA1,EXTRA2 ...]``. However some of the extras have additional
system requirements and you might need to install additional packages on your
local machine.

For example if you have trouble installing mysql client on MacOS and you have
an error similar to

.. code:: text

    ld: library not found for -lssl

you should set LIBRARY\_PATH before running ``pip install``:

.. code:: bash

    export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/opt/openssl/lib/

The full list of extras is available in `<setup.py>`_


Running individual tests
------------------------

Once you activate virtualenv (or enter docker container) as described
below you should be able to run ``run-tests`` at will (it is in the path
in Docker environment but you need to prepend it with ``./`` in local
virtualenv (``./run-tests``).

Note that this script has several flags that can be useful for your
testing.

.. code:: text

    Usage: run-tests [FLAGS] [TESTS_TO_RUN] -- <EXTRA_NOSETEST_ARGS>

    Runs tests specified (or all tests if no tests are specified)

    Flags:

    -h, --help
            Shows this help message.

    -i, --with-db-init
            Forces database initialization before tests

    -s, --nocapture
            Don't capture stdout when running the tests. This is useful if you are
            debugging with ipdb and want to drop into console with it
            by adding this line to source code:

                import ipdb; ipdb.set_trace()

    -v, --verbose
            Verbose output showing coloured output of tests being run and summary
            of the tests - in a manner similar to the tests run in the CI environment.

You can pass extra parameters to nose, by adding nose arguments after
``--``

For example, in order to just execute the "core" unit tests and add ipdb
set\_trace method, you can run the following command:

.. code:: bash

    ./run-tests tests.core:TestCore --nocapture --verbose

or a single test method without colors or debug logs:

.. code:: bash

    ./run-tests tests.core:TestCore.test_check_operators

Note that ``./run_tests`` script runs tests but the first time it runs,
it performs database initialisation. If you run further tests without
leaving the environment, the database will not be initialized, but you
can always force database initialization with ``--with-db-init``
(``-i``) switch. The scripts will inform you what you can do when they
are run.

Running tests directly from the IDE
-----------------------------------

Once you configure your tests to use the virtualenv you created. running
tests from IDE is as simple as:

.. figure:: images/run_unittests.png
   :alt: Run unittests


Running integration tests
-------------------------

Note that while most of the tests are typical "unit" tests that do not
require external components, there are a number of tests that are more
of "integration" or even "system" tests. You can technically use local
virtualenv to run those tests, but it requires to setup a number of
external components (databases/queues/kubernetes and the like) so it is
much easier to use the `Breeze development environment <BREEZE.rst>`_
for those tests.

Note - soon we will separate the integration and system tests out
so that you can clearly know which tests are unit tests and can be run in
the local virtualenv and which should be run using Breeze.