gecko-dev/build/docs/test_manifests.rst

.. _test_manifests:

==============
Test Manifests
==============

Many test suites have their test metadata defined in files called
**test manifests**.

Test manifests are divided into two flavors: :ref:`manifest_destiny_manifests`
and :ref:`reftest_manifests`.

.. _manifest_destiny_manifests:

Manifest Destiny Manifests
==========================

Manifest destiny manifests are essentially ini files that conform to a basic
set of assumptions.

The `reference documentation <http://mozbase.readthedocs.org/en/latest/manifestdestiny.html>`_
for manifest destiny manifests describes the basic format of test manifests.

In summary, manifests are ini files with section names describing test files::

    [test_foo.js]
    [test_bar.js]

Keys under sections can hold metadata about each test::

    [test_foo.js]
    skip-if = os == win

There is a special **DEFAULT** section whose keys/metadata apply to all
sections/tests::

    [DEFAULT]
    property = value

    [test_foo.js]

In the above example, **test_foo.js** inherits the metadata **property = value**
from the **DEFAULT** section.

Recognized Metadata
-------------------

Test manifests can define some common keys/metadata to influence behavior.
Those keys are as follows:

head
   List of files that will be executed before the test file. (Used in
   xpcshell tests.)

tail
   List of files that will be executed after the test file. (Used in
   xpcshell tests.)

support-files
   List of additional files required to run tests. This is typically
   defined in the **DEFAULT** section.

   Unlike other file lists, *support-files* supports a globbing mechanism
   to facilitate pulling in many files with minimal typing. This globbing
   mechanism is activated if an entry in this value contains a ``*``
   character. A single ``*`` will wildcard match all files in a directory.
   A double ``**`` will descend into child directories. For example,
   ``data/*`` will match ``data/foo`` but not ``data/subdir/bar`` where
   ``data/**`` will match ``data/foo`` and ``data/subdir/bar``.

generated-files
   List of files that are generated as part of the build and don't exist in
   the source tree.

   The build system assumes that each manifest file, test file, and file
   listed in **head**, **tail**, and **support-files** is static and
   provided by the source tree (and not automatically generated as part
   of the build). This variable tells the build system not to make this
   assumption.

   This variable will likely go away sometime once all generated files are
   accounted for in the build config.

   If a generated file is not listed in this key, a clobber build will
   likely fail.

dupe-manifest
   Record that this manifest duplicates another manifest.

   The common scenario is two manifest files will include a shared
   manifest file via the ``[include:file]`` special section. The build
   system enforces that each test file is only provided by a single
   manifest. Having this key present bypasses that check.

   The value of this key is ignored.

run-if
   Run this test only if the specified condition is true.
   See :ref:`manifest_filter_language`.

skip-if
   Skip this test if the specified condition is true.
   See :ref:`manifest_filter_language`.

fail-if
   Expect test failure if the specified condition is true.
   See :ref:`manifest_filter_language`.

run-sequentially
   If present, the test should not be run in parallel with other tests.

   Some test harnesses support parallel test execution on separate processes
   and/or threads (behavior varies by test harness). If this key is present,
   the test harness should not attempt to run this test in parallel with any
   other test.

   By convention, the value of this key is a string describing why the test
   can't be run in parallel.

.. _manifest_filter_language:

Manifest Filter Language
------------------------

Some manifest keys accept a special filter syntax as their values. These
values are essentially boolean expressions that are evaluated at test
execution time.

See
`the source <https://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/manifestdestiny/manifestparser/manifestparser.py>`_.

.. todo::

   Document manifest filter language.

.. _manifest_file_installation:

File Installation
-----------------

Files referenced by manifests are automatically installed into the object
directory into paths defined in
:py:func:`mozbuild.frontend.emitter.TreeMetadataEmitter._process_test_manifest`.

Referenced files in the manifest not in the same directory tree as the manifest
file are **not** installed.

.. _reftest_manifests:

Reftest Manifests
=================

See `MDN <https://developer.mozilla.org/en-US/docs/Creating_reftest-based_unit_tests>`_.