Bug 1502306 - Make it easier to run Selenium tests (#4194)

* Enables the display of skipped test/expected fail reasons, in
  the pytest summary.
* Skips the Selenium tests with a clear reason message, unless the
  built UI is found (preventing the annoying/confusing test timeouts).
* Removes the disabling of the `pytest-html` and `pytest-metadata`
  plugins, since they are required when passing the `--html` option
  to generate an HTML report.
* Updates the docs to mention `yarn build` and `--html`.

docs/backend_tasks.md

@@ -10,6 +10,8 @@ You can run flake8, isort and the pytest suite inside the Vagrant VM, using:
 vagrant ~/treeherder$ ./runtests.sh
 ```
 
+Note: The Selenium tests will be skipped unless `yarn build` has been manually run prior.
+
 Or for more control, run each tool individually:
 
 * [pytest](https://docs.pytest.org/en/stable/):
@@ -27,7 +29,14 @@ Or for more control, run each tool individually:
   vagrant ~/treeherder$ pytest --runslow tests/
   ```
 
-  For more options, see `pytest --help` or <https://docs.pytest.org/en/stable/usage.html>
+  For more options, see `pytest --help` or <https://docs.pytest.org/en/stable/usage.html>.
+
+  To assist with debugging Selenium test failures, an HTML reporting containing screenshots
+  can be generated using:
+
+  ```bash
+  vagrant ~/treeherder$ pytest tests/selenium/ --html report.html
+  ```
 
 * [flake8](https://flake8.readthedocs.io/):
 

setup.cfg

@@ -19,8 +19,10 @@ known_third_party = recommonmark
 testpaths = tests
 norecursedirs = __pycache__ ui
 DJANGO_SETTINGS_MODULE=tests.settings
-# Disable unused auto-loaded plugins.
+# Enable display of skipped/expected fail test reasons.
-addopts = --driver Firefox -p no:mozlog -p no:metadata -p no:html
+# Tell Selenium to use the Firefix driver.
+# Disable unused auto-loaded mozlog plugin.
+addopts = -rsx --driver Firefox -p no:mozlog
 # Make most warnings fatal (including the hidden by default DeprecationWarning):
 # https://docs.pytest.org/en/latest/warnings.html
 # https://docs.python.org/2.7/library/warnings.html#warning-categories

tests/selenium/conftest.py

@@ -1,8 +1,15 @@
+import os
+
 import pytest
+from django.conf import settings
 
 
 @pytest.fixture(scope="session")
 def base_url(live_server):
+    if not os.path.exists(settings.WHITENOISE_ROOT):
+        pytest.skip(
+            'Skipping Selenium tests since built UI not found (generate it using `yarn build`).'
+        )
     return live_server.url