treeherder/docs/installation.md

Installation
============

```eval_rst
.. note:: This section describes how to set up a fully functioning
          instance of Treeherder. If you only want to hack on the UI,
          you can just setup a standalone webserver which accesses
          the server backend using node.js, which is much simpler.
          See the :doc:`UI development section <ui/installation>`.
```

Prerequisites
-------------

* If you are new to Mozilla or the A-Team, read the [A-Team Bootcamp].
* Install [Git], [Virtualbox] and [Vagrant] (latest versions recommended).
* Clone the [treeherder repo] from GitHub.
* Linux only: An nfsd server is required. You can install this on Ubuntu by running `apt-get install nfs-common nfs-kernel-server`

Setting up Vagrant
------------------

* Open a shell, cd into the root of the Treeherder repository, and type:

  ```bash
  > vagrant up --provision
  ```

  It will typically take 5 to 30 minutes for the vagrant provision to
  complete, depending on your network performance. If you experience
  any errors, see the [troubleshooting page](troubleshooting.md).

  It is *very important* that the provisioning process complete successfully before
  trying to interact with your test instance of treeherder: some things might
  superficially seem to work a partially configured machine, but
  it is almost guaranteed that some things *will break* in
  hard-to-diagnose ways if vagrant provision is not run to completion.

* Once the virtual machine is set up, connect to it using:

  ```bash
  > vagrant ssh
  ```

  A python virtual environment will be activated on login, and the working directory will be the treeherder source directory shared from the host machine.

* For the full list of available Vagrant commands (for example, suspending the VM when you are finished for the day),
  see their [command line documentation](https://www.vagrantup.com/docs/cli/).

* If you just wish to [run the tests](common_tasks.html#running-the-tests),
  you can stop now without performing the remaining steps.

Starting a local Treeherder instance
------------------------------------

* Start a gunicorn instance inside the Vagrant VM, to serve the static UI and API requests:

  ```bash
  vagrant ~/treeherder$ ./bin/run_gunicorn
  ```

  Or for development you can use the django runserver instead of gunicorn:

  ```bash
  vagrant ~/treeherder$ ./manage.py runserver
  ```

  this is more convenient because it automatically refreshes every time there's a change in the code.

* You must also start the UI dev server. Open a new terminal window and ``vagrant ssh`` to
  the VM again, then run the following:

  ```bash
  vagrant ~/treeherder$ yarn start:local
  ```

  This will build the UI code in the ``dist/`` folder and keep watching for
  new changes (See the [UI development section](ui/installation.md) for more ways to work with the UI code).

* Visit <http://localhost:5000> in your browser (NB: port has changed). Note: There will be no data to display until the ingestion tasks are run.

Running the ingestion tasks
---------------------------

Ingestion tasks populate the database with version control push logs, queued/running/completed jobs & output from log parsing, as well as maintain a cache of intermittent failure bugs. To run these:

* Start up a celery worker to process async tasks:

  ```bash
  vagrant ~/treeherder$ celery -A treeherder worker -B --concurrency 5
  ```

  The "-B" option tells the celery worker to startup a beat service, so that periodic tasks can be executed.
  You only need one worker with the beat service enabled. Multiple beat services will result in periodic tasks being executed multiple times.

* Then in a new terminal window, run `vagrant ssh` again, and follow the steps from the [loading pulse data](pulseload.md) page.

Ingesting a single push (at a time)
-----------------------------------

```eval_rst
.. warning::
  With the end of life of buildbot, this command is no longer able to ingest jobs.
  For now after running it, you will need to manually follow the steps from the
  :doc:`loading pulse data<pulseload>` page.
```

Alternatively, instead of running a full ingestion task, you can process just
the jobs associated with any single push generated in the last 4 hours
([builds-4h]), in a synchronous manner. This is ideal for testing. For example:

[builds-4h]: http://builddata.pub.build.mozilla.org/buildjson/

```bash
vagrant ~/treeherder$ ./manage.py ingest_push mozilla-inbound 63f8a47cfdf5
```

If running this locally, replace `63f8a47cfdf5` with a recent revision (= pushed within
the last four hours) on mozilla-inbound.

Ingesting a range of pushes
---------------------------

It is also possible to ingest the last N pushes for a repository:

```bash
vagrant ~/treeherder$ ./manage.py ingest_push mozilla-central --last-n-pushes 100
```

In this mode, only the pushlog data will be ingested: additional results
associated with the pushes will not. This mode is useful to seed pushes so
they are visible on the web interface and so you can easily copy and paste
changesets from the web interface into subsequent ``ingest_push`` commands.

[A-Team Bootcamp]: https://ateam-bootcamp.readthedocs.io
[Git]: https://git-scm.com
[Vagrant]: https://www.vagrantup.com
[Virtualbox]: https://www.virtualbox.org
[treeherder repo]: https://github.com/mozilla/treeherder