There's no need to make multiple calls to peep - we can just combine
them into one. Not changing the puppet instances for Vagrant, since the
calls are made in two separate puppet modules and so would require a bit
of refactoring, which is going to occur in bug 1074151 and friends.
The Cython build process creates .c and .so files in the log_parser
directory, which only get re-created if the original source changes.
We recently updated the version of Cython, however this doesn't trigger
re-building them. To avoid surprises, we should purge them like we do
for the .pyc files.
In addition, there is no need to delete .pyc files from the UI source
directory, since that does not contain any Python files.
We're passing settings.UPDATE_REF, which is the default ref value, not
the actual value that was passed via the Chief web UI. As such, New
Relic was always displaying the revision deployed as "master", even if
it was a non-master branch deploy. Ideally we'd use the value of 'ref'
from pre_update(), but that's not available during the deploy() task
(the script is run in three independent stages & Chief unhelpfully only
passes 'ref' to the first stage), so we'll have to read it back from the
media/revision file - which will require more mangling in bug 1076886.
For now, seeing as the 'revision' property is optional, lets just omit
it, rather than setting it to a sometimes wrong value.
Prior to this change, on stage/production we didn't use virtualenvs
(unlike dev/the local Vagrant project) and instead pip installed
packages globally (when puppet ran periodically), using requirements
files that are not in the repo.
Now during deployment, a virtualenv is created and then populated using
peep (which uses hashes to verify the contents of packages before pip
installing them). The virtualenv is then made relocatable (as best as it
can, the feature isn't perfect), the lib64 symlinks are made relative,
and then the virtualenv is rsynced to all nodes, along with the source.
The one main remaining limitation of --relocatable is that the bash
activate script will not work on the other nodes - however the wrapper
scripts under treeherder-service/bin/ add venv/bin/ to PATH so using the
activate script is unnecessary for them. This just leaves running
manage.py commands locally on a node, for we can use:
|../venv/bin/python manage.py foo|, an alias or else we can fix up the
activate scripts in a follow-up bug.
The current values were copied from another project's deploy script, but
they are not working - so let's use what the latest New Relic docs say
we should use.
Apparently the current directory is only sometimes on the path & I
mis-read or understood the docs, sigh. Let's just keep the sys.path
manipulation and use #noqa to keep flake8 quiet.
Anything in the deploy script decorated with @task or @hostgroups ends
up having it's progress displayed in IRC by the bot - eg:
[2015-02-23 20:09:40] Finished deploy_log (0.454s)
However, the "deploy_X" @hostgroup functions were actually only
rsyncing the code (which is fast compared to restarting the processes),
but the name used to imply the whole deploy for that node type was
complete.
The new structure not only reduces code duplication, but makes the IRC
output more accurate, to provide a better idea as to what stage the
deploy is at, and which are the problematic steps in the case of errors.
Previously the revision file was regenerated after it had already been
rsynced to the webheads, and as such the publicly accessible file
displayed the previous revision, not the new revision.
The current fetch/reset/checkout doesn't correctly update the working
directory when switching between branches. This new approach is
recommended by:
http://stackoverflow.com/a/9589927
The directory of the currently running script is already on the path:
https://docs.python.org/2/tutorial/modules.html#the-module-search-path
...so adding it is unnecessary.
In addition, I don't believe we need the treeherder-service directory on
the path, since we don't directly import anything from there.
The @hostgroups decorator was causing us to call restart-jobs multiple
times (once for each host within a host group), when we only want to
call it once per group of hosts.
Abstracting the restart-jobs call to its own method also means the IRC
bot will output "restart_jobs complete" after each restart-jobs call,
making the progress through the deployment task clearer.
Ed Morley
Ed Morley
MikeLing
Mauro Doglio
Cameron Dawson
mdoglio
Jonathan French