mozilla
/
treeherder
зеркало из https://github.com/mozilla/treeherder.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
5 831 коммит 28 Ветки 2 Теги 541 MiB
Граф коммитов

112 Коммитов

Автор SHA1 Сообщение Дата
Ed Morley 58cc851204 Bug 1146184 - Add virtualenv's bin to PATH rather than using activate 
In order to rsync the virtualenv as part of deployment, we'll need to
use |virtualenv --relocatable|. However this does not update the
activate scripts (making them unusable), and the changes it makes in the
virtualenv bin directory require that the virtualenv's python is first
in the path:
https://virtualenv.pypa.io/en/latest/userguide.html#making-environments-relocatable

As such, we must make the Treeherder runner scripts add the virtualenv
bin to PATH instead of using the activate pattern.
 2015-04-20 23:48:26 +01:00
Ed Morley a2d66a85d1 Bug 1146184 - Clean up the directory variables in runner scripts 
The existing name "curr_dir" isn't overly clear - does it mean the
current working directory or the directory in which this script exists?
It should also be uppercase. I think the behaviour is clearer when
reworked like this :-)

There's a lot of other cleanup that could be done (eg reducing
duplication), but saving that for bug 1153971.
 2015-04-20 23:48:26 +01:00
Ed Morley 5ee006eb01 Bug 1155293 - Update to peep v2.4 
Copied verbatim from:
https://github.com/erikrose/peep/archive/2.4.zip

Changes:
https://github.com/erikrose/peep/compare/2.2...2.4
 2015-04-17 17:38:40 +01:00
Ed Morley b1f14bb458 Bug 1153966 - Remove unused Pulse consumer code 
Unlike the Pulse publishing, the code for consuming data from Pulse is
unused, being a leftover from initial attempts to ingest buildbot data
via pulse, rather than builds-{4hr,running,pending}.js
 2015-04-14 00:21:50 +01:00
Ed Morley 87f6605de1 Bug 1153186 - Split up the high_priority task queue 
...into 'classification_mirroring' and 'publish_to_pulse'. This gives
greater visibility into the relative queue sizes of each, allows us to
move one of them to another node, plus means we can pause consumption of
the classification_mirroring task for when the Elasticsearch indicies
are migrated in bug 1142538.
 2015-04-13 18:55:34 +01:00
Ed Morley e8ad64842d Bug 1140850 - Remove gevent celery worker script from the repo 
Since it's unused & we're moving away from gevent.
 2015-04-13 17:12:44 +01:00
Ed Morley 19d0b51d2a Bug 1140882 - Use prefork scheduling for pushlog workers 
Use prefork scheduling instead of gevent scheduling, to avoid issues
we've had with gevent - both with zombie tasks & also incompatibilities
with Python 2.7.9.
 2015-04-13 17:09:55 +01:00
Ed Morley 8e67030a35 Bug 1143350 - Use peep instead of pip locally, on Travis & in Docker 
We want to start using peep in production, to alleviate security
concerns with the idea of auto-updating packages from PyPI on deploy.
As a first step, we switch to using peep in the Vagrant environment,
on Travis and in the Docker build - so we can confirm the hashes are
correct.

Close bug 1143350.
 2015-03-19 12:26:07 +00:00
Ed Morley 29ca673205 Bug 1143350 - Check in peep v2.2 
We're checking this in so we have a known good starting point in the
chain of trust. It also simplifies our deployment requirements.

peep.py was taken from:
https://github.com/erikrose/peep/archive/2.2.tar.gz

The only alteration made was the addition of the licence block at the
top of the file, taken from LICENCE in the peep repo.
 2015-03-19 12:12:01 +00:00
Ed Morley 99c6a1a9d6 Bug 1070470 - Rename pure.txt to checked-in.txt 
The packages in this file are those that have been checked in to
vendor/, and the new name makes this more obvious.
 2015-03-12 19:04:07 +00:00
Cameron Dawson d4bb357d5e Bug 1113873 - Enable the structured log parser celery queue 2015-03-11 08:49:10 -07:00
Ed Morley e9780bf1a0 Bug 1133482 - Remove download_logs.py since it's always been broken 
Traceback (most recent call last):
  File "./bin/download_logs.py", line 97, in <module>
    download_logs(sys.argv[1])
  File "./bin/download_logs.py", line 74, in download_logs
    logrefs = job["job"]["log_references"]
TypeError: string indices must be integers, not str

And I don't think we have a need for it now anyway.
 2015-03-03 17:13:35 +00:00
mdoglio fd9eb17603 Bug 1123479 - add startup script for a prefork-based log parser 2015-02-02 12:45:29 +00:00
mdoglio 40b01c751d Bug 1122139 - Split common tasks to separate queues 2015-01-28 14:33:38 +00:00
mdoglio f173254fd7 Bug 1094814 - reduce buildapi worker memory consumption 2014-12-10 11:07:57 +00:00
mdoglio 78ecb0900c Bug 1105800 - increase the web service timeout limit 2014-11-27 17:04:50 +00:00
Jonathan French dbb4d11e09 Bug 1090689 - Add MPL2.0 headers to the repo 2014-11-03 13:06:03 -05:00
mdoglio 826a8a7dbc change log parser log verbosity back to info 2014-10-28 17:30:53 +00:00
mdoglio a1a53c1f6b increase log parser log verbosity 2014-10-28 16:48:52 +00:00
camd f4e979a248 Merge pull request #261 from mozilla/bug-1076769-parse-log-on-demand 
Bug 1076769 parse log on demand
 2014-10-27 10:41:56 -07:00
mdoglio ac3455521c Bug 1088188 - increase gunicorn timeout to 60s 2014-10-24 15:37:47 +01:00
mdoglio 04e54027b6 Bug 1088188 - increase gunicorn timeout to 45s 2014-10-23 19:15:02 +01:00
mdoglio b6898b6f87 Bug 1076769 - support parsing a log on demand 2014-10-23 14:34:59 +01:00
mdoglio 5df6bd4212 Unplug the default celery worker from etl queues 2014-10-21 18:21:02 +01:00
Cameron Dawson bb84435786 Merge branch 'master' into queue-missing-resultsets2 2014-10-15 11:29:29 -07:00
mdoglio b9d40fb40a Bug 1016117 - update celery project layout and worker init script 2014-10-10 18:12:16 +01:00
Cameron Dawson 3127f53e19 Bug 1077136 - removing dead socket.io code 2014-10-09 12:19:42 -07:00
mdoglio 844b6a8c9f Temporarily consume buildapi and pushlog queues from default 
This is to support the use of separate queues for etl tasks until we
have dedicated machines on production
 2014-10-06 23:40:46 +01:00
mdoglio 9555d42872 Bug 1076774 - use a separate worker for jobs ingestion 2014-10-06 16:28:00 +01:00
camd 893d89ce84 Merge pull request #236 from mozilla/bug-1076778-use-gevent-for-pushlog 
Bug 1076778 - Use a gevent worker for pushlog ingestion
 2014-10-06 08:20:53 -07:00
mdoglio cfa29aaa65 Bug 1076778 - Use a gevent worker for pushlog ingestion 2014-10-06 14:24:26 +01:00
Jonathan Eads 56005e495a Adjust max-requests and keepalive time to minimize cpu spikes 2014-10-01 14:47:09 -07:00
Jonathan Eads c20dbced12 Reduce max-requests and keepalive to minimize memory consumption 2014-10-01 14:18:27 -07:00
mdoglio e3384e2d09 add high priority queue for tbpl-related tasks 2014-07-15 18:27:32 +01:00
Jonathan Eads b2002c68c9 set log level to INFO 2014-06-20 14:33:17 -07:00
mdoglio 4f90094e7a trying to make the socketio logging work on py26 2014-06-20 16:20:59 +02:00
Mauro Doglio 518e281178 revert run_celery_worker 2014-06-19 18:27:40 +01:00
mdoglio b95cd5249b celery worker unique name 2014-06-18 19:59:46 +01:00
mdoglio feb3862db8 reduce the celery concurrency to 1 worker 2014-06-18 18:03:39 +01:00
Mauro Doglio 6a202bbf26 discard static name for gevent worker 2014-06-16 18:41:29 +01:00
Jonathan Eads ad1310fb26 fixed a bunch of stuff for production deployment 2014-06-11 16:28:06 -07:00
mdoglio 1e468c7f14 set treeherder logger to write ERROR level logs to stderr 
remove unused celerymon service
disable flash transport for socketio
 2014-06-04 14:14:41 +02:00
uberj 3809cbd985 Re-add the use of $PYTHON (I nixed this accidentally during a rebase) 2014-06-02 14:07:35 -07:00
uberj 3fa8784f64 Resolve log name conflict (with run_celery_worder) and fix bash syntax error 2014-06-02 07:55:42 -07:00
uberj 968fa002ee bash needed more spaces 2014-06-02 07:55:42 -07:00
uberj f2f3b6cbd7 fix bash style issues 2014-06-02 07:55:42 -07:00
uberj cc5de0d423 add log file/dir creation logic for all processes 
@mozjcrowe r?
 2014-06-02 07:55:42 -07:00
Mauro Doglio 413867d4e6 Update run_gunicorn 
add a missing 'fi'
 2014-06-02 11:38:05 +02:00
Jonathan Eads 4df3dce627 made LOGFILE a variable 2014-05-30 15:28:36 -07:00
Jonathan Eads 5cc1d93f8c changed python path to be set dynamically 2014-05-30 10:49:05 -07:00
Jonathan Eads e8845853cc changed python and gunicorn paths to be dynamic 2014-05-30 10:40:14 -07:00
Jonathan Eads 21bf167647 dynamically set python path 2014-05-30 10:29:34 -07:00
mdoglio 993921fbed remove unuseful print 2014-05-20 17:18:19 +01:00
mdoglio de2d694f22 add newrelic support to the worker scripts 2014-05-20 16:57:13 +01:00
Mauro Doglio 3d32797a40 Increase gunicorn max requests limit 2014-05-19 16:08:38 +01:00
Mauro Doglio 86bad9bb9d Update run_celery_worker_gevent 
Use the hostname as part of the gevent_worker name
 2014-03-11 15:27:09 +00:00
mdoglio d10506b710 fixup! fixup! remove db access from the parse_log task 2014-03-11 14:38:30 +00:00
mdoglio 0e53e70491 fix difference between supervisord v2 and v3 2014-01-07 21:28:48 +00:00
mdoglio 327ed28648 improve environment setup via puppet 2014-01-07 15:24:35 +00:00
mdoglio 3085581e2d add daemon scripts 2014-01-06 15:49:32 +00:00
Cameron Dawson 24da622b9b moved log downloader to a script in bin 2013-06-10 09:16:59 -07:00
mdoglio 81b441c527 add generate-vendor-lib.py 2013-06-10 12:14:47 +01:00