mozilla
/
treeherder
зеркало из https://github.com/mozilla/treeherder.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
7 578 коммитов 25 Ветки 2 Теги 538 MiB
Граф коммитов

161 Коммитов

Автор SHA1 Сообщение Дата
Ed Morley a2d66a85d1 Bug 1146184 - Clean up the directory variables in runner scripts 
The existing name "curr_dir" isn't overly clear - does it mean the
current working directory or the directory in which this script exists?
It should also be uppercase. I think the behaviour is clearer when
reworked like this :-)

There's a lot of other cleanup that could be done (eg reducing
duplication), but saving that for bug 1153971.
 2015-04-20 23:48:26 +01:00
Ed Morley 5ee006eb01 Bug 1155293 - Update to peep v2.4 
Copied verbatim from:
https://github.com/erikrose/peep/archive/2.4.zip

Changes:
https://github.com/erikrose/peep/compare/2.2...2.4
 2015-04-17 17:38:40 +01:00
Ed Morley b1f14bb458 Bug 1153966 - Remove unused Pulse consumer code 
Unlike the Pulse publishing, the code for consuming data from Pulse is
unused, being a leftover from initial attempts to ingest buildbot data
via pulse, rather than builds-{4hr,running,pending}.js
 2015-04-14 00:21:50 +01:00
Ed Morley 87f6605de1 Bug 1153186 - Split up the high_priority task queue 
...into 'classification_mirroring' and 'publish_to_pulse'. This gives
greater visibility into the relative queue sizes of each, allows us to
move one of them to another node, plus means we can pause consumption of
the classification_mirroring task for when the Elasticsearch indicies
are migrated in bug 1142538.
 2015-04-13 18:55:34 +01:00
Ed Morley e8ad64842d Bug 1140850 - Remove gevent celery worker script from the repo 
Since it's unused & we're moving away from gevent.
 2015-04-13 17:12:44 +01:00
Ed Morley 19d0b51d2a Bug 1140882 - Use prefork scheduling for pushlog workers 
Use prefork scheduling instead of gevent scheduling, to avoid issues
we've had with gevent - both with zombie tasks & also incompatibilities
with Python 2.7.9.
 2015-04-13 17:09:55 +01:00
Ed Morley 8e67030a35 Bug 1143350 - Use peep instead of pip locally, on Travis & in Docker 
We want to start using peep in production, to alleviate security
concerns with the idea of auto-updating packages from PyPI on deploy.
As a first step, we switch to using peep in the Vagrant environment,
on Travis and in the Docker build - so we can confirm the hashes are
correct.

Close bug 1143350.
 2015-03-19 12:26:07 +00:00
Ed Morley 29ca673205 Bug 1143350 - Check in peep v2.2 
We're checking this in so we have a known good starting point in the
chain of trust. It also simplifies our deployment requirements.

peep.py was taken from:
https://github.com/erikrose/peep/archive/2.2.tar.gz

The only alteration made was the addition of the licence block at the
top of the file, taken from LICENCE in the peep repo.
 2015-03-19 12:12:01 +00:00
Ed Morley 99c6a1a9d6 Bug 1070470 - Rename pure.txt to checked-in.txt 
The packages in this file are those that have been checked in to
vendor/, and the new name makes this more obvious.
 2015-03-12 19:04:07 +00:00
Cameron Dawson d4bb357d5e Bug 1113873 - Enable the structured log parser celery queue 2015-03-11 08:49:10 -07:00
Ed Morley e9780bf1a0 Bug 1133482 - Remove download_logs.py since it's always been broken 
Traceback (most recent call last):
  File "./bin/download_logs.py", line 97, in <module>
    download_logs(sys.argv[1])
  File "./bin/download_logs.py", line 74, in download_logs
    logrefs = job["job"]["log_references"]
TypeError: string indices must be integers, not str

And I don't think we have a need for it now anyway.
 2015-03-03 17:13:35 +00:00
mdoglio fd9eb17603 Bug 1123479 - add startup script for a prefork-based log parser 2015-02-02 12:45:29 +00:00
mdoglio 40b01c751d Bug 1122139 - Split common tasks to separate queues 2015-01-28 14:33:38 +00:00
mdoglio f173254fd7 Bug 1094814 - reduce buildapi worker memory consumption 2014-12-10 11:07:57 +00:00
mdoglio 78ecb0900c Bug 1105800 - increase the web service timeout limit 2014-11-27 17:04:50 +00:00
Jonathan French dbb4d11e09 Bug 1090689 - Add MPL2.0 headers to the repo 2014-11-03 13:06:03 -05:00
mdoglio 826a8a7dbc change log parser log verbosity back to info 2014-10-28 17:30:53 +00:00
mdoglio a1a53c1f6b increase log parser log verbosity 2014-10-28 16:48:52 +00:00
camd f4e979a248 Merge pull request #261 from mozilla/bug-1076769-parse-log-on-demand 
Bug 1076769 parse log on demand
 2014-10-27 10:41:56 -07:00
mdoglio ac3455521c Bug 1088188 - increase gunicorn timeout to 60s 2014-10-24 15:37:47 +01:00
mdoglio 04e54027b6 Bug 1088188 - increase gunicorn timeout to 45s 2014-10-23 19:15:02 +01:00
mdoglio b6898b6f87 Bug 1076769 - support parsing a log on demand 2014-10-23 14:34:59 +01:00
mdoglio 5df6bd4212 Unplug the default celery worker from etl queues 2014-10-21 18:21:02 +01:00
Cameron Dawson bb84435786 Merge branch 'master' into queue-missing-resultsets2 2014-10-15 11:29:29 -07:00
mdoglio b9d40fb40a Bug 1016117 - update celery project layout and worker init script 2014-10-10 18:12:16 +01:00
Cameron Dawson 3127f53e19 Bug 1077136 - removing dead socket.io code 2014-10-09 12:19:42 -07:00
mdoglio 844b6a8c9f Temporarily consume buildapi and pushlog queues from default 
This is to support the use of separate queues for etl tasks until we
have dedicated machines on production
 2014-10-06 23:40:46 +01:00
mdoglio 9555d42872 Bug 1076774 - use a separate worker for jobs ingestion 2014-10-06 16:28:00 +01:00
camd 893d89ce84 Merge pull request #236 from mozilla/bug-1076778-use-gevent-for-pushlog 
Bug 1076778 - Use a gevent worker for pushlog ingestion
 2014-10-06 08:20:53 -07:00
mdoglio cfa29aaa65 Bug 1076778 - Use a gevent worker for pushlog ingestion 2014-10-06 14:24:26 +01:00
Jonathan Eads 56005e495a Adjust max-requests and keepalive time to minimize cpu spikes 2014-10-01 14:47:09 -07:00
Jonathan Eads c20dbced12 Reduce max-requests and keepalive to minimize memory consumption 2014-10-01 14:18:27 -07:00
mdoglio e3384e2d09 add high priority queue for tbpl-related tasks 2014-07-15 18:27:32 +01:00
Jonathan Eads b2002c68c9 set log level to INFO 2014-06-20 14:33:17 -07:00
mdoglio 4f90094e7a trying to make the socketio logging work on py26 2014-06-20 16:20:59 +02:00
Mauro Doglio 518e281178 revert run_celery_worker 2014-06-19 18:27:40 +01:00
mdoglio b95cd5249b celery worker unique name 2014-06-18 19:59:46 +01:00
mdoglio feb3862db8 reduce the celery concurrency to 1 worker 2014-06-18 18:03:39 +01:00
Mauro Doglio 6a202bbf26 discard static name for gevent worker 2014-06-16 18:41:29 +01:00
Jonathan Eads ad1310fb26 fixed a bunch of stuff for production deployment 2014-06-11 16:28:06 -07:00
mdoglio 1e468c7f14 set treeherder logger to write ERROR level logs to stderr 
remove unused celerymon service
disable flash transport for socketio
 2014-06-04 14:14:41 +02:00
uberj 3809cbd985 Re-add the use of $PYTHON (I nixed this accidentally during a rebase) 2014-06-02 14:07:35 -07:00
uberj 3fa8784f64 Resolve log name conflict (with run_celery_worder) and fix bash syntax error 2014-06-02 07:55:42 -07:00
uberj 968fa002ee bash needed more spaces 2014-06-02 07:55:42 -07:00
uberj f2f3b6cbd7 fix bash style issues 2014-06-02 07:55:42 -07:00
uberj cc5de0d423 add log file/dir creation logic for all processes 
@mozjcrowe r?
 2014-06-02 07:55:42 -07:00
Mauro Doglio 413867d4e6 Update run_gunicorn 
add a missing 'fi'
 2014-06-02 11:38:05 +02:00
Jonathan Eads 4df3dce627 made LOGFILE a variable 2014-05-30 15:28:36 -07:00
Jonathan Eads 5cc1d93f8c changed python path to be set dynamically 2014-05-30 10:49:05 -07:00
Jonathan Eads e8845853cc changed python and gunicorn paths to be dynamic 2014-05-30 10:40:14 -07:00
Jonathan Eads 21bf167647 dynamically set python path 2014-05-30 10:29:34 -07:00
mdoglio 993921fbed remove unuseful print 2014-05-20 17:18:19 +01:00
mdoglio de2d694f22 add newrelic support to the worker scripts 2014-05-20 16:57:13 +01:00
Mauro Doglio 3d32797a40 Increase gunicorn max requests limit 2014-05-19 16:08:38 +01:00
Mauro Doglio 86bad9bb9d Update run_celery_worker_gevent 
Use the hostname as part of the gevent_worker name
 2014-03-11 15:27:09 +00:00
mdoglio d10506b710 fixup! fixup! remove db access from the parse_log task 2014-03-11 14:38:30 +00:00
mdoglio 0e53e70491 fix difference between supervisord v2 and v3 2014-01-07 21:28:48 +00:00
mdoglio 327ed28648 improve environment setup via puppet 2014-01-07 15:24:35 +00:00
mdoglio 3085581e2d add daemon scripts 2014-01-06 15:49:32 +00:00
Cameron Dawson 24da622b9b moved log downloader to a script in bin 2013-06-10 09:16:59 -07:00
mdoglio 81b441c527 add generate-vendor-lib.py 2013-06-10 12:14:47 +01:00