treeherder/bin/run_celery_worker_log_parser

#!/usr/bin/env bash

# Make non-zero exit codes & other errors fatal.
set -euo pipefail

SRC_DIR=$(dirname "$(dirname "${BASH_SOURCE[0]}")")
cd "$SRC_DIR"

source vagrant/env.sh

exec newrelic-admin run-program celery -A treeherder worker \
    -Q \
log_parser,\
log_parser_fail,\
log_store_failure_lines,\
log_store_failure_lines_fail,\
log_crossreference_error_lines,\
log_crossreference_error_lines_fail,\
log_autoclassify,\
log_autoclassify_fail\
    --concurrency=10 -l INFO \
    --maxtasksperchild=500 -n log_parser.%h
Bug 1414255 - Enable bash 'strict mode' in all shell scripts This makes some of the error checking in `runtest.sh` redundant. The shebangs have also been made more consistent. See: http://redsymbol.net/articles/unofficial-bash-strict-mode/ 2017-11-03 17:31:40 +03:00			`#!/usr/bin/env bash`

			`# Make non-zero exit codes & other errors fatal.`
			`set -euo pipefail`
Bug 1123479 - add startup script for a prefork-based log parser 2015-02-02 15:45:29 +03:00
Bug 1146184 - Clean up the directory variables in runner scripts The existing name "curr_dir" isn't overly clear - does it mean the current working directory or the directory in which this script exists? It should also be uppercase. I think the behaviour is clearer when reworked like this :-) There's a lot of other cleanup that could be done (eg reducing duplication), but saving that for bug 1153971. 2015-04-15 17:59:39 +03:00			`SRC_DIR=$(dirname "$(dirname "${BASH_SOURCE[0]}")")`
Bug 1414255 - Fix shellcheck warnings about missing quotes `SC2086: Double quote to prevent globbing and word splitting` See: https://github.com/koalaman/shellcheck/wiki/SC2086 2017-11-03 17:32:31 +03:00			`cd "$SRC_DIR"`
Bug 1146184 - Clean up the directory variables in runner scripts The existing name "curr_dir" isn't overly clear - does it mean the current working directory or the directory in which this script exists? It should also be uppercase. I think the behaviour is clearer when reworked like this :-) There's a lot of other cleanup that could be done (eg reducing duplication), but saving that for bug 1153971. 2015-04-15 17:59:39 +03:00
Bug 1414255 - Fix shellcheck warnings about sourced files ``` SC1090: Can't follow non-constant source. Use a directive to specify location. SC1091: Not following: /etc/profile.d/treeherder.sh was not specified as input (see shellcheck -x). ``` See: https://github.com/koalaman/shellcheck/wiki/SC1090 https://github.com/koalaman/shellcheck/wiki/SC1091 2017-11-03 17:51:44 +03:00			`source vagrant/env.sh`
Bug 1123479 - add startup script for a prefork-based log parser 2015-02-02 15:45:29 +03:00
Bug 1169944 - Use the New Relic wrapper when running scripts locally too Previously if the `bin/` directory scripts were run in the Vagrant environment, the processes run would not have been started via the New Relic wrapper command, since the New Relic licence key is not set. For greater consistency between Vagrant and production, the wrapper command is now always used. This works since `NEW_RELIC_DEVELOPER_MODE` is defined in the Vagrant environment (thanks to a previous commit in the same bug), which prevents the agent from trying to submit real data: https://docs.newrelic.com/docs/agents/python-agent/installation-configuration/python-agent-configuration#developer_mode 2016-05-03 20:40:10 +03:00			`exec newrelic-admin run-program celery -A treeherder worker \`
Bug 1252854 - Backend work for matching unstructured and structured log summary lines. 2016-02-05 13:44:58 +03:00			`-Q \`
			`log_parser,\`
			`log_parser_fail,\`
			`log_store_failure_lines,\`
			`log_store_failure_lines_fail,\`
			`log_crossreference_error_lines,\`
			`log_crossreference_error_lines_fail,\`
			`log_autoclassify,\`
Bug 1258861 - Move text log steps and errors into main database (#1696) This changes ingestion, the API endpoints, and the frontend to match the new structure. For now we continue to store text_log_summary artifacts, though they don't do anything anymore. 2016-09-12 19:30:36 +03:00			`log_autoclassify_fail\`
Bug 1318021 - Vagrant: Stop using log files for gunicorn/celery output Outputting to the console rather than a log file: * is more user-friendly during development * is more consistent with Heroku * means the Vagrant-specific Django LOGGING config is now closer to the one in settings.py, and so more easily combined with it Both gunicorn and celery default to outputting to stdout/stderr, so the `logfile` options can be omitted entirely. 2016-11-17 15:38:30 +03:00			`--concurrency=10 -l INFO \`
Bug 1123479 - add startup script for a prefork-based log parser 2015-02-02 15:45:29 +03:00			`--maxtasksperchild=500 -n log_parser.%h`