зеркало из https://github.com/mozilla/treeherder.git
Bug 1145606 - Setup treeherder to deploy on heroku
I added a Procfile listing all the different python services treeherder needs. Heroku provides deployment-specific settings via environment variables, so I had to modify the settings file to listen to them where that wasn't the case. I created an enviroment variable IS_HEROKU which allows to have a heroku-only configuration where needed. The db service is provided by Amazon RDS, which requires a ssl connection. To enable ssl in the MySQLdb python client I had to modify Datasource (and bump up the version used). The cache service is provided by the memcachier heroku addon. Heroku recommends to use pylibmc, so I set it up according to the docs here https://devcenter.heroku.com/articles/memcachier#python. The amqp service is provided by the CloudAMQP addon. I added a post_compile script that runs every time we deploy. We should run every build step we require in there, like static asset minification, collection, etc. To share the oauth credentials among the various services I used an environment variable. I also added an option to export_project_credentials so that the credentials can be printed to stdout. This should come handy when we will need to update the environment-stored credentials with the ones in the db.
This commit is contained in:
Mauro Doglio
Родитель
b73307e941
Коммит
597282fe58
|
|
@ -0,0 +1,8 @@
|
||||||
|
web: newrelic-admin run-program gunicorn treeherder.webapp.wsgi:application --log-file - --timeout 29 --max-requests 2000
|
||||||
|
worker_beat: newrelic-admin run-program celery -A treeherder beat
|
||||||
|
worker_pushlog: newrelic-admin run-program celery -A treeherder worker -Q pushlog,fetch_missing_push_logs --maxtasksperchild=500 --concurrency=5
|
||||||
|
worker_buildapi: newrelic-admin run-program celery -A treeherder worker -Q buildapi --maxtasksperchild=20 --concurrency=5
|
||||||
|
worker_default: newrelic-admin run-program celery -A treeherder worker -Q default,process_objects,cycle_data,calculate_eta,populate_performance_series,fetch_bugs --maxtasksperchild=50 --concurrency=3
|
||||||
|
worker_hp: newrelic-admin run-program celery -A treeherder worker -Q classification_mirroring,publish_to_pulse --maxtasksperchild=50 --concurrency=1
|
||||||
|
worker_log_parser: newrelic-admin run-program celery -A treeherder worker -Q log_parser_fail,log_parser,log_parser_hp,log_parser_json --maxtasksperchild=50 --concurrency=5
|
||||||
|
|
||||||
|
|
@ -0,0 +1,2 @@
|
||||||
|
#! /bin/bash -e
|
||||||
|
python setup.py build_ext --inplace
|
||||||
|
|
@ -0,0 +1,260 @@
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIDQzCCAqygAwIBAgIJAOd1tlfiGoEoMA0GCSqGSIb3DQEBBQUAMHUxCzAJBgNV
|
||||||
|
BAYTAlVTMRMwEQYDVQQIEwpXYXNoaW5ndG9uMRAwDgYDVQQHEwdTZWF0dGxlMRMw
|
||||||
|
EQYDVQQKEwpBbWF6b24uY29tMQwwCgYDVQQLEwNSRFMxHDAaBgNVBAMTE2F3cy5h
|
||||||
|
bWF6b24uY29tL3Jkcy8wHhcNMTAwNDA1MjI0NDMxWhcNMTUwNDA0MjI0NDMxWjB1
|
||||||
|
MQswCQYDVQQGEwJVUzETMBEGA1UECBMKV2FzaGluZ3RvbjEQMA4GA1UEBxMHU2Vh
|
||||||
|
dHRsZTETMBEGA1UEChMKQW1hem9uLmNvbTEMMAoGA1UECxMDUkRTMRwwGgYDVQQD
|
||||||
|
ExNhd3MuYW1hem9uLmNvbS9yZHMvMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB
|
||||||
|
gQDKhXGU7tizxUR5WaFoMTFcxNxa05PEjZaIOEN5ctkWrqYSRov0/nOMoZjqk8bC
|
||||||
|
med9vPFoQGD0OTakPs0jVe3wwmR735hyVwmKIPPsGlaBYj1O6llIpZeQVyupNx56
|
||||||
|
UzqtiLaDzh1KcmfqP3qP2dInzBfJQKjiRudo1FWnpPt33QIDAQABo4HaMIHXMB0G
|
||||||
|
A1UdDgQWBBT/H3x+cqSkR/ePSIinPtc4yWKe3DCBpwYDVR0jBIGfMIGcgBT/H3x+
|
||||||
|
cqSkR/ePSIinPtc4yWKe3KF5pHcwdTELMAkGA1UEBhMCVVMxEzARBgNVBAgTCldh
|
||||||
|
c2hpbmd0b24xEDAOBgNVBAcTB1NlYXR0bGUxEzARBgNVBAoTCkFtYXpvbi5jb20x
|
||||||
|
DDAKBgNVBAsTA1JEUzEcMBoGA1UEAxMTYXdzLmFtYXpvbi5jb20vcmRzL4IJAOd1
|
||||||
|
tlfiGoEoMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAvguZy/BDT66x
|
||||||
|
GfgnJlyQwnFSeVLQm9u/FIvz4huGjbq9dqnD6h/Gm56QPFdyMEyDiZWaqY6V08lY
|
||||||
|
LTBNb4kcIc9/6pc0/ojKciP5QJRm6OiZ4vgG05nF4fYjhU7WClUx7cxq1fKjNc2J
|
||||||
|
UCmmYqgiVkAGWRETVo+byOSDZ4swb10=
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIID9DCCAtygAwIBAgIBQjANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||||
|
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||||
|
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||||
|
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNTAyMDUwOTExMzFaFw0y
|
||||||
|
MDAzMDUwOTExMzFaMIGKMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||||
|
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||||
|
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEbMBkGA1UEAwwSQW1hem9uIFJE
|
||||||
|
UyBSb290IENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuD8nrZ8V
|
||||||
|
u+VA8yVlUipCZIKPTDcOILYpUe8Tct0YeQQr0uyl018StdBsa3CjBgvwpDRq1HgF
|
||||||
|
Ji2N3+39+shCNspQeE6aYU+BHXhKhIIStt3r7gl/4NqYiDDMWKHxHq0nsGDFfArf
|
||||||
|
AOcjZdJagOMqb3fF46flc8k2E7THTm9Sz4L7RY1WdABMuurpICLFE3oHcGdapOb9
|
||||||
|
T53pQR+xpHW9atkcf3pf7gbO0rlKVSIoUenBlZipUlp1VZl/OD/E+TtRhDDNdI2J
|
||||||
|
P/DSMM3aEsq6ZQkfbz/Ilml+Lx3tJYXUDmp+ZjzMPLk/+3beT8EhrwtcG3VPpvwp
|
||||||
|
BIOqsqVVTvw/CwIDAQABo2MwYTAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUw
|
||||||
|
AwEB/zAdBgNVHQ4EFgQUTgLurD72FchM7Sz1BcGPnIQISYMwHwYDVR0jBBgwFoAU
|
||||||
|
TgLurD72FchM7Sz1BcGPnIQISYMwDQYJKoZIhvcNAQEFBQADggEBAHZcgIio8pAm
|
||||||
|
MjHD5cl6wKjXxScXKtXygWH2BoDMYBJF9yfyKO2jEFxYKbHePpnXB1R04zJSWAw5
|
||||||
|
2EUuDI1pSBh9BA82/5PkuNlNeSTB3dXDD2PEPdzVWbSKvUB8ZdooV+2vngL0Zm4r
|
||||||
|
47QPyd18yPHrRIbtBtHR/6CwKevLZ394zgExqhnekYKIqqEX41xsUV0Gm6x4vpjf
|
||||||
|
2u6O/+YE2U+qyyxHE5Wd5oqde0oo9UUpFETJPVb6Q2cEeQib8PBAyi0i6KnF+kIV
|
||||||
|
A9dY7IHSubtCK/i8wxMVqfd5GtbA8mmpeJFwnDvm9rBEsHybl08qlax9syEwsUYr
|
||||||
|
/40NawZfTUU=
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIEATCCAumgAwIBAgIBRDANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||||
|
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||||
|
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||||
|
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNTAyMDUyMjAzMDZaFw0y
|
||||||
|
MDAzMDUyMjAzMDZaMIGUMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||||
|
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||||
|
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzElMCMGA1UEAwwcQW1hem9uIFJE
|
||||||
|
UyBhcC1ub3J0aGVhc3QtMSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
|
||||||
|
ggEBAMmM2B4PfTXCZjbZMWiDPyxvk/eeNwIRJAhfzesiGUiLozX6CRy3rwC1ZOPV
|
||||||
|
AcQf0LB+O8wY88C/cV+d4Q2nBDmnk+Vx7o2MyMh343r5rR3Na+4izd89tkQVt0WW
|
||||||
|
vO21KRH5i8EuBjinboOwAwu6IJ+HyiQiM0VjgjrmEr/YzFPL8MgHD/YUHehqjACn
|
||||||
|
C0+B7/gu7W4qJzBL2DOf7ub2qszGtwPE+qQzkCRDwE1A4AJmVE++/FLH2Zx78Egg
|
||||||
|
fV1sUxPtYgjGH76VyyO6GNKM6rAUMD/q5mnPASQVIXgKbupr618bnH+SWHFjBqZq
|
||||||
|
HvDGPMtiiWII41EmGUypyt5AbysCAwEAAaNmMGQwDgYDVR0PAQH/BAQDAgEGMBIG
|
||||||
|
A1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFIiKM0Q6n1K4EmLxs3ZXxINbwEwR
|
||||||
|
MB8GA1UdIwQYMBaAFE4C7qw+9hXITO0s9QXBj5yECEmDMA0GCSqGSIb3DQEBBQUA
|
||||||
|
A4IBAQBezGbE9Rw/k2e25iGjj5n8r+M3dlye8ORfCE/dijHtxqAKasXHgKX8I9Tw
|
||||||
|
JkBiGWiuzqn7gO5MJ0nMMro1+gq29qjZnYX1pDHPgsRjUX8R+juRhgJ3JSHijRbf
|
||||||
|
4qNJrnwga7pj94MhcLq9u0f6dxH6dXbyMv21T4TZMTmcFduf1KgaiVx1PEyJjC6r
|
||||||
|
M+Ru+A0eM+jJ7uCjUoZKcpX8xkj4nmSnz9NMPog3wdOSB9cAW7XIc5mHa656wr7I
|
||||||
|
WJxVcYNHTXIjCcng2zMKd1aCcl2KSFfy56sRfT7J5Wp69QSr+jq8KM55gw8uqAwi
|
||||||
|
VPrXn2899T1rcTtFYFP16WXjGuc0
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIEATCCAumgAwIBAgIBRTANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||||
|
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||||
|
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||||
|
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNTAyMDUyMjAzMTlaFw0y
|
||||||
|
MDAzMDUyMjAzMTlaMIGUMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||||
|
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||||
|
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzElMCMGA1UEAwwcQW1hem9uIFJE
|
||||||
|
UyBhcC1zb3V0aGVhc3QtMSBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
|
||||||
|
ggEBANaXElmSEYt/UtxHFsARFhSUahTf1KNJzR0Dmay6hqOXQuRVbKRwPd19u5vx
|
||||||
|
DdF1sLT7D69IK3VDnUiQScaCv2Dpu9foZt+rLx+cpx1qiQd1UHrvqq8xPzQOqCdC
|
||||||
|
RFStq6yVYZ69yfpfoI67AjclMOjl2Vph3ftVnqP0IgVKZdzeC7fd+umGgR9xY0Qr
|
||||||
|
Ubhd/lWdsbNvzK3f1TPWcfIKQnpvSt85PIEDJir6/nuJUKMtmJRwTymJf0i+JZ4x
|
||||||
|
7dJa341p2kHKcHMgOPW7nJQklGBA70ytjUV6/qebS3yIugr/28mwReflg3TJzVDl
|
||||||
|
EOvi6pqbqNbkMuEwGDCmEQIVqgkCAwEAAaNmMGQwDgYDVR0PAQH/BAQDAgEGMBIG
|
||||||
|
A1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFAu93/4k5xbWOsgdCdn+/KdiRuit
|
||||||
|
MB8GA1UdIwQYMBaAFE4C7qw+9hXITO0s9QXBj5yECEmDMA0GCSqGSIb3DQEBBQUA
|
||||||
|
A4IBAQBlcjSyscpPjf5+MgzMuAsCxByqUt+WFspwcMCpwdaBeHOPSQrXNqX2Sk6P
|
||||||
|
kth6oCivA64trWo8tFMvPYlUA1FYVD5WpN0kCK+P5pD4KHlaDsXhuhClJzp/OP8t
|
||||||
|
pOyUr5109RHLxqoKB5J5m1XA7rgcFjnMxwBSWFe3/4uMk/+4T53YfCVXuc6QV3i7
|
||||||
|
I/2LAJwFf//pTtt6fZenYfCsahnr2nvrNRNyAxcfvGZ/4Opn/mJtR6R/AjvQZHiR
|
||||||
|
bkRNKF2GW0ueK5W4FkZVZVhhX9xh1Aj2Ollb+lbOqADaVj+AT3PoJPZ3MPQHKCXm
|
||||||
|
xwG0LOLlRr/TfD6li1AfOVTAJXv9
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIEATCCAumgAwIBAgIBRjANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||||
|
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||||
|
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||||
|
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNTAyMDUyMjAzMjRaFw0y
|
||||||
|
MDAzMDUyMjAzMjRaMIGUMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||||
|
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||||
|
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzElMCMGA1UEAwwcQW1hem9uIFJE
|
||||||
|
UyBhcC1zb3V0aGVhc3QtMiBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
|
||||||
|
ggEBAJqBAJutz69hFOh3BtLHZTbwE8eejGGKayn9hu98YMDPzWzGXWCmW+ZYWELA
|
||||||
|
cY3cNWNF8K4FqKXFr2ssorBYim1UtYFX8yhydT2hMD5zgQ2sCGUpuidijuPA6zaq
|
||||||
|
Z3tdhVR94f0q8mpwpv2zqR9PcqaGDx2VR1x773FupRPRo7mEW1vC3IptHCQlP/zE
|
||||||
|
7jQiLl28bDIH2567xg7e7E9WnZToRnhlYdTaDaJsHTzi5mwILi4cihSok7Shv/ME
|
||||||
|
hnukvxeSPUpaVtFaBhfBqq055ePq9I+Ns4KGreTKMhU0O9fkkaBaBmPaFgmeX/XO
|
||||||
|
n2AX7gMouo3mtv34iDTZ0h6YCGkCAwEAAaNmMGQwDgYDVR0PAQH/BAQDAgEGMBIG
|
||||||
|
A1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFIlQnY0KHYWn1jYumSdJYfwj/Nfw
|
||||||
|
MB8GA1UdIwQYMBaAFE4C7qw+9hXITO0s9QXBj5yECEmDMA0GCSqGSIb3DQEBBQUA
|
||||||
|
A4IBAQA0wVU6/l41cTzHc4azc4CDYY2Wd90DFWiH9C/mw0SgToYfCJ/5Cfi0NT/Y
|
||||||
|
PRnk3GchychCJgoPA/k9d0//IhYEAIiIDjyFVgjbTkKV3sh4RbdldKVOUB9kumz/
|
||||||
|
ZpShplsGt3z4QQiVnKfrAgqxWDjR0I0pQKkxXa6Sjkicos9LQxVtJ0XA4ieG1E7z
|
||||||
|
zJr+6t80wmzxvkInSaWP3xNJK9azVRTrgQZQlvkbpDbExl4mNTG66VD3bAp6t3Wa
|
||||||
|
B49//uDdfZmPkqqbX+hsxp160OH0rxJppwO3Bh869PkDnaPEd/Pxw7PawC+li0gi
|
||||||
|
NRV8iCEx85aFxcyOhqn0WZOasxee
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIID/zCCAuegAwIBAgIBRzANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||||
|
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||||
|
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||||
|
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNTAyMDUyMjAzMzFaFw0y
|
||||||
|
MDAzMDUyMjAzMzFaMIGSMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||||
|
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||||
|
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEjMCEGA1UEAwwaQW1hem9uIFJE
|
||||||
|
UyBldS1jZW50cmFsLTEgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIB
|
||||||
|
AQDFtP2dhSLuaPOI4ZrrPWsK4OY9ocQBp3yApH1KJYmI9wpQKZG/KCH2E6Oo7JAw
|
||||||
|
QORU519r033T+FO2Z7pFPlmz1yrxGXyHpJs8ySx3Yo5S8ncDCdZJCLmtPiq/hahg
|
||||||
|
5/0ffexMFUCQaYicFZsrJ/cStdxUV+tSw2JQLD7UxS9J97LQWUPyyG+ZrjYVTVq+
|
||||||
|
zudnFmNSe4QoecXMhAFTGJFQXxP7nhSL9Ao5FGgdXy7/JWeWdQIAj8ku6cBDKPa6
|
||||||
|
Y6kP+ak+In+Lye8z9qsCD/afUozfWjPR2aA4JoIZVF8dNRShIMo8l0XfgfM2q0+n
|
||||||
|
ApZWZ+BjhIO5XuoUgHS3D2YFAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNV
|
||||||
|
HRMBAf8ECDAGAQH/AgEAMB0GA1UdDgQWBBRm4GsWIA/M6q+tK8WGHWDGh2gcyTAf
|
||||||
|
BgNVHSMEGDAWgBROAu6sPvYVyEztLPUFwY+chAhJgzANBgkqhkiG9w0BAQUFAAOC
|
||||||
|
AQEAHpMmeVQNqcxgfQdbDIi5UIy+E7zZykmtAygN1XQrvga9nXTis4kOTN6g5/+g
|
||||||
|
HCx7jIXeNJzAbvg8XFqBN84Quqgpl/tQkbpco9Jh1HDs558D5NnZQxNqH5qXQ3Mm
|
||||||
|
uPgCw0pYcPOa7bhs07i+MdVwPBsX27CFDtsgAIru8HvKxY1oTZrWnyIRo93tt/pk
|
||||||
|
WuItVMVHjaQZVfTCow0aDUbte6Vlw82KjUFq+n2NMSCJDiDKsDDHT6BJc4AJHIq3
|
||||||
|
/4Z52MSC9KMr0yAaaoWfW/yMEj9LliQauAgwVjArF4q78rxpfKTG9Rfd8U1BZANP
|
||||||
|
7FrFMN0ThjfA1IvmOYcgskY5bQ==
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIID/DCCAuSgAwIBAgIBSDANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||||
|
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||||
|
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||||
|
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNTAyMDUyMjAzMzVaFw0y
|
||||||
|
MDAzMDUyMjAzMzVaMIGPMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||||
|
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||||
|
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEgMB4GA1UEAwwXQW1hem9uIFJE
|
||||||
|
UyBldS13ZXN0LTEgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCx
|
||||||
|
PdbqQ0HKRj79Pmocxvjc+P6i4Ux24kgFIl+ckiir1vzkmesc3a58gjrMlCksEObt
|
||||||
|
Yihs5IhzEq1ePT0gbfS9GYFp34Uj/MtPwlrfCBWG4d2TcrsKRHr1/EXUYhWqmdrb
|
||||||
|
RhX8XqoRhVkbF/auzFSBhTzcGGvZpQ2KIaxRcQfcXlMVhj/pxxAjh8U4F350Fb0h
|
||||||
|
nX1jw4/KvEreBL0Xb2lnlGTkwVxaKGSgXEnOgIyOFdOQc61vdome0+eeZsP4jqeR
|
||||||
|
TGYJA9izJsRbe2YJxHuazD+548hsPlM3vFzKKEVURCha466rAaYAHy3rKur3HYQx
|
||||||
|
Yt+SoKcEz9PXuSGj96ejAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMB
|
||||||
|
Af8ECDAGAQH/AgEAMB0GA1UdDgQWBBTebg//h2oeXbZjQ4uuoiuLYzuiPDAfBgNV
|
||||||
|
HSMEGDAWgBROAu6sPvYVyEztLPUFwY+chAhJgzANBgkqhkiG9w0BAQUFAAOCAQEA
|
||||||
|
TikPaGeZasTPw+4RBemlsyPAjtFFQLo7ddaFdORLgdEysVf8aBqndvbA6MT/v4lj
|
||||||
|
GtEtUdF59ZcbWOrVm+fBZ2h/jYJ59dYF/xzb09nyRbdMSzB9+mkSsnOMqluq5y8o
|
||||||
|
DY/PfP2vGhEg/2ZncRC7nlQU1Dm8F4lFWEiQ2fi7O1cW852Vmbq61RIfcYsH/9Ma
|
||||||
|
kpgk10VZ75b8m3UhmpZ/2uRY+JEHImH5WpcTJ7wNiPNJsciZMznGtrgOnPzYco8L
|
||||||
|
cDleOASIZifNMQi9PKOJKvi0ITz0B/imr8KBsW0YjZVJ54HMa7W1lwugSM7aMAs+
|
||||||
|
E3Sd5lS+SHwWaOCHwhOEVA==
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIID/DCCAuSgAwIBAgIBSTANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||||
|
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||||
|
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||||
|
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNTAyMDUyMjAzNDBaFw0y
|
||||||
|
MDAzMDUyMjAzNDBaMIGPMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||||
|
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||||
|
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEgMB4GA1UEAwwXQW1hem9uIFJE
|
||||||
|
UyBzYS1lYXN0LTEgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCU
|
||||||
|
X4OBnQ5xA6TLJAiFEI6l7bUWjoVJBa/VbMdCCSs2i2dOKmqUaXu2ix2zcPILj3lZ
|
||||||
|
GMk3d/2zvTK/cKhcFrewHUBamTeVHdEmynhMQamqNmkM4ptYzFcvEUw1TGxHT4pV
|
||||||
|
Q6gSN7+/AJewQvyHexHo8D0+LDN0/Wa9mRm4ixCYH2CyYYJNKaZt9+EZfNu+PPS4
|
||||||
|
8iB0TWH0DgQkbWMBfCRgolLLitAZklZ4dvdlEBS7evN1/7ttBxUK6SvkeeSx3zBl
|
||||||
|
ww3BlXqc3bvTQL0A+RRysaVyFbvtp9domFaDKZCpMmDFAN/ntx215xmQdrSt+K3F
|
||||||
|
cXdGQYHx5q410CAclGnbAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMB
|
||||||
|
Af8ECDAGAQH/AgEAMB0GA1UdDgQWBBT6iVWnm/uakS+tEX2mzIfw+8JL0zAfBgNV
|
||||||
|
HSMEGDAWgBROAu6sPvYVyEztLPUFwY+chAhJgzANBgkqhkiG9w0BAQUFAAOCAQEA
|
||||||
|
FmDD+QuDklXn2EgShwQxV13+txPRuVdOSrutHhoCgMwFWCMtPPtBAKs6KPY7Guvw
|
||||||
|
DpJoZSehDiOfsgMirjOWjvfkeWSNvKfjWTVneX7pZD9W5WPnsDBvTbCGezm+v87z
|
||||||
|
b+ZM2ZMo98m/wkMcIEAgdSKilR2fuw8rLkAjhYFfs0A7tDgZ9noKwgHvoE4dsrI0
|
||||||
|
KZYco6DlP/brASfHTPa2puBLN9McK3v+h0JaSqqm5Ro2Bh56tZkQh8AWy/miuDuK
|
||||||
|
3+hNEVdxosxlkM1TPa1DGj0EzzK0yoeerXuH2HX7LlCrrxf6/wdKnjR12PMrLQ4A
|
||||||
|
pCqkcWw894z6bV9MAvKe6A==
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIID/DCCAuSgAwIBAgIBQzANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||||
|
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||||
|
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||||
|
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNTAyMDUyMTU0MDRaFw0y
|
||||||
|
MDAzMDUyMTU0MDRaMIGPMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||||
|
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||||
|
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEgMB4GA1UEAwwXQW1hem9uIFJE
|
||||||
|
UyB1cy1lYXN0LTEgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDI
|
||||||
|
UIuwh8NusKHk1SqPXcP7OqxY3S/M2ZyQWD3w7Bfihpyyy/fc1w0/suIpX3kbMhAV
|
||||||
|
2ESwged2/2zSx4pVnjp/493r4luhSqQYzru78TuPt9bhJIJ51WXunZW2SWkisSaf
|
||||||
|
USYUzVN9ezR/bjXTumSUQaLIouJt3OHLX49s+3NAbUyOI8EdvgBQWD68H1epsC0n
|
||||||
|
CI5s+pIktyOZ59c4DCDLQcXErQ+tNbDC++oct1ANd/q8p9URonYwGCGOBy7sbCYq
|
||||||
|
9eVHh1Iy2M+SNXddVOGw5EuruvHoCIQyOz5Lz4zSuZA9dRbrfztNOpezCNYu6NKM
|
||||||
|
n+hzcvdiyxv77uNm8EaxAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMB
|
||||||
|
Af8ECDAGAQH/AgEAMB0GA1UdDgQWBBQSQG3TmMe6Sa3KufaPBa72v4QFDzAfBgNV
|
||||||
|
HSMEGDAWgBROAu6sPvYVyEztLPUFwY+chAhJgzANBgkqhkiG9w0BAQUFAAOCAQEA
|
||||||
|
L/mOZfB3187xTmjOHMqN2G2oSKHBKiQLM9uv8+97qT+XR+TVsBT6b3yoPpMAGhHA
|
||||||
|
Pc7nxAF5gPpuzatx0OTLPcmYucFmfqT/1qA5WlgCnMNtczyNMH97lKFTNV7Njtek
|
||||||
|
jWEzAEQSyEWrkNpNlC4j6kMYyPzVXQeXUeZTgJ9FNnVZqmvfjip2N22tawMjrCn5
|
||||||
|
7KN/zN65EwY2oO9XsaTwwWmBu3NrDdMbzJnbxoWcFWj4RBwanR1XjQOVNhDwmCOl
|
||||||
|
/1Et13b8CPyj69PC8BOVU6cfTSx8WUVy0qvYOKHNY9Bqa5BDnIL3IVmUkeTlM1mt
|
||||||
|
enRpyBj+Bk9rh/ICdiRKmA==
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIID/DCCAuSgAwIBAgIBSjANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||||
|
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||||
|
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||||
|
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNTAyMDUyMjAzNDVaFw0y
|
||||||
|
MDAzMDUyMjAzNDVaMIGPMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||||
|
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||||
|
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEgMB4GA1UEAwwXQW1hem9uIFJE
|
||||||
|
UyB1cy13ZXN0LTEgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDE
|
||||||
|
Dhw+uw/ycaiIhhyu2pXFRimq0DlB8cNtIe8hdqndH8TV/TFrljNgR8QdzOgZtZ9C
|
||||||
|
zzQ2GRpInN/qJF6slEd6wO+6TaDBQkPY+07TXNt52POFUhdVkhJXHpE2BS7Xn6J7
|
||||||
|
7RFAOeG1IZmc2DDt+sR1BgXzUqHslQGfFYNS0/MBO4P+ya6W7IhruB1qfa4HiYQS
|
||||||
|
dbe4MvGWnv0UzwAqdR7OF8+8/5c58YXZIXCO9riYF2ql6KNSL5cyDPcYK5VK0+Q9
|
||||||
|
VI6vuJHSMYcF7wLePw8jtBktqAFE/wbdZiIHhZvNyiNWPPNTGUmQbaJ+TzQEHDs5
|
||||||
|
8en+/W7JKnPyBOkxxENbAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMB
|
||||||
|
Af8ECDAGAQH/AgEAMB0GA1UdDgQWBBS0nw/tFR9bCjgqWTPJkyy4oOD8bzAfBgNV
|
||||||
|
HSMEGDAWgBROAu6sPvYVyEztLPUFwY+chAhJgzANBgkqhkiG9w0BAQUFAAOCAQEA
|
||||||
|
CXGAY3feAak6lHdqj6+YWjy6yyUnLK37bRxZDsyDVXrPRQaXRzPTzx79jvDwEb/H
|
||||||
|
Q/bdQ7zQRWqJcbivQlwhuPJ4kWPUZgSt3JUUuqkMsDzsvj/bwIjlrEFDOdHGh0mi
|
||||||
|
eVIngFEjUXjMh+5aHPEF9BlQnB8LfVtKj18e15UDTXFa+xJPFxUR7wDzCfo4WI1m
|
||||||
|
sUMG4q1FkGAZgsoyFPZfF8IVvgCuGdR8z30VWKklFxttlK0eGLlPAyIO0CQxPQlo
|
||||||
|
saNJrHf4tLOgZIWk+LpDhNd9Et5EzvJ3aURUsKY4pISPPF5WdvM9OE59bERwUErd
|
||||||
|
nuOuQWQeeadMceZnauRzJQ==
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIID/DCCAuSgAwIBAgIBSzANBgkqhkiG9w0BAQUFADCBijELMAkGA1UEBhMCVVMx
|
||||||
|
EzARBgNVBAgMCldhc2hpbmd0b24xEDAOBgNVBAcMB1NlYXR0bGUxIjAgBgNVBAoM
|
||||||
|
GUFtYXpvbiBXZWIgU2VydmljZXMsIEluYy4xEzARBgNVBAsMCkFtYXpvbiBSRFMx
|
||||||
|
GzAZBgNVBAMMEkFtYXpvbiBSRFMgUm9vdCBDQTAeFw0xNTAyMDUyMjAzNTBaFw0y
|
||||||
|
MDAzMDUyMjAzNTBaMIGPMQswCQYDVQQGEwJVUzETMBEGA1UECAwKV2FzaGluZ3Rv
|
||||||
|
bjEQMA4GA1UEBwwHU2VhdHRsZTEiMCAGA1UECgwZQW1hem9uIFdlYiBTZXJ2aWNl
|
||||||
|
cywgSW5jLjETMBEGA1UECwwKQW1hem9uIFJEUzEgMB4GA1UEAwwXQW1hem9uIFJE
|
||||||
|
UyB1cy13ZXN0LTIgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDM
|
||||||
|
H58SR48U6jyERC1vYTnub34smf5EQVXyzaTmspWGWGzT31NLNZGSDFaa7yef9kdO
|
||||||
|
mzJsgebR5tXq6LdwlIoWkKYQ7ycUaadtVKVYdI40QcI3cHn0qLFlg2iBXmWp/B+i
|
||||||
|
Z34VuVlCh31Uj5WmhaBoz8t/GRqh1V/aCsf3Wc6jCezH3QfuCjBpzxdOOHN6Ie2v
|
||||||
|
xX09O5qmZTvMoRBAvPkxdaPg/Mi7fxueWTbEVk78kuFbF1jHYw8U1BLILIAhcqlq
|
||||||
|
x4u8nl73t3O3l/soNUcIwUDK0/S+Kfqhwn9yQyPlhb4Wy3pfnZLJdkyHldktnQav
|
||||||
|
9TB9u7KH5Lk0aAYslMLxAgMBAAGjZjBkMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMB
|
||||||
|
Af8ECDAGAQH/AgEAMB0GA1UdDgQWBBT8roM4lRnlFHWMPWRz0zkwFZog1jAfBgNV
|
||||||
|
HSMEGDAWgBROAu6sPvYVyEztLPUFwY+chAhJgzANBgkqhkiG9w0BAQUFAAOCAQEA
|
||||||
|
JwrxwgwmPtcdaU7O7WDdYa4hprpOMamI49NDzmE0s10oGrqmLwZygcWU0jT+fJ+Y
|
||||||
|
pJe1w0CVfKaeLYNsOBVW3X4ZPmffYfWBheZiaiEflq/P6t7/Eg81gaKYnZ/x1Dfa
|
||||||
|
sUYkzPvCkXe9wEz5zdUTOCptDt89rBR9CstL9vE7WYUgiVVmBJffWbHQLtfjv6OF
|
||||||
|
NMb0QME981kGRzc2WhgP71YS2hHd1kXtsoYP1yTu4vThSKsoN4bkiHsaC1cRkLoy
|
||||||
|
0fFA4wpB3WloMEvCDaUvvH1LZlBXTNlwi9KtcwD4tDxkkBt4tQczKLGpQ/nF/W9n
|
||||||
|
8YDWk3IIc1sd0bkZqoau2Q==
|
||||||
|
-----END CERTIFICATE-----
|
||||||
|
|
@ -0,0 +1,12 @@
|
||||||
|
# HEROKU REQUIREMENTS
|
||||||
|
|
||||||
|
-r requirements/common.txt
|
||||||
|
-r requirements/prod.txt
|
||||||
|
|
||||||
|
# Pylibmc must be in this file so that heroku knows it has to install
|
||||||
|
# libmemcached when bootstrapping the containers.
|
||||||
|
# More info here: https://devcenter.heroku.com/articles/memcachier#python
|
||||||
|
pylibmc==1.4.2
|
||||||
|
|
||||||
|
django-pylibmc==0.5.0
|
||||||
|
django-heroku-memcacheify==0.8
|
||||||
|
|
@ -68,8 +68,8 @@ jsonfield==0.9.20
|
||||||
# sha256: HYBs6IKx3VBrKM04OI_9AhHx2ASI7eeJLEwmMF1e7GA
|
# sha256: HYBs6IKx3VBrKM04OI_9AhHx2ASI7eeJLEwmMF1e7GA
|
||||||
mozlog==2.10
|
mozlog==2.10
|
||||||
|
|
||||||
# sha256: EdbpC9WxbfhNw4c3kDfij8HZ-pkkUnv5Oe07q0cSH4w
|
# sha256: nnifUs_v5v9XxzD-_XayNXIzZOVCB12Z34M7zdrOV9c
|
||||||
https://github.com/jeads/datasource/archive/v0.6.tar.gz#egg=datasource
|
https://github.com/jeads/datasource/archive/v0.7.tar.gz#egg=datasource
|
||||||
|
|
||||||
# Required by django-rest-swagger
|
# Required by django-rest-swagger
|
||||||
# sha256: 7wE5rDCRO0Po04ij53blNKIvgZdZZpaNKDaewDOKabc
|
# sha256: 7wE5rDCRO0Po04ij53blNKIvgZdZZpaNKDaewDOKabc
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
python-2.7.9
|
||||||
|
|
@ -11,6 +11,13 @@ from django.core.management.base import BaseCommand
|
||||||
from treeherder.model.derived.base import TreeherderModelBase
|
from treeherder.model.derived.base import TreeherderModelBase
|
||||||
|
|
||||||
|
|
||||||
|
DEFAULT_CREDENTIALS_PATH = os.path.join(
|
||||||
|
os.path.dirname(os.path.dirname(os.path.dirname(__file__))),
|
||||||
|
'data',
|
||||||
|
'credentials.json'
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class Command(BaseCommand):
|
class Command(BaseCommand):
|
||||||
"""Management command to export project credentials."""
|
"""Management command to export project credentials."""
|
||||||
help = "Exports the objectstore Oauth keys for etl data import tasks"
|
help = "Exports the objectstore Oauth keys for etl data import tasks"
|
||||||
|
|
@ -23,27 +30,27 @@ class Command(BaseCommand):
|
||||||
dest='safe',
|
dest='safe',
|
||||||
help="Don't overwrite credentials file if it exists."
|
help="Don't overwrite credentials file if it exists."
|
||||||
),
|
),
|
||||||
|
make_option(
|
||||||
|
'--destination',
|
||||||
|
action='store',
|
||||||
|
default=DEFAULT_CREDENTIALS_PATH,
|
||||||
|
dest='destination',
|
||||||
|
help="Don't overwrite credentials file if it exists."
|
||||||
|
),
|
||||||
|
|
||||||
)
|
)
|
||||||
|
|
||||||
def handle(self, *args, **options):
|
def handle(self, *args, **options):
|
||||||
safe = options.get("safe")
|
safe = options.get('safe')
|
||||||
|
file_path = options.get('destination')
|
||||||
file_path = os.path.join(
|
if file_path == 'stdout':
|
||||||
os.path.dirname(os.path.dirname(os.path.dirname(__file__))),
|
write_credentials(self.stdout)
|
||||||
'data',
|
|
||||||
'credentials.json'
|
|
||||||
)
|
|
||||||
|
|
||||||
if not os.path.isfile(file_path):
|
|
||||||
# If it doesn't exist create it
|
|
||||||
write_credentials(file_path)
|
|
||||||
else:
|
else:
|
||||||
# File already exists, if safe is specified don't do anything
|
if not os.path.isfile(file_path) or not safe:
|
||||||
if not safe:
|
with open(file_path, 'w') as fh:
|
||||||
write_credentials(file_path)
|
write_credentials(fh)
|
||||||
|
|
||||||
|
|
||||||
def write_credentials(file_path):
|
def write_credentials(fh):
|
||||||
immutable_credentials = TreeherderModelBase.get_oauth_credentials()
|
immutable_credentials = TreeherderModelBase.get_oauth_credentials()
|
||||||
with open(file_path, 'w') as keys_fh:
|
fh.write(json.dumps(immutable_credentials))
|
||||||
keys_fh.write(json.dumps(immutable_credentials, indent=4))
|
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,7 @@
|
||||||
# file, you can obtain one at http://mozilla.org/MPL/2.0/.
|
# file, you can obtain one at http://mozilla.org/MPL/2.0/.
|
||||||
|
|
||||||
import simplejson as json
|
import simplejson as json
|
||||||
|
import os
|
||||||
from treeherder import path
|
from treeherder import path
|
||||||
import copy
|
import copy
|
||||||
import logging
|
import logging
|
||||||
|
|
@ -40,25 +41,27 @@ class OAuthCredentials():
|
||||||
|
|
||||||
# Only get the credentials once
|
# Only get the credentials once
|
||||||
if not cls.credentials and not credentials:
|
if not cls.credentials and not credentials:
|
||||||
|
credentials_string = os.environ.get('TREEHERDER_CREDENTIALS', None)
|
||||||
|
if credentials_string:
|
||||||
|
credentials = json.loads(credentials_string)
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
with open(cls.credentials_file) as f:
|
||||||
|
credentials_string = f.read()
|
||||||
|
credentials = json.loads(credentials_string)
|
||||||
|
|
||||||
try:
|
except IOError:
|
||||||
with open(cls.credentials_file) as f:
|
msg = ('Credentials file not found at {0}.'
|
||||||
credentials = f.read()
|
' Try running `manage.py export_project_credentials`'
|
||||||
cls.credentials = json.loads(credentials)
|
' to generate them').format(cls.credentials_file)
|
||||||
|
|
||||||
except IOError:
|
logger.error(msg)
|
||||||
msg = ('Credentials file not found at {0}.'
|
|
||||||
' Try running `manage.py export_project_credentials`'
|
|
||||||
' to generate them').format(cls.credentials_file)
|
|
||||||
|
|
||||||
logger.error(msg)
|
except Exception as e:
|
||||||
|
logger.error(e)
|
||||||
|
raise e
|
||||||
|
|
||||||
except Exception as e:
|
cls.credentials = credentials
|
||||||
logger.error(e)
|
|
||||||
raise e
|
|
||||||
|
|
||||||
else:
|
|
||||||
cls.credentials = credentials
|
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def get_credentials(cls, project):
|
def get_credentials(cls, project):
|
||||||
|
|
|
||||||
|
|
@ -25,24 +25,34 @@ class RefDataManager(object):
|
||||||
procs_path = os.path.join(
|
procs_path = os.path.join(
|
||||||
os.path.dirname(os.path.dirname(__file__)),
|
os.path.dirname(os.path.dirname(__file__)),
|
||||||
'sql', 'reference.json')
|
'sql', 'reference.json')
|
||||||
|
|
||||||
|
master_host_config = {
|
||||||
|
"host": settings.DATABASES['default']['HOST'],
|
||||||
|
"user": settings.DATABASES['default']['USER'],
|
||||||
|
"passwd": settings.DATABASES['default']['PASSWORD']
|
||||||
|
}
|
||||||
|
if 'OPTIONS' in settings.DATABASES['default']:
|
||||||
|
master_host_config.update(settings.DATABASES['default']['OPTIONS'])
|
||||||
|
|
||||||
|
read_host_config = {
|
||||||
|
"host": settings.DATABASES['read_only']['HOST'],
|
||||||
|
"user": settings.DATABASES['read_only']['USER'],
|
||||||
|
"passwd": settings.DATABASES['read_only']['PASSWORD']
|
||||||
|
}
|
||||||
|
if 'OPTIONS' in settings.DATABASES['read_only']:
|
||||||
|
read_host_config.update(settings.DATABASES['read_only']['OPTIONS'])
|
||||||
|
|
||||||
data_source = {
|
data_source = {
|
||||||
'reference': {
|
'reference': {
|
||||||
"hub": "MySQL",
|
"hub": "MySQL",
|
||||||
"master_host": {
|
"master_host": master_host_config,
|
||||||
"host": settings.DATABASES['default']['HOST'],
|
"read_host": read_host_config,
|
||||||
"user": settings.DATABASES['default']['USER'],
|
|
||||||
"passwd": settings.DATABASES['default']['PASSWORD']
|
|
||||||
},
|
|
||||||
"read_host": {
|
|
||||||
"host": settings.DATABASES['read_only']['HOST'],
|
|
||||||
"user": settings.DATABASES['read_only']['USER'],
|
|
||||||
"passwd": settings.DATABASES['read_only']['PASSWORD']
|
|
||||||
},
|
|
||||||
"require_host_type": True,
|
"require_host_type": True,
|
||||||
"default_db": settings.DATABASES['default']['NAME'],
|
"default_db": settings.DATABASES['default']['NAME'],
|
||||||
"procs": [procs_path]
|
"procs": [procs_path]
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
BaseHub.add_data_source(data_source)
|
BaseHub.add_data_source(data_source)
|
||||||
self.dhub = DataHub.get("reference")
|
self.dhub = DataHub.get("reference")
|
||||||
self.DEBUG = settings.DEBUG
|
self.DEBUG = settings.DEBUG
|
||||||
|
|
|
||||||
|
|
@ -5,7 +5,6 @@
|
||||||
from __future__ import unicode_literals
|
from __future__ import unicode_literals
|
||||||
|
|
||||||
import uuid
|
import uuid
|
||||||
import subprocess
|
|
||||||
import os
|
import os
|
||||||
|
|
||||||
from collections import defaultdict
|
from collections import defaultdict
|
||||||
|
|
@ -14,7 +13,7 @@ from datasource.bases.BaseHub import BaseHub
|
||||||
from datasource.hubs.MySQL import MySQL
|
from datasource.hubs.MySQL import MySQL
|
||||||
from django.conf import settings
|
from django.conf import settings
|
||||||
from django.core.cache import cache
|
from django.core.cache import cache
|
||||||
from django.db import models
|
from django.db import models, connection
|
||||||
from django.db.models import Max, Q
|
from django.db.models import Max, Q
|
||||||
from django.contrib.auth.models import User
|
from django.contrib.auth.models import User
|
||||||
from django.utils.encoding import python_2_unicode_compatible
|
from django.utils.encoding import python_2_unicode_compatible
|
||||||
|
|
@ -324,21 +323,29 @@ class Datasource(models.Model):
|
||||||
Return a configured ``DataHub`` using the given SQL procs file.
|
Return a configured ``DataHub`` using the given SQL procs file.
|
||||||
|
|
||||||
"""
|
"""
|
||||||
|
master_host_config = {
|
||||||
|
"host": self.host,
|
||||||
|
"user": settings.TREEHERDER_DATABASE_USER,
|
||||||
|
"passwd": settings.TREEHERDER_DATABASE_PASSWORD,
|
||||||
|
}
|
||||||
|
if 'OPTIONS' in settings.DATABASES['default']:
|
||||||
|
master_host_config.update(settings.DATABASES['default']['OPTIONS'])
|
||||||
|
|
||||||
|
read_host_config = {
|
||||||
|
"host": self.read_only_host,
|
||||||
|
"user": settings.TREEHERDER_RO_DATABASE_USER,
|
||||||
|
"passwd": settings.TREEHERDER_RO_DATABASE_PASSWORD,
|
||||||
|
}
|
||||||
|
if 'OPTIONS' in settings.DATABASES['read_only']:
|
||||||
|
read_host_config.update(settings.DATABASES['read_only']['OPTIONS'])
|
||||||
|
|
||||||
data_source = {
|
data_source = {
|
||||||
self.key: {
|
self.key: {
|
||||||
# @@@ this should depend on self.type
|
# @@@ this should depend on self.type
|
||||||
# @@@ shouldn't have to specify this here and below
|
# @@@ shouldn't have to specify this here and below
|
||||||
"hub": "MySQL",
|
"hub": "MySQL",
|
||||||
"master_host": {
|
"master_host": master_host_config,
|
||||||
"host": self.host,
|
"read_host": read_host_config,
|
||||||
"user": settings.TREEHERDER_DATABASE_USER,
|
|
||||||
"passwd": settings.TREEHERDER_DATABASE_PASSWORD,
|
|
||||||
},
|
|
||||||
"read_host": {
|
|
||||||
"host": self.read_only_host,
|
|
||||||
"user": settings.TREEHERDER_RO_DATABASE_USER,
|
|
||||||
"passwd": settings.TREEHERDER_RO_DATABASE_PASSWORD,
|
|
||||||
},
|
|
||||||
"require_host_type": True,
|
"require_host_type": True,
|
||||||
"default_db": self.name,
|
"default_db": self.name,
|
||||||
"procs": [
|
"procs": [
|
||||||
|
|
@ -364,10 +371,8 @@ class Datasource(models.Model):
|
||||||
``DATABASE_PASSWORD`` exists on it and has permissions to
|
``DATABASE_PASSWORD`` exists on it and has permissions to
|
||||||
create databases.
|
create databases.
|
||||||
"""
|
"""
|
||||||
from django.conf import settings
|
|
||||||
import MySQLdb
|
import MySQLdb
|
||||||
DB_USER = settings.DATABASES["default"]["USER"]
|
|
||||||
DB_PASS = settings.DATABASES["default"]["PASSWORD"]
|
|
||||||
if self.type.lower().startswith("mysql-"):
|
if self.type.lower().startswith("mysql-"):
|
||||||
engine = self.type[len("mysql-"):]
|
engine = self.type[len("mysql-"):]
|
||||||
elif self.type.lower() == "mysql":
|
elif self.type.lower() == "mysql":
|
||||||
|
|
@ -384,63 +389,27 @@ class Datasource(models.Model):
|
||||||
"project_{0}_1.sql.tmpl".format(self.contenttype),
|
"project_{0}_1.sql.tmpl".format(self.contenttype),
|
||||||
)
|
)
|
||||||
|
|
||||||
conn = MySQLdb.connect(
|
|
||||||
host=self.host,
|
|
||||||
user=DB_USER,
|
|
||||||
passwd=DB_PASS,
|
|
||||||
)
|
|
||||||
filterwarnings('ignore', category=MySQLdb.Warning)
|
filterwarnings('ignore', category=MySQLdb.Warning)
|
||||||
cur = conn.cursor()
|
with connection.cursor() as cursor:
|
||||||
cur.execute("CREATE DATABASE IF NOT EXISTS {0}".format(self.name))
|
cursor.execute("CREATE DATABASE IF NOT EXISTS {0}".format(self.name))
|
||||||
conn.close()
|
cursor.execute("USE {0}".format(self.name))
|
||||||
|
try:
|
||||||
|
with open(schema_file) as f:
|
||||||
|
# set the engine to use
|
||||||
|
sql = f.read().format(engine=engine)
|
||||||
|
statement_list = sql.split(";")
|
||||||
|
for statement in statement_list:
|
||||||
|
cursor.execute(statement)
|
||||||
|
finally:
|
||||||
|
cursor.execute("USE {0}".format(
|
||||||
|
settings.TREEHERDER_DATABASE_NAME
|
||||||
|
))
|
||||||
|
|
||||||
resetwarnings()
|
resetwarnings()
|
||||||
|
|
||||||
# MySQLdb provides no way to execute an entire SQL file in bulk, so we
|
|
||||||
# have to shell out to the commandline client.
|
|
||||||
with open(schema_file) as f:
|
|
||||||
# set the engine to use
|
|
||||||
sql = f.read().format(engine=engine)
|
|
||||||
|
|
||||||
args = [
|
|
||||||
"mysql",
|
|
||||||
"--host={0}".format(self.host),
|
|
||||||
"--user={0}".format(DB_USER),
|
|
||||||
]
|
|
||||||
if DB_PASS:
|
|
||||||
args.append(
|
|
||||||
"--password={0}".format(
|
|
||||||
DB_PASS)
|
|
||||||
)
|
|
||||||
args.append(self.name)
|
|
||||||
proc = subprocess.Popen(
|
|
||||||
args,
|
|
||||||
stdin=subprocess.PIPE,
|
|
||||||
stdout=subprocess.PIPE,
|
|
||||||
stderr=subprocess.STDOUT,
|
|
||||||
)
|
|
||||||
(output, _) = proc.communicate(sql)
|
|
||||||
if proc.returncode:
|
|
||||||
raise IOError(
|
|
||||||
"Unable to set up schema for datasource {0}: "
|
|
||||||
"mysql returned code {1}, output follows:\n\n{2}".format(
|
|
||||||
self.key, proc.returncode, output
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
def delete_db(self):
|
def delete_db(self):
|
||||||
from django.conf import settings
|
with connection.cursor() as cursor:
|
||||||
import MySQLdb
|
cursor.execute("DROP DATABASE {0}".format(self.name))
|
||||||
DB_USER = settings.DATABASES["default"]["USER"]
|
|
||||||
DB_PASS = settings.DATABASES["default"]["PASSWORD"]
|
|
||||||
conn = MySQLdb.connect(
|
|
||||||
host=self.host,
|
|
||||||
user=DB_USER,
|
|
||||||
passwd=DB_PASS,
|
|
||||||
)
|
|
||||||
|
|
||||||
cur = conn.cursor()
|
|
||||||
cur.execute("DROP DATABASE {0}".format(self.name))
|
|
||||||
conn.close()
|
|
||||||
|
|
||||||
def delete(self, *args, **kwargs):
|
def delete(self, *args, **kwargs):
|
||||||
self.delete_db()
|
self.delete_db()
|
||||||
|
|
@ -451,33 +420,18 @@ class Datasource(models.Model):
|
||||||
Truncate all tables in the db self refers to.
|
Truncate all tables in the db self refers to.
|
||||||
Skip_list is a list of table names to skip truncation.
|
Skip_list is a list of table names to skip truncation.
|
||||||
"""
|
"""
|
||||||
from django.conf import settings
|
|
||||||
import MySQLdb
|
|
||||||
|
|
||||||
skip_list = set(skip_list or [])
|
skip_list = set(skip_list or [])
|
||||||
|
|
||||||
DB_USER = settings.DATABASES["default"]["USER"]
|
with connection.cursor() as cursor:
|
||||||
DB_PASS = settings.DATABASES["default"]["PASSWORD"]
|
cursor.execute("SET FOREIGN_KEY_CHECKS = 0")
|
||||||
|
cursor.execute("SHOW TABLES")
|
||||||
conn = MySQLdb.connect(
|
for table, in cursor.fetchall():
|
||||||
host=self.host,
|
# if there is a skip_list, then skip any table with matching name
|
||||||
user=DB_USER,
|
if table.lower() not in skip_list:
|
||||||
passwd=DB_PASS,
|
# needed to use backticks around table name, because if the
|
||||||
db=self.name,
|
# table name is a keyword (like "option") then this will fail
|
||||||
)
|
cursor.execute("TRUNCATE TABLE `{0}`".format(table))
|
||||||
cur = conn.cursor()
|
cursor.execute("SET FOREIGN_KEY_CHECKS = 1")
|
||||||
cur.execute("SET FOREIGN_KEY_CHECKS = 0")
|
|
||||||
cur.execute("SHOW TABLES")
|
|
||||||
|
|
||||||
for table, in cur.fetchall():
|
|
||||||
# if there is a skip_list, then skip any table with matching name
|
|
||||||
if table.lower() not in skip_list:
|
|
||||||
# needed to use backticks around table name, because if the
|
|
||||||
# table name is a keyword (like "option") then this will fail
|
|
||||||
cur.execute("TRUNCATE TABLE `{0}`".format(table))
|
|
||||||
|
|
||||||
cur.execute("SET FOREIGN_KEY_CHECKS = 1")
|
|
||||||
conn.close()
|
|
||||||
|
|
||||||
|
|
||||||
@python_2_unicode_compatible
|
@python_2_unicode_compatible
|
||||||
|
|
|
||||||
|
|
@ -298,7 +298,7 @@ REST_FRAMEWORK = {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
SITE_URL = "http://local.treeherder.mozilla.org"
|
SITE_URL = os.environ.get("TREEHERDER_SITE_URL", "http://local.treeherder.mozilla.org")
|
||||||
|
|
||||||
BUILDAPI_PENDING_URL = "https://secure.pub.build.mozilla.org/builddata/buildjson/builds-pending.js"
|
BUILDAPI_PENDING_URL = "https://secure.pub.build.mozilla.org/builddata/buildjson/builds-pending.js"
|
||||||
BUILDAPI_RUNNING_URL = "https://secure.pub.build.mozilla.org/builddata/buildjson/builds-running.js"
|
BUILDAPI_RUNNING_URL = "https://secure.pub.build.mozilla.org/builddata/buildjson/builds-running.js"
|
||||||
|
|
@ -379,6 +379,16 @@ DATABASES = {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Setup ssl connection for aws rds.
|
||||||
|
if 'IS_HEROKU' in os.environ:
|
||||||
|
ca_path = '/app/deployment/aws/combined-ca-bundle.pem'
|
||||||
|
for db_name in DATABASES:
|
||||||
|
DATABASES[db_name]['OPTIONS'] = {
|
||||||
|
'ssl': {
|
||||||
|
'ca': ca_path
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
# TREEHERDER_MEMCACHED is a string of comma-separated address:port pairs
|
# TREEHERDER_MEMCACHED is a string of comma-separated address:port pairs
|
||||||
MEMCACHED_LOCATION = TREEHERDER_MEMCACHED.strip(',').split(',')
|
MEMCACHED_LOCATION = TREEHERDER_MEMCACHED.strip(',').split(',')
|
||||||
|
|
||||||
|
|
@ -412,6 +422,18 @@ BROKER_URL = 'amqp://{0}:{1}@{2}:{3}/{4}'.format(
|
||||||
RABBITMQ_VHOST
|
RABBITMQ_VHOST
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# This code handles the memcachier service on heroku.
|
||||||
|
if "IS_HEROKU" in os.environ:
|
||||||
|
from memcacheify import memcacheify
|
||||||
|
CACHES['default'].update(
|
||||||
|
memcacheify().get('default')
|
||||||
|
)
|
||||||
|
|
||||||
|
if "CLOUDAMQP_URL" in os.environ:
|
||||||
|
BROKER_URL = os.environ["CLOUDAMQP_URL"]
|
||||||
|
BROKER_POOL_LIMIT = 1
|
||||||
|
|
||||||
|
|
||||||
CELERY_IGNORE_RESULT = True
|
CELERY_IGNORE_RESULT = True
|
||||||
|
|
||||||
API_HOSTNAME = SITE_URL
|
API_HOSTNAME = SITE_URL
|
||||||
|
|
|
||||||
|
|
@ -42,6 +42,10 @@ application = get_wsgi_application()
|
||||||
if newrelic:
|
if newrelic:
|
||||||
application = newrelic.agent.wsgi_application()(application)
|
application = newrelic.agent.wsgi_application()(application)
|
||||||
|
|
||||||
|
# Fix django closing connection to MemCachier after every request (#11331)
|
||||||
|
from django.core.cache.backends.memcached import BaseMemcachedCache
|
||||||
|
BaseMemcachedCache.close = lambda self, **kwargs: None
|
||||||
|
|
||||||
# Apply WSGI middleware here.
|
# Apply WSGI middleware here.
|
||||||
# from helloworld.wsgi import HelloWorldApplication
|
# from helloworld.wsgi import HelloWorldApplication
|
||||||
# application = HelloWorldApplication(application)
|
# application = HelloWorldApplication(application)
|
||||||
|
|
|
||||||
Загрузка…
Ссылка в новой задаче