зеркало из https://github.com/mozilla/coversheet.git
Make use of Jenkins for Coversheet (#22)
This commit is contained in:
Cosmin Malutan
Henrik Skupin
Родитель
5797da9f90
Коммит
438f2a081c
|
|
@ -1,3 +1,9 @@
|
|||
*.DS_STORE
|
||||
*.swp
|
||||
*.egg-info
|
||||
.DS_Store
|
||||
*.pyc
|
||||
*.bak
|
||||
jenkins-env/
|
||||
jenkins.out
|
||||
*.log
|
||||
*.war
|
||||
log/
|
||||
virtualenv-*
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
language: python
|
||||
python: 2.7
|
||||
script: ./run_tests.sh
|
||||
notifications:
|
||||
email:
|
||||
- dev-automation@lists.mozilla.org
|
||||
irc:
|
||||
- "irc.mozilla.org#automation"
|
||||
1
README
1
README
|
|
@ -1 +0,0 @@
|
|||
See https://bugzilla.mozilla.org/show_bug.cgi?id=740228
|
||||
|
|
@ -0,0 +1,106 @@
|
|||
# coversheet
|
||||
Coversheet is a CI system for TPS, which allows to run tests for each daily
|
||||
build of Firefox across all platforms.
|
||||
|
||||
## Setup
|
||||
Before you can start the system the following commands have to be performed:
|
||||
|
||||
```bash
|
||||
git clone git://github.com/mozilla/coversheet.git
|
||||
cd coversheet
|
||||
./setup.sh
|
||||
```
|
||||
|
||||
You will need to have the Python header files installed:
|
||||
|
||||
* Ubuntu: Install the package via: `apt-get install python-dev`
|
||||
* OSX, Windows: Install the latest [Python 2.7](http://www.python.org/getit/)
|
||||
|
||||
## Startup
|
||||
To start Jenkins simply run `./start.py` from the coversheet directory. You
|
||||
can tell when Jenkins is running by looking out for "Jenkins is fully up and
|
||||
running" in the console output. You will also be able to view the web dashboard
|
||||
by pointing your browser at http://localhost:8080/
|
||||
|
||||
## Jenkins URL
|
||||
If you intend to connect to this Jenkins instance from another machine (for
|
||||
example connecting additional nodes) you will need to update the `Jenkins URL`
|
||||
to the IP or DNS name. This can be found in http://localhost:8080/configure
|
||||
under the section headed "Jenkins Location".
|
||||
|
||||
## Adding new Nodes
|
||||
To add Jenkins slaves to your master you have to create new nodes. You can use
|
||||
one of the example nodes (Windows XP and Ubuntu) as a template. Once done the
|
||||
nodes have to be connected to the master. Therefore Java has to be installed on
|
||||
the node first.
|
||||
|
||||
### Windows:
|
||||
Go to [www.java.com/download/](http://www.java.com/download/) and install the
|
||||
latest version of Java JRE. Also make sure that the UAC is completely disabled,
|
||||
and the screensaver and any energy settings have been turned off.
|
||||
|
||||
### Linux (Ubuntu):
|
||||
Open the terminal or any other package manager and install the following
|
||||
packages:
|
||||
|
||||
```bash
|
||||
sudo add-apt-repository ppa:webupd8team/java
|
||||
sudo apt-get update
|
||||
sudo apt-get install oracle-java7-installer
|
||||
```
|
||||
|
||||
Also make sure that the screensaver and any energy settings have been turned
|
||||
off.
|
||||
|
||||
After Java has been installed open the appropriate node within Jenkins from the
|
||||
nodes web browser like:
|
||||
|
||||
http://IP:8080/computer/windows_xp_32_01/
|
||||
|
||||
Now click the `Launch` button and the node should automatically connect to the
|
||||
master. It will be used once a job for this type of platform has been requested
|
||||
by the Pulse consumer.
|
||||
|
||||
## Using the Jenkins master as executor
|
||||
If you want that the master node also executes jobs you will have to update its
|
||||
labels and add/modify the appropriate platforms, e.g. `master mac 10.7 64bit`
|
||||
for Mac OS X 10.7.
|
||||
|
||||
## Testing changes
|
||||
In order to check that patches will apply and no Jenkins configuration changes
|
||||
are missing from your changes you can run the `run_tests.sh` script. This uses
|
||||
[Selenium](http://code.google.com/p/selenium/) and
|
||||
[PhantomJS](http://phantomjs.org/) to save the configuration for each job and
|
||||
reports any unexpected changes. Note that you will need to
|
||||
[download](http://phantomjs.org/download.html) PhantomJS and put it in your
|
||||
path in order for these tests to run.
|
||||
|
||||
## Merging branches
|
||||
The main development on the coversheet code happens on the master branch. In
|
||||
not yet specified intervals we are merging changesets into the staging branch.
|
||||
It is used for testing all the new features before those go live on production.
|
||||
When running those merge tasks you will have to obey the following steps:
|
||||
|
||||
1. Select the appropriate target branch
|
||||
2. Run 'git rebase master' for staging or 'git rebase staging' for production
|
||||
3. Run 'git pull' for the remote branch you want to push to
|
||||
4. Ensure the merged patches are on top of the branch
|
||||
5. Ensure that the Jenkins patch can be applied by running 'patch --dry-run -p1
|
||||
<config/%BRANCH%/jenkins.patch'
|
||||
6. Run 'git push' for the remote branch
|
||||
|
||||
For emergency fixes we are using cherry-pick to port individual fixes to the
|
||||
staging and production branch:
|
||||
|
||||
1. Select the appropriate target branch
|
||||
2. Run 'git cherry-pick %changeset%' to pick the specific changeset for the
|
||||
current branch
|
||||
3. Run 'git push' for the remote branch
|
||||
|
||||
Once the changes have been landed you will have to update the staging or
|
||||
production machines. Run the following steps:
|
||||
|
||||
1. Run 'git reset --hard' to remove the locally applied patch
|
||||
2. Pull the latest changes with 'git pull'
|
||||
3. Apply the Jenkins patch with 'patch -p1 <config/%BRANCH%/jenkins.patch'
|
||||
4. Restart Jenkins
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
{
|
||||
"sync_account": {
|
||||
"username": "crossweaveservices@mozilla.com",
|
||||
"password": "crossweaveservicescrossweaveservices",
|
||||
"passphrase": "r-jwcbc-zgf42-fjn72-p5vpp-iypmi"
|
||||
},
|
||||
"fx_account": {
|
||||
"username": "crossweaveservices@restmail.net",
|
||||
"password": "crossweaveservicescrossweaveservices"
|
||||
},
|
||||
"email": {
|
||||
"username": "crossweave@mozilla.com",
|
||||
"password": "",
|
||||
"passednotificationlist": ["crossweave@mozilla.com"],
|
||||
"notificationlist": ["crossweave@mozilla.com"]
|
||||
},
|
||||
"platform": "win32",
|
||||
"os": "win7",
|
||||
"es": "localhost:9200",
|
||||
"serverURL": null,
|
||||
"testdir": "__TESTDIR__",
|
||||
"extensiondir": "__EXTENSIONDIR__"
|
||||
}
|
||||
|
|
@ -1,38 +0,0 @@
|
|||
# ***** BEGIN LICENSE BLOCK *****
|
||||
# Version: MPL 1.1/GPL 2.0/LGPL 2.1
|
||||
#
|
||||
# The contents of this file are subject to the Mozilla Public License Version
|
||||
# 1.1 (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
# http://www.mozilla.org/MPL/
|
||||
#
|
||||
# Software distributed under the License is distributed on an "AS IS" basis,
|
||||
# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
|
||||
# for the specific language governing rights and limitations under the
|
||||
# License.
|
||||
#
|
||||
# The Original Code is TPS.
|
||||
#
|
||||
# The Initial Developer of the Original Code is
|
||||
# Mozilla Foundation.
|
||||
# Portions created by the Initial Developer are Copyright (C) 2011
|
||||
# the Initial Developer. All Rights Reserved.
|
||||
#
|
||||
# Contributor(s):
|
||||
# Sam Garrett <samdgarrett@gmail.com>
|
||||
#
|
||||
# Alternatively, the contents of this file may be used under the terms of
|
||||
# either the GNU General Public License Version 2 or later (the "GPL"), or
|
||||
# the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
|
||||
# in which case the provisions of the GPL or the LGPL are applicable instead
|
||||
# of those above. If you wish to allow use of your version of this file only
|
||||
# under the terms of either the GPL or the LGPL, and not to allow others to
|
||||
# use your version of this file under the terms of the MPL, indicate your
|
||||
# decision by deleting the provisions above and replace them with the notice
|
||||
# and other provisions required by the GPL or the LGPL. If you do not delete
|
||||
# the provisions above, a recipient may use your version of this file under
|
||||
# the terms of any one of the MPL, the GPL or the LGPL.
|
||||
#
|
||||
# ***** END LICENSE BLOCK *****
|
||||
|
||||
#from pulse import TPSPulseMonitor
|
||||
|
|
@ -1,153 +0,0 @@
|
|||
# ***** BEGIN LICENSE BLOCK *****
|
||||
# Version: MPL 1.1/GPL 2.0/LGPL 2.1
|
||||
#
|
||||
# The contents of this file are subject to the Mozilla Public License Version
|
||||
# 1.1 (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
# http://www.mozilla.org/MPL/
|
||||
#
|
||||
# Software distributed under the License is distributed on an "AS IS" basis,
|
||||
# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
|
||||
# for the specific language governing rights and limitations under the
|
||||
# License.
|
||||
#
|
||||
# The Original Code is TPS.
|
||||
#
|
||||
# The Initial Developer of the Original Code is
|
||||
# Mozilla Foundation.
|
||||
# Portions created by the Initial Developer are Copyright (C) 2011
|
||||
# the Initial Developer. All Rights Reserved.
|
||||
#
|
||||
# Contributor(s):
|
||||
# Jonathan Griffin <jgriffin@mozilla.com>
|
||||
# Sam Garrett <samdgarrett@gmail.com>
|
||||
#
|
||||
# Alternatively, the contents of this file may be used under the terms of
|
||||
# either the GNU General Public License Version 2 or later (the "GPL"), or
|
||||
# the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
|
||||
# in which case the provisions of the GPL or the LGPL are applicable instead
|
||||
# of those above. If you wish to allow use of your version of this file only
|
||||
# under the terms of either the GPL or the LGPL, and not to allow others to
|
||||
# use your version of this file under the terms of the MPL, indicate your
|
||||
# decision by deleting the provisions above and replace them with the notice
|
||||
# and other provisions required by the GPL or the LGPL. If you do not delete
|
||||
# the provisions above, a recipient may use your version of this file under
|
||||
# the terms of any one of the MPL, the GPL or the LGPL.
|
||||
#
|
||||
# ***** END LICENSE BLOCK *****
|
||||
|
||||
import json
|
||||
import logging
|
||||
import optparse
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import traceback
|
||||
|
||||
from pulse import TPSPulseMonitor
|
||||
|
||||
def main():
|
||||
""" 1. Create virtualenv via virtualenv cov
|
||||
2. Run python setup.py install
|
||||
3. Call coversheet
|
||||
"""
|
||||
parser = optparse.OptionParser()
|
||||
parser.add_option("--email-results",
|
||||
action = "store_true", dest = "emailresults",
|
||||
default = False,
|
||||
help = "email the test results to the recipients defined "
|
||||
"in the config file")
|
||||
parser.add_option("--mobile",
|
||||
action = "store_true", dest = "mobile",
|
||||
default = False,
|
||||
help = "run with mobile settings")
|
||||
parser.add_option("--autolog",
|
||||
action = "store_true", dest = "autolog",
|
||||
default = False,
|
||||
help = "post results to Autolog")
|
||||
parser.add_option("--testfile",
|
||||
action = "store", type = "string", dest = "testfile",
|
||||
default = 'all_tests.json',
|
||||
help = "path to the test file to run "
|
||||
"[default: %default]")
|
||||
parser.add_option("--logfile",
|
||||
action = "store", type = "string", dest = "logfile",
|
||||
default = 'tps.log',
|
||||
help = "path to the log file [default: %default]")
|
||||
parser.add_option("--resultfile",
|
||||
action = "store", type = "string", dest = "resultfile",
|
||||
default = 'tps_result.json',
|
||||
help = "path to the result file [default: %default]")
|
||||
parser.add_option("--binary",
|
||||
action = "store", type = "string", dest = "binary",
|
||||
default = None,
|
||||
help = "path to the Firefox binary, specified either as "
|
||||
"a local file or a url; if omitted, the PATH "
|
||||
"will be searched;")
|
||||
parser.add_option("--configfile",
|
||||
action = "store", type = "string", dest = "configfile",
|
||||
default = None,
|
||||
help = "path to the config file to use "
|
||||
"[default: %default]")
|
||||
parser.add_option("--pulsefile",
|
||||
action = "store", type = "string", dest = "pulsefile",
|
||||
default = None,
|
||||
help = "path to file containing a pulse message in "
|
||||
"json format that you want to inject into the monitor")
|
||||
parser.add_option("--ignore-unused-engines",
|
||||
default=False,
|
||||
action="store_true",
|
||||
dest="ignore_unused_engines",
|
||||
help="If defined, don't load unused engines in individual tests."
|
||||
" Has no effect for pulse monitor.")
|
||||
(options, args) = parser.parse_args()
|
||||
|
||||
configfile = options.configfile
|
||||
if configfile is None:
|
||||
if os.environ.get('VIRTUAL_ENV'):
|
||||
configfile = os.path.join(os.path.dirname(__file__), 'config.json')
|
||||
if configfile is None or not os.access(configfile, os.F_OK):
|
||||
raise Exception("Unable to find config.json in a VIRTUAL_ENV; you must "
|
||||
"specify a config file using the --configfile option")
|
||||
configfile = os.path.abspath(configfile)
|
||||
|
||||
# load the config file
|
||||
f = open(configfile, 'r')
|
||||
configcontent = f.read()
|
||||
f.close()
|
||||
config = json.loads(configcontent)
|
||||
|
||||
options.resultfile = os.path.abspath(options.resultfile)
|
||||
print 'using resultfile:', options.resultfile
|
||||
|
||||
if options.binary is None:
|
||||
while True:
|
||||
try:
|
||||
# If no binary is specified, start the pulse build monitor, and wait
|
||||
# until we receive build notifications before running tests.
|
||||
monitor = TPSPulseMonitor(autolog=options.autolog,
|
||||
emailresults=options.emailresults,
|
||||
config=configfile,
|
||||
testfile=options.testfile,
|
||||
logfile=options.logfile,
|
||||
resultfile=options.resultfile,
|
||||
mobile=options.mobile,
|
||||
ignore_unused_engines=options.ignore_unused_engines)
|
||||
print "waiting for pulse build notifications"
|
||||
|
||||
if options.pulsefile:
|
||||
# For testing purposes, inject a pulse message directly into
|
||||
# the monitor.
|
||||
builddata = json.loads(open(options.pulsefile, 'r').read())
|
||||
monitor.on_build_complete(builddata)
|
||||
|
||||
monitor.listen()
|
||||
except KeyboardInterrupt:
|
||||
sys.exit()
|
||||
except:
|
||||
traceback.print_exc()
|
||||
print 'sleeping 5 minutes'
|
||||
time.sleep(300)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -1,158 +0,0 @@
|
|||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
|
||||
import datetime
|
||||
|
||||
def GenerateEmailBody(data, numpassed, numfailed, serverUrl, buildUrl):
|
||||
|
||||
now = datetime.datetime.now()
|
||||
builddate = datetime.datetime.strptime(data['productversion']['buildid'],
|
||||
'%Y%m%d%H%M%S')
|
||||
tree = data['productversion']['repository']
|
||||
|
||||
row = """
|
||||
<tr>
|
||||
<td><a href="http://hg.mozilla.org/services/services-central/file/default/services/sync/tests/tps/{name}">{name}</a></td>
|
||||
<td>{state}</td>
|
||||
<td>{message}</td>
|
||||
</tr>
|
||||
"""
|
||||
|
||||
rowWithLog = """
|
||||
<tr>
|
||||
<td><a href="http://hg.mozilla.org/services/services-central/file/default/services/sync/tests/tps/{name}">{name}</a></td>
|
||||
<td>{state}</td>
|
||||
<td>{message} [<a href="{logurl}">view log</a>]</td>
|
||||
</tr>
|
||||
"""
|
||||
|
||||
rows = ""
|
||||
for test in data['tests']:
|
||||
if test.get('logurl'):
|
||||
rows += rowWithLog.format(name=test['name'],
|
||||
state=test['state'],
|
||||
message=test['message'] if test['message'] else 'None',
|
||||
logurl=test['logurl'])
|
||||
else:
|
||||
rows += row.format(name=test['name'],
|
||||
state=test['state'],
|
||||
message=test['message'] if test['message'] else 'None')
|
||||
|
||||
firefox_version = data['productversion']['version']
|
||||
if buildUrl is not None:
|
||||
firefox_version = "<a href='%s'>%s</a>" % (buildUrl, firefox_version)
|
||||
body = """
|
||||
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
|
||||
<head>
|
||||
<title>TPS</title>
|
||||
<style type="text/css">
|
||||
#headertable {{ border: solid 1px black; margin-bottom: 2em; border-collapse: collapse; font-size: 0.8em; }}
|
||||
#headertable th {{ border: solid 1px black; background-color: lightgray; padding: 4px; }}
|
||||
#headertable td {{ border: solid 1px black; padding: 4px; }}
|
||||
.light {{ color: gray; }}
|
||||
.pass, a.pass:link, a.pass:visited {{ color: green; font-weight: bold; }}
|
||||
.fail, a.fail:link, a.fail:visited {{ color: red; font-weight: bold; }}
|
||||
.rightgray {{ text-align: right; background-color: lightgray; }}
|
||||
#summarytable {{ border: solid 1px black; margin-bottom: 2em; border-collapse: collapse; font-size: 0.8em; }}
|
||||
#summarytable th {{ border: solid 1px black; background-color: lightgray; padding: 4px; }}
|
||||
#summarytable td {{ border: solid 1px black; padding: 4px; }}
|
||||
</style>
|
||||
</head>
|
||||
|
||||
<body>
|
||||
<div id="content">
|
||||
|
||||
<h2>TPS Testrun Details</h2>
|
||||
|
||||
<table id="headertable">
|
||||
|
||||
<tr>
|
||||
<td class="rightgray">Testrun Date</td>
|
||||
<td>{date}</td>
|
||||
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rightgray">Firefox Version</td>
|
||||
<td>{firefox_version}</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rightgray">Firefox Build Date</td>
|
||||
<td>{firefox_date}</td>
|
||||
</tr>
|
||||
|
||||
<tr>
|
||||
<td class="rightgray">Firefox Sync Version / Type</td>
|
||||
<td>{sync_version} / {sync_type}
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rightgray">Firefox Sync Changeset</td>
|
||||
<td>
|
||||
|
||||
<a href="{repository}/rev/{changeset}">
|
||||
|
||||
{changeset}</a> / {sync_tree}
|
||||
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rightgray">Sync Server</td>
|
||||
<td>{server}</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rightgray">OS</td>
|
||||
<td>{os}</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rightgray">Passed Tests</td>
|
||||
|
||||
<td>
|
||||
<span class="{passclass}">{numpassed}</span>
|
||||
</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td class="rightgray">Failed Tests</td>
|
||||
<td>
|
||||
|
||||
<span class="{failclass}">{numfailed}</span>
|
||||
</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
|
||||
<table id="summarytable">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Testcase</th>
|
||||
<th>Result</th>
|
||||
<th>Message</th>
|
||||
</tr>
|
||||
</thead>
|
||||
|
||||
{rows}
|
||||
|
||||
</table>
|
||||
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
""".format(date=now.ctime(),
|
||||
firefox_version=firefox_version,
|
||||
firefox_date=builddate.ctime(),
|
||||
sync_version=data['addonversion']['version'],
|
||||
sync_type=data['synctype'],
|
||||
sync_tree=tree[tree.rfind("/") + 1:],
|
||||
repository=data['productversion']['repository'],
|
||||
changeset=data['productversion']['changeset'],
|
||||
os=data['os'],
|
||||
rows=rows,
|
||||
numpassed=numpassed,
|
||||
numfailed=numfailed,
|
||||
passclass="pass" if numpassed > 0 else "light",
|
||||
failclass="fail" if numfailed > 0 else "light",
|
||||
server=serverUrl if serverUrl != "" else "default"
|
||||
)
|
||||
|
||||
return body
|
||||
|
|
@ -1,120 +0,0 @@
|
|||
# ***** BEGIN LICENSE BLOCK *****
|
||||
# Version: MPL 1.1/GPL 2.0/LGPL 2.1
|
||||
#
|
||||
# The contents of this file are subject to the Mozilla Public License Version
|
||||
# 1.1 (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
# http://www.mozilla.org/MPL/
|
||||
#
|
||||
# Software distributed under the License is distributed on an "AS IS" basis,
|
||||
# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
|
||||
# for the specific language governing rights and limitations under the
|
||||
# License.
|
||||
#
|
||||
# The Original Code is TPS.
|
||||
#
|
||||
# The Initial Developer of the Original Code is
|
||||
# Mozilla Foundation.
|
||||
# Portions created by the Initial Developer are Copyright (C) 2011
|
||||
# the Initial Developer. All Rights Reserved.
|
||||
#
|
||||
# Contributor(s):
|
||||
# Jonathan Griffin <jgriffin@mozilla.com>
|
||||
# Sam Garrett <samdgarrett@gmail.com>
|
||||
#
|
||||
# Alternatively, the contents of this file may be used under the terms of
|
||||
# either the GNU General Public License Version 2 or later (the "GPL"), or
|
||||
# the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
|
||||
# in which case the provisions of the GPL or the LGPL are applicable instead
|
||||
# of those above. If you wish to allow use of your version of this file only
|
||||
# under the terms of either the GPL or the LGPL, and not to allow others to
|
||||
# use your version of this file under the terms of the MPL, indicate your
|
||||
# decision by deleting the provisions above and replace them with the notice
|
||||
# and other provisions required by the GPL or the LGPL. If you do not delete
|
||||
# the provisions above, a recipient may use your version of this file under
|
||||
# the terms of any one of the MPL, the GPL or the LGPL.
|
||||
#
|
||||
# ***** END LICENSE BLOCK *****
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import socket
|
||||
|
||||
from pulsebuildmonitor import PulseBuildMonitor
|
||||
from subproc import TPSSubproc
|
||||
from results import Covresults
|
||||
|
||||
|
||||
class TPSPulseMonitor(PulseBuildMonitor):
|
||||
|
||||
def __init__(self, platform='linux', config=None,
|
||||
autolog=False, emailresults=False, testfile=None,
|
||||
logfile=None, resultfile=None, mobile=False,
|
||||
ignore_unused_engines=False, **kwargs):
|
||||
self.buildtype = ['opt']
|
||||
self.autolog = autolog
|
||||
self.emailresults = emailresults
|
||||
self.testfile = testfile
|
||||
self.logfile = logfile
|
||||
self.resultfile = resultfile
|
||||
self.mobile = mobile
|
||||
self.ignore_unused_engines = ignore_unused_engines
|
||||
self.config = config
|
||||
f = open(config, 'r')
|
||||
configcontent = f.read()
|
||||
f.close()
|
||||
configjson = json.loads(configcontent)
|
||||
self.tree = configjson.get('tree', ['services-central'])
|
||||
self.platform = [configjson.get('platform', 'linux')]
|
||||
self.label=('crossweave@mozilla.com|tps_build_monitor_' +
|
||||
socket.gethostname())
|
||||
|
||||
self.logger = logging.getLogger('tps_pulse')
|
||||
self.logger.setLevel(logging.DEBUG)
|
||||
handler = logging.FileHandler('tps_pulse.log')
|
||||
self.logger.addHandler(handler)
|
||||
|
||||
self.results = Covresults(configjson, self.autolog, self.emailresults,
|
||||
self.resultfile)
|
||||
|
||||
PulseBuildMonitor.__init__(self,
|
||||
trees=self.tree,
|
||||
label=self.label,
|
||||
logger=self.logger,
|
||||
platforms=self.platform,
|
||||
buildtypes=self.buildtype,
|
||||
builds=True,
|
||||
**kwargs)
|
||||
|
||||
def on_pulse_message(self, data):
|
||||
key = data['_meta']['routing_key']
|
||||
|
||||
def on_build_complete(self, builddata):
|
||||
print "================================================================="
|
||||
print json.dumps(builddata)
|
||||
print "================================================================="
|
||||
|
||||
# Don't run tests if some conditions aren't met
|
||||
if not builddata.get('testsurl') or builddata.get('locale') != 'en-US' \
|
||||
or builddata.get('status') != 0:
|
||||
return
|
||||
|
||||
if os.access(self.resultfile, os.F_OK):
|
||||
os.remove(self.resultfile)
|
||||
|
||||
mysub = TPSSubproc(builddata=builddata,
|
||||
emailresults=self.emailresults,
|
||||
autolog=self.autolog,
|
||||
testfile=self.testfile,
|
||||
logfile=self.logfile,
|
||||
config=self.config,
|
||||
mobile=self.mobile,
|
||||
resultfile=self.resultfile,
|
||||
ignore_unused_engines=self.ignore_unused_engines)
|
||||
|
||||
mysub.get_buildAndTests()
|
||||
mysub.setup_tps()
|
||||
mysub.update_config()
|
||||
mysub.call_testrunners()
|
||||
self.results.handleResults()
|
||||
|
|
@ -1,178 +0,0 @@
|
|||
# ***** BEGIN LICENSE BLOCK *****
|
||||
# Version: MPL 1.1/GPL 2.0/LGPL 2.1
|
||||
#
|
||||
# The contents of this file are subject to the Mozilla Public License Version
|
||||
# 1.1 (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
# http://www.mozilla.org/MPL/
|
||||
#
|
||||
# Software distributed under the License is distributed on an "AS IS" basis,
|
||||
# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
|
||||
# for the specific language governing rights and limitations under the
|
||||
# License.
|
||||
#
|
||||
# The Original Code is TPS.
|
||||
#
|
||||
# The Initial Developer of the Original Code is
|
||||
# Mozilla Foundation.
|
||||
# Portions created by the Initial Developer are Copyright (C) 2011
|
||||
# the Initial Developer. All Rights Reserved.
|
||||
#
|
||||
# Contributor(s):
|
||||
# Jonathan Griffin <jgriffin@mozilla.com>
|
||||
# Sam Garrett <samdgarrett@gmail.com>
|
||||
#
|
||||
# Alternatively, the contents of this file may be used under the terms of
|
||||
# either the GNU General Public License Version 2 or later (the "GPL"), or
|
||||
# the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
|
||||
# in which case the provisions of the GPL or the LGPL are applicable instead
|
||||
# of those above. If you wish to allow use of your version of this file only
|
||||
# under the terms of either the GPL or the LGPL, and not to allow others to
|
||||
# use your version of this file under the terms of the MPL, indicate your
|
||||
# decision by deleting the provisions above and replace them with the notice
|
||||
# and other provisions required by the GPL or the LGPL. If you do not delete
|
||||
# the provisions above, a recipient may use your version of this file under
|
||||
# the terms of any one of the MPL, the GPL or the LGPL.
|
||||
#
|
||||
# ***** END LICENSE BLOCK *****
|
||||
import json
|
||||
import socket
|
||||
import traceback
|
||||
|
||||
|
||||
class Covresults(object):
|
||||
"""Class for handling coversheet test-run results."""
|
||||
|
||||
def __init__(self, config, autolog, emailresults, filename):
|
||||
self.config = config
|
||||
self.filename = filename
|
||||
self.autolog = autolog
|
||||
self.emailresults = emailresults
|
||||
|
||||
def readResults(self):
|
||||
"""Reads in the results from the tps test run json file."""
|
||||
f = open(self.filename, 'r')
|
||||
fileContents = f.read()
|
||||
f.close()
|
||||
self.test = json.loads(fileContents)
|
||||
|
||||
def handleResults(self):
|
||||
"""Handles sending of results to the appropriate destinations."""
|
||||
self.readResults()
|
||||
|
||||
for result in self.test['results']:
|
||||
print result
|
||||
self.postdata = result
|
||||
|
||||
if result.has_key('numpassed'):
|
||||
self.numpassed = result['numpassed']
|
||||
if result.has_key('numfailed'):
|
||||
self.numfailed = result['numfailed']
|
||||
self.firefoxrunnerurl = result.get('firefoxrunnerurl', 'unknown')
|
||||
self.synctype = result.get('synctype', '')
|
||||
|
||||
if result.has_key('body'):
|
||||
body = result['body']
|
||||
else:
|
||||
body = None
|
||||
sendTo = result['sendTo']
|
||||
|
||||
if self.autolog:
|
||||
self.postToAutolog()
|
||||
|
||||
if self.emailresults:
|
||||
try:
|
||||
self.sendEmail(body, sendTo)
|
||||
except:
|
||||
traceback.print_exc()
|
||||
|
||||
def sendEmail(self, body=None, sendTo=None):
|
||||
"""Send the result email"""
|
||||
if self.config.get('email') and self.config['email'].get('username') \
|
||||
and self.config['email'].get('password'):
|
||||
from sendemail import SendEmail
|
||||
from emailtemplate import GenerateEmailBody
|
||||
|
||||
if body is None:
|
||||
buildUrl = None
|
||||
if self.firefoxrunnerurl:
|
||||
buildUrl = self.firefoxrunnerurl
|
||||
body = GenerateEmailBody(self.postdata,
|
||||
self.numpassed,
|
||||
self.numfailed,
|
||||
self.config['serverURL'],
|
||||
buildUrl)
|
||||
|
||||
subj = "TPS Report: "
|
||||
if self.numfailed == 0 and self.numpassed > 0:
|
||||
subj += "YEEEAAAHHH"
|
||||
else:
|
||||
subj += "PC LOAD LETTER"
|
||||
|
||||
changeset = self.postdata['productversion']['changeset'] if \
|
||||
self.postdata and self.postdata.get('productversion') and \
|
||||
self.postdata['productversion'].get('changeset') \
|
||||
else 'unknown'
|
||||
subj +=", changeset " + changeset + "; " + str(self.numfailed) + \
|
||||
" failed, " + str(self.numpassed) + " passed"
|
||||
|
||||
SendEmail(From=self.config['email']['username'],
|
||||
To=sendTo,
|
||||
Subject=subj,
|
||||
HtmlData=body,
|
||||
Username=self.config['email']['username'],
|
||||
Password=self.config['email']['password'])
|
||||
|
||||
def postToAutolog(self):
|
||||
from mozautolog import RESTfulAutologTestGroup as AutologTestGroup
|
||||
|
||||
for server in self.config.get('es'):
|
||||
|
||||
group = AutologTestGroup(
|
||||
harness='crossweave',
|
||||
testgroup='crossweave-%s' % self.synctype,
|
||||
server=server,
|
||||
restserver=self.config.get('restserver'),
|
||||
machine=socket.gethostname(),
|
||||
platform=self.config.get('platform', None),
|
||||
os=self.config.get('os', None),
|
||||
)
|
||||
tree = self.postdata['productversion']['repository']
|
||||
group.set_primary_product(
|
||||
tree=tree[tree.rfind("/")+1:],
|
||||
version=self.postdata['productversion']['version'],
|
||||
buildid=self.postdata['productversion']['buildid'],
|
||||
buildtype='opt',
|
||||
revision=self.postdata['productversion']['changeset'],
|
||||
)
|
||||
group.add_test_suite(
|
||||
passed=self.numpassed,
|
||||
failed=self.numfailed,
|
||||
todo=0,
|
||||
)
|
||||
for test in self.postdata['tests']:
|
||||
if test['state'] != "TEST-PASS":
|
||||
# XXX FIX ME
|
||||
#errorlog = self.errorlogs.get(test['name'])
|
||||
#errorlog_filename = errorlog.filename if errorlog else None
|
||||
errorlog_filename = None
|
||||
group.add_test_failure(
|
||||
test = test['name'],
|
||||
status = test['state'],
|
||||
text = test['message'],
|
||||
logfile = errorlog_filename
|
||||
)
|
||||
try:
|
||||
group.submit()
|
||||
except:
|
||||
self.sendEmail('<pre>%s</pre>' % traceback.format_exc(),
|
||||
sendTo='crossweave@mozilla.com')
|
||||
return
|
||||
|
||||
# Iterate through all testfailure objects, and update the postdata
|
||||
# dict with the testfailure logurl's, if any.
|
||||
for tf in group.testsuites[-1].testfailures:
|
||||
result = [x for x in self.postdata['tests'] if x.get('name') == tf.test]
|
||||
if not result:
|
||||
continue
|
||||
result[0]['logurl'] = tf.logurl
|
||||
|
|
@ -1,50 +0,0 @@
|
|||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
|
||||
import smtplib
|
||||
from email.mime.multipart import MIMEMultipart
|
||||
from email.mime.text import MIMEText
|
||||
|
||||
def SendEmail(From=None, To=None, Subject='No Subject',
|
||||
TextData=None, HtmlData=None,
|
||||
Server='mail.mozilla.com', Port=465,
|
||||
Username=None, Password=None):
|
||||
"""Sends an e-mail.
|
||||
|
||||
From is an e-mail address, To is a list of e-mail adresses.
|
||||
|
||||
TextData and HtmlData are both strings. You can specify one or both.
|
||||
If you specify both, the e-mail will be sent as a MIME multipart
|
||||
alternative; i.e., the recipient will see the HTML content if his
|
||||
viewer supports it, otherwise he'll see the text content.
|
||||
"""
|
||||
|
||||
if From is None or To is None:
|
||||
raise Exception("Both From and To must be specified")
|
||||
if TextData is None and HtmlData is None:
|
||||
raise Exception("Must specify either TextData or HtmlData")
|
||||
|
||||
server = smtplib.SMTP_SSL(Server, Port)
|
||||
|
||||
if Username is not None and Password is not None:
|
||||
server.login(Username, Password)
|
||||
|
||||
if HtmlData is None:
|
||||
msg = MIMEText(TextData)
|
||||
elif TextData is None:
|
||||
msg = MIMEMultipart()
|
||||
msg.preamble = Subject
|
||||
msg.attach(MIMEText(HtmlData, 'html'))
|
||||
else:
|
||||
msg = MIMEMultipart('alternative')
|
||||
msg.attach(MIMEText(TextData, 'plain'))
|
||||
msg.attach(MIMEText(HtmlData, 'html'))
|
||||
|
||||
msg['Subject'] = Subject
|
||||
msg['From'] = From
|
||||
msg['To'] = ', '.join(To)
|
||||
|
||||
server.sendmail(From, To, msg.as_string())
|
||||
|
||||
server.quit()
|
||||
|
|
@ -1,241 +0,0 @@
|
|||
# ***** BEGIN LICENSE BLOCK *****
|
||||
# Version: MPL 1.1/GPL 2.0/LGPL 2.1
|
||||
#
|
||||
# The contents of this file are subject to the Mozilla Public License Version
|
||||
# 1.1 (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
# http://www.mozilla.org/MPL/
|
||||
#
|
||||
# Software distributed under the License is distributed on an "AS IS" basis,
|
||||
# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
|
||||
# for the specific language governing rights and limitations under the
|
||||
# License.
|
||||
#
|
||||
# The Original Code is TPS.
|
||||
#
|
||||
# The Initial Developer of the Original Code is
|
||||
# Mozilla Foundation.
|
||||
# Portions created by the Initial Developer are Copyright (C) 2011
|
||||
# the Initial Developer. All Rights Reserved.
|
||||
#
|
||||
# Contributor(s):
|
||||
# Jonathan Griffin <jgriffin@mozilla.com>
|
||||
# Sam Garrett <samdgarrett@gmail.com>
|
||||
#
|
||||
# Alternatively, the contents of this file may be used under the terms of
|
||||
# either the GNU General Public License Version 2 or later (the "GPL"), or
|
||||
# the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
|
||||
# in which case the provisions of the GPL or the LGPL are applicable instead
|
||||
# of those above. If you wish to allow use of your version of this file only
|
||||
# under the terms of either the GPL or the LGPL, and not to allow others to
|
||||
# use your version of this file under the terms of the MPL, indicate your
|
||||
# decision by deleting the provisions above and replace them with the notice
|
||||
# and other provisions required by the GPL or the LGPL. If you do not delete
|
||||
# the provisions above, a recipient may use your version of this file under
|
||||
# the terms of any one of the MPL, the GPL or the LGPL.
|
||||
#
|
||||
# ***** END LICENSE BLOCK *****
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import zipfile
|
||||
import re
|
||||
import subprocess
|
||||
import json
|
||||
import requests
|
||||
|
||||
import mozinfo
|
||||
import mozinstall
|
||||
|
||||
class TPSSubproc():
|
||||
def __init__(self, builddata=None, emailresults=False,
|
||||
testfile=None, logfile=None, config=None, autolog=False,
|
||||
mobile=False, ignore_unused_engines=False,
|
||||
resultfile=None):
|
||||
assert(builddata)
|
||||
assert(config)
|
||||
|
||||
self.url = builddata['buildurl']
|
||||
self.testurl = builddata['testsurl']
|
||||
self.emailresults = emailresults
|
||||
self.testfile = testfile
|
||||
self.logfile = logfile
|
||||
self.config = config
|
||||
self.autolog = autolog
|
||||
self.mobile = mobile
|
||||
self.resultfile = resultfile
|
||||
self.ignore_unused_engines = ignore_unused_engines
|
||||
|
||||
def update_download_progress(self, percent):
|
||||
sys.stdout.write("===== Downloaded %d%% =====\r"%percent)
|
||||
sys.stdout.flush()
|
||||
if percent >= 100:
|
||||
sys.stdout.write("\n")
|
||||
|
||||
def download_url(self, url, outputPath):
|
||||
print "Downloading %s...\r" % url
|
||||
with open(outputPath, 'wb') as handle:
|
||||
request = requests.get(url, stream=True)
|
||||
if request.status_code != 200:
|
||||
print "Error downloading file status_code=%s" % request.status_code
|
||||
|
||||
bytes_so_far = 0.0
|
||||
block_size = 16 * 1024
|
||||
total_size = int(request.headers['content-length'])
|
||||
|
||||
for block in request.iter_content(block_size):
|
||||
if not block:
|
||||
continue
|
||||
bytes_so_far += block_size
|
||||
percent = (bytes_so_far / total_size) * 100
|
||||
self.update_download_progress(percent)
|
||||
handle.write(block)
|
||||
|
||||
def prepare_build(self, installdir='downloadedbuild', appname='firefox'):
|
||||
self.installdir = os.path.abspath(installdir)
|
||||
buildName = os.path.basename(self.url)
|
||||
pathToBuild = os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
||||
buildName)
|
||||
|
||||
# delete the build if it already exists
|
||||
if os.access(pathToBuild, os.F_OK):
|
||||
os.remove(pathToBuild)
|
||||
|
||||
# download the build
|
||||
self.download_url(self.url, pathToBuild)
|
||||
|
||||
# install the build
|
||||
print "installing %s" % pathToBuild
|
||||
shutil.rmtree(self.installdir, True)
|
||||
installed_at = mozinstall.install(pathToBuild, self.installdir)
|
||||
|
||||
# remove the downloaded archive
|
||||
os.remove(pathToBuild)
|
||||
|
||||
binary = mozinstall.get_binary(installed_at, appname)
|
||||
return os.path.abspath(binary)
|
||||
|
||||
def download_tests(self, installdir='downloadedtests'):
|
||||
self.testinstalldir = os.path.abspath(installdir)
|
||||
testsName = os.path.basename(self.testurl)
|
||||
pathToTests = os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
||||
testsName)
|
||||
|
||||
# delete tests if they already exist
|
||||
if os.access(pathToTests, os.F_OK):
|
||||
os.remove(pathToTests)
|
||||
|
||||
# download the tests
|
||||
print "downloading tests from %s" % self.testurl
|
||||
self.download_url(self.testurl, pathToTests)
|
||||
|
||||
print "extracting test files to %s" % pathToTests
|
||||
tempZipFile = zipfile.ZipFile(pathToTests)
|
||||
tempZipFile.extractall(path=self.testinstalldir)
|
||||
|
||||
print "finished downloading test files"
|
||||
|
||||
return self.testinstalldir
|
||||
|
||||
def get_buildAndTests(self):
|
||||
if self.url is not None and ('http://' in self.url or 'ftp://' in self.url):
|
||||
self.binary = self.prepare_build()
|
||||
self.tests = self.download_tests()
|
||||
else:
|
||||
self.binary = self.binary
|
||||
|
||||
def setup_tps(self):
|
||||
# Setup the downloaded TPS
|
||||
self.tpswd = os.path.join(self.tests, "tps")
|
||||
self.tpsenv = os.path.join(self.testinstalldir, "tpsenv")
|
||||
if os.access(self.tpsenv, os.F_OK):
|
||||
shutil.rmtree(self.tpsenv)
|
||||
print "Installing tps in %s" % self.tpsenv
|
||||
create_venv = os.path.join(self.tpswd, 'create_venv.py')
|
||||
if os.path.exists(create_venv):
|
||||
cmd_args = ["python", create_venv, self.tpsenv]
|
||||
else:
|
||||
cmd_args = ["sh", os.path.join(self.tpswd, "INSTALL.sh"),
|
||||
self.tpsenv]
|
||||
self.run_process(cmd_args, self.tpswd, ignoreFailures=True)
|
||||
print "TPS setup complete"
|
||||
|
||||
def update_config(self):
|
||||
f = open(self.config, 'r')
|
||||
tpsconfig = f.read()
|
||||
configjson = json.loads(tpsconfig)
|
||||
configjson['testdir'] = os.path.join(self.tpswd, "tests")
|
||||
configjson['extensiondir'] = os.path.join(self.tpswd, "extensions")
|
||||
# Update our coversheet config file
|
||||
updateFile = open(self.config, 'w')
|
||||
updateFile.write(json.dumps(configjson))
|
||||
updateFile.close()
|
||||
print 'wrote config file to', self.config
|
||||
|
||||
# Update relative testfile paths to point to our downloaded tests.
|
||||
if self.testfile and not os.path.isabs(self.testfile):
|
||||
self.testfile = os.path.join(self.tpswd, "tests", self.testfile)
|
||||
print "testfile: ", self.testfile
|
||||
|
||||
def call_testrunners(self):
|
||||
# getting our python location
|
||||
bin_dir = 'Scripts' if sys.platform.startswith('win') else 'bin'
|
||||
python_exe = 'python.exe' if sys.platform.startswith('win') else 'python'
|
||||
python_path = os.path.join(self.tpsenv, bin_dir, python_exe)
|
||||
|
||||
tps_cli = os.path.join(self.tpswd, "tps", "cli.py")
|
||||
# standard call
|
||||
self.run_process([python_path, tps_cli,
|
||||
"--binary", self.binary,
|
||||
"--testfile", self.testfile,
|
||||
"--resultfile", self.resultfile,
|
||||
"--logfile", self.logfile,
|
||||
"--configfile", self.config,
|
||||
"--mobile" if self.mobile else '',
|
||||
"--ignore_unused_engines" if self.ignore_unused_engines else ''],
|
||||
self.tpswd)
|
||||
|
||||
# mobile call
|
||||
self.run_process([python_path, tps_cli,
|
||||
"--binary", self.binary,
|
||||
"--testfile", self.testfile,
|
||||
"--resultfile", self.resultfile,
|
||||
"--logfile", self.logfile,
|
||||
"--configfile", self.config,
|
||||
"--mobile",
|
||||
"--ignore_unused_engines" if self.ignore_unused_engines else ''],
|
||||
self.tpswd)
|
||||
|
||||
# ... and again via the staging server, if credentials are present
|
||||
f = open(self.config, 'r')
|
||||
configcontent = f.read()
|
||||
f.close()
|
||||
configjson = json.loads(configcontent)
|
||||
stageaccount = configjson.get('stageaccount')
|
||||
if stageaccount:
|
||||
username = stageaccount.get('username')
|
||||
password = stageaccount.get('password')
|
||||
passphrase = stageaccount.get('passphrase')
|
||||
if username and password and passphrase:
|
||||
stageconfig = configjson.copy()
|
||||
stageconfig['account'] = stageaccount.copy()
|
||||
self.run_process([python_path, tps_cli,
|
||||
"--binary", self.binary,
|
||||
"--testfile", self.testfile,
|
||||
"--resultfile", self.resultfile,
|
||||
"--logfile", self.logfile,
|
||||
"--configfile", stageconfig,
|
||||
"--ignore_unused_engines" if self.ignore_unused_engines else ''],
|
||||
self.tpswd)
|
||||
|
||||
def run_process(self, params, cwd=None, ignoreFailures=False):
|
||||
process = subprocess.Popen(params, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE, cwd=cwd)
|
||||
stdout, stderr = process.communicate() #this blocks until the subprocess is finished
|
||||
print stdout, stderr
|
||||
retcode = process.returncode
|
||||
if not ignoreFailures:
|
||||
assert(retcode == 0)
|
||||
return retcode
|
||||
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
.DS_Store
|
||||
*.key
|
||||
*.log
|
||||
*.log.*
|
||||
*.pyc
|
||||
*.tmp
|
||||
*.bak
|
||||
builds/
|
||||
lastStable
|
||||
lastSuccessful
|
||||
nextBuildNumber
|
||||
.owner
|
||||
fingerprints
|
||||
hudson.maven.*
|
||||
hudson.scm.CVSSCM.xml
|
||||
hudson.scm.SubversionSCM.xml
|
||||
hudson.tasks.Ant.xml
|
||||
hudson.tasks.Maven.xml
|
||||
hudson.tasks.Shell.xml
|
||||
hudson.triggers.SCMTrigger.xml
|
||||
jenkins.mvn.GlobalMavenConfig.xml
|
||||
jobs/*/workspace/
|
||||
!jobs/tools/workspace
|
||||
logs
|
||||
monitoring/
|
||||
nodeMonitors.xml
|
||||
plugins/*/
|
||||
plugins/ant.jpi
|
||||
plugins/antisamy-markup-formatter.jpi
|
||||
plugins/cvs.jpi
|
||||
plugins/credentials.jpi
|
||||
plugins/external-monitor-job.jpi
|
||||
plugins/javadoc.jpi
|
||||
plugins/ldap.jpi
|
||||
plugins/mailer.jpi
|
||||
plugins/matrix-auth.jpi
|
||||
plugins/maven-plugin.jpi
|
||||
plugins/pam-auth.jpi
|
||||
plugins/ssh-credentials.jpi
|
||||
plugins/ssh-slaves.jpi
|
||||
plugins/subversion.jpi
|
||||
plugins/translation.jpi
|
||||
plugins/windows-slaves.jpi
|
||||
queue.xml
|
||||
secret*
|
||||
updates
|
||||
userContent
|
||||
users/
|
||||
war/
|
||||
|
|
@ -0,0 +1,48 @@
|
|||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<hudson>
|
||||
<disabledAdministrativeMonitors/>
|
||||
<version>1.554.2</version>
|
||||
<numExecutors>1</numExecutors>
|
||||
<mode>NORMAL</mode>
|
||||
<useSecurity>true</useSecurity>
|
||||
<authorizationStrategy class="hudson.security.AuthorizationStrategy$Unsecured"/>
|
||||
<securityRealm class="hudson.security.SecurityRealm$None"/>
|
||||
<disableRememberMe>false</disableRememberMe>
|
||||
<projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
|
||||
<workspaceDir>${ITEM_ROOTDIR}/workspace</workspaceDir>
|
||||
<buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
|
||||
<jdks/>
|
||||
<viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
|
||||
<myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
|
||||
<clouds/>
|
||||
<slaves>
|
||||
<slave>
|
||||
<name>dummy</name>
|
||||
<description></description>
|
||||
<remoteFS></remoteFS>
|
||||
<numExecutors>1</numExecutors>
|
||||
<mode>NORMAL</mode>
|
||||
<retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
|
||||
<launcher class="hudson.slaves.JNLPLauncher"/>
|
||||
<label></label>
|
||||
<nodeProperties/>
|
||||
<userId>anonymous</userId>
|
||||
</slave>
|
||||
</slaves>
|
||||
<quietPeriod>5</quietPeriod>
|
||||
<scmCheckoutRetryCount>0</scmCheckoutRetryCount>
|
||||
<views>
|
||||
<hudson.model.AllView>
|
||||
<owner class="hudson" reference="../../.."/>
|
||||
<name>All</name>
|
||||
<filterExecutors>false</filterExecutors>
|
||||
<filterQueue>false</filterQueue>
|
||||
<properties class="hudson.model.View$PropertyList"/>
|
||||
</hudson.model.AllView>
|
||||
</views>
|
||||
<primaryView>All</primaryView>
|
||||
<slaveAgentPort>0</slaveAgentPort>
|
||||
<label>master</label>
|
||||
<nodeProperties/>
|
||||
<globalNodeProperties/>
|
||||
</hudson>
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<sites>
|
||||
<site>
|
||||
<id>default</id>
|
||||
<url>http://updates.jenkins-ci.org/stable/update-center.json</url>
|
||||
</site>
|
||||
</sites>
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<hudson.tasks.Mailer_-DescriptorImpl plugin="mailer@1.6">
|
||||
<useSsl>false</useSsl>
|
||||
<charset>UTF-8</charset>
|
||||
</hudson.tasks.Mailer_-DescriptorImpl>
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<jenkins.model.ArtifactManagerConfiguration>
|
||||
<artifactManagerFactories/>
|
||||
</jenkins.model.ArtifactManagerConfiguration>
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<jenkins.model.DownloadSettings>
|
||||
<useBrowser>true</useBrowser>
|
||||
</jenkins.model.DownloadSettings>
|
||||
|
|
@ -0,0 +1,5 @@
|
|||
<?xml version='1.0' encoding='UTF-8'?>
|
||||
<jenkins.model.JenkinsLocationConfiguration>
|
||||
<adminAddress>address not configured yet <nobody@nowhere></adminAddress>
|
||||
<jenkinsUrl>http://localhost:8080/</jenkinsUrl>
|
||||
</jenkins.model.JenkinsLocationConfiguration>
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this file,
|
||||
# You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
|
||||
set -e
|
||||
DIR_TEST_ENV="tests/venv"
|
||||
DIR_JENKINS_ENV=jenkins-env
|
||||
VERSION_VIRTUALENV=1.9.1
|
||||
|
||||
if [ -d "$DIR_JENKINS_ENV" ] && [ -z $CI ]; then
|
||||
echo "Jenkins environment already exists!"
|
||||
while true; do
|
||||
read -p "Would you like to recreate it? " yn
|
||||
case $yn in
|
||||
[Yy]* ) echo "Running setup"; ./setup.sh $DIR_JENKINS_ENV; break;;
|
||||
[Nn]* ) break;;
|
||||
* ) echo "Please answer yes or no.";;
|
||||
esac
|
||||
done
|
||||
else
|
||||
echo "Running setup"
|
||||
./setup.sh $DIR_JENKINS_ENV
|
||||
fi
|
||||
|
||||
echo "Starting Jenkins"
|
||||
./start.py > jenkins.out &
|
||||
sleep 60
|
||||
|
||||
# Check if environment exists, if not, create a virtualenv:
|
||||
if [ -d $DIR_TEST_ENV ]
|
||||
then
|
||||
echo "Using virtual environment in $DIR_TEST_ENV"
|
||||
else
|
||||
echo "Creating a virtual environment (version ${VERSION_VIRTUALENV}) in ${DIR_TEST_ENV}"
|
||||
curl -O https://pypi.python.org/packages/source/v/virtualenv/virtualenv-${VERSION_VIRTUALENV}.tar.gz
|
||||
tar xvfz virtualenv-${VERSION_VIRTUALENV}.tar.gz
|
||||
python virtualenv-${VERSION_VIRTUALENV}/virtualenv.py ${DIR_TEST_ENV}
|
||||
fi
|
||||
. $DIR_TEST_ENV/bin/activate || exit $?
|
||||
|
||||
pip install selenium
|
||||
python tests/configuration/save_config.py
|
||||
|
||||
echo "Killing Jenkins"
|
||||
pid=$(lsof -i:8080 -t); kill -TERM $pid || kill -KILL $pid
|
||||
|
||||
git --no-pager diff --exit-code
|
||||
79
setup.py
79
setup.py
|
|
@ -1,79 +0,0 @@
|
|||
# ***** BEGIN LICENSE BLOCK *****
|
||||
# Version: MPL 1.1/GPL 2.0/LGPL 2.1
|
||||
#
|
||||
# The contents of this file are subject to the Mozilla Public License Version
|
||||
# 1.1 (the "License"); you may not use this file except in compliance with
|
||||
# the License. You may obtain a copy of the License at
|
||||
# http://www.mozilla.org/MPL/
|
||||
#
|
||||
# Software distributed under the License is distributed on an "AS IS" basis,
|
||||
# WITHOUT WARRANTY OF ANY KIND, either express or implied. See the License
|
||||
# for the specific language governing rights and limitations under the
|
||||
# License.
|
||||
#
|
||||
# The Original Code is TPS.
|
||||
#
|
||||
# The Initial Developer of the Original Code is
|
||||
# Mozilla foundation
|
||||
# Portions created by the Initial Developer are Copyright (C) 2011
|
||||
# the Initial Developer. All Rights Reserved.
|
||||
#
|
||||
# Contributor(s):
|
||||
# Sam Garrett <samdgarrett@gmail.com>
|
||||
#
|
||||
# Alternatively, the contents of this file may be used under the terms of
|
||||
# either the GNU General Public License Version 2 or later (the "GPL"), or
|
||||
# the GNU Lesser General Public License Version 2.1 or later (the "LGPL"),
|
||||
# in which case the provisions of the GPL or the LGPL are applicable instead
|
||||
# of those above. If you wish to allow use of your version of this file only
|
||||
# under the terms of either the GPL or the LGPL, and not to allow others to
|
||||
# use your version of this file under the terms of the MPL, indicate your
|
||||
# decision by deleting the provisions above and replace them with the notice
|
||||
# and other provisions required by the GPL or the LGPL. If you do not delete
|
||||
# the provisions above, a recipient may use your version of this file under
|
||||
# the terms of any one of the MPL, the GPL or the LGPL.
|
||||
#
|
||||
# ***** END LICENSE BLOCK *****
|
||||
|
||||
import sys
|
||||
from setuptools import setup, find_packages
|
||||
|
||||
version = '0.2'
|
||||
|
||||
deps = ['requests >= 2.2.1',
|
||||
'mozinstall >= 0.10',
|
||||
'mozinfo >= 0.7',
|
||||
'mozautolog >= 0.2.4',
|
||||
'pulsebuildmonitor >= 0.80',]
|
||||
|
||||
# we only support python 2.6+ right now
|
||||
assert sys.version_info[0] == 2
|
||||
assert sys.version_info[1] >= 6
|
||||
|
||||
setup(name='coversheet',
|
||||
version=version,
|
||||
description='run automated multi-profile sync tests',
|
||||
long_description="""\
|
||||
""",
|
||||
classifiers=[], # Get strings from http://pypi.python.org/pypi?%3Aaction=list_classifiers
|
||||
keywords='',
|
||||
author='Sam Garrett',
|
||||
author_email='samdgarrett@gmail.com',
|
||||
url='http://hg.mozilla.org/services/services-central',
|
||||
license='MPL',
|
||||
dependency_links = [
|
||||
"http://people.mozilla.org/~jgriffin/packages/"
|
||||
],
|
||||
packages=find_packages(exclude=['ez_setup', 'examples', 'tests']),
|
||||
include_package_data=True,
|
||||
zip_safe=False,
|
||||
install_requires=deps,
|
||||
entry_points="""
|
||||
# -*- Entry points: -*-
|
||||
[console_scripts]
|
||||
coversheet = coversheet.cli:main
|
||||
""",
|
||||
data_files=[
|
||||
('coversheet', ['config/config.json']),
|
||||
],
|
||||
)
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
#!/usr/bin/env bash
|
||||
|
||||
# Link to the folder which contains the zip archives of virtualenv
|
||||
URL_VIRTUALENV=https://codeload.github.com/pypa/virtualenv/zip/
|
||||
|
||||
VERSION_PULSEBUILDMONITOR=0.81
|
||||
VERSION_PYTHON_JENKINS=0.2.1
|
||||
VERSION_VIRTUALENV=1.9.1
|
||||
|
||||
VERSION_PYTHON=$(python -c "import sys;print sys.version[:3]")
|
||||
|
||||
DIR_BASE=$(cd $(dirname ${BASH_SOURCE}); pwd)
|
||||
DIR_ENV=${DIR_BASE}/${1:-"jenkins-env"}
|
||||
DIR_TMP=${DIR_BASE}/tmp
|
||||
|
||||
echo "Cleaning up existent jenkins env and tmp folders"
|
||||
rm -r ${DIR_ENV} ${DIR_TMP}
|
||||
|
||||
echo "Fetching virtualenv ${VERSION_VIRTUALENV} and creating jenkins environment"
|
||||
mkdir ${DIR_TMP}
|
||||
curl ${URL_VIRTUALENV}${VERSION_VIRTUALENV} > ${DIR_TMP}/virtualenv.zip
|
||||
unzip ${DIR_TMP}/virtualenv.zip -d ${DIR_TMP}
|
||||
python ${DIR_TMP}/virtualenv-${VERSION_VIRTUALENV}/virtualenv.py ${DIR_ENV}
|
||||
|
||||
echo "Activating the new environment"
|
||||
source ${DIR_ENV}/bin/activate
|
||||
if [ ! -n "${VIRTUAL_ENV:+1}" ]; then
|
||||
echo "### Failure in activating the new virtual environment: '${DIR_ENV}'"
|
||||
rm -r ${DIR_ENV} ${DIR_TMP}
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Installing required dependencies"
|
||||
pip install --upgrade python-jenkins==${VERSION_PYTHON_JENKINS}
|
||||
pip install --upgrade pulsebuildmonitor==${VERSION_PULSEBUILDMONITOR}
|
||||
|
||||
echo "Deactivating the environment"
|
||||
deactivate
|
||||
|
||||
echo "Successfully created the Jenkins environment: '${DIR_ENV}'"
|
||||
echo "Run 'source ${DIR_ENV}/bin/activate' to activate the environment"
|
||||
|
||||
rm -r ${DIR_TMP}
|
||||
|
|
@ -0,0 +1,64 @@
|
|||
#!/usr/bin/env python
|
||||
|
||||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
|
||||
import os
|
||||
from subprocess import check_call, CalledProcessError
|
||||
import sys
|
||||
import urllib2
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
|
||||
JENKINS_VERSION = '1.554.2'
|
||||
JENKINS_URL = 'http://mirrors.jenkins-ci.org/war-stable/%s/jenkins.war' % JENKINS_VERSION
|
||||
|
||||
JENKINS_ENV = os.path.join(HERE, 'jenkins-env', 'bin', 'activate_this.py')
|
||||
JENKINS_WAR = os.path.join(HERE, 'jenkins-%s.war' % JENKINS_VERSION)
|
||||
|
||||
def download_jenkins():
|
||||
"""Downloads Jenkins.war file"""
|
||||
|
||||
if os.path.isfile(JENKINS_WAR):
|
||||
print "Jenkins already downloaded"
|
||||
else:
|
||||
print "Downloading Jenkins %s from %s" % (JENKINS_VERSION, JENKINS_URL)
|
||||
# Download starts
|
||||
tmp_file = JENKINS_WAR + ".part"
|
||||
|
||||
while True:
|
||||
try:
|
||||
r = urllib2.urlopen(JENKINS_URL)
|
||||
CHUNK = 16 * 1024
|
||||
with open(tmp_file, 'wb') as f:
|
||||
for chunk in iter(lambda: r.read(CHUNK), ''):
|
||||
f.write(chunk)
|
||||
break
|
||||
except (urllib2.HTTPError, urllib2.URLError):
|
||||
print "Download failed."
|
||||
raise
|
||||
os.rename(tmp_file, JENKINS_WAR)
|
||||
|
||||
if __name__ == "__main__":
|
||||
download_jenkins()
|
||||
|
||||
try:
|
||||
# for more info see:
|
||||
# http://www.virtualenv.org/en/latest/#using-virtualenv-without-bin-python
|
||||
execfile(JENKINS_ENV, dict(__file__=JENKINS_ENV))
|
||||
print "Virtual environment activated successfully."
|
||||
except IOError:
|
||||
print "Could not activate virtual environment."
|
||||
print "Exiting."
|
||||
sys.exit(IOError)
|
||||
|
||||
# TODO: Start Jenkins as daemon
|
||||
print "Starting Jenkins"
|
||||
os.environ['JENKINS_HOME'] = os.path.join(HERE, 'jenkins-master')
|
||||
args = ['java', '-Xms2g', '-Xmx2g', '-XX:MaxPermSize=512M',
|
||||
'-Xincgc', '-jar', JENKINS_WAR]
|
||||
try:
|
||||
check_call(args)
|
||||
except CalledProcessError as e:
|
||||
sys.exit(e.returncode)
|
||||
|
|
@ -0,0 +1 @@
|
|||
venv
|
||||
|
|
@ -0,0 +1,47 @@
|
|||
from selenium import webdriver
|
||||
|
||||
|
||||
def main():
|
||||
base_url = 'http://localhost:8080/'
|
||||
|
||||
driver = webdriver.PhantomJS()
|
||||
driver.implicitly_wait(10)
|
||||
|
||||
print 'Saving main configuration...'
|
||||
driver.get(base_url + 'configure')
|
||||
driver.find_element_by_css_selector(
|
||||
'#bottom-sticker .submit-button button').click()
|
||||
|
||||
print 'Saving node configurations...'
|
||||
driver.get(base_url + 'computer/')
|
||||
node_links = driver.find_elements_by_css_selector(
|
||||
"tr[id*='node_'] > td:nth-child(2) > a")
|
||||
nodes = [{'name': link.text, 'href': link.get_attribute('href')} for
|
||||
link in node_links]
|
||||
|
||||
for i, node in enumerate(nodes):
|
||||
driver.get(node['href'] + 'configure')
|
||||
print '[%d/%d] %s' % (i + 1, len(nodes), node['name'])
|
||||
driver.find_element_by_css_selector('.submit-button button').click()
|
||||
driver.find_element_by_css_selector('#main-panel h1')
|
||||
|
||||
print 'Saving job configurations...'
|
||||
driver.get(base_url)
|
||||
job_links = driver.find_elements_by_css_selector(
|
||||
"tr[id*='job_'] > td:nth-child(3) > a")
|
||||
jobs = [{'name': link.text, 'href': link.get_attribute('href')} for
|
||||
link in job_links]
|
||||
assert len(jobs) == 0, 'No jobs configured in Jenkins!'
|
||||
|
||||
for i, job in enumerate(jobs):
|
||||
driver.get(job['href'] + 'configure')
|
||||
print '[%d/%d] %s' % (i + 1, len(jobs), job['name'])
|
||||
driver.find_element_by_css_selector(
|
||||
'#bottom-sticker .submit-button button').click()
|
||||
driver.find_element_by_css_selector('#main-panel h1')
|
||||
|
||||
driver.quit()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Загрузка…
Ссылка в новой задаче