mozilla
/
arewefastyet
зеркало из
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

#53 #52 #51 #55 Rewrite of the slave software. Enabling running benchmarks easier.

This commit is contained in:
Hannes Verschore 2015-08-10 12:03:43 +02:00
Родитель 7a088f1f37
Коммит b1406807f9
39 изменённых файлов: 1072 добавлений и 3164 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

68
README.md
Просмотреть файл

@ -1,16 +1,43 @@
Components
==========
Slave:
1. Builder: A python driver (build.py) that can create shell builds of spidermonkey/jsc/v8.
2. Downloader: A python driver (download.py) that can download browser builds of Firefox.
3. Executor: (execute.py) is a python script that executes one or multiple benchmarks on one or more builds.
Site:
1. Database: MySQL database that stores statistics.
2. Collector: Hidden PHP script on the webserver, where stats get sent.
3. Driver: Python driver that runs on each benchmark computer, and submits stats.
4. Processor: Python aggregator that builds JSON data from the DB.
5. Website: Static HTML as the frontpage, that queries JSON via XHR.
3. Processor: Python aggregator that builds JSON data from the DB.
4. Website: Static HTML as the frontpage, that queries JSON via XHR.
5. Command center: Sends commands to the slaves on what to execute. (In construction.)
Components (2), (3), and (5) must be on the same webserver, otherwise timestamps might not be computed correctly.
Components (2) and (4) must be on the same webserver, otherwise timestamps might not be computed correctly.
Keep in mind, most of this documentation is for posterity. AWFY was never intended to be a drag-and-drop all-in-one released product, so the procedures and scripts may be pretty rough.
Benchmark locally
=================
1. Fetch the repo
2. Create a (shell) or retrieve a (browser) build to benchmark
- Creating a build:
cd slave
python build.py -s mozilla
- Pull a build:
cd slave
python download.py http://archive.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/latest/
3. Benchmark
python execute.p -b remote.octane -b remote.kraken
Installation
============
@ -25,38 +52,7 @@ Drop `website/UPDATE.PHP` and `website/internals.php` somewhere, and rename `UPD
Benchmark Computers
-------------------
Clone the AWFY repo and check out each vendor's source code. Typically this looks something like:
git clone http://github.com/dvander/arewefastyet awfy
cd awfy
mkdir repos
cd repos
# Get V8
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
PATH=`pwd`/depot_tools:$PATH fetch v8
cd v8
git checkout master
cd ..
# Get Mozilla
hg clone http://hg.mozilla.org/integration/mozilla-inbound
# Get WebKit - Mac/Linux only
svn checkout https://svn.webkit.org/repository/webkit/trunk WebKit
cd ../driver
cp awfy.config.sample awfy.config
Then,
1. Add a database entry for the machine configuration.
2. Edit `awfy.config` to match the build architecture you want, and to have the correct machine database number.
3. Set up a cronjob, service, or screen to run dostuff.py periodically. Mozilla uses `run.sh` which will run continuously, since a cronjob could run overlapping jobs. `run.sh` also lets you configure lock files in `/tmp`.
Note, interrupting `dostuff.py` can cause problems with subversion, for example, the WebKit repository may become stuck and need an `svn cleanup` or an `rm -rf` and clean checkout. For sanity, the helper script `run.sh` will pause its next run if it sees a `/tmp/awfy` lock in place, and this can be used to wait.
Note, it is not safe to share multiple AWFY instances from the same repository, since C++ object files are generally re-used and may not correctly link depending on build flags. Also, only one instance of AWFY should ever be running at a given time. For best benchmark results, no other programs should be running.
In development...
Data Processor
--------------

27
browser-driver/awfy.android.config.sample
Просмотреть файл

@ -1,27 +0,0 @@
[main]
slaveType = android
repos = /Users/mozilla/awfy/repos
tmpDir= /tmp/
timeout = 3 ; in minutes
serverURL = http://192.168.0.212:8000/
machine = 27
updateURL = http://server2.arewefastyet.com/???
[engines]
list = mozilla, chrome
[benchmarks]
dir = /Users/mozilla/awfy/benchmarks
browserList = benchmark.local.octane,
benchmark.local.sunspider,
benchmark.local.kraken,
benchmark.local.webglsamples
shellList =
[mozilla]
nightlyDir = http://archive.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-11/
[chrome]
buildInfoUrl = http://build.chromium.org/p/chromium.lkgr/builders/Android
nightlyDir = http://commondatastorage.googleapis.com/chromium-browser-continuous/Android/

34
browser-driver/awfy.mac-desktop.config.sample
Просмотреть файл

@ -1,34 +0,0 @@
[main]
slaveType = mac-desktop
repos = /Users/mozilla/awfy/repos
tmpDir= /tmp/
timeout = 20 ; in minutes
serverURL = http://localhost:8000/
machine = 16
updateURL = http://www.arewefastyet.com/???
[engines]
list = mozilla,chrome,webkit
[benchmarks]
dir = /Users/mozilla/awfy/benchmarks
browserList = benchmark.remote.octane,
benchmark.remote.massive,
benchmark.remote.jetstream,
benchmark.remote.speedometer,
benchmark.remote.kraken,
benchmark.remote.sunspider,
benchmark.remote.browsermark
shellList =
[mozilla]
nightlyDir = http://archive.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-macosx64/
[chrome]
buildInfoUrl = http://build.chromium.org/p/chromium.lkgr/builders/Mac
nightlyDir = http://commondatastorage.googleapis.com/chromium-browser-continuous/Mac/
[webkit]
buildInfoUrl = http://nightly.webkit.org/
nightlyDir = http://builds.nightly.webkit.org/files/trunk/mac/

32
browser-driver/awfy.win-desktop.config.sample
Просмотреть файл

@ -1,32 +0,0 @@
[main]
slaveType = windows-desktop
repos = /Users/mozilla/awfy/repos
tmpDir= C:\Users\mozilla\Desktop\tmp\
timeout = 20 ; in minutes
serverURL = http://localhost:8000
machine = 17
updateURL = http://www.arewefastyet.com/???
[engines]
list = mozilla,chrome,mozillapgo,mozillashell
[benchmarks]
dir = C:\cygwin\home\mozilla\arewefastyet\benchmarks
browserList = benchmark.local.octane,
benchmark.local.kraken,
benchmark.local.sunspider,
benchmark.local.webglsamples,
benchmark.remote.dromaeo,
benchmark.local.assorteddom
shellList = benchmark.shell.octane,
benchmark.shell.kraken,
benchmark.shell.sunspider
[mozilla]
nightlyDir = http://archive.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32/
pgoDir = http://archive.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-win32-pgo/
[chrome]
buildInfoUrl = http://build.chromium.org/p/chromium.lkgr/builders/Win
nightlyDir = http://commondatastorage.googleapis.com/chromium-browser-continuous/Win/

83
browser-driver/dostuff.py
Просмотреть файл

@ -1,83 +0,0 @@
import urllib2
import re
import urllib
import tarfile
import subprocess
import engine
import sys
import time
import traceback
from optparse import OptionParser
sys.path.insert(1, '../driver')
import submitter
import utils
parser = OptionParser(usage="usage: %prog [options]")
parser.add_option("-f", "--force", dest="force", action="store_true", default=False,
help="Force runs even without source changes")
parser.add_option("-n", "--no-update", dest="noupdate", action="store_true", default=False,
help="Skip updating source repositories")
parser.add_option("-c", "--config", dest="config_name", type="string", default="awfy.config",
help="Config file (default: awfy.config)")
parser.add_option("-s", "--submitter", dest="submitter", type="string", default="remote",
help="Submitter class ('remote' or 'print')")
(options, args) = parser.parse_args()
utils.config.init(options.config_name)
slaveType = utils.config.SlaveType
BrowserBenchmarks = utils.config.browserbenchmarks()
ShellBenchmarks = utils.config.shellbenchmarks()
KnownEngines = utils.config.engines()
# Update All engines
NumUpdated = 0
RunningEngines = []
for engine in KnownEngines:
try:
engine.update()
if engine.updated:
NumUpdated += 1
RunningEngines.append(engine)
except Exception, e:
import logging
logging.exception(e) # or pass an error message, see comment
class Slave:
def __init__(self, machine):
self.machine = machine
submitter = submitter.getSubmitter(options.submitter)
submit = submitter(Slave(utils.config.get('main', 'machine')))
# No updates. Report to server and wait 60 seconds, before moving on
if NumUpdated == 0 and not options.force:
submit.Awake();
time.sleep(60)
sys.exit(0)
# Report all engines
submit.Start()
for e in RunningEngines:
for modeInfo in e.modes:
submit.AddEngine(modeInfo["name"], e.cset)
ranBenchmark = False
# Run all browser benchmarks
for benchmark in BrowserBenchmarks:
for e in RunningEngines:
if hasattr(e, "isBrowser") and e.isBrowser:
if benchmark.run(e, submit):
ranBenchmark = True
# Run all shell benchmarks
for benchmark in ShellBenchmarks:
for e in RunningEngines:
if hasattr(e, "isShell") and e.isShell:
if benchmark.run(e, submit):
ranBenchmark = True
if ranBenchmark:
submit.Finish(1)

377
browser-driver/engine.py
Просмотреть файл

@ -1,377 +0,0 @@
import json
import sys
import urllib2
import urllib
import re
import subprocess
import os
import stat
import shutil
import signal
import time
import socket
socket.setdefaulttimeout(120)
sys.path.insert(1, '../driver')
import utils
class TimeException(Exception):
pass
def timeout_handler(signum, frame):
raise TimeException()
class Engine:
def __init__(self):
self.updated = False
self.tmp_dir = utils.config.get('main', 'tmpDir')
self.slaveType = utils.config.get('main', 'slaveType')
def kill(self):
self.subprocess.terminate()
for i in range(100):
time.sleep(0.1)
if self.subprocess.poll():
return
time.sleep(0.1)
try:
os.kill(int(self.pid), signal.SIGTERM)
except:
pass
class Mozilla(Engine):
def __init__(self):
Engine.__init__(self)
self.nightly_dir = utils.config.get('mozilla', 'nightlyDir')
self.isBrowser = True
self.modes = [{
'name': 'jmim',
'env': { 'JSGC_DISABLE_POISONING': '1' }
}, {
'name': 'unboxedobjects',
'env': { 'JSGC_DISABLE_POISONING': '1',
'JS_OPTION_USE_UNBOXED_ARRAYS': '1' }
}]
self.folder = "firefox"
if not os.path.isdir(self.tmp_dir + self.folder):
os.mkdir(self.tmp_dir + self.folder)
def update(self):
# Step 0: Make sure folder exists.
if not os.path.exists(self.tmp_dir+"/"+self.folder):
os.makedirs(self.tmp_dir+"/"+self.folder)
# Step 1: Get newest nightly folder
if "latest" in self.nightly_dir:
self._update(".")
else:
response = urllib2.urlopen(self.nightly_dir+"/?C=N;O=D")
html = response.read()
ids = re.findall("[0-9]{5,}", html)
for folder_id in ids[0:4]:
try:
print "trying", folder_id
self._update(folder_id)
except Exception, e:
import logging
logging.exception(e) # or pass an error message, see comment
continue
break
def _update(self, folder_id):
# Step 2: Find the correct file
response = urllib2.urlopen(self.nightly_dir+"/"+folder_id)
html = response.read()
if self.slaveType == "android":
exec_file = re.findall("fennec-[a-zA-Z0-9.]*.en-US.android-arm.apk", html)[0]
json_file = re.findall("fennec-[a-zA-Z0-9.]*.en-US.android-arm.json", html)[0]
elif self.slaveType == "mac-desktop":
exec_file = re.findall("firefox-[a-zA-Z0-9.]*.en-US.mac.dmg", html)[0]
json_file = re.findall("firefox-[a-zA-Z0-9.]*.en-US.mac.json", html)[0]
elif self.slaveType == "linux-desktop":
exec_file = re.findall("firefox-[a-zA-Z0-9.]*.en-US.linux-x86_64.tar.bz2", html)[0]
json_file = re.findall("firefox-[a-zA-Z0-9.]*.en-US.linux-x86_64.json", html)[0]
else:
exec_file = re.findall("firefox-[a-zA-Z0-9.]*.en-US.win32.zip", html)[0]
json_file = re.findall("firefox-[a-zA-Z0-9.]*.en-US.win32.json", html)[0]
# Step 3: Get build information
response = urllib2.urlopen(self.nightly_dir+"/"+folder_id+"/"+json_file)
html = response.read()
info = json.loads(html)
# Step 4: Test if there is a new revision
if self.slaveType == "android":
output = self.tmp_dir + self.folder + "/fennec.apk"
elif self.slaveType == "mac-desktop":
output = self.tmp_dir + self.folder + "/firefox.dmg"
elif self.slaveType == "linux-desktop":
output = self.tmp_dir + self.folder + "/firefox.tar.bz2"
else:
output = self.tmp_dir + self.folder + "/firefox.zip"
utils.getOrDownload(self.tmp_dir, "mozilla", info["moz_source_stamp"],
self.nightly_dir + "/" + folder_id + "/" + exec_file,
output)
# Step 5: Prepare to run
if self.slaveType == "android":
print subprocess.check_output(["adb", "install", "-r", self.tmp_dir + self.folder + "/fennec.apk"])
elif self.slaveType == "mac-desktop":
if os.path.exists("/Volumes/Nightly"):
print subprocess.check_output(["hdiutil", "detach", "-force", "/Volumes/Nightly"])
print subprocess.check_output(["hdiutil", "attach", self.tmp_dir + self.folder + "/firefox.dmg"])
elif self.slaveType == "linux-desktop":
utils.unzip(self.tmp_dir + self.folder, "firefox.tar.bz2")
else:
utils.unzip(self.tmp_dir + self.folder, "firefox.zip")
# Step 6: Save info
self.updated = True
self.cset = info["moz_source_stamp"]
def runAndroid(self, page, mode):
# To be sure.
self.kill()
# Remove profile
print subprocess.check_output(["adb", "shell", "rm -rf /storage/emulated/legacy/awfy"])
# Create profile and disable slow script dialog
print subprocess.check_output(["adb", "shell", "mkdir /storage/emulated/legacy/awfy"])
print subprocess.check_output(["adb", "shell", "echo 'user_pref(\"dom.max_script_run_time\", 0);' > /storage/emulated/legacy/awfy/prefs.js"])
# Create env
parsedenv = ""
i = 0
for env in mode["env"]:
parsedenv += "--es env"+str(i)+" "+env+"="+mode["env"][env]+" "
i += 0
# Start browser
print subprocess.check_output(["adb", "shell", "am start -a android.intent.action.VIEW -n org.mozilla.fennec/.App -d "+page+" "+parsedenv+" --es args \"--profile /storage/emulated/legacy/awfy\""])
def runDesktop(self, page, mode):
if self.slaveType == "mac-desktop":
executable = "/Volumes/Nightly/Nightly.app/Contents/MacOS/firefox"
elif self.slaveType == "linux-desktop":
executable = self.tmp_dir + self.folder + "/firefox/firefox"
else:
executable = self.tmp_dir + self.folder + "/firefox/firefox.exe"
# Step 2: Delete profile
if os.path.exists(self.tmp_dir + "profile"):
shutil.rmtree(self.tmp_dir + "profile")
# Step 3: Create new profile
output = subprocess.check_output([executable,
"-CreateProfile", "test "+self.tmp_dir+"profile"],
stderr=subprocess.STDOUT)
# Step 4: Disable slow script dialog
fp = open(self.tmp_dir + "profile/prefs.js", 'w')
fp.write('user_pref("dom.max_script_run_time", 0);');
fp.close()
# Step 5: Start browser
env = os.environ.copy()
if "env" in mode:
env.update(mode["env"])
self.subprocess = subprocess.Popen([executable,
"-P", "test", page], env=env)
self.pid = self.subprocess.pid
def run(self, page, mode):
if self.slaveType == "android":
self.runAndroid(page, mode)
else:
self.runDesktop(page, mode)
def kill(self):
if self.slaveType == "android":
print subprocess.check_output(["adb", "shell", "pm", "clear", "org.mozilla.fennec"]);
elif self.slaveType == "linux-desktop" or self.slaveType == "mac-desktop":
subprocess.Popen(["killall", "plugin-container"])
Engine.kill(self)
else:
Engine.kill(self)
class MozillaPGO(Mozilla):
def __init__(self):
Mozilla.__init__(self)
self.nightly_dir = utils.config.get('mozilla', 'pgoDir')
self.modes = [{
'name': 'pgo',
'env': { 'JSGC_DISABLE_POISONING': '1' }
}]
self.folder = "firefox-pgo"
class MozillaShell(Engine):
def __init__(self):
Engine.__init__(self)
self.nightly_dir = utils.config.get('mozilla', 'nightlyDir')
self.isShell = True
self.modes = [{
'name': 'mozshell',
'args': []
}]
def update(self):
# Step 1: Get newest nightly folder
response = urllib2.urlopen(self.nightly_dir+"/?C=N;O=D")
html = response.read()
folder_id = re.findall("[0-9]{5,}", html)[0]
# Step 2: Find the correct file
response = urllib2.urlopen(self.nightly_dir+"/"+folder_id)
html = response.read()
exec_file = re.findall("jsshell-win32.zip", html)[0]
json_file = re.findall("firefox-[a-zA-Z0-9.]*.en-US.win32.json", html)[0]
# Step 3: Get build information
response = urllib2.urlopen(self.nightly_dir+"/"+folder_id+"/"+json_file)
html = response.read()
info = json.loads(html)
# Step 4: Fetch archive
print "Retrieving", self.nightly_dir+"/"+folder_id+"/"+exec_file
urllib.urlretrieve(self.nightly_dir+"/"+folder_id+"/"+exec_file, self.tmp_dir + "shell.zip")
# Step 5: Unzip
utils.unzip(self.tmp_dir,"shell.zip")
# Step 6: Save info
self.updated = True
self.cset = info["moz_source_stamp"]
def run(self, page, mode):
pass
def shell(self):
return os.path.join(self.tmp_dir,'js.exe')
def env(self):
return {"JSGC_DISABLE_POISONING": "1"}
class Chrome(Engine):
def __init__(self):
Engine.__init__(self)
self.build_info_url = utils.config.get('chrome', 'buildInfoUrl')
self.nightly_dir = utils.config.get('chrome', 'nightlyDir')
self.isBrowser = True
self.modes = [{
'name': 'v8'
}]
if self.slaveType == "android":
self.filename = "chrome-android.zip"
elif self.slaveType == "mac-desktop":
self.filename = "chrome-mac.zip"
elif self.slaveType == "linux-desktop":
self.filename = "chrome-linux.zip"
else:
self.filename = "chrome-win32.zip"
def update(self):
# Step 1: Get latest succesfull build revision
response = urllib2.urlopen(self.nightly_dir+"LAST_CHANGE")
chromium_rev = response.read()
# Step 3: Get v8 revision
response = urllib2.urlopen(self.nightly_dir + chromium_rev + "/REVISIONS")
self.cset = re.findall('"v8_revision_git": "([a-z0-9]*)",', response.read())[0]
# Step 3: Test if there is a new revision
utils.getOrDownload(self.tmp_dir, "chrome", self.cset,
self.nightly_dir + chromium_rev + "/" + self.filename,
self.tmp_dir + self.filename)
# Step 4: Unzip
utils.unzip(self.tmp_dir, self.filename)
# Step 5: Install on device
if self.slaveType == "android":
print subprocess.check_output(["adb", "install", "-r", self.tmp_dir+"/chrome-android/apks/ChromeShell.apk"])
# Step 6: Save info
self.updated = True
def run(self, page, mode):
if self.slaveType == "android":
self.kill()
print subprocess.check_output(["adb", "shell", "am start -a android.intent.action.VIEW -n org.chromium.chrome.shell/org.chromium.chrome.shell.ChromeShellActivity -d", page])
elif self.slaveType == "mac-desktop":
execs = subprocess.check_output(["find", self.tmp_dir + "chrome-mac", "-type", "f"])
for i in execs.split("\n"):
if "/Contents/MacOS/" in i:
utils.chmodx(i)
self.subprocess = subprocess.Popen([self.tmp_dir + "chrome-mac/Chromium.app/Contents/MacOS/Chromium", page])
self.pid = self.subprocess.pid
elif self.slaveType == "linux-desktop":
utils.chmodx(self.tmp_dir + "chrome-linux/chrome")
self.subprocess = subprocess.Popen([self.tmp_dir + "chrome-linux/chrome", page])
self.pid = self.subprocess.pid
else:
self.subprocess = subprocess.Popen([self.tmp_dir + "chrome-win32/chrome.exe", page])
self.pid = self.subprocess.pid
def kill(self):
if self.slaveType == "android":
print subprocess.check_output(["adb", "shell", "pm clear org.chromium.chrome.shell"]);
else:
Engine.kill(self)
class WebKit(Engine):
def __init__(self):
Engine.__init__(self)
self.build_info_url = utils.config.get('webkit', 'buildInfoUrl')
self.nightly_dir = utils.config.get('webkit', 'nightlyDir')
self.isBrowser = True
self.modes = [{
'name': 'jsc'
}]
def update(self):
# Step 1: Get latest succesfull build revision
response = urllib2.urlopen(self.build_info_url)
self.cset = re.findall('WebKit r([0-9]*)<', response.read())[0]
# Step 2: Download the latest installation
utils.getOrDownload(self.tmp_dir, "webkit", self.cset,
self.nightly_dir + "WebKit-SVN-r" + self.cset + ".dmg",
self.tmp_dir + "WebKit.dmg")
# Step 3: Prepare running
if os.path.exists("/Volumes/WebKit"):
self.kill()
print subprocess.check_output(["hdiutil", "detach", "/Volumes/WebKit"])
print subprocess.check_output(["hdiutil", "attach", self.tmp_dir + "/WebKit.dmg"])
# Step 4: Save info
self.updated = True
def run(self, page, mode):
self.subprocess = subprocess.Popen(["open", "-F", "-a", "/Volumes/WebKit/WebKit.app/Contents/MacOS/WebKit", page])
self.pid = self.subprocess.pid
def kill(self):
try:
subprocess.check_output("kill $(ps aux | grep '/[V]olumes/WebKit/' | awk '{print $2}')", shell=True)
except:
pass
try:
subprocess.check_output("rm -Rf ~/Library/Saved\ Application\ State/com.apple.Safari.savedState", shell=True)
except:
pass
def getEngine(name):
if name == "chrome":
return Chrome()
if name == "mozillapgo":
return MozillaPGO()
if name == "mozillashell":
return MozillaShell()
if name == "mozilla":
return Mozilla()
if name == "webkit":
return WebKit()
raise Exception("Unknown engine")

186
browser-driver/kraken.html
Просмотреть файл

@ -1,186 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset=utf8>
<!--
Copyright (C) 2007 Apple Inc. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY APPLE COMPUTER, INC. ``AS IS'' AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE COMPUTER, INC. OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-->
<title>Kraken JavaScript Benchmark</title>
</head>
<body onload="start()">
<h2><span id="logo">&#x2600;</span>Kraken JavaScript Benchmark <small>(In Progress...)</small></h2>
<script src="/sunspider.js?tests=benchmarks/kraken/tests/kraken-1.1"></script>
<script>
var html_start = "<!DOCTYPE html>\n\
<head>\n\
\n\
<meta charset=utf8>\n\
\n\
<!--\n\
Copyright (C) 2007 Apple Inc. All rights reserved.\n\
\n\
Redistribution and use in source and binary forms, with or without\n\
modification, are permitted provided that the following conditions\n\
are met:\n\
1. Redistributions of source code must retain the above copyright\n\
notice, this list of conditions and the following disclaimer.\n\
2. Redistributions in binary form must reproduce the above copyright\n\
notice, this list of conditions and the following disclaimer in the\n\
documentation and/or other materials provided with the distribution.\n\
\n\
THIS SOFTWARE IS PROVIDED BY APPLE COMPUTER, INC. ``AS IS'' AND ANY\n\
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n\
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\n\
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE COMPUTER, INC. OR\n\
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,\n\
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,\n\
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR\n\
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY\n\
OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n\
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n\
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. \n\
-->\n\
\n\
<title>SunSpider test</title>\n\
</head>\n\
\n\
<body>\n\
<div id=\"console\">\n\
</div>\n\
<script>\n\
function record(time) {\n\
document.getElementById(\"console\").innerHTML = time + \"ms\";\n\
if (window.parent) {\n\
parent.recordResult(time);\n\
}\n\
}\n\
\n\
window.onerror = function(e) {\n\
console.log(\"test failed with error: \" + e);\n\
record(0 / 0);\n\
}\n\
\n\
var _sunSpiderStartDate = new Date();\n"
var html_end = "var _sunSpiderInterval = new Date() - _sunSpiderStartDate;\n\
\n\
record(_sunSpiderInterval);\n\
</scr"+"ipt>\n\
\n\
\n\
</body>\n\
</html>\n"
var testIndex = -1;
var currentRepeat = -1;
var repeatCount = 10;
var warmupMS = 8;
var output = [];
output.length = repeatCount;
for (var i = 0; i < output.length; i++) {
output[i] = {};
}
function warmup()
{
for (var start = new Date; new Date - start < warmupMS; ) {
for (var i = 0; i < 100; ++i) {
if (Math.atan(Math.acos(Math.asin(Math.random()))) > 4) // Always false.
return;
}
}
}
function start()
{
window.setTimeout(next, 10000);
}
function next()
{
document.getElementById("frameparent").innerHTML = "";
document.getElementById("frameparent").innerHTML = "<iframe id='testframe'>";
var testFrame = document.getElementById("testframe");
if (++testIndex < tests.length) {
// Warm up the CPU a little bit, in case power management clocked it down
// or put it to sleep. We're trying to strike a balance here: do enough
// work so that all browsers see the CPU at an equal clock rate, but
// not so much work that we hide performance problems caused by overly
// aggressive power management.
warmup();
testFrame.contentDocument.open();
testFrame.contentDocument.write(html_start + testContents[testIndex] + html_end);
testFrame.contentDocument.close();
window.setTimeout(next, 0);
} else if (++currentRepeat < repeatCount) {
document.getElementById("countdown").innerHTML = repeatCount - currentRepeat;
testIndex = -1;
window.setTimeout(next, 128);
} else {
finish();
}
}
function recordResult(time)
{
if (currentRepeat >= 0) // negative repeats are warmups
output[currentRepeat][tests[testIndex]] = time;
}
function finish()
{
var total = 0;
var results = [];
for (var test in output[0]) {
var time = 0
for (var i = 0; i < output.length; i++) {
time += output[i][test];
}
total += time;
var pos = test.indexOf("-");
if (pos >= 0)
test = test.substring(pos + 1)
results[results.length] = {'name': test, 'time': time / output.length}
}
results[results.length] = {'name': "__total__", 'time': total / output.length}
location.href = "/submit?results="+encodeURIComponent(JSON.stringify(results))
}
</script>
<h3 id="countdown">warmup</h3>
<div id="frameparent"></div>
</body>
</html>

55
browser-driver/octane.html
Просмотреть файл

@ -1,55 +0,0 @@
<script>
var base_dir = '';
</script>
<script type="text/javascript" src="/benchmarks/octane/base.js"></script>
<script type="text/javascript" src="/benchmarks/octane/richards.js"></script>
<script type="text/javascript" src="/benchmarks/octane/deltablue.js"></script>
<script type="text/javascript" src="/benchmarks/octane/crypto.js"></script>
<script type="text/javascript" src="/benchmarks/octane/raytrace.js"></script>
<script type="text/javascript" src="/benchmarks/octane/earley-boyer.js"></script>
<script type="text/javascript" src="/benchmarks/octane/regexp.js"></script>
<script type="text/javascript" src="/benchmarks/octane/splay.js"></script>
<script type="text/javascript" src="/benchmarks/octane/navier-stokes.js"></script>
<script type="text/javascript" src="/benchmarks/octane/pdfjs.js"></script>
<script type="text/javascript" src="/benchmarks/octane/mandreel.js"></script>
<script type="text/javascript" src="/benchmarks/octane/gbemu-part1.js"></script>
<script type="text/javascript" src="/benchmarks/octane/gbemu-part2.js"></script>
<script type="text/javascript" src="/benchmarks/octane/code-load.js"></script>
<script type="text/javascript" src="/benchmarks/octane/box2d.js"></script>
<script type="text/javascript" src="/benchmarks/octane/zlib.js"></script>
<script type="text/javascript" src="/benchmarks/octane/zlib-data.js"></script>
<script type="text/javascript" src="/benchmarks/octane/typescript.js"></script>
<script type="text/javascript" src="/benchmarks/octane/typescript-input.js"></script>
<script type="text/javascript" src="/benchmarks/octane/typescript-compiler.js"></script>
<script>
var success = true;
var results = {};
function PrintResult(name, result) {
results[name] = result;
}
function PrintError(name, error) {
PrintResult(name, error);
success = false;
}
function PrintScore(score) {
results["total"] = score;
location.href = "/submit?results="+encodeURIComponent(JSON.stringify(results))
}
window.onload = function() {
setTimeout(function() {
BenchmarkSuite.RunSuites({ NotifyResult: PrintResult,
NotifyError: PrintError,
NotifyScore: PrintScore });
}, 10000);
}
</script>

11
browser-driver/run.sh
Просмотреть файл

@ -1,11 +0,0 @@
#!/bin/bash
adb kill-server
killall adb
sudo adb start-server
while :
do
python dostuff.py
sleep 60
done

187
browser-driver/ss.html
Просмотреть файл

@ -1,187 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset=utf8>
<!--
Copyright (C) 2007 Apple Inc. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY APPLE COMPUTER, INC. ``AS IS'' AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE COMPUTER, INC. OR
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY
OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-->
<title>SunSpider 1.0.2 JavaScript Benchmark (sunspider-1.0.2 test suite - In Progress...)</title>
</head>
<body onload="start()">
<h2><span id="logo">&#x2600;</span>SunSpider JavaScript Benchmark <small>(In Progress...)</small></h2>
<h3>Content Version: sunspider-1.0.2</h3>
<script src="/sunspider.js?tests=benchmarks/SunSpider/tests/sunspider-1.0.1"></script>
<script>
var html_start = "<!DOCTYPE html>\n\
<head>\n\
\n\
<meta charset=utf8>\n\
\n\
<!--\n\
Copyright (C) 2007 Apple Inc. All rights reserved.\n\
\n\
Redistribution and use in source and binary forms, with or without\n\
modification, are permitted provided that the following conditions\n\
are met:\n\
1. Redistributions of source code must retain the above copyright\n\
notice, this list of conditions and the following disclaimer.\n\
2. Redistributions in binary form must reproduce the above copyright\n\
notice, this list of conditions and the following disclaimer in the\n\
documentation and/or other materials provided with the distribution.\n\
\n\
THIS SOFTWARE IS PROVIDED BY APPLE COMPUTER, INC. ``AS IS'' AND ANY\n\
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n\
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\n\
PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL APPLE COMPUTER, INC. OR\n\
CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,\n\
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,\n\
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR\n\
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY\n\
OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n\
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\n\
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. \n\
-->\n\
\n\
<title>SunSpider test</title>\n\
</head>\n\
\n\
<body>\n\
<div id=\"console\">\n\
</div>\n\
<script>\n\
function record(time) {\n\
document.getElementById(\"console\").innerHTML = time + \"ms\";\n\
if (window.parent) {\n\
parent.recordResult(time);\n\
}\n\
}\n\
\n\
window.onerror = function(e) {\n\
console.log(\"test failed with error: \" + e);\n\
record(0 / 0);\n\
}\n\
\n\
var _sunSpiderStartDate = new Date();\n"
var html_end = "var _sunSpiderInterval = new Date() - _sunSpiderStartDate;\n\
\n\
record(_sunSpiderInterval);\n\
</scr"+"ipt>\n\
\n\
\n\
</body>\n\
</html>\n"
var testIndex = -1;
var currentRepeat = -1;
var repeatCount = 10;
var warmupMS = 8;
var output = [];
output.length = repeatCount;
for (var i = 0; i < output.length; i++) {
output[i] = {};
}
function warmup()
{
for (var start = new Date; new Date - start < warmupMS; ) {
for (var i = 0; i < 100; ++i) {
if (Math.atan(Math.acos(Math.asin(Math.random()))) > 4) // Always false.
return;
}
}
}
function start()
{
window.setTimeout(next, 10000);
}
function next()
{
document.getElementById("frameparent").innerHTML = "";
document.getElementById("frameparent").innerHTML = "<iframe id='testframe'>";
var testFrame = document.getElementById("testframe");
if (++testIndex < tests.length) {
// Warm up the CPU a little bit, in case power management clocked it down
// or put it to sleep. We're trying to strike a balance here: do enough
// work so that all browsers see the CPU at an equal clock rate, but
// not so much work that we hide performance problems caused by overly
// aggressive power management.
warmup();
testFrame.contentDocument.open();
testFrame.contentDocument.write(html_start + testContents[testIndex] + html_end);
testFrame.contentDocument.close();
window.setTimeout(next, 0);
} else if (++currentRepeat < repeatCount) {
document.getElementById("countdown").innerHTML = repeatCount - currentRepeat;
testIndex = -1;
window.setTimeout(next, 128);
} else {
finish();
}
}
function recordResult(time)
{
if (currentRepeat >= 0) // negative repeats are warmups
output[currentRepeat][tests[testIndex]] = time;
}
function finish()
{
var total = 0;
var results = [];
for (var test in output[0]) {
var time = 0
for (var i = 0; i < output.length; i++) {
time += output[i][test];
}
total += time;
var pos = test.indexOf("-");
if (pos >= 0)
test = test.substring(pos + 1)
results[results.length] = {'name': test, 'time': time / output.length}
}
results[results.length] = {'name': "__total__", 'time': total / output.length}
location.href = "/submit?results="+encodeURIComponent(JSON.stringify(results))
}
</script>
<h3 id="countdown">warmup</h3>
<div id="frameparent"></div>
</body>
</html>

909
browser-driver/webglsamples.html
Просмотреть файл

@ -1,909 +0,0 @@
<!--
* Copyright 2009, Google Inc.
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following disclaimer
* in the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Google Inc. nor the names of its
* contributors may be used to endorse or promote products derived from
* this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
-->
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>WebGL Aquarium</title>
<style>
html, body {
width: 100%;
height: 100%;
border: 0px;
padding: 0px;
margin: 0px;
background-color: black;
font-family: sans-serif;
overflow: hidden;
color: #fff;
}
a {
color: #fff;
}
#info {
font-size: small;
position: absolute;
top: 0px; width: 100%;
padding: 5px;
text-align: center;
z-index: 2;
}
CANVAS {
background-color: gray;
}
.fpsContainer {
position: absolute;
top: 10px;
left: 10px;
z-index: 3;
color: gray;
background-color: rgba(0,0,0,0.5);
border-radius: 10px;
padding: 10px;
}
.fps {
color: white;
}
#uiContainer {
z-index: 3;
position: absolute;
top: 10px;
right: 20px;
width: 250px;
background: rgba(0,0,0,0.5);
color: white;
font-size: xx-small;
border-radius: 10px;
padding: 10px;
}
#ui {
}
.clickable {
cursor: pointer;
}
#viewContainer {
width: 100%;
height: 100%;
}
#msgContainer {
z-index: 10;
position: absolute;
bottom: 20px;
left: 10px;
background: rgba(0,0,0,0.5);
color: white;
font-size: xx-small;
border-radius: 10px;
padding: 10px;
}
</style>
<link type="text/css" href="/benchmarks/webglsamples/jquery-ui-1.8.2.custom/css/ui-lightness/jquery-ui-1.8.2.custom.css" rel="stylesheet" />
<script type="text/javascript" src="/benchmarks/webglsamples/jquery-ui-1.8.2.custom/js/jquery-1.4.2.min.js"></script>
<script type="text/javascript" src="/benchmarks/webglsamples/jquery-ui-1.8.2.custom/js/jquery-ui-1.8.2.custom.min.js"></script>
<script type="text/javascript" src="/benchmarks/webglsamples/khronos/webgl-debug.js"></script>
<script type="text/javascript" src="/benchmarks/webglsamples/tdl/base.js"></script>
<script type="text/javascript" src="/benchmarks/webglsamples/aquarium/aquarium-common.js"></script>
<script type="text/javascript" src="/benchmarks/webglsamples/aquarium/aquarium-desktop.js"></script>
</head>
<body>
<div id="info"><a href="http://threedlibrary.googlecode.com" target="_blank">tdl.js</a> - aquarium</div>
<div class="fpsContainer">
<div class="fps">fps: <span id="fps"></div>
<div id="topUI">
<div>Number of Fish</div>
<div class="clickable" id="setSetting0">1</div>
<div class="clickable" id="setSetting1">10</div>
<div class="clickable" id="setSetting2">50</div>
<div class="clickable" id="setSetting3">100</div>
<div class="clickable" id="setSetting4">250</div>
<div class="clickable" id="setSetting5">500</div>
<div class="clickable" id="setSetting6">1000</div>
<div class="clickable" id="setSetting7">2000</div>
<div class="clickable" id="setSetting8">4000</div>
<div class="clickable" id="setSetting9">Change View</div>
<div class="clickable" id="setSetting10">Advanced</div>
<div class="clickable" id="options">Options...
<div id="optionsContainer">
</div>
</div>
</div>
</div>
<div id="uiContainer">
<div id="ui"></div>
</div>
<div id="viewContainer">
<canvas id="canvas" width="1024" height="1024" style="width: 100%; height: 100%;"></canvas>
</div>
<div id="msgContainer"></div>
</body>
<!--
<script id="constVertexShader" type="text/something-not-javascript">
attribute vec4 position;
//attribute vec3 normal;
//attribute vec2 texCoord;
//varying vec2 v_texCoord;
//varying vec3 v_normal;
uniform mat4 worldViewProjection;
void main() {
//v_texCoord = texCoord;
//v_normal = normal;
gl_Position = (worldViewProjection * position);
}
</script>
<script id="constFragmentShader" type="text/something-not-javascript">
precision mediump float;
//varying vec2 v_texCoord;
//varying vec3 v_normal;
void main() {
// gl_FragColor = vec4(vec3(v_texCoord, 1) + v_normal, 1);
gl_FragColor = vec4(1, 1, 0, 1);
}
</script>
-->
<script id="texVertexShader" type="text/something-not-javascript">
attribute vec4 position;
attribute vec2 texCoord;
varying vec2 v_texCoord;
uniform mat4 worldViewProjection;
void main() {
v_texCoord = texCoord;
gl_Position = (worldViewProjection * position);
}
</script>
<script id="texFragmentShader" type="text/something-not-javascript">
precision mediump float;
varying vec2 v_texCoord;
uniform vec4 colorMult;
uniform sampler2D colorMap;
void main() {
gl_FragColor = texture2D(colorMap, v_texCoord) * colorMult;
}
</script>
<!-- ===[ Laser Shader ]=========================================== -->
<script id="laserVertexShader" type="text/something-not-javascript">
attribute vec4 position;
attribute vec2 texCoord;
varying vec2 v_texCoord;
uniform mat4 worldViewProjection;
void main() {
v_texCoord = texCoord;
gl_Position = (worldViewProjection * position);
}
</script>
<script id="laserFragmentShader" type="text/something-not-javascript">
precision mediump float;
varying vec2 v_texCoord;
uniform vec4 colorMult;
uniform sampler2D colorMap;
void main() {
gl_FragColor = texture2D(colorMap, v_texCoord) * colorMult;
}
</script>
<!-- ===[ fishNormalMap Shader ]=========================================== -->
<script id="fishVertexShader" type="text/something-not-javascript">
uniform vec3 lightWorldPos;
uniform mat4 viewInverse;
uniform mat4 viewProjection;
uniform vec3 worldPosition;
uniform vec3 nextPosition;
uniform float scale;
uniform float time;
uniform float fishLength;
uniform float fishWaveLength;
uniform float fishBendAmount;
attribute vec4 position;
attribute vec3 normal;
attribute vec2 texCoord;
attribute vec3 tangent; // #normalMap
attribute vec3 binormal; // #normalMap
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_tangent; // #normalMap
varying vec3 v_binormal; // #normalMap
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
void main() {
vec3 vz = normalize(worldPosition - nextPosition);
vec3 vx = normalize(cross(vec3(0,1,0), vz));
vec3 vy = cross(vz, vx);
mat4 orientMat = mat4(
vec4(vx, 0),
vec4(vy, 0),
vec4(vz, 0),
vec4(worldPosition, 1));
mat4 scaleMat = mat4(
vec4(scale, 0, 0, 0),
vec4(0, scale, 0, 0),
vec4(0, 0, scale, 0),
vec4(0, 0, 0, 1));
mat4 world = orientMat * scaleMat;
mat4 worldViewProjection = viewProjection * world;
mat4 worldInverseTranspose = world;
v_texCoord = texCoord;
// NOTE:If you change this you need to change the laser code to match!
float mult = position.z > 0.0 ?
(position.z / fishLength) :
(-position.z / fishLength * 2.0);
float s = sin(time + mult * fishWaveLength);
float a = sign(s);
float offset = pow(mult, 2.0) * s * fishBendAmount;
v_position = (
worldViewProjection *
(position +
vec4(offset, 0, 0, 0)));
v_normal = (worldInverseTranspose * vec4(normal, 0)).xyz;
v_surfaceToLight = lightWorldPos - (world * position).xyz;
v_surfaceToView = (viewInverse[3] - (world * position)).xyz;
v_binormal = (worldInverseTranspose * vec4(binormal, 0)).xyz; // #normalMap
v_tangent = (worldInverseTranspose * vec4(tangent, 0)).xyz; // #normalMap
gl_Position = v_position;
}
</script>
<script id="fishNormalMapFragmentShader" type="text/something-not-javascript">
precision mediump float;
uniform vec4 lightColor;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_tangent; // #normalMap
varying vec3 v_binormal; // #normalMap
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
uniform vec4 ambient;
uniform sampler2D diffuse;
uniform vec4 specular;
uniform sampler2D normalMap; // #normalMap
uniform float shininess;
uniform float specularFactor;
// #fogUniforms
vec4 lit(float l ,float h, float m) {
return vec4(1.0,
max(l, 0.0),
(l > 0.0) ? pow(max(0.0, h), m) : 0.0,
1.0);
}
void main() {
vec4 diffuseColor = texture2D(diffuse, v_texCoord);
mat3 tangentToWorld = mat3(v_tangent, // #normalMap
v_binormal, // #normalMap
v_normal); // #normalMap
vec4 normalSpec = texture2D(normalMap, v_texCoord.xy); // #normalMap
vec4 normalSpec = vec4(0,0,0,0); // #noNormalMap
vec3 tangentNormal = normalSpec.xyz - vec3(0.5, 0.5, 0.5); // #normalMap
tangentNormal = normalize(tangentNormal + vec3(0, 0, 2)); // #normalMap
vec3 normal = (tangentToWorld * tangentNormal); // #normalMap
normal = normalize(normal); // #normalMap
vec3 normal = normalize(v_normal); // #noNormalMap
vec3 surfaceToLight = normalize(v_surfaceToLight);
vec3 surfaceToView = normalize(v_surfaceToView);
vec3 halfVector = normalize(surfaceToLight + surfaceToView);
vec4 litR = lit(dot(normal, surfaceToLight),
dot(normal, halfVector), shininess);
vec4 outColor = vec4(
(lightColor * (diffuseColor * litR.y + diffuseColor * ambient +
specular * litR.z * specularFactor * normalSpec.a)).rgb,
diffuseColor.a);
// #fogCode
gl_FragColor = outColor;
}
</script>
<!-- ===[ fishReflection Shader ]=========================================== -->
<script id="fishReflectionFragmentShader" type="text/something-not-javascript">
precision mediump float;
uniform vec4 lightColor;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_tangent; // #normalMap
varying vec3 v_binormal; // #normalMap
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
uniform vec4 ambient;
uniform sampler2D diffuse;
uniform vec4 specular;
uniform sampler2D normalMap;
uniform sampler2D reflectionMap; // #reflection
uniform samplerCube skybox; // #reflecton
uniform float shininess;
uniform float specularFactor;
// #fogUniforms
vec4 lit(float l ,float h, float m) {
return vec4(1.0,
max(l, 0.0),
(l > 0.0) ? pow(max(0.0, h), m) : 0.0,
1.0);
}
void main() {
vec4 diffuseColor = texture2D(diffuse, v_texCoord);
mat3 tangentToWorld = mat3(v_tangent, // #normalMap
v_binormal, // #normalMap
v_normal); // #normalMap
vec4 normalSpec = texture2D(normalMap, v_texCoord.xy); // #normalMap
vec4 normalSpec = vec4(0,0,0,0); // #noNormalMap
vec4 reflection = texture2D(reflectionMap, v_texCoord.xy); // #reflection
vec4 reflection = vec4(0,0,0,0); // #noReflection
vec3 tangentNormal = normalSpec.xyz - vec3(0.5, 0.5, 0.5); // #normalMap
vec3 normal = (tangentToWorld * tangentNormal); // #normalMap
normal = normalize(normal); // #normalMap
vec3 normal = normalize(v_normal); // #noNormalMap
vec3 surfaceToLight = normalize(v_surfaceToLight);
vec3 surfaceToView = normalize(v_surfaceToView);
vec4 skyColor = textureCube(skybox, -reflect(surfaceToView, normal)); // #reflection
vec4 skyColor = vec4(0.5,0.5,1,1); // #noReflection
vec3 halfVector = normalize(surfaceToLight + surfaceToView);
vec4 litR = lit(dot(normal, surfaceToLight),
dot(normal, halfVector), shininess);
vec4 outColor = vec4(mix(
skyColor,
lightColor * (diffuseColor * litR.y + diffuseColor * ambient +
specular * litR.z * specularFactor * normalSpec.a),
1.0 - reflection.r).rgb,
diffuseColor.a);
// #fogCode
gl_FragColor = outColor;
}
</script>
<!-- ===[ Seaweed Shader ]============================================== -->
<script id="seaweedVertexShader" type="text/something-not-javascript">
uniform mat4 world;
uniform mat4 viewProjection;
uniform vec3 lightWorldPos;
uniform mat4 viewInverse;
uniform mat4 worldInverseTranspose;
uniform float time;
attribute vec4 position;
attribute vec3 normal;
attribute vec2 texCoord;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
void main() {
vec3 toCamera = normalize(viewInverse[3].xyz - world[3].xyz);
vec3 yAxis = vec3(0, 1, 0);
vec3 xAxis = cross(yAxis, toCamera);
vec3 zAxis = cross(xAxis, yAxis);
mat4 newWorld = mat4(
vec4(xAxis, 0),
vec4(yAxis, 0),
vec4(xAxis, 0),
world[3]);
v_texCoord = texCoord;
v_position = position + vec4(
sin(time * 0.5) * pow(position.y * 0.07, 2.0) * 1.0,
-4, // TODO(gman): remove this hack
0,
0);
v_position = (viewProjection * newWorld) * v_position;
v_normal = (newWorld * vec4(normal, 0)).xyz;
v_surfaceToLight = lightWorldPos - (world * position).xyz;
v_surfaceToView = (viewInverse[3] - (world * position)).xyz;
gl_Position = v_position;
}
</script>
<script id="seaweedFragmentShader" type="text/something-not-javascript">
precision mediump float;
uniform vec4 lightColor;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
uniform vec4 ambient;
uniform sampler2D diffuse;
uniform vec4 specular;
uniform float shininess;
uniform float specularFactor;
// #fogUniforms
vec4 lit(float l ,float h, float m) {
return vec4(1.0,
max(l, 0.0),
(l > 0.0) ? pow(max(0.0, h), m) : 0.0,
1.0);
}
void main() {
vec4 diffuseColor = texture2D(diffuse, v_texCoord);
if (diffuseColor.a < 0.3) {
discard;
}
vec3 normal = normalize(v_normal);
vec3 surfaceToLight = normalize(v_surfaceToLight);
vec3 surfaceToView = normalize(v_surfaceToView);
vec3 halfVector = normalize(surfaceToLight + surfaceToView);
vec4 litR = lit(dot(normal, surfaceToLight),
dot(normal, halfVector), shininess);
vec4 outColor = vec4((
lightColor * (diffuseColor * litR.y + diffuseColor * ambient +
specular * litR.z * specularFactor)).rgb,
diffuseColor.a);
// #fogCode
gl_FragColor = outColor;
}
</script>
<!-- ===[ Diffuse Map Shader ]============================================== -->
<script id="diffuseVertexShader" type="text/something-not-javascript">
uniform mat4 worldViewProjection;
uniform vec3 lightWorldPos;
uniform mat4 world;
uniform mat4 viewInverse;
uniform mat4 worldInverseTranspose;
attribute vec4 position;
attribute vec3 normal;
attribute vec2 texCoord;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
void main() {
v_texCoord = texCoord;
v_position = (worldViewProjection * position);
v_normal = (worldInverseTranspose * vec4(normal, 0)).xyz;
v_surfaceToLight = lightWorldPos - (world * position).xyz;
v_surfaceToView = (viewInverse[3] - (world * position)).xyz;
gl_Position = v_position;
}
</script>
<script id="diffuseFragmentShader" type="text/something-not-javascript">
precision mediump float;
uniform vec4 lightColor;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
uniform vec4 ambient;
uniform sampler2D diffuse;
uniform vec4 specular;
uniform float shininess;
uniform float specularFactor;
// #fogUniforms
vec4 lit(float l ,float h, float m) {
return vec4(1.0,
max(l, 0.0),
(l > 0.0) ? pow(max(0.0, h), m) : 0.0,
1.0);
}
void main() {
vec4 diffuseColor = texture2D(diffuse, v_texCoord);
vec3 normal = normalize(v_normal);
vec3 surfaceToLight = normalize(v_surfaceToLight);
vec3 surfaceToView = normalize(v_surfaceToView);
vec3 halfVector = normalize(surfaceToLight + surfaceToView);
vec4 litR = lit(dot(normal, surfaceToLight),
dot(normal, halfVector), shininess);
vec4 outColor = vec4((
lightColor * (diffuseColor * litR.y + diffuseColor * ambient +
specular * litR.z * specularFactor)).rgb,
diffuseColor.a);
// #fogCode
gl_FragColor = outColor;
}
</script>
<!-- ===[ Normal Map Shader ]============================================== -->
<script id="normalMapVertexShader" type="text/something-not-javascript">
uniform mat4 worldViewProjection;
uniform vec3 lightWorldPos;
uniform mat4 world;
uniform mat4 viewInverse;
uniform mat4 worldInverseTranspose;
attribute vec4 position;
attribute vec3 normal;
attribute vec2 texCoord;
attribute vec3 tangent; // #normalMap
attribute vec3 binormal; // #normalMap
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_tangent; // #normalMap
varying vec3 v_binormal; // #normalMap
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
void main() {
v_texCoord = texCoord;
v_position = (worldViewProjection * position);
v_normal = (worldInverseTranspose * vec4(normal, 0)).xyz;
v_surfaceToLight = lightWorldPos - (world * position).xyz;
v_surfaceToView = (viewInverse[3] - (world * position)).xyz;
v_binormal = (worldInverseTranspose * vec4(binormal, 0)).xyz; // #normalMap
v_tangent = (worldInverseTranspose * vec4(tangent, 0)).xyz; // #normalMap
gl_Position = v_position;
}
</script>
<script id="normalMapFragmentShader" type="text/something-not-javascript">
precision mediump float;
uniform vec4 lightColor;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_tangent; // #normalMap
varying vec3 v_binormal; // #normalMap
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
uniform vec4 ambient;
uniform sampler2D diffuse;
uniform vec4 specular;
uniform sampler2D normalMap; // #normalMap
uniform float shininess;
uniform float specularFactor;
// #fogUniforms
vec4 lit(float l ,float h, float m) {
return vec4(1.0,
max(l, 0.0),
(l > 0.0) ? pow(max(0.0, h), m) : 0.0,
1.0);
}
void main() {
vec4 diffuseColor = texture2D(diffuse, v_texCoord);
mat3 tangentToWorld = mat3(v_tangent, // #normalMap
v_binormal, // #normalMap
v_normal); // #normalMap
vec4 normalSpec = texture2D(normalMap, v_texCoord.xy); // #normalMap
vec4 normalSpec = vec4(0,0,0,0); // #noNormalMap
vec3 tangentNormal = normalSpec.xyz - // #normalMap
vec3(0.5, 0.5, 0.5); // #normalMap
vec3 normal = (tangentToWorld * tangentNormal); // #normalMap
normal = normalize(normal); // #normalMap
vec3 normal = normalize(v_normal); // #noNormalMap
vec3 surfaceToLight = normalize(v_surfaceToLight);
vec3 surfaceToView = normalize(v_surfaceToView);
vec3 halfVector = normalize(surfaceToLight + surfaceToView);
vec4 litR = lit(dot(normal, surfaceToLight),
dot(normal, halfVector), shininess);
vec4 outColor = vec4(
(lightColor * (diffuseColor * litR.y + diffuseColor * ambient +
specular * litR.z * specularFactor * normalSpec.a)).rgb,
diffuseColor.a);
// #fogCode
gl_FragColor = outColor;
}
</script>
<!-- ===[ Reflection Map Shader ]============================================== -->
<script id="reflectionMapVertexShader" type="text/something-not-javascript">
uniform mat4 worldViewProjection;
uniform vec3 lightWorldPos;
uniform mat4 world;
uniform mat4 viewInverse;
uniform mat4 worldInverseTranspose;
attribute vec4 position;
attribute vec3 normal;
attribute vec2 texCoord;
attribute vec3 tangent;
attribute vec3 binormal;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_tangent;
varying vec3 v_binormal;
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
void main() {
v_texCoord = texCoord;
v_position = (worldViewProjection * position);
v_normal = (worldInverseTranspose * vec4(normal, 0)).xyz;
v_surfaceToLight = lightWorldPos - (world * position).xyz;
v_surfaceToView = (viewInverse[3] - (world * position)).xyz;
v_binormal = (worldInverseTranspose * vec4(binormal, 0)).xyz;
v_tangent = (worldInverseTranspose * vec4(tangent, 0)).xyz;
gl_Position = v_position;
}
</script>
<script id="reflectionMapFragmentShader" type="text/something-not-javascript">
precision mediump float;
uniform vec4 lightColor;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_tangent;
varying vec3 v_binormal;
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
uniform vec4 ambient;
uniform sampler2D diffuse;
uniform vec4 specular;
uniform sampler2D normalMap;
uniform sampler2D reflectionMap;
uniform samplerCube skybox;
uniform float shininess;
uniform float specularFactor;
// #fogUniforms
vec4 lit(float l ,float h, float m) {
return vec4(1.0,
max(l, 0.0),
(l > 0.0) ? pow(max(0.0, h), m) : 0.0,
1.0);
}
void main() {
vec4 diffuseColor = texture2D(diffuse, v_texCoord);
mat3 tangentToWorld = mat3(v_tangent,
v_binormal,
v_normal);
vec4 normalSpec = texture2D(normalMap, v_texCoord.xy);
vec4 reflection = texture2D(reflectionMap, v_texCoord.xy);
vec3 tangentNormal = normalSpec.xyz - vec3(0.5, 0.5, 0.5);
vec3 normal = (tangentToWorld * tangentNormal);
normal = normalize(normal);
vec3 surfaceToLight = normalize(v_surfaceToLight);
vec3 surfaceToView = normalize(v_surfaceToView);
vec4 skyColor = textureCube(skybox, -reflect(surfaceToView, normal));
vec3 halfVector = normalize(surfaceToLight + surfaceToView);
vec4 litR = lit(dot(normal, surfaceToLight),
dot(normal, halfVector), shininess);
vec4 outColor = vec4(mix(
skyColor,
lightColor * (diffuseColor * litR.y + diffuseColor * ambient +
specular * litR.z * specularFactor * normalSpec.a),
1.0 - reflection.r).rgb,
diffuseColor.a);
// #fogCode
gl_FragColor = outColor;
}
</script>
<!-- ===[ Inner Refraction Map Shader ]==================================== -->
<script id="innerRefractionMapVertexShader" type="text/something-not-javascript">
uniform mat4 worldViewProjection;
uniform vec3 lightWorldPos;
uniform mat4 world;
uniform mat4 viewInverse;
uniform mat4 worldInverseTranspose;
attribute vec4 position;
attribute vec3 normal;
attribute vec2 texCoord;
attribute vec3 tangent;
attribute vec3 binormal;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_tangent; // #normalMap
varying vec3 v_binormal; // #normalMap
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
void main() {
v_texCoord = texCoord;
v_position = (worldViewProjection * position);
v_normal = (worldInverseTranspose * vec4(normal, 0)).xyz;
v_surfaceToLight = lightWorldPos - (world * position).xyz;
v_surfaceToView = (viewInverse[3] - (world * position)).xyz;
v_binormal = (worldInverseTranspose * vec4(binormal, 0)).xyz; // #normalMap
v_tangent = (worldInverseTranspose * vec4(tangent, 0)).xyz; // #normalMap
gl_Position = v_position;
}
</script>
<script id="innerRefractionMapFragmentShader" type="text/something-not-javascript">
precision mediump float;
uniform vec4 lightColor;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_tangent; // #normalMap
varying vec3 v_binormal; // #normalMap
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
uniform sampler2D diffuse;
uniform vec4 specular;
uniform sampler2D normalMap; // #normalMap
uniform sampler2D reflectionMap;
uniform samplerCube skybox;
uniform float shininess;
uniform float specularFactor;
uniform float refractionFudge;
uniform float eta;
uniform float tankColorFudge;
// #fogUniforms
vec4 lit(float l ,float h, float m) {
return vec4(1.0,
max(l, 0.0),
(l > 0.0) ? pow(max(0.0, h), m) : 0.0,
1.0);
}
void main() {
vec4 diffuseColor = texture2D(diffuse, v_texCoord) +
vec4(tankColorFudge, tankColorFudge, tankColorFudge, 1);
mat3 tangentToWorld = mat3(v_tangent, // #normalMap
v_binormal, // #normalMap
v_normal); // #normalMap
vec4 normalSpec = texture2D(normalMap, v_texCoord.xy); // #normalMap
vec4 normalSpec = vec4(0,0,0,0); // #noNormalMap
vec4 refraction = texture2D(reflectionMap, v_texCoord.xy);
vec3 tangentNormal = normalSpec.xyz - vec3(0.5, 0.5, 0.5); // #normalMap
tangentNormal = normalize(tangentNormal + vec3(0,0,refractionFudge)); // #normalMap
vec3 normal = (tangentToWorld * tangentNormal); // #normalMap
normal = normalize(normal); // #normalMap
vec3 normal = normalize(v_normal); // #noNormalMap
vec3 surfaceToLight = normalize(v_surfaceToLight);
vec3 surfaceToView = normalize(v_surfaceToView);
vec3 refractionVec = refract(surfaceToView, normal, eta);
vec4 skyColor = textureCube(skybox, refractionVec);
// vec4 bumpSkyColor = textureCube(skybox, refractionVec);
// vec4 nonBumpSkyColor = textureCube(
// skybox,
// refract(surfaceToView, normalize(v_normal), eta));
// vec4 skyColor = mix(nonBumpSkyColor, bumpSkyColor, normalSpec.a);
vec4 outColor = vec4(
mix(skyColor * diffuseColor, diffuseColor, refraction.r).rgb,
diffuseColor.a);
// #fogCode
gl_FragColor = outColor;
}
</script>
<!-- ===[ Outer Refraction Map Shader ]==================================== -->
<script id="outerRefractionMapVertexShader" type="text/something-not-javascript">
uniform mat4 worldViewProjection;
uniform vec3 lightWorldPos;
uniform mat4 world;
uniform mat4 viewInverse;
uniform mat4 worldInverseTranspose;
attribute vec4 position;
attribute vec3 normal;
attribute vec2 texCoord;
attribute vec3 tangent;
attribute vec3 binormal;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_tangent; // #normalMap
varying vec3 v_binormal; // #normalMap
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
void main() {
v_texCoord = texCoord;
v_position = (worldViewProjection * position);
v_normal = (worldInverseTranspose * vec4(normal, 0)).xyz;
v_surfaceToLight = lightWorldPos - (world * position).xyz;
v_surfaceToView = (viewInverse[3] - (world * position)).xyz;
v_binormal = (worldInverseTranspose * vec4(binormal, 0)).xyz; // #normalMap
v_tangent = (worldInverseTranspose * vec4(tangent, 0)).xyz; // #normalMap
gl_Position = v_position;
}
</script>
<script id="outerRefractionMapFragmentShader" type="text/something-not-javascript">
precision mediump float;
uniform vec4 lightColor;
varying vec4 v_position;
varying vec2 v_texCoord;
varying vec3 v_tangent; // #normalMap
varying vec3 v_binormal; // #normalMap
varying vec3 v_normal;
varying vec3 v_surfaceToLight;
varying vec3 v_surfaceToView;
uniform sampler2D diffuse;
uniform vec4 specular;
uniform sampler2D normalMap; // #normalMap
uniform sampler2D reflectionMap;
uniform samplerCube skybox;
uniform float shininess;
uniform float specularFactor;
vec4 lit(float l ,float h, float m) {
return vec4(1.0,
max(l, 0.0),
(l > 0.0) ? pow(max(0.0, h), m) : 0.0,
1.0);
}
void main() {
vec4 diffuseColor = texture2D(diffuse, v_texCoord);
mat3 tangentToWorld = mat3(v_tangent, // #normalMap
v_binormal, // #normalMap
v_normal); // #normalMap
vec4 normalSpec = texture2D(normalMap, v_texCoord.xy); // #normalMap
vec4 normalSpec = vec4(0,0,0,0); // #noNormalMap
vec4 reflection = texture2D(reflectionMap, v_texCoord.xy);
vec3 tangentNormal = normalSpec.xyz - vec3(0.5, 0.5, 0.5); // #normalMap
// tangentNormal = normalize(tangentNormal + vec3(0,0,refractionFudge));
vec3 normal = (tangentToWorld * tangentNormal); // #normalMap
normal = normalize(normal); // #normalMap
vec3 normal = normalize(v_normal); // #noNormalMap
vec3 surfaceToLight = normalize(v_surfaceToLight);
vec3 surfaceToView = normalize(v_surfaceToView);
vec4 skyColor = textureCube(skybox, -reflect(surfaceToView, normal));
float fudgeAmount = 1.1;
vec3 fudge = skyColor.rgb * vec3(fudgeAmount, fudgeAmount, fudgeAmount);
float bright = min(1.0, fudge.r * fudge.g * fudge.b);
vec4 reflectColor =
mix(vec4(skyColor.rgb, bright),
diffuseColor,
(1.0 - reflection.r));
float r = abs(dot(surfaceToView, normal));
gl_FragColor = vec4(mix(
skyColor,
reflectColor,
((r + 0.3) * (reflection.r))).rgb, 1.0 - r);
}
</script>
<!-- ===[ SkyBox ]============================================== -->
<script id="refractSkyboxVertexShader" type="text/something-not-javascript">
attribute vec4 position;
varying vec4 v_position;
void main() {
v_position = position;
gl_Position = position;
}
</script>
<script id="skyboxVertexShader" type="text/something-not-javascript">
attribute vec4 position;
varying vec4 v_position;
void main() {
v_position = position;
gl_Position = position;
}
</script>
<script id="skyboxFragmentShader" type="text/something-not-javascript">
precision mediump float;
uniform samplerCube skybox;
uniform mat4 viewDirectionProjectionInverse;
varying vec4 v_position;
void main() {
vec4 t = viewDirectionProjectionInverse * v_position;
gl_FragColor = textureCube(
skybox,
normalize(t.xyz));
}
</script>
</html>

35
driver/awfy-x64.config.sample
Просмотреть файл

@ -1,35 +0,0 @@
[main]
repos = /Users/hannesverschore/awfy/x64/repos
cpu = x64
machine = 29
updateURL = http://www.arewefastyet.com/?????
[benchmarks]
dir = /Users/hannesverschore/awfy/x64/benchmarks
[native]
cc = /Users/hannesverschore/awfy/clang-3.3/bin/clang
cxx = /Users/hannesverschore/awfy/clang-3.3/bin/clang++
options = -O2
mode = clang
[v8]
source = v8
cc = /Users/hannesverschore/awfy/clang-3.3/bin/clang
cxx = /Users/hannesverschore/awfy/clang-3.3/bin/clang++
cpp = /Users/hannesverschore/awfy/clang-3.3/bin/clang -E
link = /Users/hannesverschore/awfy/clang-3.3/bin/clang++
cc_host = /Users/hannesverschore/awfy/clang-3.3/bin/clang
cxx_host = /Users/hannesverschore/awfy/clang-3.3/bin/clang++
cpp_host = /Users/hannesverschore/awfy/clang-3.3/bin/clang -E
link_host = /Users/hannesverschore/awfy/clang-3.3/bin/clang++
[jsc]
source = WebKit
[jsci]
source = InterpKit
[mi]
source = mozilla-inbound
conf = CC="/Users/hannesverschore/awfy/clang-3.3/bin/clang" CXX="/Users/hannesverschore/awfy/clang-3.3/bin/clang++" AR=ar ../configure --enable-optimize --disable-debug --enable-threadsafe --disable-intl-api --without-intl-api

36
driver/awfy-x86.config.sample
Просмотреть файл

@ -1,36 +0,0 @@
[main]
repos = /Users/hannesverschore/awfy/ia32/repos
benchmarks = /Users/hannesverschore/awfy/ia32/benchmarks
cpu = x86
machine = 28
updateURL = http://www.arewefastyet.com/????
[benchmarks]
dir = /Users/hannesverschore/awfy/ia32/benchmarks
[native]
cc = /Users/hannesverschore/awfy/clang-3.3/bin/clang
cxx = /Users/hannesverschore/awfy/clang-3.3/bin/clang++
options = -O2 -m32
mode = clang
[v8]
source = v8
cc = /Users/hannesverschore/awfy/clang-3.3/bin/clang
cxx = /Users/hannesverschore/awfy/clang-3.3/bin/clang++
cpp = /Users/hannesverschore/awfy/clang-3.3/bin/clang -E
link = /Users/hannesverschore/awfy/clang-3.3/bin/clang++
cc_host = /Users/hannesverschore/awfy/clang-3.3/bin/clang
cxx_host = /Users/hannesverschore/awfy/clang-3.3/bin/clang++
cpp_host = /Users/hannesverschore/awfy/clang-3.3/bin/clang -E
link_host = /Users/hannesverschore/awfy/clang-3.3/bin/clang++
[jsc]
source = WebKit
[jsci]
source = InterpKit
[mi]
source = mozilla-inbound
conf = CC="/Users/hannesverschore/awfy/clang-3.3/bin/clang -m32" CXX="/Users/hannesverschore/awfy/clang-3.3/bin/clang++ -m32" AR=ar CROSS_COMPILE=1 ../configure --enable-optimize --disable-debug --enable-threadsafe --disable-intl-api --without-intl-api --target=i686-apple-darwin10.0.0

34
driver/awfy.service
Просмотреть файл

@ -1,34 +0,0 @@
# I am a systemd .service file, to run awfy as a service!
# you'll want to copy this file to /etc/systemd/system/awfy.service, then you can start it once via
# systemctl start awfy.service
# or enable it at boottime via
# systemctl enable awfy.service
# Logs will be avialible via
# journalctl /home/awfy/arewefastyet/driver/dostuff.py
# or
# systemctl status awfy
[Unit]
Description="Run awfy"
[Service]
# The path to the executable *must* be hardcoded. You can use a relative path for config, given we're setting WorkingDirectory below.
ExecStart=/home/awfy/arewefastyet/driver/dostuff.py --config=/home/awfy/arewefastyet/driver/awfy.config
# username may vary
User=awfy
# systemd gives a very clean environment. You may not need to set PATH, but I needed to because arch
# defaults to python3, so there is a symlink to python2 in /home/awfy/bin
# the build process gets /very/ annoyed if SHELL is undefined
Environment=PATH=/home/awfy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin SHELL=/bin/bash
# Probably not needed
WorkingDirectory=/home/awfy/arewefastyet/driver
# You can also change this to onsuccess, it may be bad if dostuff.py fails quickly.
Restart=always
[Install]
WantedBy=multi-user.target

296
driver/benchmarks.py
Просмотреть файл

@ -1,296 +0,0 @@
# vim: set ts=4 sw=4 tw=99 et:
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
import re
import os
import sys
import urllib2
import StringIO
import subprocess
import signal
import pickle
import ConfigParser
import submitter
import utils
class Benchmark(object):
def __init__(self, suite, version, folder):
self.suite = suite
self.version = suite+" "+version
self.folder = folder
def run(self, submit, native, modes):
with utils.chdir(os.path.join(utils.config.BenchmarkPath, self.folder)):
return self._run(submit, native, modes)
def omit(self, mode):
if mode.name == 'noasmjs':
return True
def _run(self, submit, native, modes):
for mode in modes:
if self.omit(mode):
continue
try:
tests = None
print('Running ' + self.version + ' under ' + mode.shell + ' ' + ' '.join(mode.args))
tests = self.benchmark(mode.shell, mode.env, mode.args)
except Exception as e:
print('Failed to run ' + self.version + '!')
print("Exception: " + repr(e))
pass
if tests:
submit.AddTests(tests, self.suite, self.version, mode.name)
class AsmJS(Benchmark):
def __init__(self, suite, version, folder):
super(AsmJS, self).__init__(suite, version, folder)
def omit(self, mode):
if mode.name == 'noasmjs':
return False
return super(AsmJS, self).omit(mode)
def _run(self, submit, native, modes):
# Run the C++ mode.
full_args = [utils.config.PythonName, 'harness.py', '--native']
full_args += ['--cc="' + native.cc + '"']
full_args += ['--cxx="' + native.cxx + '"']
full_args += ['--'] + native.args
output = utils.RunTimedCheckOutput(full_args)
tests = self.parse(output)
submit.AddTests(tests, self.suite, self.version, native.mode)
# Run normal benchmarks.
super(AsmJS, self)._run(submit, native, modes)
def benchmark(self, shell, env, args):
full_args = [utils.config.PythonName, 'harness.py', shell, '--'] + args
print(' '.join(full_args))
output = utils.RunTimedCheckOutput(full_args, env=env)
return self.parse(output)
def parse(self, output):
total = 0.0
tests = []
for line in output.splitlines():
m = re.search("(.+) - (\d+(\.\d+)?)", line)
if not m:
continue
name = m.group(1)
score = m.group(2)
total += float(score)
tests.append({ 'name': name, 'time': score })
tests.append({ 'name': '__total__', 'time': total })
return tests
class AsmJSMicro(AsmJS):
def __init__(self):
super(AsmJSMicro, self).__init__('asmjs-ubench', '0.4.3', 'asmjs-ubench')
class AsmJSApps(AsmJS):
def __init__(self):
super(AsmJSApps, self).__init__('asmjs-apps', '0.2', 'asmjs-apps')
class Octane(Benchmark):
def __init__(self):
super(Octane, self).__init__('octane', '2.0.1', 'octane')
def benchmark(self, shell, env, args):
full_args = [shell]
if args:
full_args.extend(args)
full_args.append('run.js')
print(os.getcwd())
output = utils.RunTimedCheckOutput(full_args, env=env)
tests = []
lines = output.splitlines()
for x in lines:
m = re.search("(.+): (\d+)", x)
if not m:
continue
name = m.group(1)
score = m.group(2)
if name[0:5] == "Score":
name = "__total__"
tests.append({ 'name': name, 'time': score})
print(score + ' - ' + name)
return tests
class Dart(Benchmark):
def __init__(self):
super(Dart, self).__init__('dart', '0.1', 'dart')
def benchmark(self, shell, env, args):
full_args = [shell]
if args:
full_args.extend(args)
full_args.append('run.js')
print(os.getcwd())
output = utils.RunTimedCheckOutput(full_args, env=env)
tests = []
lines = output.splitlines()
total = 0.0
for x in lines:
m = re.search("(.+)\(RunTime\): (\d+\.\d+)", x)
if not m:
continue
name = m.group(1)
score = float(m.group(2))/1000
total += score
tests.append({ 'name': name, 'time': score})
print(str(score) + ' - ' + name)
tests.append({ 'name': '__total__', 'time': total })
return tests
class SunSpiderBased(Benchmark):
def __init__(self, suite, version, folder, runs):
super(SunSpiderBased, self).__init__(suite, version, folder)
self.runs = runs
def benchmark(self, shell, env, args):
if args != None:
args = '--args=' + ' '.join(args)
else:
args = ''
output = utils.RunTimedCheckOutput(["./sunspider",
"--shell=" + shell,
"--runs=" + str(self.runs),
args],
env=env)
tests = []
lines = output.splitlines()
found = False
for x in lines:
if x == "--------------------------------------------" or \
x == "-----------------------------------------------":
found = True
if x[0:5] == "Total":
m = re.search(":\s+(\d+\.\d+)ms", x)
tests.append({ 'name': '__total__', 'time': m.group(1)})
print(m.group(1) + ' - __total__')
elif found == True and x[0:4] == " ":
m = re.search(" (.+):\s+(\d+\.\d+)ms", x)
if m != None:
tests.append({ 'name': m.group(1), 'time': m.group(2)})
print(m.group(2) + ' - ' + m.group(1))
if found == False:
print(output)
raise Exception("output marker not found")
return tests
class SunSpider(SunSpiderBased):
def __init__(self):
super(SunSpider, self).__init__('ss', '1.0.1', 'SunSpider', 20)
class Kraken(SunSpiderBased):
def __init__(self):
super(Kraken, self).__init__('kraken', '1.1', 'kraken', 5)
class Assorted(SunSpiderBased):
def __init__(self):
super(Assorted, self).__init__('misc', '0.5', 'misc', 3)
class Shumway(Benchmark):
def __init__(self):
super(Shumway, self).__init__('shumway', '0.1', 'shumway')
# Only update harness once a day:
from datetime import datetime
date = datetime.now().strftime("%Y-%m-%d")
utils.getOrDownload("/tmp/", "shumway", date,
"http://mozilla.github.io/shumway/shell/shumway-shell.zip",
"/tmp/shumway-shell.zip")
utils.unzip("/tmp/", "shumway-shell.zip")
def omit(self, mode):
if "shumway_interp" not in mode.name:
# JIT is broken atm. Disable running
return True
elif mode.name not in ["jsc_shumway_interp", "jmim_shumway_interp", "v8_shumway_interp"]:
# Only run interpreter for some modes
return True
def _run(self, submit, native, modes):
# Run the full shumway jit.
super(Shumway, self)._run(submit, native, modes)
# Run the shumway interpreter.
interp_modes = []
for mode in modes:
interp_modes.append(mode._replace(name = mode.name+"_shumway_interp"))
super(Shumway, self)._run(submit, native, interp_modes)
def benchmark(self, shell, env, args):
with utils.chdir("/tmp/"):
full_args = [shell]
if args:
full_args.extend(args)
full_args.append('build/ts/shell.js')
if "WebKit" in shell:
full_args.append('--')
if "v8" in shell:
full_args.append('--')
full_args.append('-x')
tests = []
totalscore = 0
bench_path = os.path.join(utils.config.BenchmarkPath, self.folder)
for name in ["crypto", "deltablue", "raytrace", "richards", "splay"]:
output = utils.RunTimedCheckOutput(full_args +
[os.path.join(bench_path, name+".swf")], env=env)
lines = output.splitlines()
for x in lines:
m = re.search("NotifyScore (\d+)", x)
if not m:
continue
score = m.group(1)
totalscore += int(score)
tests.append({ 'name': name, 'time': score})
print(score + ' - ' + name)
if len(tests) > 0:
tests.append({ 'name': '__total__', 'time': totalscore / len(tests)})
return tests
Benchmarks = [AsmJSApps(),
AsmJSMicro(),
SunSpider(),
Kraken(),
Assorted(),
Octane(),
Shumway(),
Dart()
]
def run(submit, native, modes):
for benchmark in Benchmarks:
benchmark.run(submit, native, modes)
submit.Finish(1)
#def run(slave, submit, native, modes):
# slave.rpc(sys.modules[__name__], submit, native, modes, async=True)
#
#default_function = run_
if __name__ == "__main__":
remote.takerpc()

286
driver/builders.py
Просмотреть файл

@ -1,286 +0,0 @@
# vim: set ts=4 sw=4 tw=99 et:
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import sys
import utils
import puller
import platform
import subprocess
from utils import Run
class Engine(object):
def __init__(self):
self.cpu = utils.config.get('main', 'cpu')
def updateAndBuild(self, update=True, forceRebuild=False, rev=None):
with utils.FolderChanger(os.path.join(utils.config.RepoPath, self.source)):
self._updateAndBuild(update, forceRebuild, rev=rev)
def _updateAndBuild(self, update, forceRebuild, rev=None):
scm = puller.get(self.puller)
shell = self.shell()
if not os.path.isfile(shell):
forceRebuild = True
if update:
self.updated = scm.Update(rev)
else:
self.updated = False
if forceRebuild or self.updated:
try:
os.unlink(shell)
except:
pass
self.build()
if not os.path.isfile(shell):
if self.reconf():
self.build()
self.updated = True
self.cset = scm.Identify()
if not os.path.isfile(shell):
print(shell)
raise Exception('could not find shell')
# Throw away any modes that fail.
modes = []
for m in self.modes:
engineArgs = self.args if self.args else []
modeArgs = m['args'] if m['args'] else []
args = engineArgs + modeArgs + ["-e","print('1234567890abcdef')"]
output = utils.RunTimedCheckOutput([shell] + args, env=self.env())
if "1234567890abcdef" in output:
modes.append(m)
else:
print "Error: Couldn't run mode "+m['mode']
self.modes = modes
def reconf(self):
return False
def env(self):
return None
class Nitro(Engine):
def __init__(self):
super(Nitro, self).__init__()
self.puller = 'svn'
self.source = utils.config.get('jsc', 'source')
self.extra = utils.config.getDefault('jsc', 'conf', "").split()
self.args = None
self.important = False # WebKit changes too frequently, we'd need to detect JSC changes.
self.modes = [
{
'mode': 'jsc',
'args': None
}
]
def env(self):
with utils.chdir(os.path.join(utils.config.RepoPath, self.source)):
env = os.environ.copy()
env['DYLD_FRAMEWORK_PATH'] = os.path.abspath(os.path.join('WebKitBuild', 'Release'))
return env
def build(self):
# Hack 1: Remove reporting errors for warnings that currently are present.
Run(["sed","-i.bac","s/GCC_TREAT_WARNINGS_AS_ERRORS = YES;/GCC_TREAT_WARNINGS_AS_ERRORS=NO;/","Source/JavaScriptCore/Configurations/Base.xcconfig"])
Run(["sed","-i.bac","s/GCC_TREAT_WARNINGS_AS_ERRORS = YES;/GCC_TREAT_WARNINGS_AS_ERRORS=NO;/","Source/bmalloc/Configurations/Base.xcconfig"])
Run(["sed","-i.bac","s/GCC_TREAT_WARNINGS_AS_ERRORS = YES;/GCC_TREAT_WARNINGS_AS_ERRORS=NO;/","Source/WTF/Configurations/Base.xcconfig"])
Run(["sed","-i.bac","s/std::numeric_limits<unsigned char>::max()/255/","Source/bmalloc/bmalloc/Line.h"])
Run(["sed","-i.bac","s/std::numeric_limits<unsigned char>::max()/255/","Source/bmalloc/bmalloc/Page.h"])
Run(["patch","Source/JavaScriptCore/jsc.cpp","../../driver/jsc.patch"])
with utils.FolderChanger(os.path.join('Tools', 'Scripts')):
# Hack 2: This check fails currently. Disable checking to still have a build.
os.rename("check-for-weak-vtables-and-externals", "check-for-weak-vtables-and-externals2");
if self.cpu == 'x86':
args = ['/usr/bin/perl', 'build-jsc', '--32-bit']
else:
args = ['/usr/bin/perl', 'build-jsc']
args.extend(self.extra)
Run(args)
os.rename("check-for-weak-vtables-and-externals2", "check-for-weak-vtables-and-externals");
Run(["svn","revert","Source/JavaScriptCore/Configurations/Base.xcconfig"])
Run(["svn","revert","Source/bmalloc/Configurations/Base.xcconfig"])
Run(["svn","revert","Source/WTF/Configurations/Base.xcconfig"])
Run(["svn","revert","Source/bmalloc/bmalloc/Line.h"])
Run(["svn","revert","Source/bmalloc/bmalloc/Page.h"])
Run(["svn","revert","Source/JavaScriptCore/jsc.cpp"])
def shell(self):
return os.path.join('WebKitBuild', 'Release', 'jsc')
class V8(Engine):
def __init__(self):
super(V8, self).__init__()
self.puller = 'v8git'
self.source = utils.config.get('v8', 'source')
self.cxx = utils.config.getDefault('v8', 'cxx', None)
self.cc = utils.config.getDefault('v8', 'cc', None)
self.cpp = utils.config.getDefault('v8', 'cpp', None)
self.link = utils.config.getDefault('v8', 'link', None)
self.cxx_host = utils.config.getDefault('v8', 'cxx_host', None)
self.cc_host = utils.config.getDefault('v8', 'cc_host', None)
self.cpp_host = utils.config.getDefault('v8', 'cpp_host', None)
self.link_host = utils.config.getDefault('v8', 'link_host', None)
self.args = ['--expose-gc']
self.important = True
self.hardfp = "hardfp" in utils.config.getDefault('main', 'flags', '')
self.modes = [{
'mode': 'v8',
'args': None
}, {
'mode': 'v8-turbofan',
'args': ['--turbo']
}]
def build(self):
env = os.environ.copy()
if self.cxx is not None:
env['CXX'] = self.cxx
if self.cc is not None:
env['CC'] = self.cc
if self.cpp is not None:
env['CPP'] = self.cpp
if self.link is not None:
env['LINK'] = self.link
if self.cxx_host is not None:
env['CXX_host'] = self.cxx_host
if self.cc_host is not None:
env['CC_host'] = self.cc_host
if self.cpp_host is not None:
env['CPP_host'] = self.cpp_host
if self.link_host is not None:
env['LINK_host'] = self.link_host
env["GYP_DEFINES"] = "clang=1"
if self.cpu == 'x64':
Run(['make', 'x64.release', '-j3'], env)
elif self.cpu == 'arm':
if self.hardfp:
Run(['make', 'arm.release', 'hardfp=on', 'i18nsupport=off', '-j3'], env)
else:
Run(['make', 'arm.release', 'i18nsupport=off', '-j3'], env)
elif self.cpu == 'x86':
Run(['make', 'ia32.release', '-j3'], env)
def shell(self):
if self.cpu == 'x64':
return os.path.join('out', 'x64.release', 'd8')
elif self.cpu == 'arm':
return os.path.join('out', 'arm.release', 'd8')
elif self.cpu == 'x86':
return os.path.join('out', 'ia32.release', 'd8')
class Mozilla(Engine):
def __init__(self, source):
super(Mozilla, self).__init__()
self.puller = 'hg'
self.source = utils.config.get(source, 'source')
self.config_line = utils.config.get(source, 'conf')
self.args = None
self.important = True
self.objdir = 'Opt'
def reconf(self):
# Step 1. autoconf.
with utils.FolderChanger(os.path.join('js', 'src')):
if platform.system() == "Darwin":
utils.Shell("autoconf213")
elif platform.system() == "Linux":
utils.Shell("autoconf2.13")
elif platform.system() == "Windows":
utils.Shell("autoconf-2.13")
# Step 2. configure
if not os.path.exists(os.path.join('js', 'src', self.objdir)):
os.mkdir(os.path.join('js', 'src', self.objdir))
with utils.FolderChanger(os.path.join('js', 'src', self.objdir)):
utils.Shell(self.config_line)
return True
def build(self):
utils.Shell("make -j 3 -C " + os.path.join('js', 'src', self.objdir))
def shell(self):
return os.path.join('js', 'src', self.objdir, 'dist', 'bin', 'js')
def env(self):
env = os.environ.copy()
env['JSGC_DISABLE_POISONING'] = '1'
return env
class MozillaInbound(Mozilla):
def __init__(self):
super(MozillaInbound, self).__init__('mi')
self.modes = [
{
'mode': 'jmim',
'args': ['--ion-offthread-compile=on', '-W']
},
{
'mode': 'noasmjs',
'args': ['--ion-offthread-compile=on', '-W', '--no-asmjs']
},
{
'mode': 'unboxedobjects',
'args': ['--ion-offthread-compile=on', '-W', '--unboxed-arrays']
},
{
'mode': 'testbed',
'args': ['--ion-offthread-compile=on', '-W', '--ion-regalloc=testbed']
}
]
class MozillaInboundGGC(Mozilla):
def __init__(self):
super(MozillaInboundGGC, self).__init__('mi')
self.config_line += ' --enable-exact-rooting --enable-gcgenerational'
self.objdir = 'OptGGC'
self.modes = [
{
'mode': 'ggc',
'args': ['--ion-offthread-compile=on', '-W']
}
]
class NativeCompiler(Engine):
def __init__(self):
super(NativeCompiler, self).__init__()
self.cc = utils.config.get('native', 'cc')
self.cxx = utils.config.get('native', 'cxx')
self.args = utils.config.get('native', 'options').split(' ')
self.mode = utils.config.get('native', 'mode')
output = Run([self.cxx, '--version'])
self.signature = output.splitlines()[0].strip()
def build(engines, updateRepo=True, forceBuild=False, rev=None):
Engines = []
NumUpdated = 0
for engine in engines:
try:
engine.updateAndBuild(updateRepo, forceBuild, rev)
except Exception as err:
print('Build failed!')
print(err)
continue
if engine.cset == None:
continue
if engine.updated and engine.important:
NumUpdated += 1
Engines.append(engine)
return Engines, NumUpdated

76
driver/dorequests.py
Просмотреть файл

@ -1,76 +0,0 @@
# vim: set ts=4 sw=4 tw=99 et:
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import sys
import resource
import utils
import time
from optparse import OptionParser
from collections import namedtuple
import benchmarks
import builders
import puller
import slaves
import submitter
from remotecontroller import RemoteController
parser = OptionParser(usage="usage: %prog [options]")
parser.add_option("-c", "--config", dest="config_name", type="string", default="awfy.config",
help="Config file (default: awfy.config)")
(options, args) = parser.parse_args()
utils.InitConfig(options.config_name)
# Set resource limits for child processes
resource.setrlimit(resource.RLIMIT_AS, (-1, -1))
resource.setrlimit(resource.RLIMIT_RSS, (-1, -1))
resource.setrlimit(resource.RLIMIT_DATA, (-1, -1))
# A mode is a configuration of an engine we just built.
Mode = namedtuple('Mode', ['shell', 'args', 'env', 'name', 'cset'])
# Set of engines that get build.
KnownEngines = [builders.MozillaInbound()]
# The native compiler is a special thing, for now.
native = builders.NativeCompiler()
# No updates. Report to server and wait 60 seconds, before moving on
for slave in slaves.init():
remotecontroller = RemoteController(slave)
revs = remotecontroller.RequestedRevs();
for rev in revs:
Engines, NumUpdated = builders.build(KnownEngines, rev = rev["cset"])
# Make a list of all modes.
modes = []
for engine in Engines:
shell = os.path.join(utils.RepoPath, engine.source, engine.shell())
env = None
with utils.chdir(os.path.join(utils.RepoPath, engine.source)):
env = engine.env()
for m in engine.modes:
engineArgs = engine.args if engine.args else []
modeArgs = m['args'] if m['args'] else []
args = engineArgs + modeArgs
mode = Mode(shell, args, env, m['mode'], engine.cset)
modes.append(mode)
slave.prepare(Engines)
# Inform AWFY of each mode we found.
submit = submitter.RemoteSubmitter(slave)
submit.Start(rev["stamp"])
for mode in modes:
submit.AddEngine(mode.name, mode.cset)
submit.AddEngine(native.mode, native.signature)
slave.benchmark(submit, native, modes)
# Wait to finish before going on.
slave.synchronize()

93
driver/dostuff.py
Просмотреть файл

@ -1,93 +0,0 @@
# vim: set ts=4 sw=4 tw=99 et:
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
import os
import sys
import resource
import utils
import time
from optparse import OptionParser
from collections import namedtuple
import benchmarks
import builders
import puller
import slaves
from submitter import getSubmitter
from remotecontroller import RemoteController
parser = OptionParser(usage="usage: %prog [options]")
parser.add_option("-f", "--force", dest="force", action="store_true", default=False,
help="Force runs even without source changes")
parser.add_option("-n", "--no-update", dest="noupdate", action="store_true", default=False,
help="Skip updating source repositories")
parser.add_option("-c", "--config", dest="config_name", type="string", default="awfy.config",
help="Config file (default: awfy.config)")
parser.add_option("-s", "--submitter", dest="submitter", type="string", default="remote",
help="Submitter class ('remote' or 'print')")
(options, args) = parser.parse_args()
utils.config.init(options.config_name)
# Set resource limits for child processes
resource.setrlimit(resource.RLIMIT_AS, (-1, -1))
resource.setrlimit(resource.RLIMIT_RSS, (-1, -1))
resource.setrlimit(resource.RLIMIT_DATA, (-1, -1))
# Set of engines that get build.
KnownEngines = [builders.MozillaInbound(),
builders.V8(),
builders.Nitro()
]
Engines, NumUpdated = builders.build(KnownEngines, not options.noupdate, options.force)
submitter = getSubmitter(options.submitter)
# No updates. Report to server and wait 60 seconds, before moving on
if NumUpdated == 0 and not options.force:
if options.submitter == 'remote':
for slave in slaves.init():
submit = submitter(slave)
submit.Awake();
time.sleep(60)
sys.exit(0)
# The native compiler is a special thing, for now.
native = builders.NativeCompiler()
# A mode is a configuration of an engine we just built.
Mode = namedtuple('Mode', ['shell', 'args', 'env', 'name', 'cset'])
# Make a list of all modes.
modes = []
for engine in Engines:
shell = os.path.join(utils.config.RepoPath, engine.source, engine.shell())
env = engine.env()
for m in engine.modes:
engineArgs = engine.args if engine.args else []
modeArgs = m['args'] if m['args'] else []
args = engineArgs + modeArgs
mode = Mode(shell, args, env, m['mode'], engine.cset)
modes.append(mode)
# Set of slaves that run the builds.
KnownSlaves = slaves.init()
for slave in KnownSlaves:
slave.prepare(Engines)
# Inform AWFY of each mode we found.
submit = submitter(slave)
submit.Start()
for mode in modes:
submit.AddEngine(mode.name, mode.cset)
submit.AddEngine(native.mode, native.signature)
slave.benchmark(submit, native, modes)
# Wait for all of the slaves to finish running before exiting.
for slave in KnownSlaves:
slave.synchronize()

55
driver/jsc.patch
Просмотреть файл

@ -1,55 +0,0 @@
Index: jsc.cpp
===================================================================
--- jsc.cpp (revision 177145)
+++ jsc.cpp (working copy)
@@ -34,6 +34,7 @@
#include "Interpreter.h"
#include "JSArray.h"
#include "JSArrayBuffer.h"
+#include "JSArrayBufferConstructor.h"
#include "JSCInlines.h"
#include "JSFunction.h"
#include "JSLock.h"
@@ -461,6 +462,7 @@
static EncodedJSValue JSC_HOST_CALL functionRun(ExecState*);
static EncodedJSValue JSC_HOST_CALL functionLoad(ExecState*);
static EncodedJSValue JSC_HOST_CALL functionReadFile(ExecState*);
+static EncodedJSValue JSC_HOST_CALL functionReadBinaryFile(ExecState*);
static EncodedJSValue JSC_HOST_CALL functionCheckSyntax(ExecState*);
static EncodedJSValue JSC_HOST_CALL functionReadline(ExecState*);
static EncodedJSValue JSC_HOST_CALL functionPreciseTime(ExecState*);
@@ -598,6 +600,7 @@
addFunction(vm, "run", functionRun, 1);
addFunction(vm, "load", functionLoad, 1);
addFunction(vm, "readFile", functionReadFile, 1);
+ addFunction(vm, "readBinaryFile", functionReadBinaryFile, 1);
addFunction(vm, "checkSyntax", functionCheckSyntax, 1);
addFunction(vm, "jscStack", functionJSCStack, 1);
addFunction(vm, "readline", functionReadline, 0);
@@ -929,6 +932,26 @@
return JSValue::encode(jsString(exec, stringFromUTF(script.data())));
}
+EncodedJSValue JSC_HOST_CALL functionReadBinaryFile(ExecState* exec)
+{
+ String fileName = exec->argument(0).toString(exec)->value(exec);
+ Vector<char> script;
+ if (!fillBufferWithContentsOfFile(fileName, script))
+ return JSValue::encode(exec->vm().throwException(exec, createError(exec, ASCIILiteral("Could not open file."))));
+
+ JSArrayBufferConstructor* constructor =
+ jsCast<JSArrayBufferConstructor*>(exec->callee());
+
+ RefPtr<ArrayBuffer> buffer = ArrayBuffer::create(script.data(), script.size());
+ if (!buffer)
+ return JSValue::encode(exec->vm().throwException(exec, createError(exec, ASCIILiteral("Out of memory"))));
+
+ JSArrayBuffer* result = JSArrayBuffer::create(
+ exec->vm(), constructor->globalObject()->arrayBufferStructure(), buffer);
+
+ return JSValue::encode(result);
+}
+
EncodedJSValue JSC_HOST_CALL functionCheckSyntax(ExecState* exec)
{
String fileName = exec->argument(0).toString(exec)->value(exec);

87
driver/puller.py
Просмотреть файл

@ -1,87 +0,0 @@
# vim: set ts=4 sw=4 tw=99 et:
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
import re
import os
import sys
import subprocess
from utils import Run
from utils import FolderChanger
class HG:
@staticmethod
def Update(rev = None):
output = Run(['hg', 'pull', '-u'])
succeeded = re.search("no changes found", output) == None
if not rev:
return succeeded
output = Run(['hg', 'update', '-r', rev])
if re.search("unknown revision", output) != None:
raise Exception('unknown revision: ' + output)
return True
@staticmethod
def Identify():
output = Run(['hg', 'id', '-i'])
m = re.match("([0-9a-z]+)\s*", output)
if m == None:
raise Exception('unknown output from hg: ' + output)
return m.group(1)
class SVN:
@staticmethod
def Update(rev = None):
output = Run(['svn', 'update'])
succeeded = re.search("At revision", output) == None
if not rev:
return succeeded
output = Run(['svn', 'update', '-r', rev])
if re.search("No such revision", output) != None:
raise Exception('unknown revision: ' + output)
return True
@staticmethod
def Identify():
output = Run(['svn', 'info'])
m = re.search("Revision: ([0-9]+)", output)
if m == None:
raise Exception('unknown output from svn: ' + output)
return m.group(1)
class GIT:
@staticmethod
def Update(rev = None):
assert rev == None
output = Run(['git', 'pull'])
return re.search("Already up-to-date", output) == None
@staticmethod
def Identify():
output = Run(['git', 'log', '-1'])
m = re.match("commit ([0-9a-z]+)\s*", output)
if m == None:
raise Exception('unknown output from git: ' + output)
return m.group(1)
class V8GIT(GIT):
@staticmethod
def Update(rev = None):
assert rev == None
env = os.environ.copy()
with FolderChanger('..'):
Run(['gclient', 'sync'], {"PATH": "depot_tools/:"+env["PATH"]})
output = Run(['git', 'pull', 'origin', 'master'])
return re.search("Already up-to-date", output) == None
def get(name):
if name == 'svn':
return SVN
elif name == 'hg':
return HG
elif name == 'git':
return GIT
elif name == 'v8git':
return V8GIT
assert False

41
driver/remotecontroller.py
Просмотреть файл

@ -1,41 +0,0 @@
# vim: set ts=4 sw=4 tw=99 et:
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
import urllib2
import utils
class RemoteController(object):
def __init__(self, slave):
self.machine = slave.machine
self.urls = utils.config.get('main', 'updateURL').split(",")
self.runIds = []
for i in range(len(self.urls)):
self.urls[i] = self.urls[i].strip()
self.runIds.append(0)
def RequestedRevs(self):
data = []
for i in range(len(self.urls)):
try:
url = self.urls[i]
url += '?requests=1'
url += '&MACHINE=' + str(self.machine)
url = urllib2.urlopen(url)
data += json.loads(url.read())
except urllib2.URLError:
pass
return data
def Awake(self):
for i in range(len(self.urls)):
try:
url = self.urls[i]
url += '?awake=yes'
url += '&MACHINE=' + str(self.machine)
urllib2.urlopen(url)
except urllib2.URLError:
pass

41
driver/run.sh
Просмотреть файл

@ -1,41 +0,0 @@
#!/bin/bash
# Protocol for stopping AWFY:
#
# (1) If you want to stop it immediately:
# screen -r
# ctrl+c
# rm /tmp/awfy-daemon
# rm /tmp/awfy
# ctrl a+d
# Remember to start it again later!
#
# (2) If you want to it to stop when possible:
# touch /tmp/awfy
# screen -r
# (wait for it to confirm that it's no longer running)
# ctrl a+d
PATH=/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
if [ -e /tmp/awfy-daemon ]
then
echo "Already running"
exit 0
fi
touch /tmp/awfy-daemon
while :
do
if [ -e /tmp/awfy ]
then
echo "/tmp/awfy lock in place"
sleep 10
else
cd /Users/mozilla/awfy/ia32/driver
python dostuff.py --config=awfy-x86.config
cd /Users/mozilla/awfy/x64/driver
python dostuff.py --config=awfy-x64.config
fi
done
rm /tmp/awfy-daemon

119
driver/slaves.py
Просмотреть файл

@ -1,119 +0,0 @@
import sys
import utils
import re
import pickle
import os
import subprocess
import __main__
from collections import namedtuple
import benchmarks
class Slave(object):
def __init__(self, name):
self.name = name
self.machine = utils.config.get(name, 'machine')
def prepare(self, engines):
pass
def synchronize(self):
pass
def benchmark(self, submit, native, modes):
benchmarks.run(submit, native, modes)
class RemoteSlave(Slave):
def __init__(self, name):
super(RemoteSlave, self).__init__(name)
self.HostName = utils.config_get_default(name, 'hostname', name)
self.delayed = None
self.delayedCommand = None
def prepare(self, engines):
self.pushRemote(utils.DriverPath + os.path.sep, self.DriverPath)
self.pushRemote(utils.BenchmarkPath + os.path.sep, self.BenchmarkPath)
for engine in engines:
shell = os.path.join(utils.RepoPath, engine.source, engine.shell())
rshell = os.path.join(self.RepoPath, engine.source, engine.shell())
self.runRemote(["mkdir", "-p", os.path.dirname(rshell)])
self.pushRemote(shell, rshell, follow=True)
def benchmark(self, submit, native, modes):
fd = open("state.p", "wb")
# dump the global state gathered from the config file
pickle.dump(utils.config, fd)
# dump out all the arguments
pickle.dump(submit, fd)
pickle.dump(native, fd)
pickle.dump(modes, fd)
fd.close()
# send the pickled data over the wire so we can make a call
self.pushRemote(os.path.join(utils.DriverPath, "state.p"), os.path.join(self.DriverPath, "state.p"))
# cd into the driver's directory, then start running the module.
self.runRemote(["cd", self.DriverPath, ";", self.PythonName, 'slaves.py', os.path.join(self.DriverPath, "state.p")], async=True)
def runRemote(self, cmds, async = False):
# no matter what, we don't want to start running a new command until the old one is gone.
self.synchronize()
fullcmd = ["ssh", self.HostName, "--"] + cmds
if async:
print ("ASYNC: " + " ".join(fullcmd))
self.delayed = subprocess.Popen(fullcmd, stderr = subprocess.STDOUT, stdout = subprocess.PIPE)
subprocess.Popen(['sed', '-e', 's/^/' + self.name + ': /'], stdin = self.delayed.stdout)
self.delayedCommand = str(fullcmd)
else:
utils.Run(fullcmd)
def pushRemote(self, file_loc, file_remote, follow = False):
rsync_flags = "-aP"
# if they asked us to follow symlinks, then add '-L' into the arguments.
if follow:
rsync_flags += "L"
utils.Run(["rsync", rsync_flags, file_loc, self.HostName + ":" + file_remote])
def synchronize(self):
if self.delayed:
print("Waiting for: "+self.delayedCommand)
retval = self.delayed.wait()
if retval != 0:
raise Exception(self.delayedCommand + ": failed with exit code" + str(retval))
self.delayed = None
self.delayedCommand = None
def init():
slaves = []
slaveNames = utils.config.getDefault('main', 'slaves', None)
if slaveNames:
slaveNames = slaveNames.split(",")
for name in slaveNames:
remote = utils.config.getDefault(name, 'remote', 1)
if remote == 1:
slaves.append(RemoteSlave(name))
else:
slaves.append(Slave(name))
if not slaves:
slaves = [Slave("main")]
return slaves
if __name__ == "__main__":
Mode = namedtuple('Mode', ['shell', 'args', 'env', 'name', 'cset'])
state = sys.argv[1]
fd = open(state, "rb")
# pull out the global configuration
utils.config = pickle.load(fd)
# pull out the pickled arguments
submit = pickle.load(fd)
native = pickle.load(fd)
modes = pickle.load(fd)
fd.close()
# call the one true function
benchmarks.run(submit, native, modes)

2
browser-driver/awfy.desktop.config.sample → slave/awfy.config
Просмотреть файл

@ -2,7 +2,7 @@
slaveType = linux-desktop
repos = /Users/mozilla/awfy/repos
tmpDir= /tmp/
timeout = 20 ; in minutes
timeout = 20*60 ; in seconds
serverURL = http://localhost:8000/
machine = 16

14
slave/benchmarks.py Normal file
Просмотреть файл

@ -0,0 +1,14 @@
def getBenchmark(benchmark):
section, name = benchmark.split(".")
if section == "local":
import benchmarks_local
return benchmarks_local.getBenchmark(name);
elif section == "remote":
import benchmarks_remote
return benchmarks_remote.getBenchmark(name);
elif section == "shell":
import benchmarks_shell
return benchmarks_shell.getBenchmark(name);
else:
raise Exception("Unknown benchmark type")

0
browser-driver/benchmarks_local.py → slave/benchmarks_local.py
Просмотреть файл

0
browser-driver/benchmarks_remote.py → slave/benchmarks_remote.py
Просмотреть файл

0
browser-driver/benchmarks_shell.py → slave/benchmarks_shell.py
Просмотреть файл

289
slave/build.py Normal file
Просмотреть файл

@ -0,0 +1,289 @@
import json
import urllib2
import urllib
import re
import os
import shutil
import socket
import utils
import puller
import platform
from utils import Run
import tarfile
import zipfile
socket.setdefaulttimeout(120)
class Environment(object):
def __init__(self):
self.env_ = os.environ.copy()
self.add("CC", "gcc")
self.add("CXX", "g++")
self.add("LINK", "g++")
self.ccoption = []
def add(self, name, data):
self.env_[name] = data
def addCCOption(self, option):
self.ccoption.append(option)
def get(self):
env = self.env_.copy()
env["CC"] += " " + " ".join(self.ccoption)
env["CXX"] += " " + " ".join(self.ccoption)
return env
class Builder(object):
def __init__(self, config, folder):
self.env = Environment()
self.config = config
self.folder = folder
if platform.system() == "Darwin":
self.installClang()
self.env.add("CC", os.path.abspath("clang-3.3/bin/clang"))
self.env.add("CXX", os.path.abspath("clang-3.3/bin/clang++"))
self.env.add("LINK", os.path.abspath("clang-3.3/bin/clang++"))
def installClang(self):
# The standard clang version on mac is outdated.
# Retrieve a better one.
if os.path.exists("clang-3.3"):
return
urllib.urlretrieve("http://llvm.org/releases/3.3/clang+llvm-3.3-x86_64-apple-darwin12.tar.gz", "./clang-3.3.tar.gz")
tar = tarfile.open("clang-3.3.tar.gz")
tar.extractall(".")
tar.close()
shutil.move("clang+llvm-3.3-x86_64-apple-darwin12", "clang-3.3")
os.unlink("clang-3.3.tar.gz")
def unlinkBinary(self):
try:
os.unlink(self.binary())
except:
pass
def unlinkObjdir(self):
try:
shutil.rmtree(self.objdir())
except:
pass
def successfullyBuild(self):
return os.path.isfile(self.binary())
def reconf(self):
return
def build(self, puller):
self.unlinkBinary()
self.make()
if not self.successfullyBuild():
self.reconf()
self.make()
assert self.successfullyBuild()
info = self.retrieveInfo()
info["revision"] = puller.identify()
info["shell"] = True
info["binary"] = os.path.abspath(self.binary())
fp = open(self.folder + "info.json", "w")
json.dump(info, fp)
fp.close()
class MozillaBuilder(Builder):
def __init__(self, config, folder):
super(MozillaBuilder, self).__init__(config, folder);
if platform.architecture()[0] == "64bit" and self.config == "32bit":
self.env.add("AR",'ar')
self.env.add("CROSS_COMPILE", '1')
self.env.addCCOption("-m32")
def retrieveInfo(self):
info = {}
info["engine_type"] = "firefox"
return info
def objdir(self):
return os.path.join(self.folder, 'js', 'src', 'Opt')
def binary(self):
return os.path.join(self.objdir(), 'dist', 'bin', 'js')
def reconf(self):
# Step 1. autoconf.
with utils.FolderChanger(os.path.join(self.folder, 'js', 'src')):
if platform.system() == "Darwin":
utils.Shell("autoconf213")
elif platform.system() == "Linux":
utils.Shell("autoconf2.13")
elif platform.system() == "Windows":
utils.Shell("autoconf-2.13")
# Step 2. configure
if not os.path.exists(os.path.join(self.folder, 'js', 'src', 'Opt')):
os.mkdir(os.path.join(self.folder, 'js', 'src', 'Opt'))
with utils.FolderChanger(os.path.join(self.folder, 'js', 'src', 'Opt')):
args = ['--enable-optimize', '--disable-debug']
if platform.architecture()[0] == "64bit" and self.config == "32bit":
if platform.system() == "Darwin":
args.append("--target=i686-apple-darwin10.0.0")
elif platform.system() == "Linux":
args.append("--target=i686-pc-linux-gnu")
else:
assert False
Run(['../configure'] + args, self.env.get())
return True
def make(self):
if not os.path.exists(os.path.join(self.folder, 'js', 'src', 'Opt')):
return
utils.Shell("make -j6 -C " + os.path.join(self.folder, 'js', 'src', 'Opt'))
class WebkitBuilder(Builder):
def retrieveInfo(self):
with utils.chdir(os.path.join(self.folder)):
objdir = os.path.abspath(os.path.join('WebKitBuild', 'Release'))
info = {}
info["engine_type"] = "webkit"
info["env"] = {'DYLD_FRAMEWORK_PATH': objdir}
return info
def make(self):
with utils.chdir(os.path.join(self.folder)):
# Hack 1: Remove reporting errors for warnings that currently are present.
Run(["sed","-i.bac","s/GCC_TREAT_WARNINGS_AS_ERRORS = YES;/GCC_TREAT_WARNINGS_AS_ERRORS=NO;/","Source/JavaScriptCore/Configurations/Base.xcconfig"])
Run(["sed","-i.bac","s/GCC_TREAT_WARNINGS_AS_ERRORS = YES;/GCC_TREAT_WARNINGS_AS_ERRORS=NO;/","Source/bmalloc/Configurations/Base.xcconfig"])
Run(["sed","-i.bac","s/GCC_TREAT_WARNINGS_AS_ERRORS = YES;/GCC_TREAT_WARNINGS_AS_ERRORS=NO;/","Source/WTF/Configurations/Base.xcconfig"])
Run(["sed","-i.bac","s/std::numeric_limits<unsigned char>::max()/255/","Source/bmalloc/bmalloc/Line.h"])
Run(["sed","-i.bac","s/std::numeric_limits<unsigned char>::max()/255/","Source/bmalloc/bmalloc/Page.h"])
Run(["patch","Source/JavaScriptCore/jsc.cpp","../../driver/jsc.patch"])
with utils.FolderChanger(os.path.join('Tools', 'Scripts')):
# Hack 2: This check fails currently. Disable checking to still have a build.
os.rename("check-for-weak-vtables-and-externals", "check-for-weak-vtables-and-externals2");
args = ['/usr/bin/perl', 'build-jsc']
if self.config == '32bit':
args += ['--32-bit']
Run(args, self.env.get())
os.rename("check-for-weak-vtables-and-externals2", "check-for-weak-vtables-and-externals");
Run(["svn","revert","Source/JavaScriptCore/Configurations/Base.xcconfig"])
Run(["svn","revert","Source/bmalloc/Configurations/Base.xcconfig"])
Run(["svn","revert","Source/WTF/Configurations/Base.xcconfig"])
Run(["svn","revert","Source/bmalloc/bmalloc/Line.h"])
Run(["svn","revert","Source/bmalloc/bmalloc/Page.h"])
Run(["svn","revert","Source/JavaScriptCore/jsc.cpp"])
def objdir(self):
return os.path.join(self.folder, 'WebKitBuild', 'Release')
def binary(self):
return os.path.join(self.objdir(), 'jsc')
class V8Builder(Builder):
def __init__(self, config, folder):
super(V8Builder, self).__init__(config, folder)
self.env.add("GYP_DEFINES", "clang=1")
def retrieveInfo(self):
info = {}
info["engine_type"] = "chrome"
info["args"] = ['--expose-gc']
return info
def make(self):
args = ['make', '-j6', '-C', self.folder]
if self.config == '32bit':
args += ['ia32.release']
elif self.config == '64bit':
args += ['x64.release']
else:
assert True
Run(args, self.env.get())
def objdir(self):
if self.config == '64bit':
return os.path.join('out', 'x64.release')
elif self.config == '32bit':
return os.path.join('out', 'ia32.release')
else:
assert False
def binary(self):
return os.path.join(self.objdir(), 'd8')
def getBuilder(config, path):
# fingerprint the known builders
if os.path.exists(os.path.join(path, "js/src/")):
return MozillaBuilder(config, path)
if os.path.exists(os.path.join(path, "Source/JavaScriptCore")):
return WebkitBuilder(config, path)
if os.path.exists(os.path.join(path, "LICENSE.v8")):
return V8Builder(config, path)
raise Exception("Unknown builder")
if __name__ == "__main__":
from optparse import OptionParser
parser = OptionParser(usage="usage: %prog [options]")
parser.add_option("-s", "--source", dest="repo",
help="The url of the repo to fetch or one of the known repos name. (mozilla, v8 and webkit are supported.)", default='mozilla')
parser.add_option("-r", "--rev", dest="revision",
help="Force this revision to get build")
parser.add_option("-o", "--output", dest="output",
help="download to DIR, default=output/", metavar="DIR", default='output')
parser.add_option("-c", "--config", dest="config",
help="default, 32bit, 64bit", default='default')
parser.add_option("-f", "--force", dest="force", action="store_true", default=False,
help="Force runs even without source changes")
(options, args) = parser.parse_args()
if options.repo is None:
print "Please provide the source repository to pull"
exit()
if not options.output.endswith("/"):
options.output += "/"
if options.config not in ["default", "32bit", "64bit"]:
print "Please provide a valid config"
exit()
if options.config == "default":
options.config, _ = platform.architecture()
if options.config == "64bit" and platform.architecture()[0] == "32bit":
print "Cannot compile a 64bit binary on 32bit architecture"
exit()
puller = puller.getPuller(options.repo, options.output)
puller.update(options.revision)
builder = getBuilder(options.config, puller.path())
if options.force:
builder.unlinkObjdir()
builder.build(puller)

71
slave/configs.py Normal file
Просмотреть файл

@ -0,0 +1,71 @@
import os
import sys
sys.path.insert(1, '../driver')
import utils
#TODO: move into builder
#with utils.chdir(os.path.join(utils.config.RepoPath, self.source)):
# self.env['DYLD_FRAMEWORK_PATH'] = os.path.abspath('WebKitBuild/Release'))
#self.omit = False
class Default(object):
def __init__(self, engine, shell):
self.args_ = []
self.env_ = {}
self.omit_ = False
if engine == "firefox":
self.env_["JSGC_DISABLE_POISONING"] = "1"
elif engine == "chrome":
pass
elif engine == "webkit":
pass
elif engine == "ie":
pass
else:
self.omit_ = True
def omit(self):
return self.omit_
def args(self):
return self.args_
def env(self):
return self.env_
class UnboxedObjects(Default):
def __init__(self, engine, shell):
super(UnboxedObjects, self).__init__(engine, shell)
if engine == "firefox":
self.env_["JS_OPTION_USE_UNBOXED_ARRAYS"] = '1'
else:
self.omit_ = True
class TurboFan(Default):
def __init__(self, engine, shell):
super(TurboFan, self).__init__(engine, shell)
if engine == "chrome"and shell:
self.args.append("--turbo");
else:
self.omit_ = True
class NoAsmjs(Default):
def __init__(self, engine, shell):
super(NoAsmjs, self).__init__(engine, shell)
if engine == "firefox" and shell:
self.args.append("--no-asmjs");
else:
self.omit_ = True
def getConfig(name, info):
if name == "default":
return Default(info["engine_type"], info["shell"])
if name == "unboxedobjects":
return UnboxedObjects(info["engine_type"], info["shell"])
if name == "turbofan":
return TurboFan(info["engine_type"], info["shell"])
if name == "noasmjs":
return NoAsmjs(info["engine_type"], info["shell"])
raise Exception("Unknown config")

143
slave/download.py Normal file
Просмотреть файл

@ -0,0 +1,143 @@
import json
import urllib2
import urllib
import re
import os
import shutil
import socket
import utils
import tarfile
import zipfile
socket.setdefaulttimeout(120)
DEBUG = True
class Retriever(object):
def __init__(self, url):
if not url.endswith("/"):
url += "/"
self.url = url
self.folder = "./"
def setOutputFolder(self, folder):
if not folder.endswith("/"):
folder += "/"
self.folder = folder
def download(self):
self.createOutputFolder()
filename = self.getfilename()
self.retrieve(filename)
self.extract(filename)
info = self.retrieveInfo()
fp = open(self.folder + "info.json", "w")
json.dump(info, fp)
fp.close()
def createOutputFolder(self):
if os.path.isdir(self.folder):
shutil.rmtree(self.folder)
os.makedirs(self.folder)
def retrieve(self, filename):
#if DEBUG:
# shutil.copy("firefox.tar.bz2", self.folder + filename)
# return
print "Retrieving", self.url + filename
urllib.urlretrieve(self.url + filename, self.folder + filename)
def extract(self, filename):
if "tar.bz2" in filename:
tar = tarfile.open(self.folder + filename)
tar.extractall(self.folder)
tar.close()
elif "zip" in filename:
zip = zipfile.ZipFile(self.folder + filename)
zip.extractall(self.folder)
zip.close()
class ArchiveMozilla(Retriever):
def getfilename(self):
response = urllib2.urlopen(self.url)
html = response.read()
possibles = re.findall(r'<a href="(firefox-[a-zA-Z0-9._-]*)">', html)
possibles = [possible for possible in possibles if "tests" not in possible]
possibles = [possible for possible in possibles if "checksum" not in possible]
possibles = [possible for possible in possibles if ".json" not in possible]
possibles = [possible for possible in possibles if "crashreporter" not in possible]
possibles = [possible for possible in possibles if "langpack" not in possible]
possibles = [possible for possible in possibles if ".txt" not in possible]
possibles = [possible for possible in possibles if ".installer." not in possible]
assert len(possibles) == 1
return possibles[0]
def getinfoname(self):
response = urllib2.urlopen(self.url)
html = response.read()
possibles = re.findall(r'<a href="(firefox-[a-zA-Z0-9._-]*)">', html)
possibles = [possible for possible in possibles if ".json" in possible]
possibles = [possible for possible in possibles if "mozinfo" not in possible]
possibles = [possible for possible in possibles if "test_packages" not in possible]
assert len(possibles) == 1
return possibles[0]
def getbinary(self):
if os.path.exists(self.folder + "firefox/firefox.exe"):
return self.folder + "firefox/firefox.exe"
if os.path.exists(self.folder + "firefox/firefox"):
return self.folder + "firefox/firefox"
files = os.listdirs()
assert len(files) == 1
if files[0].endswith(".apk"):
return files[0]
if files[0].endswith(".dmg"):
return files[0]
assert False
def retrieveInfo(self):
infoname = self.getinfoname()
response = urllib2.urlopen(self.url + infoname)
raw_info = json.loads(response.read())
info = {}
info["revision"] = raw_info["moz_source_stamp"]
info["engine_type"] = "firefox"
info["shell"] = False
info["binary"] = os.path.abspath(self.getbinary())
return info
def getRetriever(url):
if (url.startswith("http://archive.mozilla.org") or
url.startswith("https://archive.mozilla.org") or
url.startswith("http://ftp.mozilla.org") or
url.startswith("https://ftp.mozilla.org")):
return ArchiveMozilla(url)
if url.startswith("http://commondatastorage.googleapis.com"):
return GoogleAPIS(url)
if url.startswith("http://builds.nightly.webkit.org"):
return BuildsWebkit(url)
raise Exception("Unknown retriever")
if __name__ == "__main__":
from optparse import OptionParser
parser = OptionParser(usage="usage: %prog url [options]")
parser.add_option("-o", "--output", dest="output",
help="download to DIR, default=output/", metavar="DIR", default='output')
(options, args) = parser.parse_args()
retriever = getRetriever(args[0])
retriever.setOutputFolder(options.output)
retriever.download()

27
slave/engineInfo.py Normal file
Просмотреть файл

@ -0,0 +1,27 @@
import json
def getInfo(path):
if not path.endswith("/"):
path += "/"
fp = open(path + "info.json", 'r')
info = json.load(fp)
fp.close();
# which platform to execute:
if info["binary"].endswith(".apk"):
info["platform"] = "android"
elif info["binary"].endswith(".dmg"):
info["platform"] = "osx"
elif info["binary"].endswith(".exe"):
info["platform"] = "win"
else:
info["platform"] = "linux"
# default args and env
if "args" not in info:
info["args"] = []
if "env" not in info:
info["env"] = {}
return info

85
slave/execute.py Normal file
Просмотреть файл

@ -0,0 +1,85 @@
import benchmarks
import configs
import executors
import engineInfo
import submitter
import sys
sys.path.insert(1, '../driver')
import utils
from optparse import OptionParser
parser = OptionParser(usage="usage: %prog url [options]")
parser.add_option("-b", "--benchmark", action="append", dest="benchmarks",
help="Benchmark to run (the local ones are deprecated): remote.octane, remote.dromaeo, remote.massive, remote.jetstream, remote.speedometer, remote.kraken, remote.sunspider, remote.browsermark, shell.octane, shell.sunspider, shell.kraken, shell.assorted, shell.asmjsapps, shell.asmjsmicro, shell.shumway, shell.dart, local.octane, local.sunspider, local.kraken, local.weglsamples, local.assorteddom")
parser.add_option("-s", "--submitter", dest="submitter", type="string", default="print",
help="Submitter class ('remote' or 'print')")
parser.add_option("--submitter_mode", action="append", dest="mode_rules",
help="When using the remote submitter, give rules to the mode name that needs to get submitted. Format: engine,config_name:mode_name. E.g. firefox,default:jmim")
parser.add_option("-e", "--engine", action="append", dest="engines",
help="Path to the engines that need to get benchmarked")
parser.add_option("-c", "--config", action="append", dest="configs",
help="The runtime configs that need to get executed: default, unboxedobjects, turbofan, noasmjs")
(options, args) = parser.parse_args()
if options.engines is None:
options.engines = ["output"]
if options.configs is None:
options.configs = ["default"]
if options.benchmarks is None:
print "Please provide a benchmark to run."
exit()
if options.mode_rules is None:
options.mode_rules = [
"firefox,default:jmim",
"firefox,unboxedobjects:unboxedobjects",
"chrome,default:v8",
"chrome,turbofan:v8-turbofan",
"webkit,default:jsc",
"native,default:clang"
]
#TODO:remove
utils.config.init("awfy.config")
submitter = submitter.getSubmitter(options.submitter)
submitter.start()
submitter.setModeRules(options.mode_rules)
# Submit the revisions for every build.
for engine_path in options.engines:
info = engineInfo.getInfo(engine_path)
for config_name in options.configs:
config = configs.getConfig(config_name, info)
if config.omit():
continue
submitter.createBuild(info["engine_type"], config_name, info["revision"])
# Run every benchmark for every build and config
benchmarks = [benchmarks.getBenchmark(i) for i in options.benchmarks]
for benchmark in benchmarks:
for engine_path in options.engines:
info = engineInfo.getInfo(engine_path)
executor = executors.getExecutor(info)
for config_name in options.configs:
config = configs.getConfig(config_name, info)
if config.omit():
continue
results = executor.run(benchmark, config)
mode = submitter.mode(info["engine_type"], config_name)
submitter.addTests(results, benchmark.suite, benchmark.version, mode)
submitter.finish()

100
slave/executors.py Normal file
Просмотреть файл

@ -0,0 +1,100 @@
import runners
import time
import os
import sys
sys.path.insert(1, '../driver')
import utils
class ShellExecutor(object):
def __init__(self, engineInfo):
self.engineInfo = engineInfo
def run(self, benchmark, config):
env = os.environ.copy()
env.update(config.env())
env.update(self.engineInfo["env"])
args = config.args() + self.engineInfo["args"]
with utils.chdir(os.path.join(utils.config.BenchmarkPath, benchmark.folder)):
return benchmark.benchmark(self.engineInfo["binary"], env, args)
class BrowserExecutor(object):
def __init__(self, engineInfo):
self.engineInfo = engineInfo
def run(self, benchmark, config):
env = os.environ.copy()
env.update(config.env())
env.update(self.engineInfo["env"])
args = [benchmark.url] + config.args() + self.engineInfo["args"]
self.execute(benchmark, env, args)
fp = open("results", "r")
results = json.loads(fp.read())
fp.close()
return benchmark.processResults(results)
def resetResults(self):
if not os.path.exists("results"):
return
os.unlink("results")
def waitForResults(self):
timeout = int(utils.config.get('main', 'timeout')) * 60
while not os.path.exists("results") and timeout > 0:
time.sleep(10)
timeout -= 10
class FirefoxExecutor(BrowserExecutor):
def execute(self, page, env, args):
runner = runners.getRunner(self.engineInfo["platform"], {
"osx_mount_point": "/Volumes/Nightly",
"android_processname": "org.mozilla.fennec"
})
# kill all possible running instances.
runner.killAllInstances()
runner.killall("plugin-container")
# if needed install the executable
runner.install(self.engineInfo["binary"])
# delete profile
runner.rm("profile/")
# create new profile
runner.mkdir("profile/")
# Update profile to disable slow script dialog
runner.write("profile/prefs.js", "user_pref(\"dom.max_script_run_time\", 0);")
# reset the result
self.resetResults()
# start browser
process = runner.start(self.engineInfo["binary"], args + ["--profile", runner.getdir("profile")], env)
# wait for results
self.waitForResults()
# kill browser
runner.kill(process)
#class NativeExecutor(Executor):
#
# def execute(self, env, args):
# pass
def getExecutor(engineInfo):
if engineInfo["shell"]:
return ShellExecutor(engineInfo)
if engineInfo["engine_type"] == "firefox" and not engineInfo["shell"]:
return FirefoxExecutor(engineInfo)

156
slave/puller.py Normal file
Просмотреть файл

@ -0,0 +1,156 @@
import re
import os
import sys
import subprocess
from utils import Run
from utils import FolderChanger
import shutil
class Puller(object):
def __init__(self, repo, folder):
self.repo = repo
self.folder = folder
if self.sameRepo():
return
shutil.rmtree(self.folder)
self.clone()
assert self.sameRepo()
def path(self):
return self.folder
class HG(Puller):
def sameRepo(self):
if not os.path.exists(self.folder + ".hg/hgrc"):
return False
fp = open(self.folder + ".hg/hgrc")
for line in fp.readlines():
if "default = "+self.repo in line:
return True
return False
def clone(self):
Run(['hg', 'clone', self.repo, self.folder])
def update(self, rev = None):
output = Run(['hg', 'pull', '--cwd', self.folder])
if not rev:
output = Run(['hg', 'update', '--cwd', self.folder])
return
else:
output = Run(['hg', 'update', '-r', rev, '--cwd', self.folder])
if re.search("unknown revision", output) != None:
raise Exception('unknown revision: ' + output)
return
def identify(self):
output = Run(['hg', 'id', '-i', '--cwd', self.folder])
m = re.match("([0-9a-z]+)\s*", output)
if m == None:
raise Exception('unknown output from hg: ' + output)
return m.group(1)
class SVN(Puller):
def clone(self):
Run(['svn', 'co', self.repo, self.folder])
def sameRepo(self):
return False
def update(self, rev = None):
with FolderChanger(self.path()):
if not rev:
output = Run(['svn', 'update'])
return
output = Run(['svn', 'update', '-r', rev])
if re.search("No such revision", output) != None:
raise Exception('unknown revision: ' + output)
return
def identify():
with FolderChanger(self.path()):
output = Run(['svn', 'info'])
m = re.search("Revision: ([0-9]+)", output)
if m == None:
raise Exception('unknown output from svn: ' + output)
return m.group(1)
class GIT(Puller):
def clone(self):
Run(['git', 'clone', self.repo, self.folder])
def sameRepo(self):
if not os.path.exists(self.path() + ".git/config"):
return False
fp = open(self.path() + ".git/config")
for line in fp.readlines():
if "url = "+self.repo in line:
return True
return False
def update(self, rev = None):
assert rev == None
with FolderChanger(self.path()):
output = Run(['git', 'pull'])
def identify(self):
with FolderChanger(self.path()):
output = Run(['git', 'log', '-1'])
m = re.match("commit ([0-9a-z]+)\s*", output)
if m == None:
raise Exception('unknown output from git: ' + output)
return m.group(1)
class V8GIT(GIT):
def clone(self):
os.mkdir(self.folder)
with FolderChanger(self.folder):
# get depot_tools
Run(['git', 'clone', 'https://chromium.googlesource.com/chromium/tools/depot_tools.git'])
# get actual v8 source
env = os.environ.copy()
Run(['fetch', 'v8'], {"PATH": "depot_tools/:"+env["PATH"]})
#TODO: not needed?
#with FolderChanger(self.path()):
# Run(['git', 'checkout', 'master'])
def path(self):
return os.path.join(self.folder, "v8")
def sameRepo(self):
# Currently only V8 uses this puller
return os.path.exists(os.path.join(self.path(), "LICENSE.v8"))
def update(self, rev = None):
assert rev == None
Run(['git', 'pull', 'origin', 'master'])
env = os.environ.copy()
with FolderChanger(self.folder):
Run(['gclient', 'sync'], {"PATH": "depot_tools/:"+env["PATH"]})
def getPuller(repo, path):
if repo == "mozilla":
repo = "http://hg.mozilla.org/integration/mozilla-inbound"
elif repo == "webkit":
repo = "https://svn.webkit.org/repository/webkit/trunk"
if "hg." in repo:
return HG(repo, path)
if "svn." in repo:
return SVN(repo, path)
if repo.endswith(".git"):
return GIT(repo, path)
if repo == "v8":
return V8GIT(repo, path)
raise Exception("Unknown puller")

117
slave/runners.py Normal file
Просмотреть файл

@ -0,0 +1,117 @@
import subprocess
import shutil
import time
import os
class Runner(object):
def __init__(self, info):
self.info = info
def rm(self, path):
print "rm", path
if not os.path.exists(path):
return
shutil.rmtree(path)
def kill(self, process):
print "kill", process.pid
process.terminate()
for i in range(100):
time.sleep(0.1)
if process.poll():
return
time.sleep(0.1)
try:
os.kill(int(process.pid), signal.SIGTERM)
except:
pass
def write(self, path, content):
print "write", path, content
fp = open(path, 'w')
fp.write(content);
fp.close()
def mkdir(self, path):
print "mkdir", path
os.mkdir(path)
def getdir(self, path):
return path
class LinuxRunner(Runner):
def killall(self, name):
print "killall", name
subprocess.Popen(["killall", name])
def killAllInstances(self):
print "killallinstances"
self.killall("firefox")
def start(self, exe, args = [], env = {}):
print "start", exe, args, env
return subprocess.Popen([exe] + args, env=env)
def install(self, exe):
pass
class OSXRunner(Runner):
def killall(self, name):
subprocess.Popen(["killall", name])
def killAllInstances(self):
if not os.path.exists(self.info["osx_mount_point"]):
return
subprocess.check_output(["hdiutil", "detach", "-force", self.info["osx_mount_point"]])
def start(self, path, args, env):
pass
def install(self, exe):
subprocess.check_output(["hdiutil", "attach", exe])
class AndroidRunner(Runner):
def killall(self, name):
pass
def killAllInstances(self):
subprocess.check_output(["adb", "shell", "pm", "clear", self.info["android_processname"]]);
pass
def start(self, path, args, env):
pass
def rm(self, path):
subprocess.check_output(["adb", "shell", "rm -rf " + self.getdir(path)])
def install(self, exe):
subprocess.check_output(["adb", "install", "-r", exe])
def kill(self):
self.killAllInstances()
def write(self, path, content):
assert "'" not in content
subprocess.check_output(["adb", "shell", "echo '" + content + "' > " + self.getdir(path)])
def mkdir(self, path):
subprocess.check_output(["adb", "shell", "mkdir " + self.getdir(path)])
def getdir(self, path):
return "/storage/emulated/legacy/" + path
def getRunner(platform, info):
if platform == "linux":
return LinuxRunner(info)
if platform == "osx":
return OSXRunner(info)
if platform == "android":
return AndroidRunner(info)
if platform == "windows":
return WindowsRunner(info)

2
browser-driver/server.py → slave/server.py
Просмотреть файл

@ -48,7 +48,7 @@ class FakeHandler(SimpleHTTPRequestHandler):
def captureResults(self, query):
queryParsed = urlparse.parse_qs(query)
fp = open("browser-driver/results", "w");
fp = open("puppet/results", "w");
fp.write(queryParsed["results"][0]);
fp.close()

62
driver/submitter.py → slave/submitter.py
Просмотреть файл

@ -9,13 +9,20 @@ import urllib2
import json
import utils
from remotecontroller import RemoteController
class RemoteSubmitter(RemoteController):
def __init__(self, slave):
super(RemoteSubmitter, self).__init__(slave)
class Submitter(object):
def Start(self, timestamp=None):
def setModeRules(self, rules):
self.rules = {}
for rule in rules:
rule = rule.split(":")
self.rules[rule[0]] = rule[1]
def mode(self, engine_type, config):
return self.rules[engine_type + "," + config]
class RemoteSubmitter(Submitter):
def start(self, timestamp=None):
for i in range(len(self.urls)):
try:
url = self.urls[i]
@ -33,20 +40,22 @@ class RemoteSubmitter(RemoteController):
except urllib2.URLError:
self.runIds[i] = None
def AddEngine(self, name, cset):
def createBuild(self, engine_type, config, cset):
mode = self.mode(engine_type, config)
for i in range(len(self.urls)):
if not self.runIds[i]:
continue
args = { 'run': 'addEngine',
'runid': str(self.runIds[i]),
'name': name,
'name': mode,
'cset': cset
}
url = self.urls[i] + '?' + urllib.urlencode(args)
urllib2.urlopen(url)
return mode
def AddTests(self, tests, suite, suiteversion, mode, extra_info = ""):
def addTests(self, tests, suite, suiteversion, mode, extra_info = ""):
for i in range(len(self.urls)):
if not self.runIds[i]:
continue
@ -56,16 +65,16 @@ class RemoteSubmitter(RemoteController):
run = self.runIds[i]
for test in tests:
if test['name'] == "__total__":
score = self.SubmitTest(submiturl, run, suite, suiteversion, mode, test['time'], extra_info)
score = self.submitTest(submiturl, run, suite, suiteversion, mode, test['time'], extra_info)
break
if score is None:
score = self.SubmitTest(submiturl, run, suite, suiteversion, mode, 0, extra_info)
score = self.submitTest(submiturl, run, suite, suiteversion, mode, 0, extra_info)
for test in tests:
if test['name'] != "__total__":
self.SubmitBreakdown(submiturl, run, score, test['name'], suite, suiteversion, mode, test['time'])
self.submitBreakdown(submiturl, run, score, test['name'], suite, suiteversion, mode, test['time'])
def SubmitTest(self, submiturl, run, suite, suiteversion, mode, time, extra_info = ""):
def submitTest(self, submiturl, run, suite, suiteversion, mode, time, extra_info = ""):
try:
args = { 'name': '__total__',
'run': str(run),
@ -86,7 +95,7 @@ class RemoteSubmitter(RemoteController):
except urllib2.URLError:
return None
def SubmitBreakdown(self, submiturl, run, score, name, suite, suiteversion, mode, time):
def submitBreakdown(self, submiturl, run, score, name, suite, suiteversion, mode, time):
# TODO: remove mode. Breakdown doesn't need it
args = { 'name': name,
'run': str(run),
@ -99,7 +108,7 @@ class RemoteSubmitter(RemoteController):
url = submiturl + '?' + urllib.urlencode(args)
urllib2.urlopen(url)
def Finish(self, status):
def finish(self, status = 1):
for i in range(len(self.urls)):
if not self.runIds[i]:
continue
@ -110,40 +119,41 @@ class RemoteSubmitter(RemoteController):
url += '&runid=' + str(self.runIds[i])
urllib2.urlopen(url)
class PrintSubmitter(object):
def __init__(self, slave):
self.slave = slave
class PrintSubmitter(Submitter):
def __init__(self):
self.msg = ''
def log(self, msg):
self.msg += msg + '\n'
print msg
def Start(self, timestamp=None):
def start(self, timestamp=None):
msg = "Starting benchmark"
if timestamp:
msg += " at timestamp" + str(timestamp)
self.log(msg)
def AddEngine(self, name, cset):
self.log("Added engine %s (changeset: %s)" % (name, cset))
def createBuild(self, engine_type, config, cset):
mode = self.mode(engine_type, config)
self.log("Added mode %s (engine: %s, config: %s, changeset: %s)" % (mode, engine_type, config, cset))
return mode
def AddTests(self, tests, suite, suiteversion, mode, extra_info = ""):
def addTests(self, tests, suite, suiteversion, mode, extra_info = ""):
for test in tests:
self.SubmitTest(test['name'], suite, suiteversion, mode, test['time'], extra_info)
self.submitTest(test['name'], suite, suiteversion, mode, test['time'], extra_info)
def SubmitTest(self, name, suite, suiteversion, mode, time, extra_info = ""):
def submitTest(self, name, suite, suiteversion, mode, time, extra_info = ""):
self.log("%s (%s -- %s): %s" % (name, suiteversion, mode, str(time)))
def Finish(self, status):
def finish(self, status = 1):
print "\n*******************************************\nSummary: "
print self.msg
self.msg = ''
def getSubmitter(name):
if name == 'remote':
return RemoteSubmitter
return RemoteSubmitter()
elif name == 'print':
return PrintSubmitter
return PrintSubmitter()
else:
raise Exception('unknown submitter!')

0
driver/utils.py → slave/utils.py
Просмотреть файл