# The Computer Language Benchmarks Game
# $Id: README,v 1.8 2012/06/14 20:07:06 igouy-guest Exp $
bencher - The Computer Language Benchmarks Game - June 2011
AUDIENCE
Someone who wants to make repeated measurements of standalone
programs - cpu time, elapsed time, resident memory usage,
cpu load while a program is running.
1) REQUIREMENTS
2) SOFTWARE RECOMMENDATIONS
3) LINUX QUICK START [ WIN32 QUICK START ]
4) FIXING PROBLEMS
5) MAKING NEW MEASUREMENTS
6) LOOKING AT THE MEASUREMENTS
7) REMOVE (REINSTALL)
8) IT'S SIMPLER THAN IT LOOKS
9) HOW TO MEASURE A SEQUENCE OF WORKLOADS (OR JUST ONE WORKLOAD)
10) HOW TO MAKE REPEATED MEASUREMENTS (OR JUST ONE MEASUREMENT)
11) HOW TO ADD NEW VERSIONS OF PROGRAMS
12) HOW TO ADD NEW LANGUAGE IMPLEMENTATIONS
13) HOW TO ADD NEW LANGUAGE IMPLEMENTATIONS THAT REQUIRE MAKE
14) HOW TO ADD NEW KINDS OF PROGRAM
15) HOW TO SET DIFFERENT OPTIONS FOR DIFFERENT KINDS OF PROGRAM
1) REQUIREMENTS
Willingness to read the README
Willingness to write ini file name=value properties
Willingness to (sometimes) write make commands
Python 2.5+ (these are Python scripts)
[ WIN32 SPECIFIC REQUIREMENTS ]
Python Win32 Extensions (for win32 process monitoring)
see http://sourceforge.net/projects/pywin32/
(add your Python directory to the system path,
for example \Python26 )
2) SOFTWARE RECOMMENDATIONS
ndiff (to check program output more selectively)
see http://www.math.utah.edu/~beebe/software/ndiff/
highlight 3+: code & syntax highlighting
see http://www.andre-simon.de/doku/highlight/en/highlight.html
LINUX SPECIFIC SOFTWARE RECOMMENDATIONS
GNU make (to compile programs for compiled languages)
GNU diff & cmp (to check program output is as expected)
libgtop2 dev files and Python bindings
(for cpu load and resident memory measurement)
[ WIN32 SPECIFIC SOFTWARE RECOMMENDATIONS ]
GNU Make for Windows
see http://gnuwin32.sourceforge.net/packages/make.htm
GNU DiffUtils for Windows
see http://gnuwin32.sourceforge.net/packages/diffutils.htm
(add GnuWin32\bin to the system path)
(add highlight to the system path)
3) LINUX QUICK START [ WIN32 QUICK START ]
3.1) unzip in ~ directory [ unzip in c:\ ]
The default configuration only creates files in the bencher
subdirectory
3.2) $ python ~/bencher/bin/bencher.py
[ > python c:\bencher\bin\bencher.py ]
measure cpu & elapsed time & memory & cpu load
Sun 15:49:01 .....OK .....nbody.java-2.java [3]
Sun 15:49:03 .....OK .....nbody.python [2]
Sun 15:49:12 .OK .....regexdna.python [1]
4) FIXING PROBLEMS
4.1) $ python ~/bencher/bin/bencher.py
[ > python c:\bencher\bin\bencher.py ]
measure cpu & elapsed time & memory & cpu load
Sun 16:44:37 .PROGRAM FAILED nbody.java-2.java [3]
Sun 16:44:38 .....OK .....nbody.python [2]
Sun 16:44:46 .OK .....regexdna.python [1]
CHECK the log file of programs that were not OK
c:\bencher\tmp\nbody\log\nbody.2.java.log
"MAKE:
make program not found"
"PROGRAM OUTPUT:
java.lang.NoClassDefFoundError: nbody"
The Java program has not been compiled because the make
utility used to build programs has not been installed.
4.2) $ python ~/bencher/bin/bencher.py
[ > python c:\bencher\bin\bencher.py ]
measure cpu & elapsed time & memory & cpu load
Sun 17:20:55 .MAKE ERROR nbody.java-2.java [3]
Sun 17:20:55 .....OK .....nbody.python [2]
Sun 17:21:05 .OK .....regexdna.python [1]
CHECK the log file of programs that were not OK
~/bencher/tmp/nbody/log/nbody.2.java.log
"MAKE:
mv nbody.java-2.java nbody.java
javac nbody.java
make: javac: Command not found"
The Java program has not been compiled because the javac cannot
be found. CHECK the [tools] section of bencher/makefiles/my.*.ini
defines the correct path for javac.
4.3) $ python ~/bencher/bin/bencher.py
[ > python c:\bencher\bin\bencher.py ]
measure cpu & elapsed time & memory & cpu load
Sun 17:33:41 .PROGRAM FAILED nbody.python [3]
Sun 17:33:41 .....OK .....nbody.java-2.java [2]
Sun 17:33:43 .OK .....regexdna.python [1]
CHECK the log file of programs that were not OK
~/bencher/tmp/nbody/log/nbody.1.python.log
"PROGRAM OUTPUT:
File "nbody.python", line 12
DAYS_PER_YEAR = 365.24 z
^
SyntaxError: invalid syntax"
There's a syntax error in the python program.
5) MAKING NEW MEASUREMENTS
5.1) $ python ~/bencher/bin/bencher.py
[ > python c:\bencher\bin\bencher.py ]
measure cpu & elapsed time & memory & cpu load
nothing to be done - measurements are up-to-date
New measurements are only made when there are no measurements to
match every program source code file, or when the measurement file
is older than the program source code file.
5.2) $ python ~/bencher/bin/bencher.py
[ > python c:\bencher\bin\bencher.py ]
measure cpu & elapsed time & memory & cpu load
Sun 18:08:31 .....OK .....nbody.python [1]
Fixing the syntax error in the python program, made the program
source code file newer than the measurement file. The measurement
was out-of-date so the python program was measured again.
5.3) $ python ~/bencher/bin/bencher.py nbody
[ > python c:\bencher\bin\bencher.py nbody ]
measure cpu & elapsed time & memory & cpu load
Sun 18:12:50 .....OK .....nbody.java-2.java [2]
Sun 18:13:01 .....OK .....nbody.python [1]
New measurements can be forced by explicitly listing the tests
that should be measured again - in this case "nbody".
5.4) $ python ~/bencher/bin/bencher.py python
[ > python c:\bencher\bin\bencher.py python ]
measure cpu & elapsed time & memory & cpu load
Sun 18:19:22 .....OK .....nbody.python [2]
Sun 18:19:31 .OK .....regexdna.python [1]
New measurements can be forced by explicitly listing the language
implementations (the program source code file extensions) that should
be measured again - in this case "python".
5.5) $ python ~/bencher/bin/bencher.py
[ > python c:\bencher\bin\bencher.py ]
measure cpu & elapsed time & memory & cpu load
Sun 18:22:41 .....OK .....nbody.python [1]
New measurement of just one program can be forced by deleting the
corresponding measurement file and running the script - in this case
"~/bencher/tmp/nbody/dat/nbody.1.python_dat" was deleted before running
the script.
6) LOOKING AT THE MEASUREMENTS
~/bencher/summary/all_measurements.csv
[ c:\bencher\summary\all_measurements.csv ]
Open all_measurements.csv with a text editor.
6.1) The first 4 comma separated columns identify each measurement record
1) test name 2) file extension 3) id# 4) data value
For example, the measurement for the nbody.java-2.java program run
with [testrange] value 20000 on the command line begins -
nbody,java,2,20000
6.2) The last 6 comma separated columns provide
5) size of the GZip compressed program source-code file in bytes
6) program usr+sys rusage time in seconds
7) memory used in KB
8) status value, if OK then 0 otherwise a negative status value
9) approximate CPU load on each core during measurement
10) program elapsed time in seconds
6.3) Notice there are repeated measurements at some data values.
See section #9 "HOW TO MAKE REPEATED MEASUREMENTS (OR JUST ONE
MEASUREMENT)" below.
6.4) For more details about how the measurements were made see -
http://shootout.alioth.debian.org/help.php#measure
6.5) Filter and process the comma separated measurement files with
a spreadsheet program or statistical analysis program or custom script.
7) REMOVE (REINSTALL)
7.1) To remove - delete the bencher directory and bencher.zip file
7.2) To reinstall - delete the bencher directory and then unzip
the bencher.zip file again
8) IT'S SIMPLER THAN IT LOOKS
8.1) The source code file extensions and the names for different kinds
of program link together source code, make command lines, run
command lines, measurement records, measurement logs and source code
markup.
8.2) Many ini file properties define values that will be substituted when
the corresponding names appear in other ini file properties, or in
Makefile commands.
8.3) Other ini file properties allow you to control which programs will
be measured, how many times they will be measured, and when to timeout.
9) HOW TO MEASURE A SEQUENCE OF WORKLOADS (OR JUST ONE WORKLOAD)
9.1) Set one or more input values in the [testrange] section
of the ini file.
For example, 5 input values are set for nbody programs and 1 input
value is set for regexdna programs -
nbody = 10000 20000 30000 40000 50000
regexdna = 10000
9.2) To provide program input from a data file redirected to stdin,
set a path to the data file in the [testdata] section of the ini
file.
Each of the input values set in the [testrange] section will
successively be substituted into the data file path - so programs
can be measured working on different size data files that provide
increasing workloads.
For example
[testrange]
regexdna = 10000 20000
[testdata]
regexdna = ../regexdna-input.txt
will measure a regexdna program firstly with the contents of this
data file redirected on the command line to program stdin
../regexdna-input10000.txt
and then measure a regexdna program with the contents of this data
file redirected on the command line to program stdin
../regexdna-input20000.txt
9.3) To provide program input from one command line argument, set the
input data value in the [testrange] section of the ini file.
For example
[testrange]
nbody = 10000 20000 30000 40000 50000
[testdata]
will measure an nbody program firstly with the string '10000'
substituted for %A in the command line set in the [commandlines]
section of the ini file.
Then measure an nbody program with the string '20000' substituted
for %A in the command line, and then '30000' and then '40000' and
then '50000'.
9.4) To measure at just one workload, set just one value in the
[testrange] section of the ini file.
For example
[testrange]
nbody = 50000
regexdna = 20000
10) HOW TO MAKE REPEATED MEASUREMENTS (OR JUST ONE MEASUREMENT)
10.1) For example, to make 50 repeated measurements for every input
value use these settings in the [measure] section of the ini file:
[measure]
runs = 50
repeatevery = True
10.2) For example, to make just one measurement for every input value
use these settings in the [measure] section of the ini file:
[measure]
runs = 1
repeatevery = True
10.3) For example, to make 50 repeated measurements for the last input
value but just one measurement for every other (smaller) input
value use these settings:
[measure]
runs = 50
repeatevery = False
11) HOW TO ADD NEW VERSIONS OF PROGRAMS
Add the new versions of the program source code files to the appropriate
subdirectories of bencher/programs, and run the bencher.py script.
11.1) For example, in the bencher/programs/nbody directory, copy and rename
the program source code file bencher/programs/nbody.java-2.java
~/bencher/programs/nbody$ cp nbody.java-2.java ./nbody.java-3.java
11.2) $ python ~/bencher/bin/bencher.py
[ > python c:\bencher\bin\bencher.py ]
measure cpu & elapsed time & memory & cpu load
Tue 19:55:07 .....OK .....nbody.java-3.java [1]
11.3) Or edit the old versions of the program source code files and run the
bencher script:
measure cpu & elapsed time & memory & cpu load
Tue 19:58:28 .....OK .....nbody.java-2.java [1]
12) HOW TO ADD NEW LANGUAGE IMPLEMENTATIONS
12.1) In the [tools] section of the ini file, define a name that will
substitute for the location of the new language implementation.
For example
[tools]
PYTHON3 = /usr/local/src/Python-3.2/bin/python3.2
12.2) Choose a file extension to identify programs and measurements made
with the new language implementation, for example - python3.
12.3) In the [commandlines] section of the ini file, define a command line
that will be used to run program source code files that have the new file
extension you chose.
For example, for file extension python3
[commandlines]
python3 = $PYTHON3 %X %A
12.4.1) EITHER alias existing source code files that have a different file
extension with the new file extension, in the [alias] section of the ini
file.
For example, re-use all source code files with file extension python but
make measurements identified with file extension python3
[alias]
python = python3
12.4.2) OR add new program source code files written for the new language
implementation to the appropriate subdirectories of bencher/programs
For example, to bencher/programs/nbody add
nbody.python3
12.5) Run the bencher.py script.
$ python ~/bencher/bin/bencher.py
[ > python c:\bencher\bin\bencher.py ]
measure cpu & elapsed time & memory & cpu load
Wed 18:58:12 .....OK .....nbody.python3 [1]
13) HOW TO ADD NEW LANGUAGE IMPLEMENTATIONS THAT REQUIRE MAKE
13.1) In the [tools] section of the ini file, define a name that will
substitute for the location of the new language implementation.
For example
[tools]
JDKRUN = /usr/local/src/jdk1.6.0_25/bin/java
JDKC = /usr/local/src/jdk1.6.0_25/bin/javac
13.2) Choose a file extension to identify programs and measurements made
with the new language implementation, for example - javaxint.
13.3) In the [commandlines] section of the ini file, define a command line
that will be used to run program source code files that have the new file
extension you chose.
For example, for file extension javaxint
[commandlines]
javaxint = $JDKRUN $JDKFLAGS -server -Xint %T %A
13.4) In the [build] section of the ini file, add the new file extension to
the list of source code files that require make.
For example, for file extension javaxint
[build]
make =
java javaxint
13.5) In the my.linux.Makefile [ my.win32.Makefile ] file, add a make target
with the commands required to prepare source code files for the new language
implementation.
For example, for file extension javaxint
%.javaxint_run: %.javaxint
-mv $< $(TEST).java # change .javaxint to .java
-$(JDKC) $(TEST).java # compile java source code
13.6.1) EITHER alias existing source code files that have a different file
extension with the new file extension, in the [alias] section of the ini
file.
For example, re-use all source code files with file extension java but
make measurements identified with file extension javaxint
[alias]
java = javaxint
13.6.2) OR add new program source code files written for the new language
implementation to the appropriate subdirectories of bencher/programs
For example, to bencher/programs/nbody add
nbody.javaxint-2.javaxint
13.7) Run the bencher.py script.
$ python ~/bencher/bin/bencher.py
[ > python c:\bencher\bin\bencher.py ]
measure cpu & elapsed time & memory & cpu load
Wed 20:26:38 .....OK .....nbody.javaxint-3.javaxint [2]
Wed 20:26:42 .....OK .....nbody.javaxint-2.javaxint [1]
14) HOW TO ADD NEW KINDS OF PROGRAM
14.1) Add a new subdirectory for program source code files to the
bencher/programs directory
14.2) Add new program source code files with matching filenames to the
new subdirectory
For example, for a new kind of program named fannkuchredux
bencher
programs
fannkuchredux (new subdirectory)
fannkuchredux.java (new program source code file)
14.3) In the [filters] section of the ini file, add the new subdirectory to
the list of onlydirs that will be searched for source code files.
For example, add the subdirectory fannkuchredux
[filters]
onlydirs =
nbody
regexdna
fannkuchredux
14.4) Input for the new kind of program can be either a single argument on
the command line or a data file redirected to the programs stdin.
14.4.1) To provide a single argument on the command line, set the input values
for the new kind of program in the [testrange] section of the ini file.
For example, add input values for fannkuchredux
[testrange]
nbody = 10000 20000 30000 40000 50000
regexdna = 10000
fannkuchredux = 10 11 12
14.4.2) To provide a data file redirected to the programs stdin, set both
a path to the data file in the [testdata] section of the ini file AND
values that will be substituted into the filename AND add the data files
to a subdirectory of bencher/tmp.
For example, for a new kind of program called knucleotide
[testdata]
knucleotide = ../knucleotide-input.txt
[testrange]
nbody = 10000 20000 30000 40000 50000
regexdna = 10000
knucleotide = 250000 2500000 25000000
Add the new subdirectory and data files
bencher
tmp
knucleotide (new subdirectory)
knucleotide-input250000.txt (new data file)
knucleotide-input2500000.txt (new data file)
knucleotide-input25000000.txt (new data file)
14.5) Run the bencher.py script.
$ python ~/bencher/bin/bencher.py
[ > python c:\bencher\bin\bencher.py ]
measure cpu & elapsed time & memory & cpu load
Thu 09:05:31 ...OK .....fannkuchredux.java [2]
Thu 09:07:18 ...OK .....knucleotide.java-3.java [1]
15) HOW TO SET DIFFERENT OPTIONS FOR DIFFERENT KINDS OF PROGRAM
15.1) Add a section to the ini file
For example, for the kind of program called nbody
; ENVIRONMENT for specific tests
[nbody]
15.2) Add a property to set a name and value, for example
; ENVIRONMENT for specific tests
[nbody]
JDKFLAGS = -Xmx8m
GCCOPTS = -lm -mfpmath=sse -msse3
15.3) Use that property name in a command line
For example, for the kind of program called nbody substitute the
value -Xmx8m into the command line set in the ini file for programs
with file extension java
[commandlines]
java = $JDKRUN $JDKFLAGS -server %T %A
In this case, nbody programs with file extension java will be run with
command lines like
$ java -Xmx8m -server nbody 50000000
But other kinds of program with file extension java will still be run
with command lines like this
$ java -server fannkuchredux 12
15.4) Use that property name in a make command
For example, for the kind of program called nbody substitute the
value -lm -mfpmath=sse -msse3 into the make command set in the Makefile
for programs with file extension gcc
STD_COPTS := -O3 -fomit-frame-pointer -march=native
%.c: %.gcc $(GCC)
-@mv $< $@
%.gcc_run: %.c $(GCC)
-$(GCC) -pipe -Wall $(STD_COPTS) $(GCCOPTS) $< -o $@
In this case, nbody programs with file extension gcc will be compiled
with command lines like
$ gcc -pipe -Wall -O3 -fomit-frame-pointer -march=native
-lm -mfpmath=sse -msse3 nbody.gcc-5.c -o nbody.gcc-5.gcc_run
and run with command lines like
./nbody.gcc-5.gcc_run 50000000
But other kinds of program with file extension gcc will still be
compiled with command lines like this
$ gcc -pipe -Wall -O3 -fomit-frame-pointer -march=native
fannkuchredux.c -o fannkuchredux.gcc_run
and run with command lines like
./fannkuchredux.gcc_run 12
*** TO BE CONTINUED ***
* Meanwhile ask questions in the benchmarks game discussion forum *
* debian issue their own security certificate - your browser will complain *
* https://alioth.debian.org/forum/forum.php?forum_id=999 *
BASIC ASSUMPTIONS
Program source files are organized by 'benchmark' in different subdirectories,
and program source files have the same name as the 'benchmark' distinguished
by file extension, for example:
programs
nbody subdirectory
nbody.python program source file
nbody.gcc program source file
nbody.gcc-2.gcc program source file
regexdna subdirectory
regexdna.python program source file
regexdna.gcc program source file
regexdna.perl-2.perl program source file
For each source code subdirectory in programs a matching directory
will be created under tmp, containing -
data subdirectory - compressed measurement files
log subdirectory - log text files recording each program build and run
tmp subdirectory - files needed to build and measure the current program
expected output files for various data input sizes *_out
c41a03ab2e