pjs/webtools/tinderbox2/Install

Preparations you will need to make and 
policies you will need to set:
-----------------------------------


To install tinderbox you will need some information about your
existing computer systems and some idea about what your goals are.
Here is a list of questions to help get you started.


The webserver will serve the tinderbox pages.  
Webserver configuration is a bit of an art and you will need to
understand the policies which are used to administer your webserver.

*) You will need to decide the directory where tinderbox should write
the static HTML pages.  This will depend on how your webserver is
configured. The default location is based on the RedHat 7.1
(apache-1.3.19-5) installation and is: /var/www/html/tinderbox2.  You
will also need to know what the URL browsers will need to use to find
this directory. Since tinderbox generates static web pages, it is
possible to run tinderbox and not run a web server.  One way this
could be done is if you have a network file system and all users have
browsers which can read from the HTML directories.  In this case all
URL's should begin with "file:/" instead of the usual "http://".

*) Project level administration is done via cgi scripts.  These
scripts allow administrators to set the message of the day, and the
state of the tree (open, closed, restricted).  Also all users can post
notices to the web pages via a cgi script.  CGI programs are often
restricted to a portion of the file system which is disjoint from the
HTML files. You will need to figure out where the CGI programs will
go.  Tinderbox takes its defaults from RedHat 7.1 and uses:
/var/www/cgi-bin/tinderbox2. You will also need to know what the URL
browsers will need to use to find this directory.

*) CGI scripts will run as an unauthenticated user on your system.
You will need to decide which user will run the tinderbox CGI scripts.
The same user id must be used for running the scripts as for tinderbox
mail delivery.  The Tinderbox Configuration files will define this
user id and as a security precaution check that it is running as the
required id.  It is suggested that this id not be a privileged id
(higher ids are better, please make this number be grater then 10 and
bigger then 100 is recommended).  Smaller ids are often assumed to
have more privileges on a Unix box then larger ids.  It is not a good
idea for an unauthenticated user to have any privileges so a large id
is recommended. It is also recommended that you not use the id 'nobody'
as this id is over used and it would be better to partition the
unauthenticated user into separate ids in case of security problems.
RedHat runs all its CGI scripts as the user 'apache', this is an
acceptable user.  I would prefer to have a separate user to run the
tinderbox CGI scripts but this would require recompiling apache to
enable suEXEC, and it is more effort then most groups can afford.

*) Tinderbox Files. There are other tinderbox files which need to be
placed on the webserver.  These include libraries and non-cgi
programs. You will need to decide where to place these files.  Most
users put them in /home/tinderbox2.

*) Tinderbox Data. Tinderbox stores its data in the file system.  For
security it is often a good idea to keep this data out of the HTML and
CGI directories so that malicious users can not directly access this
data.  The compressed build logs can grow quite large, so it is
recommended to put the data on a file system with room.  The default
is to put them in the directory /home/tinderbox2/data.


Mail
----

*) Many of the tinderbox modules (Bug Ticket, Build, CVS) receive
their data via mail.  The mail system on you web server machine must
be configured to deliver the mail into the tinderbox mail processing
programs.  You should spend some time understanding how your mail
delivery system can be configured to allow user mail to be delivered
into a program and how to set the user id under which this delivery
occurs.  If you do not wish to configure your mail delivery program
then you can use fetchmail to pull the mail out of a mail box and push
it into the programs on a periodic basis.


Production Version Control
-------------------------

One of the biggest responsibilities which a "buildmaster" has is the
requirement that all code should be reproducible.  That is that at
any point in the future, even more than one year later, the current
binaries should be able to be rebuilt byte for byte from sources.
This requirement can be broken down as follows:

1) The build machine must be reproducible.  

We must be able to get back the same build machine we had at any point
in the past.  This means that all OS libraries, all header files, all
compilers, all build tools (make, grep, sed) must have some mechanism
to roll back.  It is common to use a backup of the build machine to
reconstruct it.  Most OS will give you a list of the software packages
which are installed on the machine and their version numbers.  I like
to keep the list of software packages which are installed on the
machine checked into version control.  This allows me to compare the
state of the build machine at any two points in time.  I have tools to
recreate the build-machine from just a list of packages with version
numbers. It is considered a best practice to limit the amount of
software which is available on the build machine.  A build machine
with too much installed will only make it difficult to reproduce older
builds should the need arise.  I recommend not installing any
web servers or graphical window managers on your build machine.  It
should be clear that the build machine should not be the same machine
where the tinderbox server runs.

2) The build process must be reproducible.  That is all the steps
which are used to create the application must be reproducible.

*) Build Interface: We must be able to run exactly the same build
process in the future including: all commands with command line
arguments, all environmental variables.  I recommend that the entire
build process be viewed as something outside of the build master
control.  Developers are responsible for ensuring that there is a
simple build master interface to construct all the software products
which go into a build.  Typically there is a makefile in a standard
place where the buildmaster can run something like "make all; make
install;" and be guaranteed that this will build the product.  The
build interface should be viewed as something which never changes and
 are part of the build machine, like the OS and are changed only
rarely.  It is hard enough to track all the parts of the build process
which we expect to change, we should not need to track complex build
procedures.  The build procedures should have a standard interface.
By keeping the build instructions in one makefile which is checked
into the same version control system as the sources it is easy to
recreate any previous build even if the commands used to build the
software fluctuate rapidly between releases.  There must be a simple
interface to construct the software which will hide all the complexity
of the actual construction.  

*) Build Environment: The makefile will code all the build commands
and all the environmental variables (PATH, UMASK, LD_LIBRARY_PATH,
CLASSPATH) needed to build the software though it may rely on some
well defined command line arguments (PREFIX, CCFLAGS, JAVA_LIBS) to
make these prematurely.  These command line arguments should not
change between versions of the software but should be a fixed set of
build parameters.  The parameters may be needed to specify where some
files are found on the build machine (Ideally the build machine is set
up the same as developers machines so these directories can be
hard-coded into the makefiles but often there is a need for some
directories to be specified at build time) or where files are to be
created/installed on the build machine (typically a subdirectory of
/var/tmp but there may be several builds running at once and each will
need a different directory) or what kind of build is being created.
Each part of the build which needs a particular environmental variable
set or a special header file in some path should have tests which
ensure that the build environment is valid.  I keep my build scripts
installed on the build machine and they are always started by running
/etc/rc.d/init.d/build start this ensures that I am not relying on any
build environmental variables which are set by logging into the build
account and are thus not tracked and versioned.

*) Environmental safety issues: 

If the build environment can not be used to build the software then a
human readable error message should be generated.  My makefiles often
run various checks on the environmental variables before they
construct the code.  They check that all required environmental
variables are set, that the required libraries are found, that
directories which must be disjoint (build and install directories) do
not overlap. This test suite becomes a build regression test and as I
discover additional possible build problems I add new tests to the
makefile.  I make it a habit to explicit set all environmental
variables so that there is no doubt as to their expected values.  It
is important for the QA group to only use Builds which were created by
an automated process so that we are sure that there are no
undocumented steps in either the test builds or the released build.

3) Track the Build numbers.  Given a clean install of your product you
should have all the information necessary to reproduce the executable
from sources.  If a customer shows you the application binaries you
must be able to get the source code which build the application,
reconstruct the build machine which created the application and
possibly rerun the build exactly the same way as the application was
created before, this may include making some minor source code changes
before the build is run. I like to keep a file which contains:

	The product release name

	The sources 'as of date'. (I always checkout my sources using
		cvs -D 'date time' so that exactly the same sources
		can be recovered knowing only the 'data time' which
		was used to check them out. I am sure a similar trick
		could be used with a perforce 'change set number'.)

	The branch name.

	The module name.

This can be stored as a file in the product (encrypted if necessary)
or may be stored in some secure build master database where the data
can be looked up by release name.  My preference is to keep all data
necessary to reproduce a build in the build output and delivered as
part of the product.  This means that I can generate as many builds as
I want automatically and not need to keep track of any of them.  When
the QA team deems that a certain build is 'important', by making a
particular build the official released copy then I can take a look at
its contents and tag/branch the code at the sources which I used to
build it.

4) Build Prefix: It is a good idea to familiarize yourself with the
makefile conventions regarding the make variable PREFIX.  It is
easiest to understand if you think about what RedHat does when they
build their distribution of RPM's but this will apply in many
different systems including the Andrew File System (AFS) and most
packaging systems. This variable is used during the build process
"make all PREFIX=/home/apache" to tell the package where it will be
installed (examples include /usr, /usr/local, /home/apache).  I
suggest reading a few RedHat Spec files to see how this works in
practice.  The application may need to hard-code this value into its
object code.  When the application is installed it must not be
installed into its proper place on the build machine.  The package we
are constructing could cause the build machine to stop working
correctly if it is a buggy version of a system library or major OS
application.  Instead the makefile will install "make install
PREFIX=/var/tmp/build-root/home/apache" the package into some other
directory with a similar tree structure to its final destination.  The
packaging system will then move the files into the correct place
during an installation step on the target machine.  The installation
step only moves files and sets permissions.  The makefile is not
supposed to use the installation directories to hardcode values into
the application since the application will never be run from this
installation directory.  The hard part of the build including any
PREFIX magic is in the build section.  Notice the clear separation
between build machine / target machine and installation on the build
machine and installation on the target machine and construction of the
application binaries and installation of the application binaries.
This is one of the reasons why building an application on a build
machine is different from the way in which developers build their code
on their personal development machines.  This PREFIX issue will arise
when you try and build the Tinderbox system and also when you
construct the makefiles for your own application.  Since the build
machine is not the target machine it can not be assumed that files
will always be in the same places on both (for example perl).

5) Application Architecture: 

*) The build process should mimic the architecture of the code.  It should
be a final test that the code was coded to the same specifications
that it was designed.  It is a common problem for code to turn into
spaghetti with each piece of code using functions and creating
dependencies on every other piece of code.  For example it is probably
a mistake for code in the database abstraction layer to be implemented
in terms of code in the HTML generation layer.  These two libraries
should probably be independent of each other, though they both might
depend on a common string library.  The code architecture should limit
the dependency graph between code modules.  The BuildMaster must
enforce the restrictions on information flow between components. Thus
no libraries should be in the path unless the architecture allows this
module to depend on those libraries.

*) The architecture must not have circular dependencies.  Circular
dependencies not only make upgrading individual libraries difficult
but also make testing components nearly impossible.  That is it should
be possible to build some set of libraries L0 which depend on no
libraries and then build some other set of libraries L1 which depend
only on L0 libraries then build L2 which depend only on the L0 and L1
libraries.  This "build chain" will prevent circular dependencies and
help keep your code testable and the dependencies understandable.

*) I enforce the convention that developers are not allowed to overload
standard system libraries.  I always put standard libraries in the
path before any library our company develops.  I build the application
in stages to ensure that parts of the application which are not
intended to depend on other code will not have other header files on
the build machine at the time that they are constructed.  Build
dependencies between modules which are expected are explicitly
controlled with build scripts and version numbers.


Overview of the Tinderbox System
--------------------------------

Tinderbox is an information display system. It runs on a machine with
a webserver and will periodically write static HTML files to the disk
so that the webserver can serve these documents. Tinderbox is run out
of cron every five minutes.  It gathers up information from various
databases including: CVS Logs, Bonsai, and Perforce.  It will also
process mail which is sent to it. Mail is sent from Bug Ticketing
software and Build/Test Machines.  All this information is combined to
produce the HTML pages.


Since no two companies will structure their development processes the
same way, the tinderbox code has to be highly configurable to account
for most possible uses.  There is a main configuration file which
allows most of the major user configurable variables to be set.
Additionally each library has been broken into two parts.  One part is
the library specific configurations.  This file is expected to need
modifications in some installations.  I have put all the library
configurations into one directory to make it easy to find the parts of
tinderbox which are easy to modify.  Each configuration library can be
thought of as a table which might need to be edited or extended for
use at your company.  I have provided a working system but the
defaults may not suit your needs.  These tables can be easily changed
in small ways by simply looking at the file and making obvious
changes.  I have also allowed for the possibility of making complex
changes that only a competent perl programmer could define.  Changes
are not made to the files which I have provided.  Rather the changes
are made to copies of the files which are stored in a local
configuration directory.  This ensures that you can easily version the
Tinderbox code as it is provided to you from the official distribution
and you can separately version the local configurations which you make.
It is also easy to see the local configurations since you have both
the original and the modified code on the same server and can
difference the two.  I suggest that you read through the files to see
how they are laid out and what types of changes are possible.


The build machines are not considered part of the tinderbox server.
They are clients just like Bug Ticketing systems and Version control
systems are clients.  Build machines mail their build logs to the
server in a special format.  This format specifies that name/value
pairs must appear at the top of the mail message followed by the
complete build log.  Scripts for setting up a tinderbox build client
can be found in the clientbin directory but you may have other build
needs and may use any build methods you choose.

The central concept of the Tinderbox system is the notion of a 'Tree'.
When several different groups are working out of the same version
control system often the files are partitioned into separate modules
with each group working on one or more disjoint modules.  Over time
the developers need to branch their code because several different
versions of the files are under development at the same time.  A tree
is a module/branch pair.  This corresponds to a set of files which can
be checked out and built.  Tinderbox makes one page for each tree and
displays what work is being done on that tree.  CVS has a notion of
branches and of modules but not of trees.  It is not possible to give
a branch/module pair a name.  The tinderbox TreeData provides the
mappings between treenames and branch/module pairs.  Tinderbox
displays the updates to bug tickets on the appropriate tree page.
This requires an easy mapping between bug tickets and trees. One
example of a complex function to determine tree name would be if each
of the product product types listed in the bug tracking data base
refers to one development project, except for a particular
feature/platform of one particular project which is being developed by
a separate group of developers.  So the version control notion of
trees (a set of modules on a branch) may not have a direct map into
the bug tracking database at all times.  In large projects it is
sometimes convenient to have a tree called 'ALL' which is used to
display all checkins performed on any trees and all bug tickets worked
on by any programmers.  It is not possible to build or test the 'ALL'
tree and neither the version control nor bug ticketing system knows of
its existence.


The Bug Tracking code was intended to be as general as possible.
Most bug ticketing systems send mail when tickets change state.  The
mail is often of the same form.  It is a name/value pair which the
separator being the string ": ".  Tinderbox will parse mail of this
form and display the interesting fields on the appropriate tree page.
The configuration of this module involves specifying which bug ticket
names are interesting and should be displayed.  Also you will need to
specify how to map a bug ticket into a a tree.  This could be very
simple if each bug ticket has a field which represents the tree it is
applicable to (in this case tree could equal project) or can be very
complex if the tree must be computed by the values of a set of fields.
Also tinderbox keeps track of which bugs are "reopened" and displays
them in a different column.  The idea is that some bugs are moving
backwards and creating duplicate work.  These tickets are particularly
troublesome and should be watched specially.  So possible all ticket
status are partitioned into "progress" or "slippage" categories.  You
will need to specify what status values are possible for your ticket
system and you will also need to specify the set of columns which you
would like to see on the status page.

status table

Headers


Tinderbox Documentation:
------------------------

*) Every directory has a README file which tells about all the files
in that directory.

*) Every executable can be run with --help so that you can see what
the program does and what its arguments are.  If you have trouble
running the file just look at the file in a text editor and find
the string "usage".

*) The top of every file (both code and configuration) has a short
explanation of what this file does.

*) The code has lots of documentation so you can read the code if you
need more details about how something works.


To install:
-----------

*) The src/default_conf directory contains perl libraries which are
specific to an individual users site configuration.  No two users of
Tinderbox will have the identical sets of files.

The files found in default_conf are sample files which are used at
Mozilla.org.  Other users are expected to customize these libraries and
install the modified libraries in local_conf.  Files found in
local_conf will be used before any file found in default_conf.
The distribution of Tinderbox will never place any files in the
local_conf directory as this is assumed to be under local control.

A quick overview of the files:

	TinderConfig.pm: general configuration settings 
		(HTML directories, Log files, implementations of libraries 
		 to use, etc).
	TreeData.pm: version control (CVS, Bonsai) configuration.
	Error_Parse.pm: the regular expressions for identifying
		 errors in build logs.
	BTData.pm: bug tracking configuration
	FileStructure.pm: filesystem (storage) and global/per-project 
		settings. Most users will not have to change this.


In particular you will need a TinderConfig.pm and a TreeData.pm which
describes your local setup.  You may need to make local versions of
the other files depending on how you wish to have Tinderbox configured.

I install Tinderbox via RPM.  I have three RPMS. One tracks changes in
the Tinderbox server source code just as I get it from CVS. One RPM
contains my local_conf files.  The clientbin files get put in their
own RPM which is installed on the buildmachine along with the
local_conf files.  I do not currently have all the other configuration
details worked out (crontab files, /etc/rc.d/init.d, .cvspass,) but
most of the state of my machines is under RPM control.

*) Run ./configure. When configure is done you will have a Makefile
and a config.out.  Although Tinderbox consists entirely of perl
scripts we must substitute some values into the source code to make it
executable.  You may wish to change the default directories in
configure for some of the makefile variables.  Please read config.out
and make any changes which need to be made for your system. Configure
also excepts command line options to change some default variables.
Please look at the configure source code for variable details, but the
most common changes are:

	prefix: the directory for most of the tinderbox files,
		defaults to /home/tinderbox2

	cgibin_prefix: The directory where cgi scripts will be
		run. This depends on how your webserver is configured.  
		For security you may wish to ensure that this
		directory is disjoint from the prefix directory.  This
		defaults to /var/www/cgi-bin/tinderbox.

	htmldir: The directory where html files should be
		written. This depends on how your webserver is configured. 
		This is where the webserver will pick up tinderbox2
		output.  This defaults to /home/httpd/html/tinderbox.


*) run 'make' to create executable versions of the source code in the
./build directory.

*) run 'make compile_bin_code' to ensure that the code will compile when
it is installed.

*) If you wish run the test programs as described in
/build/test/ReadMe to ensure that you have configured the program
correctly. This will require a TinderConfig.pm file in your local_conf
directory.  See the ReadMe file in the test directory for detailed
information on how the tests work.

*) run make install, to install Tinderbox on your system.  You may
wish to run this with a different 'prefix'

	make install prefix=/opt/tinderbox

It is currently assumed that the bin directory of your installation
will be where the webserver will run the tinderbox cgi's from.  You
can copy the *.cgi files to another directory if this is not the case.


*) set up a cron job to run $prefix/bin/tinder.cgi --daemon-mode 
	every five minutes.  This generates the static tinderbox pages
	which users see.
   set up a cron job to run $prefix/bin/rmlogs at least once a day.
     six am is a good time to run this as the machine load is usually light
      and will avoids any day light savings problems.  This keeps the
	archive of compressed build logs from growing without limit.

*) set up the $prefix/bin/processmail* programs to receive the
incoming tinderbox mail.  The process id which receives and process
the mail must be the the same id which runs the tinderbox cron job to
prepare the web pages.  Usually this set up is accomplished by having
the MTA (Sendmail) pass mail for particular accounts into a script.
This can be configued via a global configuration file (Sendmail alias
file) or via a .forward file (each account gets the same user id but a
different home directory, each home directory gets a .forward to cause
incomming mail to be delivered through the correct tinderbox mail
processing program).  

I have used the following configurations for the mail server Postfix.
The postfix aliases file /etc/postfix/aliases contains the following
lines.  This file must be processed with a /usr/sbin/postalias command
which must be run by the same id which will run the tinderbox cgi scripts.

	tinderbox_builds: |/home/tinderbox2/bin/processmail_builds
	tinderbox_bugzilla: |/home/tinderbox2/bin/processmail_bugs



If it is not possible to have mail delivery on your web machine and to
have your web machine deliver specific mail through a program then I
suggest you use fetchmail to simulate this delivery format.  Download
fetchmail (from http://tuxedo.org/~esr/fetchmail) and install it on
your webserver.  Have the mail sent to any pop/imap mail server which
is accessible from the webserver.  Configure fetchmail to gather the
mail from the mail server and pass it through to the correct mail
processing program.  You should keep the polling interval short (30-200
seconds) as long polling intervals will cause delays in the tinderbox
system and limit how quickly users see the new data.  If the polling
interval is too short then you will overload your pop server with
needless queries.  Fetchmail should run under the same id as the
tinderbox server.  The fetchmail configuration file will look
something like:

	poll mail.provider.net with proto pop3:
               user "tinderbox_builds" there has password "u can't krak this"
               is tinder here and wants mda "$prefix/bin/processmail_builds"




We use the mail address 'tinderbox_builds' for build information
destined for the webserver. Similarly the bug tracking system should
send mail to 'tinderbox_bugs'.


*) If you are using VC_CVS.pm then you will need to put a ~/.cvspass
for tinderbox to use.  Log into the CVS repository yourself, once for
each tree you have defined.  The login command must use the
hostname/modules exactly as you wrote it in VCData.  Then copy your
~/.cvspass into the tinderbox server user id's home directory.  This
must be the REAL home of the Tinderbox daemon, as listed in
/etc/passwd/ and set in the $HOME environmental variable for
tinder.cgi.  The file must not be world readble or writeable or
executable.


*) Setup the build machines to mail their build logs (with tinderbox
variables on the top) to the web server machine.  New builds must not
start earlier then 6 minutes after the last build started.  Each build
machine mails the build log of each build and puts some build data at
the top of the log.  Build information includes whether the build was
a success, which error parsers to use on the log file and what build
this is.  Run 'processmail_builds --help' to learn about how the mail
processing system works and what the mail messages are expected to
look like.  There is an example mail in the test directory called
samplelog.  The clientbin directory contains code which can be used on
the buildmachine.  See the README to help setup your buildmachine.


*) Set up the bug tracking system to send mail to 'tinderbox_bugs' on
the webserver machine whenever a Bug ticket changes state. It is not
interesting to see which tickets are being worked on, so restrict
mailing to changes in ticket state not updates (edit) of a ticket.

*) Check that the time on your webserver, your version control
machine, your bug tracking machine and your build machines are all in
sync.  Check that mail if build mail bounces on any of the above
machines that it will be recieved by someone who can act on it.




I keep my email up to date in the source code.  If you have trouble
with these instructions please drop me a line.  

Ken Estes.