# -*- mode: indented-text -*- # # Author: Artem Belevich # # (Changes have been made to Artem's original doc, as things evolve.) # # ********************************************************************** As it's said in README "This is not very well packaged code. It's not packaged at all. Don't come here expecting something you plop in a directory, twiddle a few things, and you're off and using it. Much work has to be done to get there." This file is intended to make some things *easier* but not easy. You are still required to make some changes on your own. There is no guaranteed solution yet and it's unlikely that there will be one in the nearest future. ********************************************************************** 0. OVERVIEW This document describes how to install Bonsai and make it work with LXR. If you are only installing Bonsai, you can ignore the mentions about LXR and Tinderbox. You will still probably want to get registry. Some time ago I've seen Linux Source Navigator (LSN) at http://sunsite.unc.edu/linux-source. I was impressed. It was and is a wonderful tool to explore Linux kernel source code. Then Mozilla.org came up with a more elaborate tool that includes source browser with crossreferencing (LXR http://lxr.linux.no) and CVS tree control (Bonsai - http://www.mozilla.org/bonsai.html). While LXR formatting is not as pretty as LSN's one, it has a huge advantage - it lets you see where the identifier is defined and used. And Bonsai brings nice and easy (though sometimes incompatible with browsers other but Netscape's own) interface to the CVS history. This includes getting list of changes, diffs between revisions, etc. All in all LXR+Bonsai+other stuff beneath is a useful tool capable of handling huge projects. It's not that easy to make it work with other source tree but Mozilla's own but it's possible. And there are a lot of things to improve. Now I'm going to concentrate on the first goal - to make it work. 1. GETTING IT UP First of all you have to get all the tools in mozilla's mozilla/webtools CVS repository. This includes lxr,bonsai,registry and tinderbox. You're likely will not need neither tinderbox but get it just in case. To get the sources you have to follow instructions on http://www.mozilla.org/bonsai.html. OK, now you've got the sources but don't rush to try it right away. It's likely that you will not be able to even start most of the scripts. There are more things you will have to get and install. The short list of the things you will need: 1) MySQL database server. 2) Perl 5.004+ plus modules: 2a) Date::Parse 2b) Mail::Mailer 2c) DBI 2d) DBD::mysql 3) Some kind of HTTP server so you could use CGI scripts You could try running the ./configure script to see what tools it complains about right now. Mind you, it won't check for the MySQL database. 1.1 Getting and setting up MySQL database Visit MySQL homepage at http://www.mysql.com and grab the latest stable binary release of the server. Sure, you can get sources and compile them yourself, but binaries are the easiest and the fastest way to get it up and running. Follow instructions found in manual. There is a section about installing binary-only distributions. You should create database bonsai, and the user and password for it. 1.2 Perl + Mysql You will need Perl 5.004 with DB and Mysql extensions. DB is required to use LXR browser and crossreferencer for storing its database. Mysql is used by Bonsai. If you have Perl already installed, try to run genxref program from LXR suite. If it complains that it misses DB terribly then you're probably will have to get and install DB 1.86 distribution from one of the CPAN (www.cpan.org) mirrors in src/misc directory. I personally got it from http://www.cpan.org/src/misc/db.1.86.tar.gz. Having DB compiled and installed you will also have to rebuild and reinstall Perl itself so It would recognize and compile DB module in. This can be tricky if you have DB installed in some strange place as I did. I've got an error during linking phase - there was a function missing in hash/ndbm.c file, so I just commented it out. It may potentially cause troubles, but I think it does not matter in our case as this was intended only for DBM compatibility - the feature we don't really use. Now you hopefully have Perl + DB compiled installed and working. Time to set up Mysql module. This one is easy. Just follow instructions in MySQL manual. You have to read manuals sometimes.. I think I'm getting older.. 8-) Next step is to get TimeDate module from one of the CPAN mirrors. Go to CPAN search page (http://theory.uwinnipeg.ca/search/cpan-search.html) and search for the "TimeDate" module. Then get it and install. You also need to get the libnet and MailTools CPAN modules. They can both be found on CPAN at CPAN/modules/by-authors/id/GBARR. 1.3 HTTP server You have a freedom of choice here - Apache, Netscape or any other server on UNIX would do. The only thing - to make configuration easier you'd better run HTTP daemon on the same machine that you run MySQL server on. Make sure that you can access 'bonsai' database with user id you're running the daemon with. Disable web access to the Bonsai data directory and its subdirectories. In Apache you would write a section in the config file, something like: AllowOverride None Options None Order deny,allow Deny from All 2. TWEAKING THE TOOLS Now you should have all necessary tools to be able to run LXR and Bonsai scripts and see why the wouldn't work for you right now. 2.1 LXR You can skip this section if you are not planning on installing LXR. The first thing to set up is LXR tool. All it needs is the source tree (not CVS tree). It's relatively easy and works almost right of the box. Follow instructions in LXR README file. Having set LXR you will see that regardless what your source tree contains you will see that everything refers to it as Mozilla. Mozilla is a great thing and this tool was primarily tailored to mozilla tree but you'd like to control your own tree. First step is to edit your Here is the short list of changes I had to make file: ident 1) change "&root=/cvsroot" to your CVSROOT path 2) change "file=/mozilla/" to the directory under CVSROOT where your sources are. In my case it is just "/" file: index.html Nothing vital here but probably worth changing to reflect your own environment file: lxr.conf Changes to this file are described in LXR README file and are quite simple. file: source You may find it useful to uncomment "$img = "/icons/..." lines if you use Explorer as it does not have internal-gopher-* images built in. Actually Bonsai contains a lot of netscapism that will make your IE4 unhappy anyway. You'd better stick with Netscape if you are going to use LXR/Bonsai file: template-* Here you will probably want to watch closely at the places where you see the word 'mozilla' near '.cgi'. There are a lot of mozilla-specific paths hardcoded change/get rid of banner that loads straight from mozilla.org that may be very dangerous if you're working for micro$oft and your boss comes by.. 8-) 2.2 Bonsai This stuff sometimes gets very specific about your CVS repository setup. You have to make a lot of changes until more portable configuration mechanism is introduced. These steps should create a basic Bonsai install: ./configure make install You might want to give the option --prefix= to configure to install Bonsai in another place than /usr/local, e.g. /var/www. It will make a new directory named "bonsai" in the prefix directory you specify. Ensure that the bonsai cgi programs can write and create files in the data directory. Typically this means making the data directory owned by the web cgi id. Bonsai does not need to change the executable files in the main bonsai directory so these can be owned as root. Test using your web browser that you will not be able to access the data directory (you should get "access denied"). Edit data/treeconfig.pl file: treeconfig.pl defines @::TreeList, a list of trees you want to track, and %::TreeInfo, information about each of those trees. A sample treeconfig.pl: @::TreeList = ('default', 'other'); %::TreeInfo = ( default => { branch => '', description => 'My CVS repository', module => 'All', repository => '/d2/cvsroot', shortdesc => 'Mine', }, other => { branch => '', description => 'Other CVS repository', module => 'All', repository => '/d2/otherroot', shortdesc => 'Other', }, ); 1; Create data/XXX directory for each tree you defined in treeconfig (data/default and data/other using the example above). This file maps the names of trees to branch/module combinations. You will need to have at least one module in your CVS repository to run Bonsai. Typically users create a module called All which contains all the directories in the CVS repository. All repositories must be written as if they were local repositories (eg '/cvsroot') without hostnames or ':pserver:'. The cgi-bin scripts will access these directories on the web machine and they must contain the ',v' files which match cvsroot as listed in the checkin mail from the real CVS machine. Run createlegaldirs.pl to create legaldirs for your module. Using the sample treeconfig above you would run createlegaldirs.pl like this: perl createlegaldirs.pl default other Go to the data directory and run trapdoor >passwd it will set up admin's password. Bonsai should now be accessible via a web browser but not all functionality is installed yet. Visit admin.cgi and set all the parameters. That's basically it. With some luck and persistence you will have 90% working system at this point. A lot of these things are just asking to be fixed in near feature. And I hope they will be. 3. Setting up database This is quite simple but time consuming operation. First create database structure by running: maketable.sh Edit it to use the user and password you want for the bonsai database. Set file permissions so that only the Bonsai administrator can run this file (typically owner and group are set to root, and access to all but owner denied). You must ensure that your web machine can access the CVS repositories raw data files (',v' files). If the CVS repository is on another machine then the web machine must be configured to be able to read the files as if they were stored with the same pathes on the Web machine. Uually this is accomplished via an NFS read only mount of the cvsroot. You can check this configuration by looking at the file $CVSROOT/modules,v (perhaps this needs the prefix trimmed from this string to make a vaild path name). This file should be readable on both the CVS machine and on the web machine. Then go to Bonsai administration page and press "Rebuild CVS history" button. Then you may go to the theater and watch a movie or two. It will take a lot of time. It takes several seconds to process one file. The more revisions in file the more time it will take. My SUN workstation with 2x200Mhz UltraSPARC processors run about an hour to process about 4K files with 20K+ revisions. Your mileage may vary. If you need to do this more then once you may wish to purge the legaldirs file in the data directory. This is a cache file which holds the names of the directories in CVS, if a directory is not listed here it will not be loaded into the database. Changes to the modules file shoud probably be followed by a deletion of the legaldirs file. I have also found it useful to rerun maketables.sh before reloading the CVS information. If I forget to do this step occasionally the load will fail in the middle because of duplicate data in the table. Copy "dolog.pl" to your CVSROOT directory, and check it in. Add "dolog.pl" to CVSROOT/checkoutlist, and check it in. Then, add a line to your CVSROOT/loginfo file that says something like: ALL $CVSROOT/CVSROOT/dolog.pl -r /cvsroot bonsai-checkin-daemon@my.bonsai.machine Replace "/cvsroot" with the name of the CVS root directory, and "my.bonsai.machine" with the name of the machine Bonsai runs on. Now, on my.bonsai.machine, add a mail alias so that mail sent to "bonsai-checkin-daemon" will get piped to handleCheckinMail.pl. The first argument to handleCheckinMail.pl is the directory that bonsai is installed in. E.g. in /etc/aliases, add bonsai-checkin-daemon: "|/usr/local/bonsai/handleCheckinMail.pl /usr/local/bonsai" or whatever is appropriate for your mail transport agent. Note that if you are using smrsh with Sendmail, you will need to list handleCheckinMail.pl in /etc/smrsh. For example: cd /etc/smrsh ln -s /usr/local/bonsai/handleCheckinMail.pl handleCheckinMail.pl and change the bonsai-checkin-daemon in /etc/aliases to point to /etc/smrsh/handleCheckinMail.pl 4. Registry The Bonsai administrator interface will let you specify where the registry tools are located relative to bonsai. The default is ../registry. Copy the registry directory into this location. One of the registry files has a hardcoded netscape.com domain name in it. Open who.cgi in your favorite editor and change that as needed. 5. Things to do a) There should be better way to track CVS tree changes. Now it's done by making CVS send e-mail about each checkin. (See the comments at the top of dolog.pl for some clues.) One alternative theory would be to take advantage of the CVS history command, which provides all necessary information to get the list of recently committed files, so there is no need to send/process email. Just set up a cron job that will periodically look for CVS tree changes and update database. On the other hand, it's not at all clear how efficient the cvs history command is for large, active repositories. b) Better configuration. One should not hardcode CVS tree <-> Source tree translations. Another thing to configure - banners. c) LXR could be improved in a number of ways. Using MySQL database instead of DB would probably be a good idea. It's unclear what impact it will have on performance though. Incremental database updates would be nice. It might also be nice to borrow syntax highlighting from LSN. 6. Conclusion. OK. This may or may not work for you. But I hope you had a great time trying. Or just reading. Any suggestions/additions are welcome. 7. APPENDIX: Permisions 7.1 mySQL Permissions If you have trouble with the database, make bonsai database writable by all users on your machine and change access level later. This could save you a lot of time trying to guess whether it's permissions or a mistake in the script that make things fail. 7.2 File System Permissions Some symptoms that may be caused by wrong file permissions: pages do not show up, or they show up only partially; new checkins do not show up. The bonsai installation directory needs to be accessible by the web server process and mail process that runs handleCheckinMail.pl. These are typically "apache" and "mail", respectively. make install will set permissions to allow everybody access. Note that maketables.sh should only be available to Bonsai administrator, and you must change this by hand! Everything in the data directory and its subdirectories needs to be accessible by the web server process. Some of the files will also need to be accessible by the mail process. Some files will need to be accessible to all. Below is a sample that is known to work: drwxrwxrwx apache mail ./ -rw-rw-rw- apache apache batch-1.pl -rw-rw---- apache mail batchid.pl -rw-rw---- apache apache cachedstartdates drwxrwx--- apache mail checkinlog/ -rw-rw---- apache mail cvsgraph.conf drwxrw---- apache mail default/ -rw-rw---- apache mail hidelist -rw-rw-rw- apache apache hooklist -rw-rw---- apache mail legaldirs -rw-rw-rw- apache apache lock -rw-rw-rw- apache mail log -rw-rw-rw- apache apache motd.pl -rw-rw-rw- apache apache params -rw-rw-r-- apache apache passwd -rwxrw---- apache apache trapdoor* -rw-rw---- apache mail treeconfig.pl -rw-rw-rw- apache apache whiteboard 7.3 Disable web access to data directory You should make it so that web users can not browse the data directory, or anybody can read data meant only for administrators. In Apache you would typically write the following section in http.conf and restart the server: AllowOverride None Options None Order deny,allow Deny from All 8. APPENDIX: MySQL Replication If you have a really high-traffic site and bonsai is getting queried a lot (this typically happens if you have tinderbox set up with it, and there are a lot of tinderbox trees - since every report from a tinderbox will query bonsai), you can get a performance boost by having bonsai use the slave database (which is typically configured to give queries priority over updates - thus avoiding many lock contentions) for everything except write operations. To do this, set the 'shadowdbiparam' parameter to point at the slave database. The username and password used to access it must be the same. Setting up replication in MySQL is beyond the scope of this document. MySQL has plenty of docs on this subject on their website.