зеркало из https://github.com/mozilla/pjs.git
7013 строки
337 KiB
HTML
7013 строки
337 KiB
HTML
<html lang="en">
|
|
<head>
|
|
<title>BuildBot Manual 0.7.5</title>
|
|
<meta http-equiv="Content-Type" content="text/html">
|
|
<meta name="description" content="BuildBot Manual 0.7.5">
|
|
<meta name="generator" content="makeinfo 4.8">
|
|
<link title="Top" rel="top" href="#Top">
|
|
<link href="http://www.gnu.org/software/texinfo/" rel="generator-home" title="Texinfo Homepage">
|
|
<!--
|
|
This is the BuildBot manual.
|
|
|
|
Copyright (C) 2005,2006 Brian Warner
|
|
|
|
Copying and distribution of this file, with or without
|
|
modification, are permitted in any medium without royalty
|
|
provided the copyright notice and this notice are preserved.-->
|
|
<meta http-equiv="Content-Style-Type" content="text/css">
|
|
<style type="text/css"><!--
|
|
pre.display { font-family:inherit }
|
|
pre.format { font-family:inherit }
|
|
pre.smalldisplay { font-family:inherit; font-size:smaller }
|
|
pre.smallformat { font-family:inherit; font-size:smaller }
|
|
pre.smallexample { font-size:smaller }
|
|
pre.smalllisp { font-size:smaller }
|
|
span.sc { font-variant:small-caps }
|
|
span.roman { font-family:serif; font-weight:normal; }
|
|
span.sansserif { font-family:sans-serif; font-weight:normal; }
|
|
--></style>
|
|
</head>
|
|
<body>
|
|
<h1 class="settitle">BuildBot Manual 0.7.5</h1>
|
|
<div class="contents">
|
|
<h2>Table of Contents</h2>
|
|
<ul>
|
|
<li><a name="toc_Top" href="#Top">BuildBot</a>
|
|
<li><a name="toc_Introduction" href="#Introduction">1 Introduction</a>
|
|
<ul>
|
|
<li><a href="#History-and-Philosophy">1.1 History and Philosophy</a>
|
|
<li><a href="#System-Architecture">1.2 System Architecture</a>
|
|
<ul>
|
|
<li><a href="#BuildSlave-Connections">1.2.1 BuildSlave Connections</a>
|
|
<li><a href="#Buildmaster-Architecture">1.2.2 Buildmaster Architecture</a>
|
|
<li><a href="#Status-Delivery-Architecture">1.2.3 Status Delivery Architecture</a>
|
|
</li></ul>
|
|
<li><a href="#Control-Flow">1.3 Control Flow</a>
|
|
</li></ul>
|
|
<li><a name="toc_Installation" href="#Installation">2 Installation</a>
|
|
<ul>
|
|
<li><a href="#Requirements">2.1 Requirements</a>
|
|
<li><a href="#Installing-the-code">2.2 Installing the code</a>
|
|
<li><a href="#Creating-a-buildmaster">2.3 Creating a buildmaster</a>
|
|
<li><a href="#Creating-a-buildslave">2.4 Creating a buildslave</a>
|
|
<ul>
|
|
<li><a href="#Buildslave-Options">2.4.1 Buildslave Options</a>
|
|
</li></ul>
|
|
<li><a href="#Launching-the-daemons">2.5 Launching the daemons</a>
|
|
<li><a href="#Logfiles">2.6 Logfiles</a>
|
|
<li><a href="#Shutdown">2.7 Shutdown</a>
|
|
<li><a href="#Maintenance">2.8 Maintenance</a>
|
|
<li><a href="#Troubleshooting">2.9 Troubleshooting</a>
|
|
<ul>
|
|
<li><a href="#Starting-the-buildslave">2.9.1 Starting the buildslave</a>
|
|
<li><a href="#Connecting-to-the-buildmaster">2.9.2 Connecting to the buildmaster</a>
|
|
<li><a href="#Forcing-Builds">2.9.3 Forcing Builds</a>
|
|
</li></ul>
|
|
</li></ul>
|
|
<li><a name="toc_Concepts" href="#Concepts">3 Concepts</a>
|
|
<ul>
|
|
<li><a href="#Version-Control-Systems">3.1 Version Control Systems</a>
|
|
<ul>
|
|
<li><a href="#Generalizing-VC-Systems">3.1.1 Generalizing VC Systems</a>
|
|
<li><a href="#Source-Tree-Specifications">3.1.2 Source Tree Specifications</a>
|
|
<li><a href="#How-Different-VC-Systems-Specify-Sources">3.1.3 How Different VC Systems Specify Sources</a>
|
|
<li><a href="#Attributes-of-Changes">3.1.4 Attributes of Changes</a>
|
|
</li></ul>
|
|
<li><a href="#Schedulers">3.2 Schedulers</a>
|
|
<li><a href="#BuildSet">3.3 BuildSet</a>
|
|
<li><a href="#BuildRequest">3.4 BuildRequest</a>
|
|
<li><a href="#Builder">3.5 Builder</a>
|
|
<li><a href="#Users">3.6 Users</a>
|
|
<ul>
|
|
<li><a href="#Doing-Things-With-Users">3.6.1 Doing Things With Users</a>
|
|
<li><a href="#Email-Addresses">3.6.2 Email Addresses</a>
|
|
<li><a href="#IRC-Nicknames">3.6.3 IRC Nicknames</a>
|
|
<li><a href="#Live-Status-Clients">3.6.4 Live Status Clients</a>
|
|
</li></ul>
|
|
</li></ul>
|
|
<li><a name="toc_Configuration" href="#Configuration">4 Configuration</a>
|
|
<ul>
|
|
<li><a href="#Config-File-Format">4.1 Config File Format</a>
|
|
<li><a href="#Loading-the-Config-File">4.2 Loading the Config File</a>
|
|
<li><a href="#Defining-the-Project">4.3 Defining the Project</a>
|
|
<li><a href="#Listing-Change-Sources-and-Schedulers">4.4 Listing Change Sources and Schedulers</a>
|
|
<ul>
|
|
<li><a href="#Scheduler-Types">4.4.1 Scheduler Types</a>
|
|
<li><a href="#Build-Dependencies">4.4.2 Build Dependencies</a>
|
|
</li></ul>
|
|
<li><a href="#Setting-the-slaveport">4.5 Setting the slaveport</a>
|
|
<li><a href="#Buildslave-Specifiers">4.6 Buildslave Specifiers</a>
|
|
<li><a href="#Defining-Builders">4.7 Defining Builders</a>
|
|
<li><a href="#Defining-Status-Targets">4.8 Defining Status Targets</a>
|
|
<li><a href="#Debug-options">4.9 Debug options</a>
|
|
</li></ul>
|
|
<li><a name="toc_Getting-Source-Code-Changes" href="#Getting-Source-Code-Changes">5 Getting Source Code Changes</a>
|
|
<ul>
|
|
<li><a href="#Change-Sources">5.1 Change Sources</a>
|
|
<ul>
|
|
<li><a href="#Choosing-ChangeSources">5.1.1 Choosing ChangeSources</a>
|
|
<li><a href="#CVSToys-_002d-PBService">5.1.2 CVSToys - PBService</a>
|
|
<li><a href="#CVSToys-_002d-mail-notification">5.1.3 CVSToys - mail notification</a>
|
|
<li><a href="#Other-mail-notification-ChangeSources">5.1.4 Other mail notification ChangeSources</a>
|
|
<li><a href="#PBChangeSource">5.1.5 PBChangeSource</a>
|
|
<li><a href="#P4Source">5.1.6 P4Source</a>
|
|
<li><a href="#BonsaiPoller">5.1.7 BonsaiPoller</a>
|
|
<li><a href="#SVNPoller">5.1.8 SVNPoller</a>
|
|
</li></ul>
|
|
</li></ul>
|
|
<li><a name="toc_Build-Process" href="#Build-Process">6 Build Process</a>
|
|
<ul>
|
|
<li><a href="#Build-Steps">6.1 Build Steps</a>
|
|
<ul>
|
|
<li><a href="#Common-Parameters">6.1.1 Common Parameters</a>
|
|
<li><a href="#Source-Checkout">6.1.2 Source Checkout</a>
|
|
<ul>
|
|
<li><a href="#CVS">6.1.2.1 CVS</a>
|
|
<li><a href="#SVN">6.1.2.2 SVN</a>
|
|
<li><a href="#Darcs">6.1.2.3 Darcs</a>
|
|
<li><a href="#Mercurial">6.1.2.4 Mercurial</a>
|
|
<li><a href="#Arch">6.1.2.5 Arch</a>
|
|
<li><a href="#Bazaar">6.1.2.6 Bazaar</a>
|
|
<li><a href="#P4">6.1.2.7 P4</a>
|
|
</li></ul>
|
|
<li><a href="#ShellCommand">6.1.3 ShellCommand</a>
|
|
<li><a href="#Simple-ShellCommand-Subclasses">6.1.4 Simple ShellCommand Subclasses</a>
|
|
<ul>
|
|
<li><a href="#Configure">6.1.4.1 Configure</a>
|
|
<li><a href="#Compile">6.1.4.2 Compile</a>
|
|
<li><a href="#Test">6.1.4.3 Test</a>
|
|
<li><a href="#Build-Properties">6.1.4.4 Build Properties</a>
|
|
</li></ul>
|
|
<li><a href="#Python-BuildSteps">6.1.5 Python BuildSteps</a>
|
|
<ul>
|
|
<li><a href="#BuildEPYDoc">6.1.5.1 BuildEPYDoc</a>
|
|
<li><a href="#PyFlakes">6.1.5.2 PyFlakes</a>
|
|
</li></ul>
|
|
<li><a href="#Transferring-Files">6.1.6 Transferring Files</a>
|
|
<li><a href="#Writing-New-BuildSteps">6.1.7 Writing New BuildSteps</a>
|
|
<ul>
|
|
<li><a href="#BuildStep-LogFiles">6.1.7.1 BuildStep LogFiles</a>
|
|
<li><a href="#Adding-LogObservers">6.1.7.2 Adding LogObservers</a>
|
|
<li><a href="#BuildStep-URLs">6.1.7.3 BuildStep URLs</a>
|
|
</li></ul>
|
|
</li></ul>
|
|
<li><a href="#Interlocks">6.2 Interlocks</a>
|
|
<li><a href="#Build-Factories">6.3 Build Factories</a>
|
|
<ul>
|
|
<li><a href="#BuildStep-Objects">6.3.1 BuildStep Objects</a>
|
|
<li><a href="#BuildFactory">6.3.2 BuildFactory</a>
|
|
<ul>
|
|
<li><a href="#BuildFactory-Attributes">6.3.2.1 BuildFactory Attributes</a>
|
|
<li><a href="#Quick-builds">6.3.2.2 Quick builds</a>
|
|
</li></ul>
|
|
<li><a href="#Process_002dSpecific-build-factories">6.3.3 Process-Specific build factories</a>
|
|
<ul>
|
|
<li><a href="#GNUAutoconf">6.3.3.1 GNUAutoconf</a>
|
|
<li><a href="#CPAN">6.3.3.2 CPAN</a>
|
|
<li><a href="#Python-distutils">6.3.3.3 Python distutils</a>
|
|
<li><a href="#Python_002fTwisted_002ftrial-projects">6.3.3.4 Python/Twisted/trial projects</a>
|
|
</li></ul>
|
|
</li></ul>
|
|
</li></ul>
|
|
<li><a name="toc_Status-Delivery" href="#Status-Delivery">7 Status Delivery</a>
|
|
<ul>
|
|
<li><a href="#HTML-Waterfall">7.1 HTML Waterfall</a>
|
|
<li><a href="#IRC-Bot">7.2 IRC Bot</a>
|
|
<li><a href="#PBListener">7.3 PBListener</a>
|
|
<li><a href="#Writing-New-Status-Plugins">7.4 Writing New Status Plugins</a>
|
|
</li></ul>
|
|
<li><a name="toc_Command_002dline-tool" href="#Command_002dline-tool">8 Command-line tool</a>
|
|
<ul>
|
|
<li><a href="#Administrator-Tools">8.1 Administrator Tools</a>
|
|
<li><a href="#Developer-Tools">8.2 Developer Tools</a>
|
|
<ul>
|
|
<li><a href="#statuslog">8.2.1 statuslog</a>
|
|
<li><a href="#statusgui">8.2.2 statusgui</a>
|
|
<li><a href="#try">8.2.3 try</a>
|
|
</li></ul>
|
|
<li><a href="#Other-Tools">8.3 Other Tools</a>
|
|
<ul>
|
|
<li><a href="#sendchange">8.3.1 sendchange</a>
|
|
<li><a href="#debugclient">8.3.2 debugclient</a>
|
|
</li></ul>
|
|
<li><a href="#_002ebuildbot-config-directory">8.4 .buildbot config directory</a>
|
|
</li></ul>
|
|
<li><a name="toc_Resources" href="#Resources">9 Resources</a>
|
|
<li><a name="toc_Developer_0027s-Appendix" href="#Developer_0027s-Appendix">Developer's Appendix</a>
|
|
<li><a name="toc_Index-of-Useful-Classes" href="#Index-of-Useful-Classes">Index of Useful Classes</a>
|
|
<li><a name="toc_Index-of-master_002ecfg-keys" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>
|
|
<li><a name="toc_Index" href="#Index">Index</a>
|
|
</li></ul>
|
|
</div>
|
|
|
|
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Top"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Introduction">Introduction</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#dir">(dir)</a>,
|
|
Up: <a rel="up" accesskey="u" href="#dir">(dir)</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="unnumbered">BuildBot</h2>
|
|
|
|
<p>This is the BuildBot manual.
|
|
|
|
<p>Copyright (C) 2005,2006 Brian Warner
|
|
|
|
<p>Copying and distribution of this file, with or without
|
|
modification, are permitted in any medium without royalty
|
|
provided the copyright notice and this notice are preserved.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Introduction">Introduction</a>: What the BuildBot does.
|
|
<li><a accesskey="2" href="#Installation">Installation</a>: Creating a buildmaster and buildslaves,
|
|
running them.
|
|
<li><a accesskey="3" href="#Concepts">Concepts</a>: What goes on in the buildbot's little mind.
|
|
<li><a accesskey="4" href="#Configuration">Configuration</a>: Controlling the buildbot.
|
|
<li><a accesskey="5" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>: Discovering when to run a build.
|
|
<li><a accesskey="6" href="#Build-Process">Build Process</a>: Controlling how each build is run.
|
|
<li><a accesskey="7" href="#Status-Delivery">Status Delivery</a>: Telling the world about the build's results.
|
|
<li><a accesskey="8" href="#Command_002dline-tool">Command-line tool</a>
|
|
<li><a accesskey="9" href="#Resources">Resources</a>: Getting help.
|
|
<li><a href="#Developer_0027s-Appendix">Developer's Appendix</a>
|
|
<li><a href="#Index-of-Useful-Classes">Index of Useful Classes</a>
|
|
<li><a href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>
|
|
<li><a href="#Index">Index</a>: Complete index.
|
|
|
|
</li></ul>
|
|
<p>--- The Detailed Node Listing ---
|
|
|
|
<p>Introduction
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#History-and-Philosophy">History and Philosophy</a>
|
|
<li><a href="#System-Architecture">System Architecture</a>
|
|
<li><a href="#Control-Flow">Control Flow</a>
|
|
|
|
</li></ul>
|
|
<p>System Architecture
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#BuildSlave-Connections">BuildSlave Connections</a>
|
|
<li><a href="#Buildmaster-Architecture">Buildmaster Architecture</a>
|
|
<li><a href="#Status-Delivery-Architecture">Status Delivery Architecture</a>
|
|
|
|
</li></ul>
|
|
<p>Installation
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Requirements">Requirements</a>
|
|
<li><a href="#Installing-the-code">Installing the code</a>
|
|
<li><a href="#Creating-a-buildmaster">Creating a buildmaster</a>
|
|
<li><a href="#Creating-a-buildslave">Creating a buildslave</a>
|
|
<li><a href="#Launching-the-daemons">Launching the daemons</a>
|
|
<li><a href="#Logfiles">Logfiles</a>
|
|
<li><a href="#Shutdown">Shutdown</a>
|
|
<li><a href="#Maintenance">Maintenance</a>
|
|
<li><a href="#Troubleshooting">Troubleshooting</a>
|
|
|
|
</li></ul>
|
|
<p>Creating a buildslave
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Buildslave-Options">Buildslave Options</a>
|
|
|
|
</li></ul>
|
|
<p>Troubleshooting
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Starting-the-buildslave">Starting the buildslave</a>
|
|
<li><a href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>
|
|
<li><a href="#Forcing-Builds">Forcing Builds</a>
|
|
|
|
</li></ul>
|
|
<p>Concepts
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Version-Control-Systems">Version Control Systems</a>
|
|
<li><a href="#Schedulers">Schedulers</a>
|
|
<li><a href="#BuildSet">BuildSet</a>
|
|
<li><a href="#BuildRequest">BuildRequest</a>
|
|
<li><a href="#Builder">Builder</a>
|
|
<li><a href="#Users">Users</a>
|
|
|
|
</li></ul>
|
|
<p>Version Control Systems
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Generalizing-VC-Systems">Generalizing VC Systems</a>
|
|
<li><a href="#Source-Tree-Specifications">Source Tree Specifications</a>
|
|
<li><a href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>
|
|
<li><a href="#Attributes-of-Changes">Attributes of Changes</a>
|
|
|
|
</li></ul>
|
|
<p>Users
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Doing-Things-With-Users">Doing Things With Users</a>
|
|
<li><a href="#Email-Addresses">Email Addresses</a>
|
|
<li><a href="#IRC-Nicknames">IRC Nicknames</a>
|
|
<li><a href="#Live-Status-Clients">Live Status Clients</a>
|
|
|
|
</li></ul>
|
|
<p>Configuration
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Config-File-Format">Config File Format</a>
|
|
<li><a href="#Loading-the-Config-File">Loading the Config File</a>
|
|
<li><a href="#Defining-the-Project">Defining the Project</a>
|
|
<li><a href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>
|
|
<li><a href="#Setting-the-slaveport">Setting the slaveport</a>
|
|
<li><a href="#Buildslave-Specifiers">Buildslave Specifiers</a>
|
|
<li><a href="#Defining-Builders">Defining Builders</a>
|
|
<li><a href="#Defining-Status-Targets">Defining Status Targets</a>
|
|
<li><a href="#Debug-options">Debug options</a>
|
|
|
|
</li></ul>
|
|
<p>Listing Change Sources and Schedulers
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Scheduler-Types">Scheduler Types</a>
|
|
<li><a href="#Build-Dependencies">Build Dependencies</a>
|
|
|
|
</li></ul>
|
|
<p>Getting Source Code Changes
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Change-Sources">Change Sources</a>
|
|
|
|
</li></ul>
|
|
<p>Change Sources
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Choosing-ChangeSources">Choosing ChangeSources</a>
|
|
<li><a href="#CVSToys-_002d-PBService">CVSToys - PBService</a>
|
|
<li><a href="#CVSToys-_002d-mail-notification">CVSToys - mail notification</a>
|
|
<li><a href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a>
|
|
<li><a href="#PBChangeSource">PBChangeSource</a>
|
|
<li><a href="#P4Source">P4Source</a>
|
|
<li><a href="#BonsaiPoller">BonsaiPoller</a>
|
|
<li><a href="#SVNPoller">SVNPoller</a>
|
|
|
|
</li></ul>
|
|
<p>Build Process
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Build-Steps">Build Steps</a>
|
|
<li><a href="#Interlocks">Interlocks</a>
|
|
<li><a href="#Build-Factories">Build Factories</a>
|
|
|
|
</li></ul>
|
|
<p>Build Steps
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Common-Parameters">Common Parameters</a>
|
|
<li><a href="#Source-Checkout">Source Checkout</a>
|
|
<li><a href="#ShellCommand">ShellCommand</a>
|
|
<li><a href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>
|
|
<li><a href="#Python-BuildSteps">Python BuildSteps</a>
|
|
<li><a href="#Transferring-Files">Transferring Files</a>
|
|
<li><a href="#Writing-New-BuildSteps">Writing New BuildSteps</a>
|
|
|
|
</li></ul>
|
|
<p>Source Checkout
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#CVS">CVS</a>
|
|
<li><a href="#SVN">SVN</a>
|
|
<li><a href="#Darcs">Darcs</a>
|
|
<li><a href="#Mercurial">Mercurial</a>
|
|
<li><a href="#Arch">Arch</a>
|
|
<li><a href="#Bazaar">Bazaar</a>
|
|
<li><a href="#P4">P4</a>
|
|
|
|
</li></ul>
|
|
<p>Simple ShellCommand Subclasses
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Configure">Configure</a>
|
|
<li><a href="#Compile">Compile</a>
|
|
<li><a href="#Test">Test</a>
|
|
<li><a href="#Build-Properties">Build Properties</a>
|
|
|
|
</li></ul>
|
|
<p>Python BuildSteps
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#BuildEPYDoc">BuildEPYDoc</a>
|
|
<li><a href="#PyFlakes">PyFlakes</a>
|
|
|
|
</li></ul>
|
|
<p>Writing New BuildSteps
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#BuildStep-LogFiles">BuildStep LogFiles</a>
|
|
<li><a href="#Adding-LogObservers">Adding LogObservers</a>
|
|
<li><a href="#BuildStep-URLs">BuildStep URLs</a>
|
|
|
|
</li></ul>
|
|
<p>Build Factories
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#BuildStep-Objects">BuildStep Objects</a>
|
|
<li><a href="#BuildFactory">BuildFactory</a>
|
|
<li><a href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>
|
|
|
|
</li></ul>
|
|
<p>BuildStep Objects
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#BuildFactory-Attributes">BuildFactory Attributes</a>
|
|
<li><a href="#Quick-builds">Quick builds</a>
|
|
|
|
</li></ul>
|
|
<p>BuildFactory
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#BuildFactory-Attributes">BuildFactory Attributes</a>
|
|
<li><a href="#Quick-builds">Quick builds</a>
|
|
|
|
</li></ul>
|
|
<p>Process-Specific build factories
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#GNUAutoconf">GNUAutoconf</a>
|
|
<li><a href="#CPAN">CPAN</a>
|
|
<li><a href="#Python-distutils">Python distutils</a>
|
|
<li><a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a>
|
|
|
|
</li></ul>
|
|
<p>Status Delivery
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#HTML-Waterfall">HTML Waterfall</a>
|
|
<li><a href="#IRC-Bot">IRC Bot</a>
|
|
<li><a href="#PBListener">PBListener</a>
|
|
<li><a href="#Writing-New-Status-Plugins">Writing New Status Plugins</a>
|
|
|
|
</li></ul>
|
|
<p>Command-line tool
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#Administrator-Tools">Administrator Tools</a>
|
|
<li><a href="#Developer-Tools">Developer Tools</a>
|
|
<li><a href="#Other-Tools">Other Tools</a>
|
|
<li><a href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>
|
|
|
|
</li></ul>
|
|
<p>Developer Tools
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#statuslog">statuslog</a>
|
|
<li><a href="#statusgui">statusgui</a>
|
|
<li><a href="#try">try</a>
|
|
|
|
</li></ul>
|
|
<p>Other Tools
|
|
|
|
</p>
|
|
<ul class="menu">
|
|
<li><a href="#sendchange">sendchange</a>
|
|
<li><a href="#debugclient">debugclient</a>
|
|
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Introduction"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Installation">Installation</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Top">Top</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="chapter">1 Introduction</h2>
|
|
|
|
<p><a name="index-introduction-1"></a>
|
|
The BuildBot is a system to automate the compile/test cycle required by most
|
|
software projects to validate code changes. By automatically rebuilding and
|
|
testing the tree each time something has changed, build problems are
|
|
pinpointed quickly, before other developers are inconvenienced by the
|
|
failure. The guilty developer can be identified and harassed without human
|
|
intervention. By running the builds on a variety of platforms, developers
|
|
who do not have the facilities to test their changes everywhere before
|
|
checkin will at least know shortly afterwards whether they have broken the
|
|
build or not. Warning counts, lint checks, image size, compile time, and
|
|
other build parameters can be tracked over time, are more visible, and
|
|
are therefore easier to improve.
|
|
|
|
<p>The overall goal is to reduce tree breakage and provide a platform to
|
|
run tests or code-quality checks that are too annoying or pedantic for
|
|
any human to waste their time with. Developers get immediate (and
|
|
potentially public) feedback about their changes, encouraging them to
|
|
be more careful about testing before checkin.
|
|
|
|
<p>Features:
|
|
|
|
<ul>
|
|
<li>run builds on a variety of slave platforms
|
|
<li>arbitrary build process: handles projects using C, Python, whatever
|
|
<li>minimal host requirements: python and Twisted
|
|
<li>slaves can be behind a firewall if they can still do checkout
|
|
<li>status delivery through web page, email, IRC, other protocols
|
|
<li>track builds in progress, provide estimated completion time
|
|
<li>flexible configuration by subclassing generic build process classes
|
|
<li>debug tools to force a new build, submit fake Changes, query slave status
|
|
<li>released under the GPL
|
|
</ul>
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#History-and-Philosophy">History and Philosophy</a>
|
|
<li><a accesskey="2" href="#System-Architecture">System Architecture</a>
|
|
<li><a accesskey="3" href="#Control-Flow">Control Flow</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="History-and-Philosophy"></a>
|
|
Next: <a rel="next" accesskey="n" href="#System-Architecture">System Architecture</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Introduction">Introduction</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Introduction">Introduction</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">1.1 History and Philosophy</h3>
|
|
|
|
<p><a name="index-Philosophy-of-operation-2"></a>
|
|
The Buildbot was inspired by a similar project built for a development
|
|
team writing a cross-platform embedded system. The various components
|
|
of the project were supposed to compile and run on several flavors of
|
|
unix (linux, solaris, BSD), but individual developers had their own
|
|
preferences and tended to stick to a single platform. From time to
|
|
time, incompatibilities would sneak in (some unix platforms want to
|
|
use <code>string.h</code>, some prefer <code>strings.h</code>), and then the tree
|
|
would compile for some developers but not others. The buildbot was
|
|
written to automate the human process of walking into the office,
|
|
updating a tree, compiling (and discovering the breakage), finding the
|
|
developer at fault, and complaining to them about the problem they had
|
|
introduced. With multiple platforms it was difficult for developers to
|
|
do the right thing (compile their potential change on all platforms);
|
|
the buildbot offered a way to help.
|
|
|
|
<p>Another problem was when programmers would change the behavior of a
|
|
library without warning its users, or change internal aspects that
|
|
other code was (unfortunately) depending upon. Adding unit tests to
|
|
the codebase helps here: if an application's unit tests pass despite
|
|
changes in the libraries it uses, you can have more confidence that
|
|
the library changes haven't broken anything. Many developers
|
|
complained that the unit tests were inconvenient or took too long to
|
|
run: having the buildbot run them reduces the developer's workload to
|
|
a minimum.
|
|
|
|
<p>In general, having more visibility into the project is always good,
|
|
and automation makes it easier for developers to do the right thing.
|
|
When everyone can see the status of the project, developers are
|
|
encouraged to keep the tree in good working order. Unit tests that
|
|
aren't run on a regular basis tend to suffer from bitrot just like
|
|
code does: exercising them on a regular basis helps to keep them
|
|
functioning and useful.
|
|
|
|
<p>The current version of the Buildbot is additionally targeted at
|
|
distributed free-software projects, where resources and platforms are
|
|
only available when provided by interested volunteers. The buildslaves
|
|
are designed to require an absolute minimum of configuration, reducing
|
|
the effort a potential volunteer needs to expend to be able to
|
|
contribute a new test environment to the project. The goal is for
|
|
anyone who wishes that a given project would run on their favorite
|
|
platform should be able to offer that project a buildslave, running on
|
|
that platform, where they can verify that their portability code
|
|
works, and keeps working.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="System-Architecture"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Control-Flow">Control Flow</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#History-and-Philosophy">History and Philosophy</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Introduction">Introduction</a>
|
|
|
|
</div>
|
|
|
|
<!-- node-name, next, previous, up -->
|
|
<h3 class="section">1.2 System Architecture</h3>
|
|
|
|
<p>The Buildbot consists of a single <code>buildmaster</code> and one or more
|
|
<code>buildslaves</code>, connected in a star topology. The buildmaster
|
|
makes all decisions about what, when, and how to build. It sends
|
|
commands to be run on the build slaves, which simply execute the
|
|
commands and return the results. (certain steps involve more local
|
|
decision making, where the overhead of sending a lot of commands back
|
|
and forth would be inappropriate, but in general the buildmaster is
|
|
responsible for everything).
|
|
|
|
<p>The buildmaster is usually fed <code>Changes</code> by some sort of version
|
|
control system (see <a href="#Change-Sources">Change Sources</a>), which may cause builds to be
|
|
run. As the builds are performed, various status messages are
|
|
produced, which are then sent to any registered Status Targets
|
|
(see <a href="#Status-Delivery">Status Delivery</a>).
|
|
|
|
<!-- @image{FILENAME, WIDTH, HEIGHT, ALTTEXT, EXTENSION} -->
|
|
<div class="block-image"><img src="images/overview.png" alt="Overview Diagram"></div>
|
|
|
|
<p>The buildmaster is configured and maintained by the “buildmaster
|
|
admin”, who is generally the project team member responsible for
|
|
build process issues. Each buildslave is maintained by a “buildslave
|
|
admin”, who do not need to be quite as involved. Generally slaves are
|
|
run by anyone who has an interest in seeing the project work well on
|
|
their favorite platform.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#BuildSlave-Connections">BuildSlave Connections</a>
|
|
<li><a accesskey="2" href="#Buildmaster-Architecture">Buildmaster Architecture</a>
|
|
<li><a accesskey="3" href="#Status-Delivery-Architecture">Status Delivery Architecture</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="BuildSlave-Connections"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Buildmaster-Architecture">Buildmaster Architecture</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#System-Architecture">System Architecture</a>,
|
|
Up: <a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">1.2.1 BuildSlave Connections</h4>
|
|
|
|
<p>The buildslaves are typically run on a variety of separate machines,
|
|
at least one per platform of interest. These machines connect to the
|
|
buildmaster over a TCP connection to a publically-visible port. As a
|
|
result, the buildslaves can live behind a NAT box or similar
|
|
firewalls, as long as they can get to buildmaster. The TCP connections
|
|
are initiated by the buildslave and accepted by the buildmaster, but
|
|
commands and results travel both ways within this connection. The
|
|
buildmaster is always in charge, so all commands travel exclusively
|
|
from the buildmaster to the buildslave.
|
|
|
|
<p>To perform builds, the buildslaves must typically obtain source code
|
|
from a CVS/SVN/etc repository. Therefore they must also be able to
|
|
reach the repository. The buildmaster provides instructions for
|
|
performing builds, but does not provide the source code itself.
|
|
|
|
<div class="block-image"><img src="images/slaves.png" alt="BuildSlave Connections"></div>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Buildmaster-Architecture"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Status-Delivery-Architecture">Status Delivery Architecture</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#BuildSlave-Connections">BuildSlave Connections</a>,
|
|
Up: <a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">1.2.2 Buildmaster Architecture</h4>
|
|
|
|
<p>The Buildmaster consists of several pieces:
|
|
|
|
<div class="block-image"><img src="images/master.png" alt="BuildMaster Architecture"></div>
|
|
|
|
<ul>
|
|
<li>Change Sources, which create a Change object each time something is
|
|
modified in the VC repository. Most ChangeSources listen for messages
|
|
from a hook script of some sort. Some sources actively poll the
|
|
repository on a regular basis. All Changes are fed to the Schedulers.
|
|
|
|
<li>Schedulers, which decide when builds should be performed. They collect
|
|
Changes into BuildRequests, which are then queued for delivery to
|
|
Builders until a buildslave is available.
|
|
|
|
<li>Builders, which control exactly <em>how</em> each build is performed
|
|
(with a series of BuildSteps, configured in a BuildFactory). Each
|
|
Build is run on a single buildslave.
|
|
|
|
<li>Status plugins, which deliver information about the build results
|
|
through protocols like HTTP, mail, and IRC.
|
|
|
|
</ul>
|
|
|
|
<div class="block-image"><img src="images/slavebuilder.png" alt="SlaveBuilders"></div>
|
|
|
|
<p>Each Builder is configured with a list of BuildSlaves that it will use
|
|
for its builds. These buildslaves are expected to behave identically:
|
|
the only reason to use multiple BuildSlaves for a single Builder is to
|
|
provide a measure of load-balancing.
|
|
|
|
<p>Within a single BuildSlave, each Builder creates its own SlaveBuilder
|
|
instance. These SlaveBuilders operate independently from each other.
|
|
Each gets its own base directory to work in. It is quite common to
|
|
have many Builders sharing the same buildslave. For example, there
|
|
might be two buildslaves: one for i386, and a second for PowerPC.
|
|
There may then be a pair of Builders that do a full compile/test run,
|
|
one for each architecture, and a lone Builder that creates snapshot
|
|
source tarballs if the full builders complete successfully. The full
|
|
builders would each run on a single buildslave, whereas the tarball
|
|
creation step might run on either buildslave (since the platform
|
|
doesn't matter when creating source tarballs). In this case, the
|
|
mapping would look like:
|
|
|
|
<pre class="example"> Builder(full-i386) -> BuildSlaves(slave-i386)
|
|
Builder(full-ppc) -> BuildSlaves(slave-ppc)
|
|
Builder(source-tarball) -> BuildSlaves(slave-i386, slave-ppc)
|
|
</pre>
|
|
<p>and each BuildSlave would have two SlaveBuilders inside it, one for a
|
|
full builder, and a second for the source-tarball builder.
|
|
|
|
<p>Once a SlaveBuilder is available, the Builder pulls one or more
|
|
BuildRequests off its incoming queue. (It may pull more than one if it
|
|
determines that it can merge the requests together; for example, there
|
|
may be multiple requests to build the current HEAD revision). These
|
|
requests are merged into a single Build instance, which includes the
|
|
SourceStamp that describes what exact version of the source code
|
|
should be used for the build. The Build is then assigned to a
|
|
SlaveBuilder and the build begins.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Status-Delivery-Architecture"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Buildmaster-Architecture">Buildmaster Architecture</a>,
|
|
Up: <a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">1.2.3 Status Delivery Architecture</h4>
|
|
|
|
<p>The buildmaster maintains a central Status object, to which various
|
|
status plugins are connected. Through this Status object, a full
|
|
hierarchy of build status objects can be obtained.
|
|
|
|
<div class="block-image"><img src="images/status.png" alt="Status Delivery"></div>
|
|
|
|
<p>The configuration file controls which status plugins are active. Each
|
|
status plugin gets a reference to the top-level Status object. From
|
|
there they can request information on each Builder, Build, Step, and
|
|
LogFile. This query-on-demand interface is used by the html.Waterfall
|
|
plugin to create the main status page each time a web browser hits the
|
|
main URL.
|
|
|
|
<p>The status plugins can also subscribe to hear about new Builds as they
|
|
occur: this is used by the MailNotifier to create new email messages
|
|
for each recently-completed Build.
|
|
|
|
<p>The Status object records the status of old builds on disk in the
|
|
buildmaster's base directory. This allows it to return information
|
|
about historical builds.
|
|
|
|
<p>There are also status objects that correspond to Schedulers and
|
|
BuildSlaves. These allow status plugins to report information about
|
|
upcoming builds, and the online/offline status of each buildslave.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Control-Flow"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#System-Architecture">System Architecture</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Introduction">Introduction</a>
|
|
|
|
</div>
|
|
|
|
<!-- node-name, next, previous, up -->
|
|
<h3 class="section">1.3 Control Flow</h3>
|
|
|
|
<p>A day in the life of the buildbot:
|
|
|
|
<ul>
|
|
<li>A developer commits some source code changes to the repository. A hook
|
|
script or commit trigger of some sort sends information about this
|
|
change to the buildmaster through one of its configured Change
|
|
Sources. This notification might arrive via email, or over a network
|
|
connection (either initiated by the buildmaster as it “subscribes”
|
|
to changes, or by the commit trigger as it pushes Changes towards the
|
|
buildmaster). The Change contains information about who made the
|
|
change, what files were modified, which revision contains the change,
|
|
and any checkin comments.
|
|
|
|
<li>The buildmaster distributes this change to all of its configured
|
|
Schedulers. Any “important” changes cause the “tree-stable-timer”
|
|
to be started, and the Change is added to a list of those that will go
|
|
into a new Build. When the timer expires, a Build is started on each
|
|
of a set of configured Builders, all compiling/testing the same source
|
|
code. Unless configured otherwise, all Builds run in parallel on the
|
|
various buildslaves.
|
|
|
|
<li>The Build consists of a series of Steps. Each Step causes some number
|
|
of commands to be invoked on the remote buildslave associated with
|
|
that Builder. The first step is almost always to perform a checkout of
|
|
the appropriate revision from the same VC system that produced the
|
|
Change. The rest generally perform a compile and run unit tests. As
|
|
each Step runs, the buildslave reports back command output and return
|
|
status to the buildmaster.
|
|
|
|
<li>As the Build runs, status messages like “Build Started”, “Step
|
|
Started”, “Build Finished”, etc, are published to a collection of
|
|
Status Targets. One of these targets is usually the HTML “Waterfall”
|
|
display, which shows a chronological list of events, and summarizes
|
|
the results of the most recent build at the top of each column.
|
|
Developers can periodically check this page to see how their changes
|
|
have fared. If they see red, they know that they've made a mistake and
|
|
need to fix it. If they see green, they know that they've done their
|
|
duty and don't need to worry about their change breaking anything.
|
|
|
|
<li>If a MailNotifier status target is active, the completion of a build
|
|
will cause email to be sent to any developers whose Changes were
|
|
incorporated into this Build. The MailNotifier can be configured to
|
|
only send mail upon failing builds, or for builds which have just
|
|
transitioned from passing to failing. Other status targets can provide
|
|
similar real-time notification via different communication channels,
|
|
like IRC.
|
|
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Installation"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Concepts">Concepts</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Introduction">Introduction</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="chapter">2 Installation</h2>
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Requirements">Requirements</a>
|
|
<li><a accesskey="2" href="#Installing-the-code">Installing the code</a>
|
|
<li><a accesskey="3" href="#Creating-a-buildmaster">Creating a buildmaster</a>
|
|
<li><a accesskey="4" href="#Creating-a-buildslave">Creating a buildslave</a>
|
|
<li><a accesskey="5" href="#Launching-the-daemons">Launching the daemons</a>
|
|
<li><a accesskey="6" href="#Logfiles">Logfiles</a>
|
|
<li><a accesskey="7" href="#Shutdown">Shutdown</a>
|
|
<li><a accesskey="8" href="#Maintenance">Maintenance</a>
|
|
<li><a accesskey="9" href="#Troubleshooting">Troubleshooting</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Requirements"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Installing-the-code">Installing the code</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Installation">Installation</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Installation">Installation</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">2.1 Requirements</h3>
|
|
|
|
<p>At a bare minimum, you'll need the following (for both the buildmaster
|
|
and a buildslave):
|
|
|
|
<ul>
|
|
<li>Python: http://www.python.org
|
|
|
|
<p>Buildbot requires python-2.2 or later, and is primarily developed
|
|
against python-2.3. The buildmaster uses generators, a feature which
|
|
is not available in python-2.1, and both master and slave require a
|
|
version of Twisted which only works with python-2.2 or later. Certain
|
|
features (like the inclusion of build logs in status emails) require
|
|
python-2.2.2 or later. The IRC “force build” command requires
|
|
python-2.3 (for the shlex.split function).
|
|
|
|
<li>Twisted: http://twistedmatrix.com
|
|
|
|
<p>Both the buildmaster and the buildslaves require Twisted-1.3.0 or
|
|
later. It has been mainly developed against Twisted-2.0.1, but has
|
|
been tested against Twisted-2.1.0 (the most recent as of this
|
|
writing), and might even work on versions as old as Twisted-1.1.0, but
|
|
as always the most recent version is recommended.
|
|
|
|
<p>Twisted-1.3.0 and earlier were released as a single monolithic
|
|
package. When you run Buildbot against Twisted-2.0.0 or later (which
|
|
are split into a number of smaller subpackages), you'll need at least
|
|
"Twisted" (the core package), and you'll also want TwistedMail,
|
|
TwistedWeb, and TwistedWords (for sending email, serving a web status
|
|
page, and delivering build status via IRC, respectively).
|
|
</ul>
|
|
|
|
<p>Certain other packages may be useful on the system running the
|
|
buildmaster:
|
|
|
|
<ul>
|
|
<li>CVSToys: http://purl.net/net/CVSToys
|
|
|
|
<p>If your buildmaster uses FreshCVSSource to receive change notification
|
|
from a cvstoys daemon, it will require CVSToys be installed (tested
|
|
with CVSToys-1.0.10). If the it doesn't use that source (i.e. if you
|
|
only use a mail-parsing change source, or the SVN notification
|
|
script), you will not need CVSToys.
|
|
|
|
</ul>
|
|
|
|
<p>And of course, your project's build process will impose additional
|
|
requirements on the buildslaves. These hosts must have all the tools
|
|
necessary to compile and test your project's source code.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Installing-the-code"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Creating-a-buildmaster">Creating a buildmaster</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Requirements">Requirements</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Installation">Installation</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">2.2 Installing the code</h3>
|
|
|
|
<p><a name="index-installation-3"></a>
|
|
The Buildbot is installed using the standard python <code>distutils</code>
|
|
module. After unpacking the tarball, the process is:
|
|
|
|
<pre class="example"> python setup.py build
|
|
python setup.py install
|
|
</pre>
|
|
<p>where the install step may need to be done as root. This will put the
|
|
bulk of the code in somewhere like
|
|
/usr/lib/python2.3/site-packages/buildbot . It will also install the
|
|
<code>buildbot</code> command-line tool in /usr/bin/buildbot.
|
|
|
|
<p>To test this, shift to a different directory (like /tmp), and run:
|
|
|
|
<pre class="example"> buildbot --version
|
|
</pre>
|
|
<p>If it shows you the versions of Buildbot and Twisted, the install went
|
|
ok. If it says <code>no such command</code> or it gets an <code>ImportError</code>
|
|
when it tries to load the libaries, then something went wrong.
|
|
<code>pydoc buildbot</code> is another useful diagnostic tool.
|
|
|
|
<p>Windows users will find these files in other places. You will need to
|
|
make sure that python can find the libraries, and will probably find
|
|
it convenient to have <code>buildbot</code> on your PATH.
|
|
|
|
<p>If you wish, you can run the buildbot unit test suite like this:
|
|
|
|
<pre class="example"> PYTHONPATH=. trial buildbot.test
|
|
</pre>
|
|
<p>This should run up to 192 tests, depending upon what VC tools you have
|
|
installed. On my desktop machine it takes about five minutes to
|
|
complete. Nothing should fail, a few might be skipped. If any of the
|
|
tests fail, you should stop and investigate the cause before
|
|
continuing the installation process, as it will probably be easier to
|
|
track down the bug early.
|
|
|
|
<p>If you cannot or do not wish to install the buildbot into a site-wide
|
|
location like <samp><span class="file">/usr</span></samp> or <samp><span class="file">/usr/local</span></samp>, you can also install
|
|
it into the account's home directory. Do the install command like
|
|
this:
|
|
|
|
<pre class="example"> python setup.py install --home=~
|
|
</pre>
|
|
<p>That will populate <samp><span class="file">~/lib/python</span></samp> and create
|
|
<samp><span class="file">~/bin/buildbot</span></samp>. Make sure this lib directory is on your
|
|
<code>PYTHONPATH</code>.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Creating-a-buildmaster"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Creating-a-buildslave">Creating a buildslave</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Installing-the-code">Installing the code</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Installation">Installation</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">2.3 Creating a buildmaster</h3>
|
|
|
|
<p>As you learned earlier (see <a href="#System-Architecture">System Architecture</a>), the buildmaster
|
|
runs on a central host (usually one that is publically visible, so
|
|
everybody can check on the status of the project), and controls all
|
|
aspects of the buildbot system. Let us call this host
|
|
<code>buildbot.example.org</code>.
|
|
|
|
<p>You may wish to create a separate user account for the buildmaster,
|
|
perhaps named <code>buildmaster</code>. This can help keep your personal
|
|
configuration distinct from that of the buildmaster and is useful if
|
|
you have to use a mail-based notification system (see <a href="#Change-Sources">Change Sources</a>). However, the Buildbot will work just fine with your regular
|
|
user account.
|
|
|
|
<p>You need to choose a directory for the buildmaster, called the
|
|
<code>basedir</code>. This directory will be owned by the buildmaster, which
|
|
will use configuration files therein, and create status files as it
|
|
runs. <samp><span class="file">~/Buildbot</span></samp> is a likely value. If you run multiple
|
|
buildmasters in the same account, or if you run both masters and
|
|
slaves, you may want a more distinctive name like
|
|
<samp><span class="file">~/Buildbot/master/gnomovision</span></samp> or
|
|
<samp><span class="file">~/Buildmasters/fooproject</span></samp>. If you are using a separate user
|
|
account, this might just be <samp><span class="file">~buildmaster/masters/fooproject</span></samp>.
|
|
|
|
<p>Once you've picked a directory, use the <samp><span class="command">buildbot
|
|
create-master</span></samp> command to create the directory and populate it with
|
|
startup files:
|
|
|
|
<pre class="example"> buildbot create-master <var>basedir</var>
|
|
</pre>
|
|
<p>You will need to create a configuration file (see <a href="#Configuration">Configuration</a>)
|
|
before starting the buildmaster. Most of the rest of this manual is
|
|
dedicated to explaining how to do this. A sample configuration file is
|
|
placed in the working directory, named <samp><span class="file">master.cfg.sample</span></samp>, which
|
|
can be copied to <samp><span class="file">master.cfg</span></samp> and edited to suit your purposes.
|
|
|
|
<p>(Internal details: This command creates a file named
|
|
<samp><span class="file">buildbot.tac</span></samp> that contains all the state necessary to create
|
|
the buildmaster. Twisted has a tool called <code>twistd</code> which can use
|
|
this .tac file to create and launch a buildmaster instance. twistd
|
|
takes care of logging and daemonization (running the program in the
|
|
background). <samp><span class="file">/usr/bin/buildbot</span></samp> is a front end which runs twistd
|
|
for you.)
|
|
|
|
<p>In addition to <samp><span class="file">buildbot.tac</span></samp>, a small <samp><span class="file">Makefile.sample</span></samp> is
|
|
installed. This can be used as the basis for customized daemon startup,
|
|
See <a href="#Launching-the-daemons">Launching the daemons</a>.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Creating-a-buildslave"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Launching-the-daemons">Launching the daemons</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Creating-a-buildmaster">Creating a buildmaster</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Installation">Installation</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">2.4 Creating a buildslave</h3>
|
|
|
|
<p>Typically, you will be adding a buildslave to an existing buildmaster,
|
|
to provide additional architecture coverage. The buildbot
|
|
administrator will give you several pieces of information necessary to
|
|
connect to the buildmaster. You should also be somewhat familiar with
|
|
the project being tested, so you can troubleshoot build problems
|
|
locally.
|
|
|
|
<p>The buildbot exists to make sure that the project's stated “how to
|
|
build it” process actually works. To this end, the buildslave should
|
|
run in an environment just like that of your regular developers.
|
|
Typically the project build process is documented somewhere
|
|
(<samp><span class="file">README</span></samp>, <samp><span class="file">INSTALL</span></samp>, etc), in a document that should
|
|
mention all library dependencies and contain a basic set of build
|
|
instructions. This document will be useful as you configure the host
|
|
and account in which the buildslave runs.
|
|
|
|
<p>Here's a good checklist for setting up a buildslave:
|
|
|
|
<ol type=1 start=1>
|
|
<li>Set up the account
|
|
|
|
<p>It is recommended (although not mandatory) to set up a separate user
|
|
account for the buildslave. This account is frequently named
|
|
<code>buildbot</code> or <code>buildslave</code>. This serves to isolate your
|
|
personal working environment from that of the slave's, and helps to
|
|
minimize the security threat posed by letting possibly-unknown
|
|
contributors run arbitrary code on your system. The account should
|
|
have a minimum of fancy init scripts.
|
|
|
|
<li>Install the buildbot code
|
|
|
|
<p>Follow the instructions given earlier (see <a href="#Installing-the-code">Installing the code</a>).
|
|
If you use a separate buildslave account, and you didn't install the
|
|
buildbot code to a shared location, then you will need to install it
|
|
with <code>--home=~</code> for each account that needs it.
|
|
|
|
<li>Set up the host
|
|
|
|
<p>Make sure the host can actually reach the buildmaster. Usually the
|
|
buildmaster is running a status webserver on the same machine, so
|
|
simply point your web browser at it and see if you can get there.
|
|
Install whatever additional packages or libraries the project's
|
|
INSTALL document advises. (or not: if your buildslave is supposed to
|
|
make sure that building without optional libraries still works, then
|
|
don't install those libraries).
|
|
|
|
<p>Again, these libraries don't necessarily have to be installed to a
|
|
site-wide shared location, but they must be available to your build
|
|
process. Accomplishing this is usually very specific to the build
|
|
process, so installing them to <samp><span class="file">/usr</span></samp> or <samp><span class="file">/usr/local</span></samp> is
|
|
usually the best approach.
|
|
|
|
<li>Test the build process
|
|
|
|
<p>Follow the instructions in the INSTALL document, in the buildslave's
|
|
account. Perform a full CVS (or whatever) checkout, configure, make,
|
|
run tests, etc. Confirm that the build works without manual fussing.
|
|
If it doesn't work when you do it by hand, it will be unlikely to work
|
|
when the buildbot attempts to do it in an automated fashion.
|
|
|
|
<li>Choose a base directory
|
|
|
|
<p>This should be somewhere in the buildslave's account, typically named
|
|
after the project which is being tested. The buildslave will not touch
|
|
any file outside of this directory. Something like <samp><span class="file">~/Buildbot</span></samp>
|
|
or <samp><span class="file">~/Buildslaves/fooproject</span></samp> is appropriate.
|
|
|
|
<li>Get the buildmaster host/port, botname, and password
|
|
|
|
<p>When the buildbot admin configures the buildmaster to accept and use
|
|
your buildslave, they will provide you with the following pieces of
|
|
information:
|
|
|
|
<ul>
|
|
<li>your buildslave's name
|
|
<li>the password assigned to your buildslave
|
|
<li>the hostname and port number of the buildmaster, i.e. buildbot.example.org:8007
|
|
</ul>
|
|
|
|
<li>Create the buildslave
|
|
|
|
<p>Now run the 'buildbot' command as follows:
|
|
|
|
<pre class="example"> buildbot create-slave <var>BASEDIR</var> <var>MASTERHOST</var>:<var>PORT</var> <var>SLAVENAME</var> <var>PASSWORD</var>
|
|
</pre>
|
|
<p>This will create the base directory and a collection of files inside,
|
|
including the <samp><span class="file">buildbot.tac</span></samp> file that contains all the
|
|
information you passed to the <code>buildbot</code> command.
|
|
|
|
<li>Fill in the hostinfo files
|
|
|
|
<p>When it first connects, the buildslave will send a few files up to the
|
|
buildmaster which describe the host that it is running on. These files
|
|
are presented on the web status display so that developers have more
|
|
information to reproduce any test failures that are witnessed by the
|
|
buildbot. There are sample files in the <samp><span class="file">info</span></samp> subdirectory of
|
|
the buildbot's base directory. You should edit these to correctly
|
|
describe you and your host.
|
|
|
|
<p><samp><span class="file">BASEDIR/info/admin</span></samp> should contain your name and email address.
|
|
This is the “buildslave admin address”, and will be visible from the
|
|
build status page (so you may wish to munge it a bit if
|
|
address-harvesting spambots are a concern).
|
|
|
|
<p><samp><span class="file">BASEDIR/info/host</span></samp> should be filled with a brief description of
|
|
the host: OS, version, memory size, CPU speed, versions of relevant
|
|
libraries installed, and finally the version of the buildbot code
|
|
which is running the buildslave.
|
|
|
|
<p>If you run many buildslaves, you may want to create a single
|
|
<samp><span class="file">~buildslave/info</span></samp> file and share it among all the buildslaves
|
|
with symlinks.
|
|
|
|
</ol>
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Buildslave-Options">Buildslave Options</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Buildslave-Options"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Creating-a-buildslave">Creating a buildslave</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Creating-a-buildslave">Creating a buildslave</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">2.4.1 Buildslave Options</h4>
|
|
|
|
<p>There are a handful of options you might want to use when creating the
|
|
buildslave with the <samp><span class="command">buildbot create-slave <options> DIR <params></span></samp>
|
|
command. You can type <samp><span class="command">buildbot create-slave --help</span></samp> for a summary.
|
|
To use these, just include them on the <samp><span class="command">buildbot create-slave</span></samp>
|
|
command line, like this:
|
|
|
|
<pre class="example"> buildbot create-slave --umask=022 ~/buildslave buildmaster.example.org:42012 myslavename mypasswd
|
|
</pre>
|
|
<dl>
|
|
<dt><code>--usepty</code><dd>This is a boolean flag that tells the buildslave whether to launch
|
|
child processes in a PTY (the default) or with regular pipes. The
|
|
advantage of using a PTY is that “grandchild” processes are more
|
|
likely to be cleaned up if the build is interrupted or times out
|
|
(since it enables the use of a “process group” in which all child
|
|
processes will be placed). The disadvantages: some forms of Unix have
|
|
problems with PTYs, some of your unit tests may behave differently
|
|
when run under a PTY (generally those which check to see if they are
|
|
being run interactively), and PTYs will merge the stdout and stderr
|
|
streams into a single output stream (which means the red-vs-black
|
|
coloring in the logfiles will be lost). If you encounter problems, you
|
|
can add <code>--usepty=0</code> to disable the use of PTYs. Note that
|
|
windows buildslaves never use PTYs.
|
|
|
|
<br><dt><code>--umask</code><dd>This is a string (generally an octal representation of an integer)
|
|
which will cause the buildslave process' “umask” value to be set
|
|
shortly after initialization. The “twistd” daemonization utility
|
|
forces the umask to 077 at startup (which means that all files created
|
|
by the buildslave or its child processes will be unreadable by any
|
|
user other than the buildslave account). If you want build products to
|
|
be readable by other accounts, you can add <code>--umask=022</code> to tell
|
|
the buildslave to fix the umask after twistd clobbers it. If you want
|
|
build products to be <em>writable</em> by other accounts too, use
|
|
<code>--umask=000</code>, but this is likely to be a security problem.
|
|
|
|
<br><dt><code>--keepalive</code><dd>This is a number that indicates how frequently “keepalive” messages
|
|
should be sent from the buildslave to the buildmaster, expressed in
|
|
seconds. The default (600) causes a message to be sent to the
|
|
buildmaster at least once every 10 minutes. To set this to a lower
|
|
value, use e.g. <code>--keepalive=120</code>.
|
|
|
|
<p>If the buildslave is behind a NAT box or stateful firewall, these
|
|
messages may help to keep the connection alive: some NAT boxes tend to
|
|
forget about a connection if it has not been used in a while. When
|
|
this happens, the buildmaster will think that the buildslave has
|
|
disappeared, and builds will time out. Meanwhile the buildslave will
|
|
not realize than anything is wrong.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Launching-the-daemons"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Logfiles">Logfiles</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Creating-a-buildslave">Creating a buildslave</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Installation">Installation</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">2.5 Launching the daemons</h3>
|
|
|
|
<p>Both the buildmaster and the buildslave run as daemon programs. To
|
|
launch them, pass the working directory to the <code>buildbot</code>
|
|
command:
|
|
|
|
<pre class="example"> buildbot start <var>BASEDIR</var>
|
|
</pre>
|
|
<p>This command will start the daemon and then return, so normally it
|
|
will not produce any output. To verify that the programs are indeed
|
|
running, look for a pair of files named <samp><span class="file">twistd.log</span></samp> and
|
|
<samp><span class="file">twistd.pid</span></samp> that should be created in the working directory.
|
|
<samp><span class="file">twistd.pid</span></samp> contains the process ID of the newly-spawned daemon.
|
|
|
|
<p>When the buildslave connects to the buildmaster, new directories will
|
|
start appearing in its base directory. The buildmaster tells the slave
|
|
to create a directory for each Builder which will be using that slave.
|
|
All build operations are performed within these directories: CVS
|
|
checkouts, compiles, and tests.
|
|
|
|
<p>Once you get everything running, you will want to arrange for the
|
|
buildbot daemons to be started at boot time. One way is to use
|
|
<code>cron</code>, by putting them in a @reboot crontab entry<a rel="footnote" href="#fn-1" name="fnd-1"><sup>1</sup></a>:
|
|
|
|
<pre class="example"> @reboot buildbot start <var>BASEDIR</var>
|
|
</pre>
|
|
<p>When you run <samp><span class="command">crontab</span></samp> to set this up, remember to do it as
|
|
the buildmaster or buildslave account! If you add this to your crontab
|
|
when running as your regular account (or worse yet, root), then the
|
|
daemon will run as the wrong user, quite possibly as one with more
|
|
authority than you intended to provide.
|
|
|
|
<p>It is important to remember that the environment provided to cron jobs
|
|
and init scripts can be quite different that your normal runtime.
|
|
There may be fewer environment variables specified, and the PATH may
|
|
be shorter than usual. It is a good idea to test out this method of
|
|
launching the buildslave by using a cron job with a time in the near
|
|
future, with the same command, and then check <samp><span class="file">twistd.log</span></samp> to
|
|
make sure the slave actually started correctly. Common problems here
|
|
are for <samp><span class="file">/usr/local</span></samp> or <samp><span class="file">~/bin</span></samp> to not be on your
|
|
<code>PATH</code>, or for <code>PYTHONPATH</code> to not be set correctly.
|
|
Sometimes <code>HOME</code> is messed up too.
|
|
|
|
<p>To modify the way the daemons are started (perhaps you want to set
|
|
some environment variables first, or perform some cleanup each time),
|
|
you can create a file named <samp><span class="file">Makefile.buildbot</span></samp> in the base
|
|
directory. When the <samp><span class="file">buildbot</span></samp> front-end tool is told to
|
|
<samp><span class="command">start</span></samp> the daemon, and it sees this file (and
|
|
<samp><span class="file">/usr/bin/make</span></samp> exists), it will do <samp><span class="command">make -f
|
|
Makefile.buildbot start</span></samp> instead of its usual action (which involves
|
|
running <samp><span class="command">twistd</span></samp>). When the buildmaster or buildslave is
|
|
installed, a <samp><span class="file">Makefile.sample</span></samp> is created which implements the
|
|
same behavior as the the <samp><span class="file">buildbot</span></samp> tool uses, so if you want to
|
|
customize the process, just copy <samp><span class="file">Makefile.sample</span></samp> to
|
|
<samp><span class="file">Makefile.buildbot</span></samp> and edit it as necessary.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Logfiles"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Shutdown">Shutdown</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Launching-the-daemons">Launching the daemons</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Installation">Installation</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">2.6 Logfiles</h3>
|
|
|
|
<p><a name="index-logfiles-4"></a>
|
|
While a buildbot daemon runs, it emits text to a logfile, named
|
|
<samp><span class="file">twistd.log</span></samp>. A command like <code>tail -f twistd.log</code> is useful
|
|
to watch the command output as it runs.
|
|
|
|
<p>The buildmaster will announce any errors with its configuration file
|
|
in the logfile, so it is a good idea to look at the log at startup
|
|
time to check for any problems. Most buildmaster activities will cause
|
|
lines to be added to the log.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Shutdown"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Maintenance">Maintenance</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Logfiles">Logfiles</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Installation">Installation</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">2.7 Shutdown</h3>
|
|
|
|
<p>To stop a buildmaster or buildslave manually, use:
|
|
|
|
<pre class="example"> buildbot stop <var>BASEDIR</var>
|
|
</pre>
|
|
<p>This simply looks for the <samp><span class="file">twistd.pid</span></samp> file and kills whatever
|
|
process is identified within.
|
|
|
|
<p>At system shutdown, all processes are sent a <code>SIGKILL</code>. The
|
|
buildmaster and buildslave will respond to this by shutting down
|
|
normally.
|
|
|
|
<p>The buildmaster will respond to a <code>SIGHUP</code> by re-reading its
|
|
config file. The following shortcut is available:
|
|
|
|
<pre class="example"> buildbot reconfig <var>BASEDIR</var>
|
|
</pre>
|
|
<p>When you update the Buildbot code to a new release, you will need to
|
|
restart the buildmaster and/or buildslave before it can take advantage
|
|
of the new code. You can do a <code>buildbot stop </code><var>BASEDIR</var> and
|
|
<code>buildbot start </code><var>BASEDIR</var> in quick succession, or you can
|
|
use the <code>restart</code> shortcut, which does both steps for you:
|
|
|
|
<pre class="example"> buildbot restart <var>BASEDIR</var>
|
|
</pre>
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Maintenance"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Troubleshooting">Troubleshooting</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Shutdown">Shutdown</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Installation">Installation</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">2.8 Maintenance</h3>
|
|
|
|
<p>It is a good idea to check the buildmaster's status page every once in
|
|
a while, to see if your buildslave is still online. Eventually the
|
|
buildbot will probably be enhanced to send you email (via the
|
|
<samp><span class="file">info/admin</span></samp> email address) when the slave has been offline for
|
|
more than a few hours.
|
|
|
|
<p>If you find you can no longer provide a buildslave to the project, please
|
|
let the project admins know, so they can put out a call for a
|
|
replacement.
|
|
|
|
<p>The Buildbot records status and logs output continually, each time a
|
|
build is performed. The status tends to be small, but the build logs
|
|
can become quite large. Each build and log are recorded in a separate
|
|
file, arranged hierarchically under the buildmaster's base directory.
|
|
To prevent these files from growing without bound, you should
|
|
periodically delete old build logs. A simple cron job to delete
|
|
anything older than, say, two weeks should do the job. The only trick
|
|
is to leave the <samp><span class="file">buildbot.tac</span></samp> and other support files alone, for
|
|
which find's <code>-mindepth</code> argument helps skip everything in the
|
|
top directory. You can use something like the following:
|
|
|
|
<pre class="example"> @weekly cd BASEDIR && find . -mindepth 2 -type f -mtime +14 -exec rm {} \;
|
|
@weekly cd BASEDIR && find twistd.log* -mtime +14 -exec rm {} \;
|
|
</pre>
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Troubleshooting"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Maintenance">Maintenance</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Installation">Installation</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">2.9 Troubleshooting</h3>
|
|
|
|
<p>Here are a few hints on diagnosing common problems.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Starting-the-buildslave">Starting the buildslave</a>
|
|
<li><a accesskey="2" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>
|
|
<li><a accesskey="3" href="#Forcing-Builds">Forcing Builds</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Starting-the-buildslave"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Troubleshooting">Troubleshooting</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">2.9.1 Starting the buildslave</h4>
|
|
|
|
<p>Cron jobs are typically run with a minimal shell (<samp><span class="file">/bin/sh</span></samp>, not
|
|
<samp><span class="file">/bin/bash</span></samp>), and tilde expansion is not always performed in such
|
|
commands. You may want to use explicit paths, because the <code>PATH</code>
|
|
is usually quite short and doesn't include anything set by your
|
|
shell's startup scripts (<samp><span class="file">.profile</span></samp>, <samp><span class="file">.bashrc</span></samp>, etc). If
|
|
you've installed buildbot (or other python libraries) to an unusual
|
|
location, you may need to add a <code>PYTHONPATH</code> specification (note
|
|
that python will do tilde-expansion on <code>PYTHONPATH</code> elements by
|
|
itself). Sometimes it is safer to fully-specify everything:
|
|
|
|
<pre class="example"> @reboot PYTHONPATH=~/lib/python /usr/local/bin/buildbot start /usr/home/buildbot/basedir
|
|
</pre>
|
|
<p>Take the time to get the @reboot job set up. Otherwise, things will work
|
|
fine for a while, but the first power outage or system reboot you have will
|
|
stop the buildslave with nothing but the cries of sorrowful developers to
|
|
remind you that it has gone away.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Connecting-to-the-buildmaster"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Forcing-Builds">Forcing Builds</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Starting-the-buildslave">Starting the buildslave</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">2.9.2 Connecting to the buildmaster</h4>
|
|
|
|
<p>If the buildslave cannot connect to the buildmaster, the reason should
|
|
be described in the <samp><span class="file">twistd.log</span></samp> logfile. Some common problems
|
|
are an incorrect master hostname or port number, or a mistyped bot
|
|
name or password. If the buildslave loses the connection to the
|
|
master, it is supposed to attempt to reconnect with an
|
|
exponentially-increasing backoff. Each attempt (and the time of the
|
|
next attempt) will be logged. If you get impatient, just manually stop
|
|
and re-start the buildslave.
|
|
|
|
<p>When the buildmaster is restarted, all slaves will be disconnected,
|
|
and will attempt to reconnect as usual. The reconnect time will depend
|
|
upon how long the buildmaster is offline (i.e. how far up the
|
|
exponential backoff curve the slaves have travelled). Again,
|
|
<code>buildbot stop </code><var>BASEDIR</var><code>; buildbot start </code><var>BASEDIR</var> will
|
|
speed up the process.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Forcing-Builds"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">2.9.3 Forcing Builds</h4>
|
|
|
|
<p>From the buildmaster's main status web page, you can force a build to
|
|
be run on your build slave. Figure out which column is for a builder
|
|
that runs on your slave, click on that builder's name, and the page
|
|
that comes up will have a “Force Build” button. Fill in the form,
|
|
hit the button, and a moment later you should see your slave's
|
|
<samp><span class="file">twistd.log</span></samp> filling with commands being run. Using <code>pstree</code>
|
|
or <code>top</code> should also reveal the cvs/make/gcc/etc processes being
|
|
run by the buildslave. Note that the same web page should also show
|
|
the <samp><span class="file">admin</span></samp> and <samp><span class="file">host</span></samp> information files that you configured
|
|
earlier.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Concepts"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Configuration">Configuration</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Installation">Installation</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="chapter">3 Concepts</h2>
|
|
|
|
<p>This chapter defines some of the basic concepts that the Buildbot
|
|
uses. You'll need to understand how the Buildbot sees the world to
|
|
configure it properly.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Version-Control-Systems">Version Control Systems</a>
|
|
<li><a accesskey="2" href="#Schedulers">Schedulers</a>
|
|
<li><a accesskey="3" href="#BuildSet">BuildSet</a>
|
|
<li><a accesskey="4" href="#BuildRequest">BuildRequest</a>
|
|
<li><a accesskey="5" href="#Builder">Builder</a>
|
|
<li><a accesskey="6" href="#Users">Users</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Version-Control-Systems"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Schedulers">Schedulers</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Concepts">Concepts</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">3.1 Version Control Systems</h3>
|
|
|
|
<p><a name="index-Version-Control-5"></a>
|
|
These source trees come from a Version Control System of some kind.
|
|
CVS and Subversion are two popular ones, but the Buildbot supports
|
|
others. All VC systems have some notion of an upstream
|
|
<code>repository</code> which acts as a server<a rel="footnote" href="#fn-2" name="fnd-2"><sup>2</sup></a>, from which clients
|
|
can obtain source trees according to various parameters. The VC
|
|
repository provides source trees of various projects, for different
|
|
branches, and from various points in time. The first thing we have to
|
|
do is to specify which source tree we want to get.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Generalizing-VC-Systems">Generalizing VC Systems</a>
|
|
<li><a accesskey="2" href="#Source-Tree-Specifications">Source Tree Specifications</a>
|
|
<li><a accesskey="3" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>
|
|
<li><a accesskey="4" href="#Attributes-of-Changes">Attributes of Changes</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Generalizing-VC-Systems"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Source-Tree-Specifications">Source Tree Specifications</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Version-Control-Systems">Version Control Systems</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">3.1.1 Generalizing VC Systems</h4>
|
|
|
|
<p>For the purposes of the Buildbot, we will try to generalize all VC
|
|
systems as having repositories that each provide sources for a variety
|
|
of projects. Each project is defined as a directory tree with source
|
|
files. The individual files may each have revisions, but we ignore
|
|
that and treat the project as a whole as having a set of revisions.
|
|
Each time someone commits a change to the project, a new revision
|
|
becomes available. These revisions can be described by a tuple with
|
|
two items: the first is a branch tag, and the second is some kind of
|
|
timestamp or revision stamp. Complex projects may have multiple branch
|
|
tags, but there is always a default branch. The timestamp may be an
|
|
actual timestamp (such as the -D option to CVS), or it may be a
|
|
monotonically-increasing transaction number (such as the change number
|
|
used by SVN and P4, or the revision number used by Arch, or a labeled
|
|
tag used in CVS)<a rel="footnote" href="#fn-3" name="fnd-3"><sup>3</sup></a>. The SHA1 revision ID used by Monotone and
|
|
Mercurial is also a kind of revision stamp, in that it specifies a
|
|
unique copy of the source tree, as does a Darcs “context” file.
|
|
|
|
<p>When we aren't intending to make any changes to the sources we check out
|
|
(at least not any that need to be committed back upstream), there are two
|
|
basic ways to use a VC system:
|
|
|
|
<ul>
|
|
<li>Retrieve a specific set of source revisions: some tag or key is used
|
|
to index this set, which is fixed and cannot be changed by subsequent
|
|
developers committing new changes to the tree. Releases are built from
|
|
tagged revisions like this, so that they can be rebuilt again later
|
|
(probably with controlled modifications).
|
|
<li>Retrieve the latest sources along a specific branch: some tag is used
|
|
to indicate which branch is to be used, but within that constraint we want
|
|
to get the latest revisions.
|
|
</ul>
|
|
|
|
<p>Build personnel or CM staff typically use the first approach: the
|
|
build that results is (ideally) completely specified by the two
|
|
parameters given to the VC system: repository and revision tag. This
|
|
gives QA and end-users something concrete to point at when reporting
|
|
bugs. Release engineers are also reportedly fond of shipping code that
|
|
can be traced back to a concise revision tag of some sort.
|
|
|
|
<p>Developers are more likely to use the second approach: each morning
|
|
the developer does an update to pull in the changes committed by the
|
|
team over the last day. These builds are not easy to fully specify: it
|
|
depends upon exactly when you did a checkout, and upon what local
|
|
changes the developer has in their tree. Developers do not normally
|
|
tag each build they produce, because there is usually significant
|
|
overhead involved in creating these tags. Recreating the trees used by
|
|
one of these builds can be a challenge. Some VC systems may provide
|
|
implicit tags (like a revision number), while others may allow the use
|
|
of timestamps to mean “the state of the tree at time X” as opposed
|
|
to a tree-state that has been explicitly marked.
|
|
|
|
<p>The Buildbot is designed to help developers, so it usually works in
|
|
terms of <em>the latest</em> sources as opposed to specific tagged
|
|
revisions. However, it would really prefer to build from reproducible
|
|
source trees, so implicit revisions are used whenever possible.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Source-Tree-Specifications"></a>
|
|
Next: <a rel="next" accesskey="n" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Generalizing-VC-Systems">Generalizing VC Systems</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">3.1.2 Source Tree Specifications</h4>
|
|
|
|
<p>So for the Buildbot's purposes we treat each VC system as a server
|
|
which can take a list of specifications as input and produce a source
|
|
tree as output. Some of these specifications are static: they are
|
|
attributes of the builder and do not change over time. Others are more
|
|
variable: each build will have a different value. The repository is
|
|
changed over time by a sequence of Changes, each of which represents a
|
|
single developer making changes to some set of files. These Changes
|
|
are cumulative<a rel="footnote" href="#fn-4" name="fnd-4"><sup>4</sup></a>.
|
|
|
|
<p>For normal builds, the Buildbot wants to get well-defined source trees
|
|
that contain specific Changes, and exclude other Changes that may have
|
|
occurred after the desired ones. We assume that the Changes arrive at
|
|
the buildbot (through one of the mechanisms described in see <a href="#Change-Sources">Change Sources</a>) in the same order in which they are committed to the
|
|
repository. The Buildbot waits for the tree to become “stable”
|
|
before initiating a build, for two reasons. The first is that
|
|
developers frequently make multiple related commits in quick
|
|
succession, even when the VC system provides ways to make atomic
|
|
transactions involving multiple files at the same time. Running a
|
|
build in the middle of these sets of changes would use an inconsistent
|
|
set of source files, and is likely to fail (and is certain to be less
|
|
useful than a build which uses the full set of changes). The
|
|
tree-stable-timer is intended to avoid these useless builds that
|
|
include some of the developer's changes but not all. The second reason
|
|
is that some VC systems (i.e. CVS) do not provide repository-wide
|
|
transaction numbers, so that timestamps are the only way to refer to
|
|
a specific repository state. These timestamps may be somewhat
|
|
ambiguous, due to processing and notification delays. By waiting until
|
|
the tree has been stable for, say, 10 minutes, we can choose a
|
|
timestamp from the middle of that period to use for our source
|
|
checkout, and then be reasonably sure that any clock-skew errors will
|
|
not cause the build to be performed on an inconsistent set of source
|
|
files.
|
|
|
|
<p>The Schedulers always use the tree-stable-timer, with a timeout that
|
|
is configured to reflect a reasonable tradeoff between build latency
|
|
and change frequency. When the VC system provides coherent
|
|
repository-wide revision markers (such as Subversion's revision
|
|
numbers, or in fact anything other than CVS's timestamps), the
|
|
resulting Build is simply performed against a source tree defined by
|
|
that revision marker. When the VC system does not provide this, a
|
|
timestamp from the middle of the tree-stable period is used to
|
|
generate the source tree<a rel="footnote" href="#fn-5" name="fnd-5"><sup>5</sup></a>.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="How-Different-VC-Systems-Specify-Sources"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Attributes-of-Changes">Attributes of Changes</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Source-Tree-Specifications">Source Tree Specifications</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">3.1.3 How Different VC Systems Specify Sources</h4>
|
|
|
|
<p>For CVS, the static specifications are <code>repository</code> and
|
|
<code>module</code>. In addition to those, each build uses a timestamp (or
|
|
omits the timestamp to mean <code>the latest</code>) and <code>branch tag</code>
|
|
(which defaults to HEAD). These parameters collectively specify a set
|
|
of sources from which a build may be performed.
|
|
|
|
<p><a href="http://subversion.tigris.org">Subversion</a> combines the
|
|
repository, module, and branch into a single <code>Subversion URL</code>
|
|
parameter. Within that scope, source checkouts can be specified by a
|
|
numeric <code>revision number</code> (a repository-wide
|
|
monotonically-increasing marker, such that each transaction that
|
|
changes the repository is indexed by a different revision number), or
|
|
a revision timestamp. When branches are used, the repository and
|
|
module form a static <code>baseURL</code>, while each build has a
|
|
<code>revision number</code> and a <code>branch</code> (which defaults to a
|
|
statically-specified <code>defaultBranch</code>). The <code>baseURL</code> and
|
|
<code>branch</code> are simply concatenated together to derive the
|
|
<code>svnurl</code> to use for the checkout.
|
|
|
|
<p><a href="http://www.perforce.com/">Perforce</a> is similar. The server
|
|
is specified through a <code>P4PORT</code> parameter. Module and branch
|
|
are specified in a single depot path, and revisions are
|
|
depot-wide. When branches are used, the <code>p4base</code> and
|
|
<code>defaultBranch</code> are concatenated together to produce the depot
|
|
path.
|
|
|
|
<p><a href="http://wiki.gnuarch.org/">Arch</a> and
|
|
<a href="http://bazaar.canonical.com/">Bazaar</a> specify a repository by
|
|
URL, as well as a <code>version</code> which is kind of like a branch name.
|
|
Arch uses the word <code>archive</code> to represent the repository. Arch
|
|
lets you push changes from one archive to another, removing the strict
|
|
centralization required by CVS and SVN. It retains the distinction
|
|
between repository and working directory that most other VC systems
|
|
use. For complex multi-module directory structures, Arch has a
|
|
built-in <code>build config</code> layer with which the checkout process has
|
|
two steps. First, an initial bootstrap checkout is performed to
|
|
retrieve a set of build-config files. Second, one of these files is
|
|
used to figure out which archives/modules should be used to populate
|
|
subdirectories of the initial checkout.
|
|
|
|
<p>Builders which use Arch and Bazaar therefore have a static archive
|
|
<code>url</code>, and a default “branch” (which is a string that specifies
|
|
a complete category–branch–version triple). Each build can have its
|
|
own branch (the category–branch–version string) to override the
|
|
default, as well as a revision number (which is turned into a
|
|
–patch-NN suffix when performing the checkout).
|
|
|
|
<p><a href="http://abridgegame.org/darcs/">Darcs</a> doesn't really have the
|
|
notion of a single master repository. Nor does it really have
|
|
branches. In Darcs, each working directory is also a repository, and
|
|
there are operations to push and pull patches from one of these
|
|
<code>repositories</code> to another. For the Buildbot's purposes, all you
|
|
need to do is specify the URL of a repository that you want to build
|
|
from. The build slave will then pull the latest patches from that
|
|
repository and build them. Multiple branches are implemented by using
|
|
multiple repositories (possibly living on the same server).
|
|
|
|
<p>Builders which use Darcs therefore have a static <code>repourl</code> which
|
|
specifies the location of the repository. If branches are being used,
|
|
the source Step is instead configured with a <code>baseURL</code> and a
|
|
<code>defaultBranch</code>, and the two strings are simply concatenated
|
|
together to obtain the repository's URL. Each build then has a
|
|
specific branch which replaces <code>defaultBranch</code>, or just uses the
|
|
default one. Instead of a revision number, each build can have a
|
|
“context”, which is a string that records all the patches that are
|
|
present in a given tree (this is the output of <samp><span class="command">darcs changes
|
|
--context</span></samp>, and is considerably less concise than, e.g. Subversion's
|
|
revision number, but the patch-reordering flexibility of Darcs makes
|
|
it impossible to provide a shorter useful specification).
|
|
|
|
<p><a href="http://selenic.com/mercurial">Mercurial</a> is like Darcs, in that
|
|
each branch is stored in a separate repository. The <code>repourl</code>,
|
|
<code>baseURL</code>, and <code>defaultBranch</code> arguments are all handled the
|
|
same way as with Darcs. The “revision”, however, is the hash
|
|
identifier returned by <samp><span class="command">hg identify</span></samp>.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Attributes-of-Changes"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">3.1.4 Attributes of Changes</h4>
|
|
|
|
<h3 class="heading">Who</h3>
|
|
|
|
<p>Each Change has a <code>who</code> attribute, which specifies which
|
|
developer is responsible for the change. This is a string which comes
|
|
from a namespace controlled by the VC repository. Frequently this
|
|
means it is a username on the host which runs the repository, but not
|
|
all VC systems require this (Arch, for example, uses a fully-qualified
|
|
<code>Arch ID</code>, which looks like an email address, as does Darcs).
|
|
Each StatusNotifier will map the <code>who</code> attribute into something
|
|
appropriate for their particular means of communication: an email
|
|
address, an IRC handle, etc.
|
|
|
|
<h3 class="heading">Files</h3>
|
|
|
|
<p>It also has a list of <code>files</code>, which are just the tree-relative
|
|
filenames of any files that were added, deleted, or modified for this
|
|
Change. These filenames are used by the <code>isFileImportant</code>
|
|
function (in the Scheduler) to decide whether it is worth triggering a
|
|
new build or not, e.g. the function could use
|
|
<code>filename.endswith(".c")</code> to only run a build if a C file were
|
|
checked in. Certain BuildSteps can also use the list of changed files
|
|
to run a more targeted series of tests, e.g. the
|
|
<code>python_twisted.Trial</code> step can run just the unit tests that
|
|
provide coverage for the modified .py files instead of running the
|
|
full test suite.
|
|
|
|
<h3 class="heading">Comments</h3>
|
|
|
|
<p>The Change also has a <code>comments</code> attribute, which is a string
|
|
containing any checkin comments.
|
|
|
|
<h3 class="heading">Revision</h3>
|
|
|
|
<p>Each Change can have a <code>revision</code> attribute, which describes how
|
|
to get a tree with a specific state: a tree which includes this Change
|
|
(and all that came before it) but none that come after it. If this
|
|
information is unavailable, the <code>.revision</code> attribute will be
|
|
<code>None</code>. These revisions are provided by the ChangeSource, and
|
|
consumed by the <code>computeSourceRevision</code> method in the appropriate
|
|
<code>step.Source</code> class.
|
|
|
|
<dl>
|
|
<dt>`<samp><span class="samp">CVS</span></samp>'<dd><code>revision</code> is an int, seconds since the epoch
|
|
<br><dt>`<samp><span class="samp">SVN</span></samp>'<dd><code>revision</code> is an int, a transation number (r%d)
|
|
<br><dt>`<samp><span class="samp">Darcs</span></samp>'<dd><code>revision</code> is a large string, the output of <code>darcs changes --context</code>
|
|
<br><dt>`<samp><span class="samp">Mercurial</span></samp>'<dd><code>revision</code> is a short string (a hash ID), the output of <code>hg identify</code>
|
|
<br><dt>`<samp><span class="samp">Arch/Bazaar</span></samp>'<dd><code>revision</code> is the full revision ID (ending in –patch-%d)
|
|
<br><dt>`<samp><span class="samp">P4</span></samp>'<dd><code>revision</code> is an int, the transaction number
|
|
</dl>
|
|
|
|
<h3 class="heading">Branches</h3>
|
|
|
|
<p>The Change might also have a <code>branch</code> attribute. This indicates
|
|
that all of the Change's files are in the same named branch. The
|
|
Schedulers get to decide whether the branch should be built or not.
|
|
|
|
<p>For VC systems like CVS, Arch, and Monotone, the <code>branch</code> name is
|
|
unrelated to the filename. (that is, the branch name and the filename
|
|
inhabit unrelated namespaces). For SVN, branches are expressed as
|
|
subdirectories of the repository, so the file's “svnurl” is a
|
|
combination of some base URL, the branch name, and the filename within
|
|
the branch. (In a sense, the branch name and the filename inhabit the
|
|
same namespace). Darcs branches are subdirectories of a base URL just
|
|
like SVN. Mercurial branches are the same as Darcs.
|
|
|
|
<dl>
|
|
<dt>`<samp><span class="samp">CVS</span></samp>'<dd>branch='warner-newfeature', files=['src/foo.c']
|
|
<br><dt>`<samp><span class="samp">SVN</span></samp>'<dd>branch='branches/warner-newfeature', files=['src/foo.c']
|
|
<br><dt>`<samp><span class="samp">Darcs</span></samp>'<dd>branch='warner-newfeature', files=['src/foo.c']
|
|
<br><dt>`<samp><span class="samp">Mercurial</span></samp>'<dd>branch='warner-newfeature', files=['src/foo.c']
|
|
<br><dt>`<samp><span class="samp">Arch/Bazaar</span></samp>'<dd>branch='buildbot–usebranches–0', files=['buildbot/master.py']
|
|
</dl>
|
|
|
|
<h3 class="heading">Links</h3>
|
|
|
|
<!-- TODO: who is using 'links'? how is it being used? -->
|
|
<p>Finally, the Change might have a <code>links</code> list, which is intended
|
|
to provide a list of URLs to a <em>viewcvs</em>-style web page that
|
|
provides more detail for this Change, perhaps including the full file
|
|
diffs.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Schedulers"></a>
|
|
Next: <a rel="next" accesskey="n" href="#BuildSet">BuildSet</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Version-Control-Systems">Version Control Systems</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">3.2 Schedulers</h3>
|
|
|
|
<p><a name="index-Scheduler-6"></a>
|
|
Each Buildmaster has a set of <code>Scheduler</code> objects, each of which
|
|
gets a copy of every incoming Change. The Schedulers are responsible
|
|
for deciding when Builds should be run. Some Buildbot installations
|
|
might have a single Scheduler, while others may have several, each for
|
|
a different purpose.
|
|
|
|
<p>For example, a “quick” scheduler might exist to give immediate
|
|
feedback to developers, hoping to catch obvious problems in the code
|
|
that can be detected quickly. These typically do not run the full test
|
|
suite, nor do they run on a wide variety of platforms. They also
|
|
usually do a VC update rather than performing a brand-new checkout
|
|
each time. You could have a “quick” scheduler which used a 30 second
|
|
timeout, and feeds a single “quick” Builder that uses a VC
|
|
<code>mode='update'</code> setting.
|
|
|
|
<p>A separate “full” scheduler would run more comprehensive tests a
|
|
little while later, to catch more subtle problems. This scheduler
|
|
would have a longer tree-stable-timer, maybe 30 minutes, and would
|
|
feed multiple Builders (with a <code>mode=</code> of <code>'copy'</code>,
|
|
<code>'clobber'</code>, or <code>'export'</code>).
|
|
|
|
<p>The <code>tree-stable-timer</code> and <code>isFileImportant</code> decisions are
|
|
made by the Scheduler. Dependencies are also implemented here.
|
|
Periodic builds (those which are run every N seconds rather than after
|
|
new Changes arrive) are triggered by a special <code>Periodic</code>
|
|
Scheduler subclass. The default Scheduler class can also be told to
|
|
watch for specific branches, ignoring Changes on other branches. This
|
|
may be useful if you have a trunk and a few release branches which
|
|
should be tracked, but when you don't want to have the Buildbot pay
|
|
attention to several dozen private user branches.
|
|
|
|
<p>Some Schedulers may trigger builds for other reasons, other than
|
|
recent Changes. For example, a Scheduler subclass could connect to a
|
|
remote buildmaster and watch for builds of a library to succeed before
|
|
triggering a local build that uses that library.
|
|
|
|
<p>Each Scheduler creates and submits <code>BuildSet</code> objects to the
|
|
<code>BuildMaster</code>, which is then responsible for making sure the
|
|
individual <code>BuildRequests</code> are delivered to the target
|
|
<code>Builders</code>.
|
|
|
|
<p><code>Scheduler</code> instances are activated by placing them in the
|
|
<code>c['schedulers']</code> list in the buildmaster config file. Each
|
|
Scheduler has a unique name.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="BuildSet"></a>
|
|
Next: <a rel="next" accesskey="n" href="#BuildRequest">BuildRequest</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Schedulers">Schedulers</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">3.3 BuildSet</h3>
|
|
|
|
<p><a name="index-BuildSet-7"></a>
|
|
A <code>BuildSet</code> is the name given to a set of Builds that all
|
|
compile/test the same version of the tree on multiple Builders. In
|
|
general, all these component Builds will perform the same sequence of
|
|
Steps, using the same source code, but on different platforms or
|
|
against a different set of libraries.
|
|
|
|
<p>The <code>BuildSet</code> is tracked as a single unit, which fails if any of
|
|
the component Builds have failed, and therefore can succeed only if
|
|
<em>all</em> of the component Builds have succeeded. There are two kinds
|
|
of status notification messages that can be emitted for a BuildSet:
|
|
the <code>firstFailure</code> type (which fires as soon as we know the
|
|
BuildSet will fail), and the <code>Finished</code> type (which fires once
|
|
the BuildSet has completely finished, regardless of whether the
|
|
overall set passed or failed).
|
|
|
|
<p>A <code>BuildSet</code> is created with a <em>source stamp</em> tuple of
|
|
(branch, revision, changes, patch), some of which may be None, and a
|
|
list of Builders on which it is to be run. They are then given to the
|
|
BuildMaster, which is responsible for creating a separate
|
|
<code>BuildRequest</code> for each Builder.
|
|
|
|
<p>There are a couple of different likely values for the
|
|
<code>SourceStamp</code>:
|
|
|
|
<dl>
|
|
<dt><code>(revision=None, changes=[CHANGES], patch=None)</code><dd>This is a <code>SourceStamp</code> used when a series of Changes have
|
|
triggered a build. The VC step will attempt to check out a tree that
|
|
contains CHANGES (and any changes that occurred before CHANGES, but
|
|
not any that occurred after them).
|
|
|
|
<br><dt><code>(revision=None, changes=None, patch=None)</code><dd>This builds the most recent code on the default branch. This is the
|
|
sort of <code>SourceStamp</code> that would be used on a Build that was
|
|
triggered by a user request, or a Periodic scheduler. It is also
|
|
possible to configure the VC Source Step to always check out the
|
|
latest sources rather than paying attention to the Changes in the
|
|
SourceStamp, which will result in same behavior as this.
|
|
|
|
<br><dt><code>(branch=BRANCH, revision=None, changes=None, patch=None)</code><dd>This builds the most recent code on the given BRANCH. Again, this is
|
|
generally triggered by a user request or Periodic build.
|
|
|
|
<br><dt><code>(revision=REV, changes=None, patch=(LEVEL, DIFF))</code><dd>This checks out the tree at the given revision REV, then applies a
|
|
patch (using <code>diff -pLEVEL <DIFF</code>). The <a href="#try">try</a> feature uses
|
|
this kind of <code>SourceStamp</code>. If <code>patch</code> is None, the patching
|
|
step is bypassed.
|
|
|
|
</dl>
|
|
|
|
<p>The buildmaster is responsible for turning the <code>BuildSet</code> into a
|
|
set of <code>BuildRequest</code> objects and queueing them on the
|
|
appropriate Builders.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="BuildRequest"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Builder">Builder</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#BuildSet">BuildSet</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">3.4 BuildRequest</h3>
|
|
|
|
<p><a name="index-BuildRequest-8"></a>
|
|
A <code>BuildRequest</code> is a request to build a specific set of sources
|
|
on a single specific Builder. Each Builder runs the
|
|
<code>BuildRequest</code> as soon as it can (i.e. when an associated
|
|
buildslave becomes free).
|
|
|
|
<p>The <code>BuildRequest</code> contains the <code>SourceStamp</code> specification.
|
|
The actual process of running the build (the series of Steps that will
|
|
be executed) is implemented by the <code>Build</code> object. In this future
|
|
this might be changed, to have the <code>Build</code> define <em>what</em>
|
|
gets built, and a separate <code>BuildProcess</code> (provided by the
|
|
Builder) to define <em>how</em> it gets built.
|
|
|
|
<p>The <code>BuildRequest</code> may be mergeable with other compatible
|
|
<code>BuildRequest</code>s. Builds that are triggered by incoming Changes
|
|
will generally be mergeable. Builds that are triggered by user
|
|
requests are generally not, unless they are multiple requests to build
|
|
the <em>latest sources</em> of the same branch.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Builder"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Users">Users</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#BuildRequest">BuildRequest</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">3.5 Builder</h3>
|
|
|
|
<p><a name="index-Builder-9"></a>
|
|
The <code>Builder</code> is a long-lived object which controls all Builds of
|
|
a given type. Each one is created when the config file is first
|
|
parsed, and lives forever (or rather until it is removed from the
|
|
config file). It mediates the connections to the buildslaves that do
|
|
all the work, and is responsible for creating the <code>Build</code> objects
|
|
that decide <em>how</em> a build is performed (i.e., which steps are
|
|
executed in what order).
|
|
|
|
<p>Each <code>Builder</code> gets a unique name, and the path name of a
|
|
directory where it gets to do all its work (there is a
|
|
buildmaster-side directory for keeping status information, as well as
|
|
a buildslave-side directory where the actual checkout/compile/test
|
|
commands are executed). It also gets a <code>BuildFactory</code>, which is
|
|
responsible for creating new <code>Build</code> instances: because the
|
|
<code>Build</code> instance is what actually performs each build, choosing
|
|
the <code>BuildFactory</code> is the way to specify what happens each time a
|
|
build is done.
|
|
|
|
<p>Each <code>Builder</code> is associated with one of more <code>BuildSlaves</code>.
|
|
A <code>Builder</code> which is used to perform OS-X builds (as opposed to
|
|
Linux or Solaris builds) should naturally be associated with an
|
|
OS-X-based buildslave.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Users"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Builder">Builder</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">3.6 Users</h3>
|
|
|
|
<p><a name="index-Users-10"></a>
|
|
Buildbot has a somewhat limited awareness of <em>users</em>. It assumes
|
|
the world consists of a set of developers, each of whom can be
|
|
described by a couple of simple attributes. These developers make
|
|
changes to the source code, causing builds which may succeed or fail.
|
|
|
|
<p>Each developer is primarily known through the source control system. Each
|
|
Change object that arrives is tagged with a <code>who</code> field that
|
|
typically gives the account name (on the repository machine) of the user
|
|
responsible for that change. This string is the primary key by which the
|
|
User is known, and is displayed on the HTML status pages and in each Build's
|
|
“blamelist”.
|
|
|
|
<p>To do more with the User than just refer to them, this username needs to
|
|
be mapped into an address of some sort. The responsibility for this mapping
|
|
is left up to the status module which needs the address. The core code knows
|
|
nothing about email addresses or IRC nicknames, just user names.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Doing-Things-With-Users">Doing Things With Users</a>
|
|
<li><a accesskey="2" href="#Email-Addresses">Email Addresses</a>
|
|
<li><a accesskey="3" href="#IRC-Nicknames">IRC Nicknames</a>
|
|
<li><a accesskey="4" href="#Live-Status-Clients">Live Status Clients</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Doing-Things-With-Users"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Email-Addresses">Email Addresses</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Users">Users</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Users">Users</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">3.6.1 Doing Things With Users</h4>
|
|
|
|
<p>Each Change has a single User who is responsible for that Change. Most
|
|
Builds have a set of Changes: the Build represents the first time these
|
|
Changes have been built and tested by the Buildbot. The build has a
|
|
“blamelist” that consists of a simple union of the Users responsible
|
|
for all the Build's Changes.
|
|
|
|
<p>The Build provides (through the IBuildStatus interface) a list of Users
|
|
who are “involved” in the build. For now this is equal to the
|
|
blamelist, but in the future it will be expanded to include a “build
|
|
sheriff” (a person who is “on duty” at that time and responsible for
|
|
watching over all builds that occur during their shift), as well as
|
|
per-module owners who simply want to keep watch over their domain (chosen by
|
|
subdirectory or a regexp matched against the filenames pulled out of the
|
|
Changes). The Involved Users are those who probably have an interest in the
|
|
results of any given build.
|
|
|
|
<p>In the future, Buildbot will acquire the concept of “Problems”,
|
|
which last longer than builds and have beginnings and ends. For example, a
|
|
test case which passed in one build and then failed in the next is a
|
|
Problem. The Problem lasts until the test case starts passing again, at
|
|
which point the Problem is said to be “resolved”.
|
|
|
|
<p>If there appears to be a code change that went into the tree at the
|
|
same time as the test started failing, that Change is marked as being
|
|
resposible for the Problem, and the user who made the change is added
|
|
to the Problem's “Guilty” list. In addition to this user, there may
|
|
be others who share responsibility for the Problem (module owners,
|
|
sponsoring developers). In addition to the Responsible Users, there
|
|
may be a set of Interested Users, who take an interest in the fate of
|
|
the Problem.
|
|
|
|
<p>Problems therefore have sets of Users who may want to be kept aware of
|
|
the condition of the problem as it changes over time. If configured, the
|
|
Buildbot can pester everyone on the Responsible list with increasing
|
|
harshness until the problem is resolved, with the most harshness reserved
|
|
for the Guilty parties themselves. The Interested Users may merely be told
|
|
when the problem starts and stops, as they are not actually responsible for
|
|
fixing anything.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Email-Addresses"></a>
|
|
Next: <a rel="next" accesskey="n" href="#IRC-Nicknames">IRC Nicknames</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Doing-Things-With-Users">Doing Things With Users</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Users">Users</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">3.6.2 Email Addresses</h4>
|
|
|
|
<p>The <code>buildbot.status.mail.MailNotifier</code> class provides a
|
|
status target which can send email about the results of each build. It
|
|
accepts a static list of email addresses to which each message should be
|
|
delivered, but it can also be configured to send mail to the Build's
|
|
Interested Users. To do this, it needs a way to convert User names into
|
|
email addresses.
|
|
|
|
<p>For many VC systems, the User Name is actually an account name on the
|
|
system which hosts the repository. As such, turning the name into an
|
|
email address is a simple matter of appending
|
|
“@repositoryhost.com”. Some projects use other kinds of mappings
|
|
(for example the preferred email address may be at “project.org”
|
|
despite the repository host being named “cvs.project.org”), and some
|
|
VC systems have full separation between the concept of a user and that
|
|
of an account on the repository host (like Perforce). Some systems
|
|
(like Arch) put a full contact email address in every change.
|
|
|
|
<p>To convert these names to addresses, the MailNotifier uses an EmailLookup
|
|
object. This provides a .getAddress method which accepts a name and
|
|
(eventually) returns an address. The default <code>MailNotifier</code>
|
|
module provides an EmailLookup which simply appends a static string,
|
|
configurable when the notifier is created. To create more complex behaviors
|
|
(perhaps using an LDAP lookup, or using “finger” on a central host to
|
|
determine a preferred address for the developer), provide a different object
|
|
as the <code>lookup</code> argument.
|
|
|
|
<p>In the future, when the Problem mechanism has been set up, the Buildbot
|
|
will need to send mail to arbitrary Users. It will do this by locating a
|
|
MailNotifier-like object among all the buildmaster's status targets, and
|
|
asking it to send messages to various Users. This means the User-to-address
|
|
mapping only has to be set up once, in your MailNotifier, and every email
|
|
message the buildbot emits will take advantage of it.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="IRC-Nicknames"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Live-Status-Clients">Live Status Clients</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Email-Addresses">Email Addresses</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Users">Users</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">3.6.3 IRC Nicknames</h4>
|
|
|
|
<p>Like MailNotifier, the <code>buildbot.status.words.IRC</code> class
|
|
provides a status target which can announce the results of each build. It
|
|
also provides an interactive interface by responding to online queries
|
|
posted in the channel or sent as private messages.
|
|
|
|
<p>In the future, the buildbot can be configured map User names to IRC
|
|
nicknames, to watch for the recent presence of these nicknames, and to
|
|
deliver build status messages to the interested parties. Like
|
|
<code>MailNotifier</code> does for email addresses, the <code>IRC</code> object
|
|
will have an <code>IRCLookup</code> which is responsible for nicknames. The
|
|
mapping can be set up statically, or it can be updated by online users
|
|
themselves (by claiming a username with some kind of “buildbot: i am
|
|
user warner” commands).
|
|
|
|
<p>Once the mapping is established, the rest of the buildbot can ask the
|
|
<code>IRC</code> object to send messages to various users. It can report on
|
|
the likelihood that the user saw the given message (based upon how long the
|
|
user has been inactive on the channel), which might prompt the Problem
|
|
Hassler logic to send them an email message instead.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Live-Status-Clients"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#IRC-Nicknames">IRC Nicknames</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Users">Users</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">3.6.4 Live Status Clients</h4>
|
|
|
|
<p>The Buildbot also offers a PB-based status client interface which can
|
|
display real-time build status in a GUI panel on the developer's desktop.
|
|
This interface is normally anonymous, but it could be configured to let the
|
|
buildmaster know <em>which</em> developer is using the status client. The
|
|
status client could then be used as a message-delivery service, providing an
|
|
alternative way to deliver low-latency high-interruption messages to the
|
|
developer (like “hey, you broke the build”).
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Configuration"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Concepts">Concepts</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="chapter">4 Configuration</h2>
|
|
|
|
<p><a name="index-Configuration-11"></a>
|
|
The buildbot's behavior is defined by the “config file”, which
|
|
normally lives in the <samp><span class="file">master.cfg</span></samp> file in the buildmaster's base
|
|
directory (but this can be changed with an option to the
|
|
<code>buildbot create-master</code> command). This file completely specifies
|
|
which Builders are to be run, which slaves they should use, how
|
|
Changes should be tracked, and where the status information is to be
|
|
sent. The buildmaster's <samp><span class="file">buildbot.tac</span></samp> file names the base
|
|
directory; everything else comes from the config file.
|
|
|
|
<p>A sample config file was installed for you when you created the
|
|
buildmaster, but you will need to edit it before your buildbot will do
|
|
anything useful.
|
|
|
|
<p>This chapter gives an overview of the format of this file and the
|
|
various sections in it. You will need to read the later chapters to
|
|
understand how to fill in each section properly.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Config-File-Format">Config File Format</a>
|
|
<li><a accesskey="2" href="#Loading-the-Config-File">Loading the Config File</a>
|
|
<li><a accesskey="3" href="#Defining-the-Project">Defining the Project</a>
|
|
<li><a accesskey="4" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>
|
|
<li><a accesskey="5" href="#Setting-the-slaveport">Setting the slaveport</a>
|
|
<li><a accesskey="6" href="#Buildslave-Specifiers">Buildslave Specifiers</a>
|
|
<li><a accesskey="7" href="#Defining-Builders">Defining Builders</a>
|
|
<li><a accesskey="8" href="#Defining-Status-Targets">Defining Status Targets</a>
|
|
<li><a accesskey="9" href="#Debug-options">Debug options</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Config-File-Format"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Loading-the-Config-File">Loading the Config File</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Configuration">Configuration</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">4.1 Config File Format</h3>
|
|
|
|
<p>The config file is, fundamentally, just a piece of Python code which
|
|
defines a dictionary named <code>BuildmasterConfig</code>, with a number of
|
|
keys that are treated specially. You don't need to know Python to do
|
|
basic configuration, though, you can just copy the syntax of the
|
|
sample file. If you <em>are</em> comfortable writing Python code,
|
|
however, you can use all the power of a full programming language to
|
|
achieve more complicated configurations.
|
|
|
|
<p>The <code>BuildmasterConfig</code> name is the only one which matters: all
|
|
other names defined during the execution of the file are discarded.
|
|
When parsing the config file, the Buildmaster generally compares the
|
|
old configuration with the new one and performs the minimum set of
|
|
actions necessary to bring the buildbot up to date: Builders which are
|
|
not changed are left untouched, and Builders which are modified get to
|
|
keep their old event history.
|
|
|
|
<p>Basic Python syntax: comments start with a hash character (“#”),
|
|
tuples are defined with <code>(parenthesis, pairs)</code>, arrays are
|
|
defined with <code>[square, brackets]</code>, tuples and arrays are mostly
|
|
interchangeable. Dictionaries (data structures which map “keys” to
|
|
“values”) are defined with curly braces: <code>{'key1': 'value1',
|
|
'key2': 'value2'} </code>. Function calls (and object instantiation) can use
|
|
named parameters, like <code>w = html.Waterfall(http_port=8010)</code>.
|
|
|
|
<p>The config file starts with a series of <code>import</code> statements,
|
|
which make various kinds of Steps and Status targets available for
|
|
later use. The main <code>BuildmasterConfig</code> dictionary is created,
|
|
then it is populated with a variety of keys. These keys are broken
|
|
roughly into the following sections, each of which is documented in
|
|
the rest of this chapter:
|
|
|
|
<ul>
|
|
<li>Project Definitions
|
|
<li>Change Sources / Schedulers
|
|
<li>Slaveport
|
|
<li>Buildslave Configuration
|
|
<li>Builders / Interlocks
|
|
<li>Status Targets
|
|
<li>Debug options
|
|
</ul>
|
|
|
|
<p>The config file can use a few names which are placed into its namespace:
|
|
|
|
<dl>
|
|
<dt><code>basedir</code><dd>the base directory for the buildmaster. This string has not been
|
|
expanded, so it may start with a tilde. It needs to be expanded before
|
|
use. The config file is located in
|
|
<code>os.path.expanduser(os.path.join(basedir, 'master.cfg'))</code>
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Loading-the-Config-File"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Defining-the-Project">Defining the Project</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Config-File-Format">Config File Format</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">4.2 Loading the Config File</h3>
|
|
|
|
<p>The config file is only read at specific points in time. It is first
|
|
read when the buildmaster is launched. Once it is running, there are
|
|
various ways to ask it to reload the config file. If you are on the
|
|
system hosting the buildmaster, you can send a <code>SIGHUP</code> signal to
|
|
it: the <samp><span class="command">buildbot</span></samp> tool has a shortcut for this:
|
|
|
|
<pre class="example"> buildbot reconfig <var>BASEDIR</var>
|
|
</pre>
|
|
<p>This command will show you all of the lines from <samp><span class="file">twistd.log</span></samp>
|
|
that relate to the reconfiguration. If there are any problems during
|
|
the config-file reload, they will be displayed in these lines.
|
|
|
|
<p>The debug tool (<code>buildbot debugclient --master HOST:PORT</code>) has a
|
|
“Reload .cfg” button which will also trigger a reload. In the
|
|
future, there will be other ways to accomplish this step (probably a
|
|
password-protected button on the web page, as well as a privileged IRC
|
|
command).
|
|
|
|
<p>When reloading the config file, the buildmaster will endeavor to
|
|
change as little as possible about the running system. For example,
|
|
although old status targets may be shut down and new ones started up,
|
|
any status targets that were not changed since the last time the
|
|
config file was read will be left running and untouched. Likewise any
|
|
Builders which have not been changed will be left running. If a
|
|
Builder is modified (say, the build process is changed) while a Build
|
|
is currently running, that Build will keep running with the old
|
|
process until it completes. Any previously queued Builds (or Builds
|
|
which get queued after the reconfig) will use the new process.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Defining-the-Project"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Loading-the-Config-File">Loading the Config File</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">4.3 Defining the Project</h3>
|
|
|
|
<p>There are a couple of basic settings that you use to tell the buildbot
|
|
what project it is working on. This information is used by status
|
|
reporters to let users find out more about the codebase being
|
|
exercised by this particular Buildbot installation.
|
|
|
|
<pre class="example"> c['projectName'] = "Buildbot"
|
|
c['projectURL'] = "http://buildbot.sourceforge.net/"
|
|
c['buildbotURL'] = "http://localhost:8010/"
|
|
</pre>
|
|
<p><a name="index-c_005b_0027projectName_0027_005d-12"></a><code>projectName</code> is a short string will be used to describe the
|
|
project that this buildbot is working on. For example, it is used as
|
|
the title of the waterfall HTML page.
|
|
|
|
<p><a name="index-c_005b_0027projectURL_0027_005d-13"></a><code>projectURL</code> is a string that gives a URL for the project as a
|
|
whole. HTML status displays will show <code>projectName</code> as a link to
|
|
<code>projectURL</code>, to provide a link from buildbot HTML pages to your
|
|
project's home page.
|
|
|
|
<p><a name="index-c_005b_0027buildbotURL_0027_005d-14"></a>The <code>buildbotURL</code> string should point to the location where the
|
|
buildbot's internal web server (usually the <code>html.Waterfall</code>
|
|
page) is visible. This typically uses the port number set when you
|
|
create the <code>Waterfall</code> object: the buildbot needs your help to
|
|
figure out a suitable externally-visible host name.
|
|
|
|
<p>When status notices are sent to users (either by email or over IRC),
|
|
<code>buildbotURL</code> will be used to create a URL to the specific build
|
|
or problem that they are being notified about. It will also be made
|
|
available to queriers (over IRC) who want to find out where to get
|
|
more information about this buildbot.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Listing-Change-Sources-and-Schedulers"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Setting-the-slaveport">Setting the slaveport</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Defining-the-Project">Defining the Project</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">4.4 Listing Change Sources and Schedulers</h3>
|
|
|
|
<p><a name="index-c_005b_0027sources_0027_005d-15"></a>The <code>c['sources']</code> key is a list of ChangeSource
|
|
instances<a rel="footnote" href="#fn-6" name="fnd-6"><sup>6</sup></a>.
|
|
This defines how the buildmaster learns about source code changes.
|
|
More information about what goes here is available in See <a href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>.
|
|
|
|
<pre class="example"> import buildbot.changes.pb
|
|
c['sources'] = [buildbot.changes.pb.PBChangeSource()]
|
|
</pre>
|
|
<p><a name="index-c_005b_0027schedulers_0027_005d-16"></a><code>c['schedulers']</code> is a list of Scheduler instances, each of which
|
|
causes builds to be started on a particular set of Builders. The two
|
|
basic Scheduler classes you are likely to start with are
|
|
<code>Scheduler</code> and <code>Periodic</code>, but you can write a customized
|
|
subclass to implement more complicated build scheduling.
|
|
|
|
<p>The docstring for <code>buildbot.scheduler.Scheduler</code> is the best
|
|
place to see all the options that can be used. Type <code>pydoc
|
|
buildbot.scheduler.Scheduler</code> to see it, or look in
|
|
<samp><span class="file">buildbot/scheduler.py</span></samp> directly.
|
|
|
|
<p>The basic Scheduler takes four arguments:
|
|
|
|
<dl>
|
|
<dt><code>name</code><dd>Each Scheduler must have a unique name. This is only used in status
|
|
displays.
|
|
|
|
<br><dt><code>branch</code><dd>This Scheduler will pay attention to a single branch, ignoring Changes
|
|
that occur on other branches. Setting <code>branch</code> equal to the
|
|
special value of <code>None</code> means it should only pay attention to the
|
|
default branch. Note that <code>None</code> is a keyword, not a string, so
|
|
you want to use <code>None</code> and not <code>"None"</code>.
|
|
|
|
<br><dt><code>treeStableTimer</code><dd>The Scheduler will wait for this many seconds before starting the
|
|
build. If new changes are made during this interval, the timer will be
|
|
restarted, so really the build will be started after a change and then
|
|
after this many seconds of inactivity.
|
|
|
|
<br><dt><code>builderNames</code><dd>When the tree-stable-timer finally expires, builds will be started on
|
|
these Builders. Each Builder gets a unique name: these strings must
|
|
match.
|
|
|
|
</dl>
|
|
|
|
<pre class="example"> from buildbot import scheduler
|
|
quick = scheduler.Scheduler("quick", None, 60,
|
|
["quick-linux", "quick-netbsd"])
|
|
full = scheduler.Scheduler("full", None, 5*60,
|
|
["full-linux", "full-netbsd", "full-OSX"])
|
|
nightly = scheduler.Periodic("nightly", ["full-solaris"], 24*60*60)
|
|
c['schedulers'] = [quick, full, nightly]
|
|
</pre>
|
|
<p>In this example, the two “quick” builds are triggered 60 seconds
|
|
after the tree has been changed. The “full” builds do not run quite
|
|
so quickly (they wait 5 minutes), so hopefully if the quick builds
|
|
fail due to a missing file or really simple typo, the developer can
|
|
discover and fix the problem before the full builds are started. Both
|
|
Schedulers only pay attention to the default branch: any changes on
|
|
other branches are ignored by these Schedulers. Each Scheduler
|
|
triggers a different set of Builders, referenced by name.
|
|
|
|
<p>The third Scheduler in this example just runs the full solaris build
|
|
once per day. (note that this Scheduler only lets you control the time
|
|
between builds, not the absolute time-of-day of each Build, so this
|
|
could easily wind up a “daily” or “every afternoon” scheduler
|
|
depending upon when it was first activated).
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Scheduler-Types">Scheduler Types</a>
|
|
<li><a accesskey="2" href="#Build-Dependencies">Build Dependencies</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Scheduler-Types"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Build-Dependencies">Build Dependencies</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">4.4.1 Scheduler Types</h4>
|
|
|
|
<p><a name="index-buildbot_002escheduler_002eScheduler-17"></a><a name="index-buildbot_002escheduler_002eAnyBranchScheduler-18"></a><a name="index-buildbot_002escheduler_002ePeriodic-19"></a><a name="index-buildbot_002escheduler_002eNightly-20"></a>
|
|
Here is a brief catalog of the available Scheduler types. All these
|
|
Schedulers are classes in <code>buildbot.scheduler</code>, and the
|
|
docstrings there are the best source of documentation on the arguments
|
|
taken by each one.
|
|
|
|
<dl>
|
|
<dt><code>Scheduler</code><dd>This is the default Scheduler class. It follows exactly one branch,
|
|
and starts a configurable tree-stable-timer after each change on that
|
|
branch. When the timer expires, it starts a build on some set of
|
|
Builders. The Scheduler accepts a <code>fileIsImportant</code> function
|
|
which can be used to ignore some Changes if they do not affect any
|
|
“important” files.
|
|
|
|
<br><dt><code>AnyBranchScheduler</code><dd>This scheduler uses a tree-stable-timer like the default one, but
|
|
follows multiple branches at once. Each branch gets a separate timer.
|
|
|
|
<br><dt><code>Dependent</code><dd>This scheduler watches an “upstream” Builder. When that Builder
|
|
successfully builds a particular set of Changes, it triggers builds of
|
|
the same code on a configured set of “downstream” builders. The next
|
|
section (see <a href="#Build-Dependencies">Build Dependencies</a>) describes this scheduler in more
|
|
detail.
|
|
|
|
<br><dt><code>Periodic</code><dd>This simple scheduler just triggers a build every N seconds.
|
|
|
|
<br><dt><code>Nightly</code><dd>This is highly configurable periodic build scheduler, which triggers a
|
|
build at particular times of day, week, month, or year. The
|
|
configuration syntax is very similar to the well-known <code>crontab</code>
|
|
format, in which you provide values for minute, hour, day, and month
|
|
(some of which can be wildcards), and a build is triggered whenever
|
|
the current time matches the given constraints. This can run a build
|
|
every night, every morning, every weekend, alternate Thursdays, on
|
|
your boss's birthday, etc.
|
|
|
|
<br><dt><code>Try_Jobdir / Try_Userpass</code><dd>This scheduler allows developers to use the <code>buildbot try</code>
|
|
command to trigger builds of code they have not yet committed. See
|
|
<a href="#try">try</a> for complete details.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Build-Dependencies"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Scheduler-Types">Scheduler Types</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">4.4.2 Build Dependencies</h4>
|
|
|
|
<p><a name="index-Dependent-21"></a><a name="index-Dependencies-22"></a><a name="index-buildbot_002escheduler_002eDependent-23"></a>
|
|
It is common to wind up with one kind of build which should only be
|
|
performed if the same source code was successfully handled by some
|
|
other kind of build first. An example might be a packaging step: you
|
|
might only want to produce .deb or RPM packages from a tree that was
|
|
known to compile successfully and pass all unit tests. You could put
|
|
the packaging step in the same Build as the compile and testing steps,
|
|
but there might be other reasons to not do this (in particular you
|
|
might have several Builders worth of compiles/tests, but only wish to
|
|
do the packaging once). Another example is if you want to skip the
|
|
“full” builds after a failing “quick” build of the same source
|
|
code. Or, if one Build creates a product (like a compiled library)
|
|
that is used by some other Builder, you'd want to make sure the
|
|
consuming Build is run <em>after</em> the producing one.
|
|
|
|
<p>You can use <code>Dependencies</code> to express this relationship to the
|
|
Buildbot. There is a special kind of Scheduler named
|
|
<code>scheduler.Dependent</code> that will watch an “upstream” Scheduler
|
|
for builds to complete successfully (on all of its Builders). Each
|
|
time that happens, the same source code (i.e. the same
|
|
<code>SourceStamp</code>) will be used to start a new set of builds, on a
|
|
different set of Builders. This “downstream” scheduler doesn't pay
|
|
attention to Changes at all, it only pays attention to the upstream
|
|
scheduler.
|
|
|
|
<p>If the SourceStamp fails on any of the Builders in the upstream set,
|
|
the downstream builds will not fire.
|
|
|
|
<pre class="example"> from buildbot import scheduler
|
|
tests = scheduler.Scheduler("tests", None, 5*60,
|
|
["full-linux", "full-netbsd", "full-OSX"])
|
|
package = scheduler.Dependent("package",
|
|
tests, # upstream scheduler
|
|
["make-tarball", "make-deb", "make-rpm"])
|
|
c['schedulers'] = [tests, package]
|
|
</pre>
|
|
<p>Note that <code>Dependent</code>'s upstream scheduler argument is given as a
|
|
<code>Scheduler</code> <em>instance</em>, not a name. This makes it impossible
|
|
to create circular dependencies in the config file.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Setting-the-slaveport"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Buildslave-Specifiers">Buildslave Specifiers</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">4.5 Setting the slaveport</h3>
|
|
|
|
<p><a name="index-c_005b_0027slavePortnum_0027_005d-24"></a>
|
|
The buildmaster will listen on a TCP port of your choosing for
|
|
connections from buildslaves. It can also use this port for
|
|
connections from remote Change Sources, status clients, and debug
|
|
tools. This port should be visible to the outside world, and you'll
|
|
need to tell your buildslave admins about your choice.
|
|
|
|
<p>It does not matter which port you pick, as long it is externally
|
|
visible, however you should probably use something larger than 1024,
|
|
since most operating systems don't allow non-root processes to bind to
|
|
low-numbered ports. If your buildmaster is behind a firewall or a NAT
|
|
box of some sort, you may have to configure your firewall to permit
|
|
inbound connections to this port.
|
|
|
|
<pre class="example"> c['slavePortnum'] = 10000
|
|
</pre>
|
|
<p><code>c['slavePortnum']</code> is a <em>strports</em> specification string,
|
|
defined in the <code>twisted.application.strports</code> module (try
|
|
<samp><span class="command">pydoc twisted.application.strports</span></samp> to get documentation on
|
|
the format). This means that you can have the buildmaster listen on a
|
|
localhost-only port by doing:
|
|
|
|
<pre class="example"> c['slavePortnum'] = "tcp:10000:interface=127.0.0.1"
|
|
</pre>
|
|
<p>This might be useful if you only run buildslaves on the same machine,
|
|
and they are all configured to contact the buildmaster at
|
|
<code>localhost:10000</code>.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Buildslave-Specifiers"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Defining-Builders">Defining Builders</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Setting-the-slaveport">Setting the slaveport</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">4.6 Buildslave Specifiers</h3>
|
|
|
|
<p><a name="index-c_005b_0027bots_0027_005d-25"></a>
|
|
The <code>c['bots']</code> key is a list of known buildslaves. Each
|
|
buildslave is defined by a tuple of (slavename, slavepassword). These
|
|
are the same two values that need to be provided to the buildslave
|
|
administrator when they create the buildslave.
|
|
|
|
<pre class="example"> c['bots'] = [('bot-solaris', 'solarispasswd'),
|
|
('bot-bsd', 'bsdpasswd'),
|
|
]
|
|
</pre>
|
|
<p>The slavenames must be unique, of course. The password exists to
|
|
prevent evildoers from interfering with the buildbot by inserting
|
|
their own (broken) buildslaves into the system and thus displacing the
|
|
real ones.
|
|
|
|
<p>Buildslaves with an unrecognized slavename or a non-matching password
|
|
will be rejected when they attempt to connect, and a message
|
|
describing the problem will be put in the log file (see <a href="#Logfiles">Logfiles</a>).
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Defining-Builders"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Defining-Status-Targets">Defining Status Targets</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Buildslave-Specifiers">Buildslave Specifiers</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">4.7 Defining Builders</h3>
|
|
|
|
<p><a name="index-c_005b_0027builders_0027_005d-26"></a>
|
|
The <code>c['builders']</code> key is a list of dictionaries which specify
|
|
the Builders. The Buildmaster runs a collection of Builders, each of
|
|
which handles a single type of build (e.g. full versus quick), on a
|
|
single build slave. A Buildbot which makes sure that the latest code
|
|
(“HEAD”) compiles correctly across four separate architecture will
|
|
have four Builders, each performing the same build but on different
|
|
slaves (one per platform).
|
|
|
|
<p>Each Builder gets a separate column in the waterfall display. In
|
|
general, each Builder runs independently (although various kinds of
|
|
interlocks can cause one Builder to have an effect on another).
|
|
|
|
<p>Each Builder specification dictionary has several required keys:
|
|
|
|
<dl>
|
|
<dt><code>name</code><dd>This specifies the Builder's name, which is used in status
|
|
reports.
|
|
|
|
<br><dt><code>slavename</code><dd>This specifies which buildslave will be used by this Builder.
|
|
<code>slavename</code> must appear in the <code>c['bots']</code> list. Each
|
|
buildslave can accomodate multiple Builders.
|
|
|
|
<br><dt><code>slavenames</code><dd>If you provide <code>slavenames</code> instead of <code>slavename</code>, you can
|
|
give a list of buildslaves which are capable of running this Builder.
|
|
If multiple buildslaves are available for any given Builder, you will
|
|
have some measure of redundancy: in case one slave goes offline, the
|
|
others can still keep the Builder working. In addition, multiple
|
|
buildslaves will allow multiple simultaneous builds for the same
|
|
Builder, which might be useful if you have a lot of forced or “try”
|
|
builds taking place.
|
|
|
|
<p>If you use this feature, it is important to make sure that the
|
|
buildslaves are all, in fact, capable of running the given build. The
|
|
slave hosts should be configured similarly, otherwise you will spend a
|
|
lot of time trying (unsuccessfully) to reproduce a failure that only
|
|
occurs on some of the buildslaves and not the others. Different
|
|
platforms, operating systems, versions of major programs or libraries,
|
|
all these things mean you should use separate Builders.
|
|
|
|
<br><dt><code>builddir</code><dd>This specifies the name of a subdirectory (under the base directory)
|
|
in which everything related to this builder will be placed. On the
|
|
buildmaster, this holds build status information. On the buildslave,
|
|
this is where checkouts, compiles, and tests are run.
|
|
|
|
<br><dt><code>factory</code><dd>This is a <code>buildbot.process.factory.BuildFactory</code> instance which
|
|
controls how the build is performed. Full details appear in their own
|
|
chapter, See <a href="#Build-Process">Build Process</a>. Parameters like the location of the CVS
|
|
repository and the compile-time options used for the build are
|
|
generally provided as arguments to the factory's constructor.
|
|
|
|
</dl>
|
|
|
|
<p>Other optional keys may be set on each Builder:
|
|
|
|
<dl>
|
|
<dt><code>category</code><dd>If provided, this is a string that identifies a category for the
|
|
builder to be a part of. Status clients can limit themselves to a
|
|
subset of the available categories. A common use for this is to add
|
|
new builders to your setup (for a new module, or for a new buildslave)
|
|
that do not work correctly yet and allow you to integrate them with
|
|
the active builders. You can put these new builders in a test
|
|
category, make your main status clients ignore them, and have only
|
|
private status clients pick them up. As soon as they work, you can
|
|
move them over to the active category.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Defining-Status-Targets"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Debug-options">Debug options</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Defining-Builders">Defining Builders</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">4.8 Defining Status Targets</h3>
|
|
|
|
<p>The Buildmaster has a variety of ways to present build status to
|
|
various users. Each such delivery method is a “Status Target” object
|
|
in the configuration's <code>status</code> list. To add status targets, you
|
|
just append more objects to this list:
|
|
|
|
<p><a name="index-c_005b_0027status_0027_005d-27"></a>
|
|
<pre class="example"> c['status'] = []
|
|
|
|
from buildbot.status import html
|
|
c['status'].append(html.Waterfall(http_port=8010))
|
|
|
|
from buildbot.status import mail
|
|
m = mail.MailNotifier(fromaddr="buildbot@localhost",
|
|
extraRecipients=["builds@lists.example.com"],
|
|
sendToInterestedUsers=False)
|
|
c['status'].append(m)
|
|
|
|
from buildbot.status import words
|
|
c['status'].append(words.IRC(host="irc.example.com", nick="bb",
|
|
channels=["#example"]))
|
|
</pre>
|
|
<p>Status delivery has its own chapter, See <a href="#Status-Delivery">Status Delivery</a>, in which
|
|
all the built-in status targets are documented.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Debug-options"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Defining-Status-Targets">Defining Status Targets</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">4.9 Debug options</h3>
|
|
|
|
<p><a name="index-c_005b_0027debugPassword_0027_005d-28"></a>If you set <code>c['debugPassword']</code>, then you can connect to the
|
|
buildmaster with the diagnostic tool launched by <code>buildbot
|
|
debugclient MASTER:PORT</code>. From this tool, you can reload the config
|
|
file, manually force builds, and inject changes, which may be useful
|
|
for testing your buildmaster without actually commiting changes to
|
|
your repository (or before you have the Change Sources set up). The
|
|
debug tool uses the same port number as the slaves do:
|
|
<code>c['slavePortnum']</code>, and is authenticated with this password.
|
|
|
|
<pre class="example"> c['debugPassword'] = "debugpassword"
|
|
</pre>
|
|
<p><a name="index-c_005b_0027manhole_0027_005d-29"></a>If you set <code>c['manhole']</code> to an instance of one of the classes in
|
|
<code>buildbot.manhole</code>, you can telnet or ssh into the buildmaster
|
|
and get an interactive Python shell, which may be useful for debugging
|
|
buildbot internals. It is probably only useful for buildbot
|
|
developers. It exposes full access to the buildmaster's account
|
|
(including the ability to modify and delete files), so it should not
|
|
be enabled with a weak or easily guessable password.
|
|
|
|
<p>There are three separate <code>Manhole</code> classes. Two of them use SSH,
|
|
one uses unencrypted telnet. Two of them use a username+password
|
|
combination to grant access, one of them uses an SSH-style
|
|
<samp><span class="file">authorized_keys</span></samp> file which contains a list of ssh public keys.
|
|
|
|
<dl>
|
|
<dt><code>manhole.AuthorizedKeysManhole</code><dd>You construct this with the name of a file that contains one SSH
|
|
public key per line, just like <samp><span class="file">~/.ssh/authorized_keys</span></samp>. If you
|
|
provide a non-absolute filename, it will be interpreted relative to
|
|
the buildmaster's base directory.
|
|
|
|
<br><dt><code>manhole.PasswordManhole</code><dd>This one accepts SSH connections but asks for a username and password
|
|
when authenticating. It accepts only one such pair.
|
|
|
|
<br><dt><code>manhole.TelnetManhole</code><dd>This accepts regular unencrypted telnet connections, and asks for a
|
|
username/password pair before providing access. Because this
|
|
username/password is transmitted in the clear, and because Manhole
|
|
access to the buildmaster is equivalent to granting full shell
|
|
privileges to both the buildmaster and all the buildslaves (and to all
|
|
accounts which then run code produced by the buildslaves), it is
|
|
highly recommended that you use one of the SSH manholes instead.
|
|
|
|
</dl>
|
|
|
|
<pre class="example"> # some examples:
|
|
from buildbot import manhole
|
|
c['manhole'] = manhole.AuthorizedKeysManhole(1234, "authorized_keys")
|
|
c['manhole'] = manhole.PasswordManhole(1234, "alice", "mysecretpassword")
|
|
c['manhole'] = manhole.TelnetManhole(1234, "bob", "snoop_my_password_please")
|
|
</pre>
|
|
<p>The <code>Manhole</code> instance can be configured to listen on a specific
|
|
port. You may wish to have this listening port bind to the loopback
|
|
interface (sometimes known as “lo0”, “localhost”, or 127.0.0.1) to
|
|
restrict access to clients which are running on the same host.
|
|
|
|
<pre class="example"> from buildbot.manhole import PasswordManhole
|
|
c['manhole'] = PasswordManhole("tcp:9999:interface=127.0.0.1","admin","passwd")
|
|
</pre>
|
|
<p>To have the <code>Manhole</code> listen on all interfaces, use
|
|
<code>"tcp:9999"</code> or simply 9999. This port specification uses
|
|
<code>twisted.application.strports</code>, so you can make it listen on SSL
|
|
or even UNIX-domain sockets if you want.
|
|
|
|
<p>Note that using any Manhole requires that the TwistedConch package be
|
|
installed, and that you be using Twisted version 2.0 or later.
|
|
|
|
<p>The buildmaster's SSH server will use a different host key than the
|
|
normal sshd running on a typical unix host. This will cause the ssh
|
|
client to complain about a “host key mismatch”, because it does not
|
|
realize there are two separate servers running on the same host. To
|
|
avoid this, use a clause like the following in your <samp><span class="file">.ssh/config</span></samp>
|
|
file:
|
|
|
|
<pre class="example"> Host remotehost-buildbot
|
|
HostName remotehost
|
|
HostKeyAlias remotehost-buildbot
|
|
Port 9999
|
|
# use 'user' if you use PasswordManhole and your name is not 'admin'.
|
|
# if you use AuthorizedKeysManhole, this probably doesn't matter.
|
|
User admin
|
|
</pre>
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Getting-Source-Code-Changes"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Build-Process">Build Process</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Configuration">Configuration</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="chapter">5 Getting Source Code Changes</h2>
|
|
|
|
<p>The most common way to use the Buildbot is centered around the idea of
|
|
<code>Source Trees</code>: a directory tree filled with source code of some form
|
|
which can be compiled and/or tested. Some projects use languages that don't
|
|
involve any compilation step: nevertheless there may be a <code>build</code> phase
|
|
where files are copied or rearranged into a form that is suitable for
|
|
installation. Some projects do not have unit tests, and the Buildbot is
|
|
merely helping to make sure that the sources can compile correctly. But in
|
|
all of these cases, the thing-being-tested is a single source tree.
|
|
|
|
<p>A Version Control System mantains a source tree, and tells the
|
|
buildmaster when it changes. The first step of each Build is typically
|
|
to acquire a copy of some version of this tree.
|
|
|
|
<p>This chapter describes how the Buildbot learns about what Changes have
|
|
occurred. For more information on VC systems and Changes, see
|
|
<a href="#Version-Control-Systems">Version Control Systems</a>.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Change-Sources">Change Sources</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Change-Sources"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">5.1 Change Sources</h3>
|
|
|
|
<!-- TODO: rework this, the one-buildmaster-one-tree thing isn't quite -->
|
|
<!-- so narrow-minded anymore -->
|
|
<p>Each Buildmaster watches a single source tree. Changes can be provided
|
|
by a variety of ChangeSource types, however any given project will
|
|
typically have only a single ChangeSource active. This section
|
|
provides a description of all available ChangeSource types and
|
|
explains how to set up each of them.
|
|
|
|
<p>There are a variety of ChangeSources available, some of which are
|
|
meant to be used in conjunction with other tools to deliver Change
|
|
events from the VC repository to the buildmaster.
|
|
|
|
<ul>
|
|
<li>CVSToys
|
|
This ChangeSource opens a TCP connection from the buildmaster to a
|
|
waiting FreshCVS daemon that lives on the repository machine, and
|
|
subscribes to hear about Changes.
|
|
|
|
<li>MaildirSource
|
|
This one watches a local maildir-format inbox for email sent out by
|
|
the repository when a change is made. When a message arrives, it is
|
|
parsed to create the Change object. A variety of parsing functions are
|
|
available to accomodate different email-sending tools.
|
|
|
|
<li>PBChangeSource
|
|
This ChangeSource listens on a local TCP socket for inbound
|
|
connections from a separate tool. Usually, this tool would be run on
|
|
the VC repository machine in a commit hook. It is expected to connect
|
|
to the TCP socket and send a Change message over the network
|
|
connection. The <samp><span class="command">buildbot sendchange</span></samp> command is one example
|
|
of a tool that knows how to send these messages, so you can write a
|
|
commit script for your VC system that calls it to deliver the Change.
|
|
There are other tools in the contrib/ directory that use the same
|
|
protocol.
|
|
|
|
</ul>
|
|
|
|
<p>As a quick guide, here is a list of VC systems and the ChangeSources
|
|
that might be useful with them. All of these ChangeSources are in the
|
|
<code>buildbot.changes</code> module.
|
|
|
|
<dl>
|
|
<dt><code>CVS</code><dd>
|
|
<ul>
|
|
<li>freshcvs.FreshCVSSource (connected via TCP to the freshcvs daemon)
|
|
<li>mail.FCMaildirSource (watching for email sent by a freshcvs daemon)
|
|
<li>mail.BonsaiMaildirSource (watching for email sent by Bonsai)
|
|
<li>mail.SyncmailMaildirSource (watching for email sent by syncmail)
|
|
<li>pb.PBChangeSource (listening for connections from <code>buildbot
|
|
sendchange</code> run in a loginfo script)
|
|
<li>pb.PBChangeSource (listening for connections from a long-running
|
|
<code>contrib/viewcvspoll.py</code> polling process which examines the ViewCVS
|
|
database directly
|
|
</ul>
|
|
|
|
<br><dt><code>SVN</code><dd>
|
|
<ul>
|
|
<li>pb.PBChangeSource (listening for connections from
|
|
<code>contrib/svn_buildbot.py</code> run in a postcommit script)
|
|
<li>pb.PBChangeSource (listening for connections from a long-running
|
|
<code>contrib/svn_watcher.py</code> or <code>contrib/svnpoller.py</code> polling
|
|
process
|
|
<li>svnpoller.SVNPoller (polling the SVN repository)
|
|
</ul>
|
|
|
|
<br><dt><code>Darcs</code><dd>
|
|
<ul>
|
|
<li>pb.PBChangeSource (listening for connections from
|
|
<code>contrib/darcs_buildbot.py</code> in a commit script
|
|
</ul>
|
|
|
|
<br><dt><code>Mercurial</code><dd>
|
|
<ul>
|
|
<li>pb.PBChangeSource (listening for connections from
|
|
<code>contrib/hg_buildbot.py</code> run in an 'incoming' hook)
|
|
</ul>
|
|
|
|
<br><dt><code>Arch/Bazaar</code><dd>
|
|
<ul>
|
|
<li>pb.PBChangeSource (listening for connections from
|
|
<code>contrib/arch_buildbot.py</code> run in a commit hook)
|
|
</ul>
|
|
|
|
</dl>
|
|
|
|
<p>All VC systems can be driven by a PBChangeSource and the
|
|
<code>buildbot sendchange</code> tool run from some form of commit script.
|
|
If you write an email parsing function, they can also all be driven by
|
|
a suitable <code>MaildirSource</code>.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Choosing-ChangeSources">Choosing ChangeSources</a>
|
|
<li><a accesskey="2" href="#CVSToys-_002d-PBService">CVSToys - PBService</a>
|
|
<li><a accesskey="3" href="#CVSToys-_002d-mail-notification">CVSToys - mail notification</a>
|
|
<li><a accesskey="4" href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a>
|
|
<li><a accesskey="5" href="#PBChangeSource">PBChangeSource</a>
|
|
<li><a accesskey="6" href="#P4Source">P4Source</a>
|
|
<li><a accesskey="7" href="#BonsaiPoller">BonsaiPoller</a>
|
|
<li><a accesskey="8" href="#SVNPoller">SVNPoller</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Choosing-ChangeSources"></a>
|
|
Next: <a rel="next" accesskey="n" href="#CVSToys-_002d-PBService">CVSToys - PBService</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Change-Sources">Change Sources</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">5.1.1 Choosing ChangeSources</h4>
|
|
|
|
<p>The <code>master.cfg</code> configuration file has a dictionary key named
|
|
<code>BuildmasterConfig['sources']</code>, which holds a list of
|
|
<code>IChangeSource</code> objects. The config file will typically create an
|
|
object from one of the classes described below and stuff it into the
|
|
list.
|
|
|
|
<pre class="example"> s = FreshCVSSourceNewcred(host="host", port=4519,
|
|
user="alice", passwd="secret",
|
|
prefix="Twisted")
|
|
BuildmasterConfig['sources'] = [s]
|
|
</pre>
|
|
<p>Each source tree has a nominal <code>top</code>. Each Change has a list of
|
|
filenames, which are all relative to this top location. The
|
|
ChangeSource is responsible for doing whatever is necessary to
|
|
accomplish this. Most sources have a <code>prefix</code> argument: a partial
|
|
pathname which is stripped from the front of all filenames provided to
|
|
that <code>ChangeSource</code>. Files which are outside this sub-tree are
|
|
ignored by the changesource: it does not generate Changes for those
|
|
files.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="CVSToys---PBService"></a>
|
|
<a name="CVSToys-_002d-PBService"></a>
|
|
Next: <a rel="next" accesskey="n" href="#CVSToys-_002d-mail-notification">CVSToys - mail notification</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Choosing-ChangeSources">Choosing ChangeSources</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">5.1.2 CVSToys - PBService</h4>
|
|
|
|
<p><a name="index-buildbot_002echanges_002efreshcvs_002eFreshCVSSource-30"></a>
|
|
The <a href="http://purl.net/net/CVSToys">CVSToys</a> package provides a
|
|
server which runs on the machine that hosts the CVS repository it
|
|
watches. It has a variety of ways to distribute commit notifications,
|
|
and offers a flexible regexp-based way to filter out uninteresting
|
|
changes. One of the notification options is named <code>PBService</code> and
|
|
works by listening on a TCP port for clients. These clients subscribe
|
|
to hear about commit notifications.
|
|
|
|
<p>The buildmaster has a CVSToys-compatible <code>PBService</code> client built
|
|
in. There are two versions of it, one for old versions of CVSToys
|
|
(1.0.9 and earlier) which used the <code>oldcred</code> authentication
|
|
framework, and one for newer versions (1.0.10 and later) which use
|
|
<code>newcred</code>. Both are classes in the
|
|
<code>buildbot.changes.freshcvs</code> package.
|
|
|
|
<p><code>FreshCVSSourceNewcred</code> objects are created with the following
|
|
parameters:
|
|
|
|
<dl>
|
|
<dt>`<samp><code>host</code><span class="samp"> and </span><code>port</code></samp>'<dd>these specify where the CVSToys server can be reached
|
|
|
|
<br><dt>`<samp><code>user</code><span class="samp"> and </span><code>passwd</code></samp>'<dd>these specify the login information for the CVSToys server
|
|
(<code>freshcvs</code>). These must match the server's values, which are
|
|
defined in the <code>freshCfg</code> configuration file (which lives in the
|
|
CVSROOT directory of the repository).
|
|
|
|
<br><dt>`<samp><code>prefix</code></samp>'<dd>this is the prefix to be found and stripped from filenames delivered
|
|
by the CVSToys server. Most projects live in sub-directories of the
|
|
main repository, as siblings of the CVSROOT sub-directory, so
|
|
typically this prefix is set to that top sub-directory name.
|
|
|
|
</dl>
|
|
|
|
<h3 class="heading">Example</h3>
|
|
|
|
<p>To set up the freshCVS server, add a statement like the following to
|
|
your <samp><span class="file">freshCfg</span></samp> file:
|
|
|
|
<pre class="example"> pb = ConfigurationSet([
|
|
(None, None, None, PBService(userpass=('foo', 'bar'), port=4519)),
|
|
])
|
|
</pre>
|
|
<p>This will announce all changes to a client which connects to port 4519
|
|
using a username of 'foo' and a password of 'bar'.
|
|
|
|
<p>Then add a clause like this to your buildmaster's <samp><span class="file">master.cfg</span></samp>:
|
|
|
|
<pre class="example"> BuildmasterConfig['sources'] = [FreshCVSSource("cvs.example.com", 4519,
|
|
"foo", "bar",
|
|
prefix="glib/")]
|
|
</pre>
|
|
<p>where "cvs.example.com" is the host that is running the FreshCVS daemon, and
|
|
"glib" is the top-level directory (relative to the repository's root) where
|
|
all your source code lives. Most projects keep one or more projects in the
|
|
same repository (along with CVSROOT/ to hold admin files like loginfo and
|
|
freshCfg); the prefix= argument tells the buildmaster to ignore everything
|
|
outside that directory, and to strip that common prefix from all pathnames
|
|
it handles.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="CVSToys---mail-notification"></a>
|
|
<a name="CVSToys-_002d-mail-notification"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#CVSToys-_002d-PBService">CVSToys - PBService</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">5.1.3 CVSToys - mail notification</h4>
|
|
|
|
<p><a name="index-buildbot_002echanges_002email_002eFCMaildirSource-31"></a>
|
|
CVSToys also provides a <code>MailNotification</code> action which will send
|
|
email to a list of recipients for each commit. This tends to work
|
|
better than using <code>/bin/mail</code> from within the CVSROOT/loginfo
|
|
file directly, as CVSToys will batch together all files changed during
|
|
the same CVS invocation, and can provide more information (like
|
|
creating a ViewCVS URL for each file changed).
|
|
|
|
<p>The Buildbot's <code>FCMaildirSource</code> is a ChangeSource which knows
|
|
how to parse these CVSToys messages and turn them into Change objects.
|
|
It watches a Maildir for new messages. The usually installation
|
|
process looks like:
|
|
|
|
<ol type=1 start=1>
|
|
<li>Create a mailing list, <code>projectname-commits</code>.
|
|
<li>In CVSToys' freshCfg file, use a <code>MailNotification</code> action to
|
|
send commit mail to this mailing list.
|
|
<li>Subscribe the buildbot user to the mailing list.
|
|
<li>Configure your .qmail or .forward file to deliver these messages into
|
|
a maildir.
|
|
<li>In the Buildbot's master.cfg file, use a <code>FCMaildirSource</code> to
|
|
watch the maildir for commit messages.
|
|
</ol>
|
|
|
|
<p>The <code>FCMaildirSource</code> is created with two parameters: the
|
|
directory name of the maildir root, and the prefix to strip.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Other-mail-notification-ChangeSources"></a>
|
|
Next: <a rel="next" accesskey="n" href="#PBChangeSource">PBChangeSource</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#CVSToys-_002d-mail-notification">CVSToys - mail notification</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">5.1.4 Other mail notification ChangeSources</h4>
|
|
|
|
<p><a name="index-buildbot_002echanges_002email_002eSyncmailMaildirSource-32"></a><a name="index-buildbot_002echanges_002email_002eBonsaiMaildirSource-33"></a>
|
|
There are other types of maildir-watching ChangeSources, which only
|
|
differ in the function used to parse the message body.
|
|
|
|
<p><code>SyncmailMaildirSource</code> knows how to parse the message format
|
|
used in mail sent by Syncmail.
|
|
|
|
<p><code>BonsaiMaildirSource</code> parses messages sent out by Bonsai.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="PBChangeSource"></a>
|
|
Next: <a rel="next" accesskey="n" href="#P4Source">P4Source</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">5.1.5 PBChangeSource</h4>
|
|
|
|
<p><a name="index-buildbot_002echanges_002epb_002ePBChangeSource-34"></a>
|
|
The last kind of ChangeSource actually listens on a TCP port for
|
|
clients to connect and push change notices <em>into</em> the
|
|
Buildmaster. This is used by the built-in <code>buildbot sendchange</code>
|
|
notification tool, as well as the VC-specific
|
|
<samp><span class="file">contrib/svn_buildbot.py</span></samp> and <samp><span class="file">contrib/arch_buildbot.py</span></samp>
|
|
tools. These tools are run by the repository (in a commit hook
|
|
script), and connect to the buildmaster directly each time a file is
|
|
comitted. This is also useful for creating new kinds of change sources
|
|
that work on a <code>push</code> model instead of some kind of subscription
|
|
scheme, for example a script which is run out of an email .forward
|
|
file.
|
|
|
|
<p>This ChangeSource can be configured to listen on its own TCP port, or
|
|
it can share the port that the buildmaster is already using for the
|
|
buildslaves to connect. (This is possible because the
|
|
<code>PBChangeSource</code> uses the same protocol as the buildslaves, and
|
|
they can be distinguished by the <code>username</code> attribute used when
|
|
the initial connection is established). It might be useful to have it
|
|
listen on a different port if, for example, you wanted to establish
|
|
different firewall rules for that port. You could allow only the SVN
|
|
repository machine access to the <code>PBChangeSource</code> port, while
|
|
allowing only the buildslave machines access to the slave port. Or you
|
|
could just expose one port and run everything over it. <em>Note:
|
|
this feature is not yet implemented, the PBChangeSource will always
|
|
share the slave port and will always have a </em><code>user</code><em> name of
|
|
</em><code>change</code><em>, and a passwd of </em><code>changepw</code><em>. These limitations will
|
|
be removed in the future.</em>.
|
|
|
|
<p>The <code>PBChangeSource</code> is created with the following arguments. All
|
|
are optional.
|
|
|
|
<dl>
|
|
<dt>`<samp><code>port</code></samp>'<dd>which port to listen on. If <code>None</code> (which is the default), it
|
|
shares the port used for buildslave connections. <em>Not
|
|
Implemented, always set to </em><code>None</code>.
|
|
|
|
<br><dt>`<samp><code>user</code><span class="samp"> and </span><code>passwd</code></samp>'<dd>The user/passwd account information that the client program must use
|
|
to connect. Defaults to <code>change</code> and <code>changepw</code>. <em>Not
|
|
Implemented, </em><code>user</code><em> is currently always set to </em><code>change</code><em>,
|
|
</em><code>passwd</code><em> is always set to </em><code>changepw</code>.
|
|
|
|
<br><dt>`<samp><code>prefix</code></samp>'<dd>The prefix to be found and stripped from filenames delivered over the
|
|
connection. Any filenames which do not start with this prefix will be
|
|
removed. If all the filenames in a given Change are removed, the that
|
|
whole Change will be dropped. This string should probably end with a
|
|
directory separator.
|
|
|
|
<p>This is useful for changes coming from version control systems that
|
|
represent branches as parent directories within the repository (like
|
|
SVN and Perforce). Use a prefix of 'trunk/' or
|
|
'project/branches/foobranch/' to only follow one branch and to get
|
|
correct tree-relative filenames. Without a prefix, the PBChangeSource
|
|
will probably deliver Changes with filenames like <samp><span class="file">trunk/foo.c</span></samp>
|
|
instead of just <samp><span class="file">foo.c</span></samp>. Of course this also depends upon the
|
|
tool sending the Changes in (like <samp><span class="command">buildbot sendchange</span></samp>) and
|
|
what filenames it is delivering: that tool may be filtering and
|
|
stripping prefixes at the sending end.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="P4Source"></a>
|
|
Next: <a rel="next" accesskey="n" href="#BonsaiPoller">BonsaiPoller</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#PBChangeSource">PBChangeSource</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">5.1.6 P4Source</h4>
|
|
|
|
<p><a name="index-buildbot_002echanges_002ep4poller_002eP4Source-35"></a>
|
|
The <code>P4Source</code> periodically polls a <a href="http://www.perforce.com/">Perforce</a> depot for changes. It accepts the following arguments:
|
|
|
|
<dl>
|
|
<dt>`<samp><code>p4base</code></samp>'<dd>The base depot path to watch, without the trailing '/...'.
|
|
|
|
<br><dt>`<samp><code>p4port</code></samp>'<dd>The Perforce server to connect to (as host:port).
|
|
|
|
<br><dt>`<samp><code>p4user</code></samp>'<dd>The Perforce user.
|
|
|
|
<br><dt>`<samp><code>p4passwd</code></samp>'<dd>The Perforce password.
|
|
|
|
<br><dt>`<samp><code>split_file</code></samp>'<dd>A function that maps a pathname, without the leading <code>p4base</code>, to a
|
|
(branch, filename) tuple. The default just returns (None, branchfile),
|
|
which effectively disables branch support. You should supply a function
|
|
which understands your repository structure.
|
|
|
|
<br><dt>`<samp><code>pollinterval</code></samp>'<dd>How often to poll, in seconds. Defaults to 600 (10 minutes).
|
|
|
|
<br><dt>`<samp><code>histmax</code></samp>'<dd>The maximum number of changes to inspect at a time. If more than this
|
|
number occur since the last poll, older changes will be silently
|
|
ignored.
|
|
</dl>
|
|
|
|
<h3 class="heading">Example</h3>
|
|
|
|
<p>This configuration uses the <code>P4PORT</code>, <code>P4USER</code>, and <code>P4PASSWD</code>
|
|
specified in the buildmaster's environment. It watches a project in which the
|
|
branch name is simply the next path component, and the file is all path
|
|
components after.
|
|
|
|
<pre class="example"> import buildbot.changes.p4poller
|
|
c['sources'].append(p4poller.P4Source(
|
|
p4base='//depot/project/',
|
|
split_file=lambda branchfile: branchfile.split('/',1)
|
|
))
|
|
</pre>
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="BonsaiPoller"></a>
|
|
Next: <a rel="next" accesskey="n" href="#SVNPoller">SVNPoller</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#P4Source">P4Source</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">5.1.7 BonsaiPoller</h4>
|
|
|
|
<p><a name="index-buildbot_002echanges_002ebonsaipoller_002eBonsaiPoller-36"></a>
|
|
The <code>BonsaiPoller</code> periodically polls a Bonsai server. This is a
|
|
CGI script accessed through a web server that provides information
|
|
about a CVS tree, for example the Mozilla bonsai server at
|
|
<a href="http://bonsai.mozilla.org">http://bonsai.mozilla.org</a>. Bonsai servers are usable by both
|
|
humans and machines. In this case, the buildbot's change source forms
|
|
a query which asks about any files in the specified branch which have
|
|
changed since the last query.
|
|
|
|
<p>Please take a look at the BonsaiPoller docstring for details about the
|
|
arguments it accepts.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="SVNPoller"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#BonsaiPoller">BonsaiPoller</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Change-Sources">Change Sources</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">5.1.8 SVNPoller</h4>
|
|
|
|
<p><a name="index-buildbot_002echanges_002esvnpoller_002eSVNPoller-37"></a>
|
|
The <code>buildbot.changes.svnpoller.SVNPoller</code> is a ChangeSource
|
|
which periodically polls a <a href="http://subversion.tigris.org/">Subversion</a> repository for new revisions, by running the <code>svn
|
|
log</code> command in a subshell. It can watch a single branch or multiple
|
|
branches.
|
|
|
|
<p><code>SVNPoller</code> accepts the following arguments:
|
|
|
|
<dl>
|
|
<dt><code>svnurl</code><dd>The base URL path to watch, like
|
|
<code>svn://svn.twistedmatrix.com/svn/Twisted/trunk</code>, or
|
|
<code>http://divmod.org/svn/Divmod/</code>, or even
|
|
<code>file:///home/svn/Repository/ProjectA/branches/1.5/</code>. This must
|
|
include the access scheme, the location of the repository (both the
|
|
hostname for remote ones, and any additional directory names necessary
|
|
to get to the repository), and the sub-path within the repository's
|
|
virtual filesystem for the project and branch of interest.
|
|
|
|
<p>The <code>SVNPoller</code> will only pay attention to files inside the
|
|
subdirectory specified by the complete svnurl.
|
|
|
|
<br><dt><code>split_file</code><dd>A function to convert pathnames into (branch, relative_pathname)
|
|
tuples. Use this to explain your repository's branch-naming policy to
|
|
<code>SVNPoller</code>. This function must accept a single string and return
|
|
a two-entry tuple. There are a few utility functions in
|
|
<code>buildbot.changes.svnpoller</code> that can be used as a
|
|
<code>split_file</code> function, see below for details.
|
|
|
|
<p>The default value always returns (None, path), which indicates that
|
|
all files are on the trunk.
|
|
|
|
<p>Subclasses of <code>SVNPoller</code> can override the <code>split_file</code>
|
|
method instead of using the <code>split_file=</code> argument.
|
|
|
|
<br><dt><code>svnuser</code><dd>An optional string parameter. If set, the <code>--user</code> argument will
|
|
be added to all <code>svn</code> commands. Use this if you have to
|
|
authenticate to the svn server before you can do <code>svn info</code> or
|
|
<code>svn log</code> commands.
|
|
|
|
<br><dt><code>svnpasswd</code><dd>Like <code>svnuser</code>, this will cause a <code>--password</code> argument to
|
|
be passed to all svn commands.
|
|
|
|
<br><dt><code>pollinterval</code><dd>How often to poll, in seconds. Defaults to 600 (checking once every 10
|
|
minutes). Lower this if you want the buildbot to notice changes
|
|
faster, raise it if you want to reduce the network and CPU load on
|
|
your svn server. Please be considerate of public SVN repositories by
|
|
using a large interval when polling them.
|
|
|
|
<br><dt><code>histmax</code><dd>The maximum number of changes to inspect at a time. Every POLLINTERVAL
|
|
seconds, the <code>SVNPoller</code> asks for the last HISTMAX changes and
|
|
looks through them for any ones it does not already know about. If
|
|
more than HISTMAX revisions have been committed since the last poll,
|
|
older changes will be silently ignored. Larger values of histmax will
|
|
cause more time and memory to be consumed on each poll attempt.
|
|
<code>histmax</code> defaults to 100.
|
|
|
|
<br><dt><code>svnbin</code><dd>This controls the <code>svn</code> executable to use. If subversion is
|
|
installed in a weird place on your system (outside of the
|
|
buildmaster's <code>$PATH</code>), use this to tell <code>SVNPoller</code> where
|
|
to find it. The default value of “svn” will almost always be
|
|
sufficient.
|
|
|
|
</dl>
|
|
|
|
<h3 class="heading">Branches</h3>
|
|
|
|
<p>Each source file that is tracked by a Subversion repository has a
|
|
fully-qualified SVN URL in the following form:
|
|
(REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the
|
|
<code>SVNPoller</code>, you give it a <code>svnurl</code> value that includes all
|
|
of the REPOURL and possibly some portion of the PROJECT-plus-BRANCH
|
|
string. The <code>SVNPoller</code> is responsible for producing Changes that
|
|
contain a branch name and a FILEPATH (which is relative to the top of
|
|
a checked-out tree). The details of how these strings are split up
|
|
depend upon how your repository names its branches.
|
|
|
|
<h4 class="subheading">PROJECT/BRANCHNAME/FILEPATH repositories</h4>
|
|
|
|
<p>One common layout is to have all the various projects that share a
|
|
repository get a single top-level directory each. Then under a given
|
|
project's directory, you get two subdirectories, one named “trunk”
|
|
and another named “branches”. Under “branches” you have a bunch of
|
|
other directories, one per branch, with names like “1.5.x” and
|
|
“testing”. It is also common to see directories like “tags” and
|
|
“releases” next to “branches” and “trunk”.
|
|
|
|
<p>For example, the Twisted project has a subversion server on
|
|
“svn.twistedmatrix.com” that hosts several sub-projects. The
|
|
repository is available through a SCHEME of “svn:”. The primary
|
|
sub-project is Twisted, of course, with a repository root of
|
|
“svn://svn.twistedmatrix.com/svn/Twisted”. Another sub-project is
|
|
Informant, with a root of
|
|
“svn://svn.twistedmatrix.com/svn/Informant”, etc. Inside any
|
|
checked-out Twisted tree, there is a file named bin/trial (which is
|
|
used to run unit test suites).
|
|
|
|
<p>The trunk for Twisted is in
|
|
“svn://svn.twistedmatrix.com/svn/Twisted/trunk”, and the
|
|
fully-qualified SVN URL for the trunk version of <code>trial</code> would be
|
|
“svn://svn.twistedmatrix.com/svn/Twisted/trunk/bin/trial”. The same
|
|
SVNURL for that file on a branch named “1.5.x” would be
|
|
“svn://svn.twistedmatrix.com/svn/Twisted/branches/1.5.x/bin/trial”.
|
|
|
|
<p>To set up a <code>SVNPoller</code> that watches the Twisted trunk (and
|
|
nothing else), we would use the following:
|
|
|
|
<pre class="example"> from buildbot.changes.svnpoller import SVNPoller
|
|
s = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted/trunk")
|
|
c['sources'].append(ss)
|
|
</pre>
|
|
<p>In this case, every Change that our <code>SVNPoller</code> produces will
|
|
have <code>.branch=None</code>, to indicate that the Change is on the trunk.
|
|
No other sub-projects or branches will be tracked.
|
|
|
|
<p>If we want our ChangeSource to follow multiple branches, we have to do
|
|
two things. First we have to change our <code>svnurl=</code> argument to
|
|
watch more than just “.../Twisted/trunk”. We will set it to
|
|
“.../Twisted” so that we'll see both the trunk and all the branches.
|
|
Second, we have to tell <code>SVNPoller</code> how to split the
|
|
(PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository
|
|
out into (BRANCH) and (FILEPATH) pairs.
|
|
|
|
<p>We do the latter by providing a “split_file” function. This function
|
|
is responsible for splitting something like
|
|
“branches/1.5.x/bin/trial” into <code>branch</code>=”branches/1.5.x” and
|
|
<code>filepath</code>=”bin/trial”. This function is always given a string
|
|
that names a file relative to the subdirectory pointed to by the
|
|
<code>SVNPoller</code>'s <code>svnurl=</code> argument. It is expected to return a
|
|
(BRANCHNAME, FILEPATH) tuple (in which FILEPATH is relative to the
|
|
branch indicated), or None to indicate that the file is outside any
|
|
project of interest.
|
|
|
|
<p>(note that we want to see “branches/1.5.x” rather than just
|
|
“1.5.x” because when we perform the SVN checkout, we will probably
|
|
append the branch name to the baseURL, which requires that we keep the
|
|
“branches” component in there. Other VC schemes use a different
|
|
approach towards branches and may not require this artifact.)
|
|
|
|
<p>If your repository uses this same PROJECT/BRANCH/FILEPATH naming
|
|
scheme, the following function will work:
|
|
|
|
<pre class="example"> def split_file_branches(path):
|
|
pieces = path.split('/')
|
|
if pieces[0] == 'trunk':
|
|
return (None, '/'.join(pieces[1:]))
|
|
elif pieces[0] == 'branches':
|
|
return ('/'.join(pieces[0:2]),
|
|
'/'.join(pieces[2:]))
|
|
else:
|
|
return None
|
|
</pre>
|
|
<p>This function is provided as
|
|
<code>buildbot.changes.svnpoller.split_file_branches</code> for your
|
|
convenience. So to have our Twisted-watching <code>SVNPoller</code> follow
|
|
multiple branches, we would use this:
|
|
|
|
<pre class="example"> from buildbot.changes.svnpoller import SVNPoller, split_file_branches
|
|
s = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted",
|
|
split_file=split_file_branches)
|
|
c['sources'].append(ss)
|
|
</pre>
|
|
<p>Changes for all sorts of branches (with names like “branches/1.5.x”,
|
|
and None to indicate the trunk) will be delivered to the Schedulers.
|
|
Each Scheduler is then free to use or ignore each branch as it sees
|
|
fit.
|
|
|
|
<h4 class="subheading">BRANCHNAME/PROJECT/FILEPATH repositories</h4>
|
|
|
|
<p>Another common way to organize a Subversion repository is to put the
|
|
branch name at the top, and the projects underneath. This is
|
|
especially frequent when there are a number of related sub-projects
|
|
that all get released in a group.
|
|
|
|
<p>For example, Divmod.org hosts a project named “Nevow” as well as one
|
|
named “Quotient”. In a checked-out Nevow tree there is a directory
|
|
named “formless” that contains a python source file named
|
|
“webform.py”. This repository is accessible via webdav (and thus
|
|
uses an “http:” scheme) through the divmod.org hostname. There are
|
|
many branches in this repository, and they use a
|
|
(BRANCHNAME)/(PROJECT) naming policy.
|
|
|
|
<p>The fully-qualified SVN URL for the trunk version of webform.py is
|
|
<code>http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py</code>.
|
|
You can do an <code>svn co</code> with that URL and get a copy of the latest
|
|
version. The 1.5.x branch version of this file would have a URL of
|
|
<code>http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py</code>.
|
|
The whole Nevow trunk would be checked out with
|
|
<code>http://divmod.org/svn/Divmod/trunk/Nevow</code>, while the Quotient
|
|
trunk would be checked out using
|
|
<code>http://divmod.org/svn/Divmod/trunk/Quotient</code>.
|
|
|
|
<p>Now suppose we want to have an <code>SVNPoller</code> that only cares about
|
|
the Nevow trunk. This case looks just like the PROJECT/BRANCH layout
|
|
described earlier:
|
|
|
|
<pre class="example"> from buildbot.changes.svnpoller import SVNPoller
|
|
s = SVNPoller("http://divmod.org/svn/Divmod/trunk/Nevow")
|
|
c['sources'].append(ss)
|
|
</pre>
|
|
<p>But what happens when we want to track multiple Nevow branches? We
|
|
have to point our <code>svnurl=</code> high enough to see all those
|
|
branches, but we also don't want to include Quotient changes (since
|
|
we're only building Nevow). To accomplish this, we must rely upon the
|
|
<code>split_file</code> function to help us tell the difference between
|
|
files that belong to Nevow and those that belong to Quotient, as well
|
|
as figuring out which branch each one is on.
|
|
|
|
<pre class="example"> from buildbot.changes.svnpoller import SVNPoller
|
|
s = SVNPoller("http://divmod.org/svn/Divmod",
|
|
split_file=my_file_splitter)
|
|
c['sources'].append(ss)
|
|
</pre>
|
|
<p>The <code>my_file_splitter</code> function will be called with
|
|
repository-relative pathnames like:
|
|
|
|
<dl>
|
|
<dt><code>trunk/Nevow/formless/webform.py</code><dd>This is a Nevow file, on the trunk. We want the Change that includes this
|
|
to see a filename of <code>formless/webform.py"</code>, and a branch of None
|
|
|
|
<br><dt><code>branches/1.5.x/Nevow/formless/webform.py</code><dd>This is a Nevow file, on a branch. We want to get
|
|
branch=”branches/1.5.x” and filename=”formless/webform.py”.
|
|
|
|
<br><dt><code>trunk/Quotient/setup.py</code><dd>This is a Quotient file, so we want to ignore it by having
|
|
<code>my_file_splitter</code> return None.
|
|
|
|
<br><dt><code>branches/1.5.x/Quotient/setup.py</code><dd>This is also a Quotient file, which should be ignored.
|
|
</dl>
|
|
|
|
<p>The following definition for <code>my_file_splitter</code> will do the job:
|
|
|
|
<pre class="example"> def my_file_splitter(path):
|
|
pieces = path.split('/')
|
|
if pieces[0] == 'trunk':
|
|
branch = None
|
|
pieces.pop(0) # remove 'trunk'
|
|
elif pieces[0] == 'branches':
|
|
pieces.pop(0) # remove 'branches'
|
|
# grab branch name
|
|
branch = 'branches/' + pieces.pop(0)
|
|
else:
|
|
return None # something weird
|
|
projectname = pieces.pop(0)
|
|
if projectname != 'Nevow':
|
|
return None # wrong project
|
|
return (branch, '/'.join(pieces))
|
|
</pre>
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Build-Process"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Status-Delivery">Status Delivery</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="chapter">6 Build Process</h2>
|
|
|
|
<p>A <code>Build</code> object is responsible for actually performing a build.
|
|
It gets access to a remote <code>SlaveBuilder</code> where it may run
|
|
commands, and a <code>BuildStatus</code> object where it must emit status
|
|
events. The <code>Build</code> is created by the Builder's
|
|
<code>BuildFactory</code>.
|
|
|
|
<p>The default <code>Build</code> class is made up of a fixed sequence of
|
|
<code>BuildSteps</code>, executed one after another until all are complete
|
|
(or one of them indicates that the build should be halted early). The
|
|
default <code>BuildFactory</code> creates instances of this <code>Build</code>
|
|
class with a list of <code>BuildSteps</code>, so the basic way to configure
|
|
the build is to provide a list of <code>BuildSteps</code> to your
|
|
<code>BuildFactory</code>.
|
|
|
|
<p>More complicated <code>Build</code> subclasses can make other decisions:
|
|
execute some steps only if certain files were changed, or if certain
|
|
previous steps passed or failed. The base class has been written to
|
|
allow users to express basic control flow without writing code, but
|
|
you can always subclass and customize to achieve more specialized
|
|
behavior.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Build-Steps">Build Steps</a>
|
|
<li><a accesskey="2" href="#Interlocks">Interlocks</a>
|
|
<li><a accesskey="3" href="#Build-Factories">Build Factories</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Build-Steps"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Interlocks">Interlocks</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Build-Process">Build Process</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Process">Build Process</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">6.1 Build Steps</h3>
|
|
|
|
<p><code>BuildStep</code>s are usually specified in the buildmaster's
|
|
configuration file, in a list of “step specifications” that is used
|
|
to create the <code>BuildFactory</code>. These “step specifications” are
|
|
not actual steps, but rather a tuple of the <code>BuildStep</code> subclass
|
|
to be created and a dictionary of arguments. (the actual
|
|
<code>BuildStep</code> instances are not created until the Build is started,
|
|
so that each Build gets an independent copy of each BuildStep). The
|
|
preferred way to create these step specifications is with the
|
|
<code>BuildFactory</code>'s <code>addStep</code> method:
|
|
|
|
<pre class="example"> from buildbot.steps import source, shell
|
|
from buildbot.process import factory
|
|
|
|
f = factory.BuildFactory()
|
|
f.addStep(source.SVN, svnurl="http://svn.example.org/Trunk/")
|
|
f.addStep(shell.ShellCommand, command=["make", "all"])
|
|
f.addStep(shell.ShellCommand, command=["make", "test"])
|
|
</pre>
|
|
<p>The rest of this section lists all the standard BuildStep objects
|
|
available for use in a Build, and the parameters which can be used to
|
|
control each.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Common-Parameters">Common Parameters</a>
|
|
<li><a accesskey="2" href="#Source-Checkout">Source Checkout</a>
|
|
<li><a accesskey="3" href="#ShellCommand">ShellCommand</a>
|
|
<li><a accesskey="4" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>
|
|
<li><a accesskey="5" href="#Python-BuildSteps">Python BuildSteps</a>
|
|
<li><a accesskey="6" href="#Transferring-Files">Transferring Files</a>
|
|
<li><a accesskey="7" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Common-Parameters"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Source-Checkout">Source Checkout</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Build-Steps">Build Steps</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">6.1.1 Common Parameters</h4>
|
|
|
|
<p>The standard <code>Build</code> runs a series of <code>BuildStep</code>s in order,
|
|
only stopping when it runs out of steps or if one of them requests
|
|
that the build be halted. It collects status information from each one
|
|
to create an overall build status (of SUCCESS, WARNINGS, or FAILURE).
|
|
|
|
<p>All BuildSteps accept some common parameters. Some of these control
|
|
how their individual status affects the overall build. Others are used
|
|
to specify which <code>Locks</code> (see see <a href="#Interlocks">Interlocks</a>) should be
|
|
acquired before allowing the step to run.
|
|
|
|
<p>Arguments common to all <code>BuildStep</code> subclasses:
|
|
|
|
<dl>
|
|
<dt><code>name</code><dd>the name used to describe the step on the status display. It is also
|
|
used to give a name to any LogFiles created by this step.
|
|
|
|
<br><dt><code>haltOnFailure</code><dd>if True, a FAILURE of this build step will cause the build to halt
|
|
immediately with an overall result of FAILURE.
|
|
|
|
<br><dt><code>flunkOnWarnings</code><dd>when True, a WARNINGS or FAILURE of this build step will mark the
|
|
overall build as FAILURE. The remaining steps will still be executed.
|
|
|
|
<br><dt><code>flunkOnFailure</code><dd>when True, a FAILURE of this build step will mark the overall build as
|
|
a FAILURE. The remaining steps will still be executed.
|
|
|
|
<br><dt><code>warnOnWarnings</code><dd>when True, a WARNINGS or FAILURE of this build step will mark the
|
|
overall build as having WARNINGS. The remaining steps will still be
|
|
executed.
|
|
|
|
<br><dt><code>warnOnFailure</code><dd>when True, a FAILURE of this build step will mark the overall build as
|
|
having WARNINGS. The remaining steps will still be executed.
|
|
|
|
<br><dt><code>locks</code><dd>a list of Locks (instances of <code>buildbot.locks.SlaveLock</code> or
|
|
<code>buildbot.locks.MasterLock</code>) that should be acquired before
|
|
starting this Step. The Locks will be released when the step is
|
|
complete. Note that this is a list of actual Lock instances, not
|
|
names. Also note that all Locks must have unique names.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Source-Checkout"></a>
|
|
Next: <a rel="next" accesskey="n" href="#ShellCommand">ShellCommand</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Common-Parameters">Common Parameters</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">6.1.2 Source Checkout</h4>
|
|
|
|
<p>The first step of any build is typically to acquire the source code
|
|
from which the build will be performed. There are several classes to
|
|
handle this, one for each of the different source control system that
|
|
Buildbot knows about. For a description of how Buildbot treats source
|
|
control in general, see <a href="#Version-Control-Systems">Version Control Systems</a>.
|
|
|
|
<p>All source checkout steps accept some common parameters to control how
|
|
they get the sources and where they should be placed. The remaining
|
|
per-VC-system parameters are mostly to specify where exactly the
|
|
sources are coming from.
|
|
|
|
<dl>
|
|
<dt><code>mode</code><dd>
|
|
a string describing the kind of VC operation that is desired. Defaults
|
|
to <code>update</code>.
|
|
|
|
<dl>
|
|
<dt><code>update</code><dd>specifies that the CVS checkout/update should be performed directly
|
|
into the workdir. Each build is performed in the same directory,
|
|
allowing for incremental builds. This minimizes disk space, bandwidth,
|
|
and CPU time. However, it may encounter problems if the build process
|
|
does not handle dependencies properly (sometimes you must do a “clean
|
|
build” to make sure everything gets compiled), or if source files are
|
|
deleted but generated files can influence test behavior (e.g. python's
|
|
.pyc files), or when source directories are deleted but generated
|
|
files prevent CVS from removing them. Builds ought to be correct
|
|
regardless of whether they are done “from scratch” or incrementally,
|
|
but it is useful to test both kinds: this mode exercises the
|
|
incremental-build style.
|
|
|
|
<br><dt><code>copy</code><dd>specifies that the CVS workspace should be maintained in a separate
|
|
directory (called the 'copydir'), using checkout or update as
|
|
necessary. For each build, a new workdir is created with a copy of the
|
|
source tree (rm -rf workdir; cp -r copydir workdir). This doubles the
|
|
disk space required, but keeps the bandwidth low (update instead of a
|
|
full checkout). A full 'clean' build is performed each time. This
|
|
avoids any generated-file build problems, but is still occasionally
|
|
vulnerable to CVS problems such as a repository being manually
|
|
rearranged, causing CVS errors on update which are not an issue with a
|
|
full checkout.
|
|
|
|
<!-- TODO: something is screwy about this, revisit. Is it the source -->
|
|
<!-- directory or the working directory that is deleted each time? -->
|
|
<br><dt><code>clobber</code><dd>specifes that the working directory should be deleted each time,
|
|
necessitating a full checkout for each build. This insures a clean
|
|
build off a complete checkout, avoiding any of the problems described
|
|
above. This mode exercises the “from-scratch” build style.
|
|
|
|
<br><dt><code>export</code><dd>this is like <code>clobber</code>, except that the 'cvs export' command is
|
|
used to create the working directory. This command removes all CVS
|
|
metadata files (the CVS/ directories) from the tree, which is
|
|
sometimes useful for creating source tarballs (to avoid including the
|
|
metadata in the tar file).
|
|
</dl>
|
|
|
|
<br><dt><code>workdir</code><dd>like all Steps, this indicates the directory where the build will take
|
|
place. Source Steps are special in that they perform some operations
|
|
outside of the workdir (like creating the workdir itself).
|
|
|
|
<br><dt><code>alwaysUseLatest</code><dd>if True, bypass the usual “update to the last Change” behavior, and
|
|
always update to the latest changes instead.
|
|
|
|
<br><dt><code>retry</code><dd>If set, this specifies a tuple of <code>(delay, repeats)</code> which means
|
|
that when a full VC checkout fails, it should be retried up to
|
|
<var>repeats</var> times, waiting <var>delay</var> seconds between attempts. If
|
|
you don't provide this, it defaults to <code>None</code>, which means VC
|
|
operations should not be retried. This is provided to make life easier
|
|
for buildslaves which are stuck behind poor network connections.
|
|
|
|
</dl>
|
|
|
|
<p>My habit as a developer is to do a <code>cvs update</code> and <code>make</code> each
|
|
morning. Problems can occur, either because of bad code being checked in, or
|
|
by incomplete dependencies causing a partial rebuild to fail where a
|
|
complete from-scratch build might succeed. A quick Builder which emulates
|
|
this incremental-build behavior would use the <code>mode='update'</code>
|
|
setting.
|
|
|
|
<p>On the other hand, other kinds of dependency problems can cause a clean
|
|
build to fail where a partial build might succeed. This frequently results
|
|
from a link step that depends upon an object file that was removed from a
|
|
later version of the tree: in the partial tree, the object file is still
|
|
around (even though the Makefiles no longer know how to create it).
|
|
|
|
<p>“official” builds (traceable builds performed from a known set of
|
|
source revisions) are always done as clean builds, to make sure it is
|
|
not influenced by any uncontrolled factors (like leftover files from a
|
|
previous build). A “full” Builder which behaves this way would want
|
|
to use the <code>mode='clobber'</code> setting.
|
|
|
|
<p>Each VC system has a corresponding source checkout class: their
|
|
arguments are described on the following pages.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#CVS">CVS</a>
|
|
<li><a accesskey="2" href="#SVN">SVN</a>
|
|
<li><a accesskey="3" href="#Darcs">Darcs</a>
|
|
<li><a accesskey="4" href="#Mercurial">Mercurial</a>
|
|
<li><a accesskey="5" href="#Arch">Arch</a>
|
|
<li><a accesskey="6" href="#Bazaar">Bazaar</a>
|
|
<li><a accesskey="7" href="#P4">P4</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="CVS"></a>
|
|
Next: <a rel="next" accesskey="n" href="#SVN">SVN</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Source-Checkout">Source Checkout</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.2.1 CVS</h5>
|
|
|
|
<p><a name="index-CVS-Checkout-38"></a><a name="index-buildbot_002esteps_002esource_002eCVS-39"></a>
|
|
|
|
<p>The <code>CVS</code> build step performs a <a href="http://www.nongnu.org/cvs/">CVS</a> checkout or update. It takes the following arguments:
|
|
|
|
<dl>
|
|
<dt><code>cvsroot</code><dd>(required): specify the CVSROOT value, which points to a CVS
|
|
repository, probably on a remote machine. For example, the cvsroot
|
|
value you would use to get a copy of the Buildbot source code is
|
|
<code>:pserver:anonymous@cvs.sourceforge.net:/cvsroot/buildbot</code>
|
|
|
|
<br><dt><code>cvsmodule</code><dd>(required): specify the cvs <code>module</code>, which is generally a
|
|
subdirectory of the CVSROOT. The cvsmodule for the Buildbot source
|
|
code is <code>buildbot</code>.
|
|
|
|
<br><dt><code>branch</code><dd>a string which will be used in a <code>-r</code> argument. This is most
|
|
useful for specifying a branch to work on. Defaults to <code>HEAD</code>.
|
|
|
|
<br><dt><code>global_options</code><dd>a list of flags to be put before the verb in the CVS command.
|
|
|
|
<br><dt><code>checkoutDelay</code><dd>if set, the number of seconds to put between the timestamp of the last
|
|
known Change and the value used for the <code>-D</code> option. Defaults to
|
|
half of the parent Build's treeStableTimer.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="SVN"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Darcs">Darcs</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#CVS">CVS</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.2.2 SVN</h5>
|
|
|
|
<p><a name="index-SVN-Checkout-40"></a><a name="index-buildbot_002esteps_002esource_002eSVN-41"></a>
|
|
|
|
<p>The <code>SVN</code> build step performs a
|
|
<a href="http://subversion.tigris.org">Subversion</a> checkout or update.
|
|
There are two basic ways of setting up the checkout step, depending
|
|
upon whether you are using multiple branches or not.
|
|
|
|
<p>If all of your builds use the same branch, then you should create the
|
|
<code>SVN</code> step with the <code>svnurl</code> argument:
|
|
|
|
<dl>
|
|
<dt><code>svnurl</code><dd>(required): this specifies the <code>URL</code> argument that will be given
|
|
to the <code>svn checkout</code> command. It dictates both where the
|
|
repository is located and which sub-tree should be extracted. In this
|
|
respect, it is like a combination of the CVS <code>cvsroot</code> and
|
|
<code>cvsmodule</code> arguments. For example, if you are using a remote
|
|
Subversion repository which is accessible through HTTP at a URL of
|
|
<code>http://svn.example.com/repos</code>, and you wanted to check out the
|
|
<code>trunk/calc</code> sub-tree, you would use
|
|
<code>svnurl="http://svn.example.com/repos/trunk/calc"</code> as an argument
|
|
to your <code>SVN</code> step.
|
|
</dl>
|
|
|
|
<p>If, on the other hand, you are building from multiple branches, then
|
|
you should create the <code>SVN</code> step with the <code>baseURL</code> and
|
|
<code>defaultBranch</code> arguments instead:
|
|
|
|
<dl>
|
|
<dt><code>baseURL</code><dd>(required): this specifies the base repository URL, to which a branch
|
|
name will be appended. It should probably end in a slash.
|
|
|
|
<br><dt><code>defaultBranch</code><dd>this specifies the name of the branch to use when a Build does not
|
|
provide one of its own. This will be appended to <code>baseURL</code> to
|
|
create the string that will be passed to the <code>svn checkout</code>
|
|
command.
|
|
</dl>
|
|
|
|
<p>If you are using branches, you must also make sure your
|
|
<code>ChangeSource</code> will report the correct branch names.
|
|
|
|
<h3 class="heading">branch example</h3>
|
|
|
|
<p>Let's suppose that the “MyProject” repository uses branches for the
|
|
trunk, for various users' individual development efforts, and for
|
|
several new features that will require some amount of work (involving
|
|
multiple developers) before they are ready to merge onto the trunk.
|
|
Such a repository might be organized as follows:
|
|
|
|
<pre class="example"> svn://svn.example.org/MyProject/trunk
|
|
svn://svn.example.org/MyProject/branches/User1/foo
|
|
svn://svn.example.org/MyProject/branches/User1/bar
|
|
svn://svn.example.org/MyProject/branches/User2/baz
|
|
svn://svn.example.org/MyProject/features/newthing
|
|
svn://svn.example.org/MyProject/features/otherthing
|
|
</pre>
|
|
<p>Further assume that we want the Buildbot to run tests against the
|
|
trunk and against all the feature branches (i.e., do a
|
|
checkout/compile/build of branch X when a file has been changed on
|
|
branch X, when X is in the set [trunk, features/newthing,
|
|
features/otherthing]). We do not want the Buildbot to automatically
|
|
build any of the user branches, but it should be willing to build a
|
|
user branch when explicitly requested (most likely by the user who
|
|
owns that branch).
|
|
|
|
<p>There are three things that need to be set up to accomodate this
|
|
system. The first is a ChangeSource that is capable of identifying the
|
|
branch which owns any given file. This depends upon a user-supplied
|
|
function, in an external program that runs in the SVN commit hook and
|
|
connects to the buildmaster's <code>PBChangeSource</code> over a TCP
|
|
connection. (you can use the “<code>buildbot sendchange</code>” utility
|
|
for this purpose, but you will still need an external program to
|
|
decide what value should be passed to the <code>--branch=</code> argument).
|
|
For example, a change to a file with the SVN url of
|
|
“svn://svn.example.org/MyProject/features/newthing/src/foo.c” should
|
|
be broken down into a Change instance with
|
|
<code>branch='features/newthing'</code> and <code>file='src/foo.c'</code>.
|
|
|
|
<p>The second piece is an <code>AnyBranchScheduler</code> which will pay
|
|
attention to the desired branches. It will not pay attention to the
|
|
user branches, so it will not automatically start builds in response
|
|
to changes there. The AnyBranchScheduler class requires you to
|
|
explicitly list all the branches you want it to use, but it would not
|
|
be difficult to write a subclass which used
|
|
<code>branch.startswith('features/'</code> to remove the need for this
|
|
explicit list. Or, if you want to build user branches too, you can use
|
|
AnyBranchScheduler with <code>branches=None</code> to indicate that you want
|
|
it to pay attention to all branches.
|
|
|
|
<p>The third piece is an <code>SVN</code> checkout step that is configured to
|
|
handle the branches correctly, with a <code>baseURL</code> value that
|
|
matches the way the ChangeSource splits each file's URL into base,
|
|
branch, and file.
|
|
|
|
<pre class="example"> from buildbot.changes.pb import PBChangeSource
|
|
from buildbot.scheduler import AnyBranchScheduler
|
|
from buildbot.process import source, factory
|
|
from buildbot.steps import source, shell
|
|
|
|
c['sources'] = [PBChangeSource()]
|
|
s1 = AnyBranchScheduler('main',
|
|
['trunk', 'features/newthing', 'features/otherthing'],
|
|
10*60, ['test-i386', 'test-ppc'])
|
|
c['schedulers'] = [s1]
|
|
|
|
f = factory.BuildFactory()
|
|
f.addStep(source.SVN, mode='update',
|
|
baseURL='svn://svn.example.org/MyProject/',
|
|
defaultBranch='trunk')
|
|
f.addStep(shell.Compile, command="make all")
|
|
f.addStep(shell.Test, command="make test")
|
|
|
|
c['builders'] = [
|
|
{'name':'test-i386', 'slavename':'bot-i386', 'builddir':'test-i386',
|
|
'factory':f },
|
|
{'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc',
|
|
'factory':f },
|
|
]
|
|
</pre>
|
|
<p>In this example, when a change arrives with a <code>branch</code> attribute
|
|
of “trunk”, the resulting build will have an SVN step that
|
|
concatenates “svn://svn.example.org/MyProject/” (the baseURL) with
|
|
“trunk” (the branch name) to get the correct svn command. If the
|
|
“newthing” branch has a change to “src/foo.c”, then the SVN step
|
|
will concatenate “svn://svn.example.org/MyProject/” with
|
|
“features/newthing” to get the svnurl for checkout.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Darcs"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Mercurial">Mercurial</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#SVN">SVN</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.2.3 Darcs</h5>
|
|
|
|
<p><a name="index-Darcs-Checkout-42"></a><a name="index-buildbot_002esteps_002esource_002eDarcs-43"></a>
|
|
|
|
<p>The <code>Darcs</code> build step performs a
|
|
<a href="http://abridgegame.org/darcs/">Darcs</a> checkout or update.
|
|
|
|
<p>Like See <a href="#SVN">SVN</a>, this step can either be configured to always check
|
|
out a specific tree, or set up to pull from a particular branch that
|
|
gets specified separately for each build. Also like SVN, the
|
|
repository URL given to Darcs is created by concatenating a
|
|
<code>baseURL</code> with the branch name, and if no particular branch is
|
|
requested, it uses a <code>defaultBranch</code>. The only difference in
|
|
usage is that each potential Darcs repository URL must point to a
|
|
fully-fledged repository, whereas SVN URLs usually point to sub-trees
|
|
of the main Subversion repository. In other words, doing an SVN
|
|
checkout of <code>baseURL</code> is legal, but silly, since you'd probably
|
|
wind up with a copy of every single branch in the whole repository.
|
|
Doing a Darcs checkout of <code>baseURL</code> is just plain wrong, since
|
|
the parent directory of a collection of Darcs repositories is not
|
|
itself a valid repository.
|
|
|
|
<p>The Darcs step takes the following arguments:
|
|
|
|
<dl>
|
|
<dt><code>repourl</code><dd>(required unless <code>baseURL</code> is provided): the URL at which the
|
|
Darcs source repository is available.
|
|
|
|
<br><dt><code>baseURL</code><dd>(required unless <code>repourl</code> is provided): the base repository URL,
|
|
to which a branch name will be appended. It should probably end in a
|
|
slash.
|
|
|
|
<br><dt><code>defaultBranch</code><dd>(allowed if and only if <code>baseURL</code> is provided): this specifies
|
|
the name of the branch to use when a Build does not provide one of its
|
|
own. This will be appended to <code>baseURL</code> to create the string that
|
|
will be passed to the <code>darcs get</code> command.
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Mercurial"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Arch">Arch</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Darcs">Darcs</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.2.4 Mercurial</h5>
|
|
|
|
<p><a name="index-Mercurial-Checkout-44"></a><a name="index-buildbot_002esteps_002esource_002eMercurial-45"></a>
|
|
|
|
<p>The <code>Mercurial</code> build step performs a
|
|
<a href="http://selenic.com/mercurial">Mercurial</a> (aka “hg”) checkout
|
|
or update.
|
|
|
|
<p>Branches are handled just like See <a href="#Darcs">Darcs</a>.
|
|
|
|
<p>The Mercurial step takes the following arguments:
|
|
|
|
<dl>
|
|
<dt><code>repourl</code><dd>(required unless <code>baseURL</code> is provided): the URL at which the
|
|
Mercurial source repository is available.
|
|
|
|
<br><dt><code>baseURL</code><dd>(required unless <code>repourl</code> is provided): the base repository URL,
|
|
to which a branch name will be appended. It should probably end in a
|
|
slash.
|
|
|
|
<br><dt><code>defaultBranch</code><dd>(allowed if and only if <code>baseURL</code> is provided): this specifies
|
|
the name of the branch to use when a Build does not provide one of its
|
|
own. This will be appended to <code>baseURL</code> to create the string that
|
|
will be passed to the <code>hg clone</code> command.
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Arch"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Bazaar">Bazaar</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Mercurial">Mercurial</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.2.5 Arch</h5>
|
|
|
|
<p><a name="index-Arch-Checkout-46"></a><a name="index-buildbot_002esteps_002esource_002eArch-47"></a>
|
|
|
|
<p>The <code>Arch</code> build step performs an <a href="http://gnuarch.org/">Arch</a> checkout or update using the <code>tla</code> client. It takes the
|
|
following arguments:
|
|
|
|
<dl>
|
|
<dt><code>url</code><dd>(required): this specifies the URL at which the Arch source archive is
|
|
available.
|
|
|
|
<br><dt><code>version</code><dd>(required): this specifies which “development line” (like a branch)
|
|
should be used. This provides the default branch name, but individual
|
|
builds may specify a different one.
|
|
|
|
<br><dt><code>archive</code><dd>(optional): Each repository knows its own archive name. If this
|
|
parameter is provided, it must match the repository's archive name.
|
|
The parameter is accepted for compatibility with the <code>Bazaar</code>
|
|
step, below.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Bazaar"></a>
|
|
Next: <a rel="next" accesskey="n" href="#P4">P4</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Arch">Arch</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.2.6 Bazaar</h5>
|
|
|
|
<p><a name="index-Bazaar-Checkout-48"></a><a name="index-buildbot_002esteps_002esource_002eBazaar-49"></a>
|
|
|
|
<p><code>Bazaar</code> is an alternate implementation of the Arch VC system,
|
|
which uses a client named <code>baz</code>. The checkout semantics are just
|
|
different enough from <code>tla</code> that there is a separate BuildStep for
|
|
it.
|
|
|
|
<p>It takes exactly the same arguments as <code>Arch</code>, except that the
|
|
<code>archive=</code> parameter is required. (baz does not emit the archive
|
|
name when you do <code>baz register-archive</code>, so we must provide it
|
|
ourselves).
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="P4"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Bazaar">Bazaar</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.2.7 P4</h5>
|
|
|
|
<p><a name="index-Perforce-Update-50"></a><a name="index-buildbot_002esteps_002esource_002eP4-51"></a><!-- TODO @bsindex buildbot.steps.source.P4Sync -->
|
|
|
|
<p>The <code>P4</code> build step creates a <a href="http://www.perforce.com/">Perforce</a> client specification and performs an update.
|
|
|
|
<dl>
|
|
<dt><code>p4base</code><dd>A view into the Perforce depot without branch name or trailing "...".
|
|
Typically "//depot/proj/".
|
|
<br><dt><code>defaultBranch</code><dd>A branch name to append on build requests if none is specified.
|
|
Typically "trunk".
|
|
<br><dt><code>p4port</code><dd>(optional): the host:port string describing how to get to the P4 Depot
|
|
(repository), used as the -p argument for all p4 commands.
|
|
<br><dt><code>p4user</code><dd>(optional): the Perforce user, used as the -u argument to all p4
|
|
commands.
|
|
<br><dt><code>p4passwd</code><dd>(optional): the Perforce password, used as the -p argument to all p4
|
|
commands.
|
|
<br><dt><code>p4extra_views</code><dd>(optional): a list of (depotpath, clientpath) tuples containing extra
|
|
views to be mapped into the client specification. Both will have
|
|
"/..." appended automatically. The client name and source directory
|
|
will be prepended to the client path.
|
|
<br><dt><code>p4client</code><dd>(optional): The name of the client to use. In mode='copy' and
|
|
mode='update', it's particularly important that a unique name is used
|
|
for each checkout directory to avoid incorrect synchronization. For
|
|
this reason, Python percent substitution will be performed on this value
|
|
to replace %(slave)s with the slave name and %(builder)s with the
|
|
builder name. The default is "buildbot_%(slave)s_%(build)s".
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="ShellCommand"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Source-Checkout">Source Checkout</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">6.1.3 ShellCommand</h4>
|
|
|
|
<p><a name="index-buildbot_002esteps_002eshell_002eShellCommand-52"></a><!-- TODO @bsindex buildbot.steps.shell.TreeSize -->
|
|
|
|
<p>This is a useful base class for just about everything you might want
|
|
to do during a build (except for the initial source checkout). It runs
|
|
a single command in a child shell on the buildslave. All stdout/stderr
|
|
is recorded into a LogFile. The step finishes with a status of FAILURE
|
|
if the command's exit code is non-zero, otherwise it has a status of
|
|
SUCCESS.
|
|
|
|
<p>The preferred way to specify the command is with a list of argv strings,
|
|
since this allows for spaces in filenames and avoids doing any fragile
|
|
shell-escaping. You can also specify the command with a single string, in
|
|
which case the string is given to '/bin/sh -c COMMAND' for parsing.
|
|
|
|
<p>All ShellCommands are run by default in the “workdir”, which
|
|
defaults to the “<samp><span class="file">build</span></samp>” subdirectory of the slave builder's
|
|
base directory. The absolute path of the workdir will thus be the
|
|
slave's basedir (set as an option to <code>buildbot create-slave</code>,
|
|
see <a href="#Creating-a-buildslave">Creating a buildslave</a>) plus the builder's basedir (set in the
|
|
builder's <code>c['builddir']</code> key in master.cfg) plus the workdir
|
|
itself (a class-level attribute of the BuildFactory, defaults to
|
|
“<samp><span class="file">build</span></samp>”).
|
|
|
|
<p><code>ShellCommand</code> arguments:
|
|
|
|
<dl>
|
|
<dt><code>command</code><dd>a list of strings (preferred) or single string (discouraged) which
|
|
specifies the command to be run. A list of strings is preferred
|
|
because it can be used directly as an argv array. Using a single
|
|
string (with embedded spaces) requires the buildslave to pass the
|
|
string to /bin/sh for interpretation, which raises all sorts of
|
|
difficult questions about how to escape or interpret shell
|
|
metacharacters.
|
|
|
|
<br><dt><code>env</code><dd>a dictionary of environment strings which will be added to the child
|
|
command's environment. For example, to run tests with a different i18n
|
|
language setting, you might use
|
|
|
|
<pre class="example"> f.addStep(ShellCommand, command=["make", "test"],
|
|
env={'LANG': 'fr_FR'})
|
|
</pre>
|
|
<p>These variable settings will override any existing ones in the
|
|
buildslave's environment. The exception is PYTHONPATH, which is merged
|
|
with (actually prepended to) any existing $PYTHONPATH setting. The
|
|
value is treated as a list of directories to prepend, and a single
|
|
string is treated like a one-item list. For example, to prepend both
|
|
<samp><span class="file">/usr/local/lib/python2.3</span></samp> and <samp><span class="file">/home/buildbot/lib/python</span></samp>
|
|
to any existing $PYTHONPATH setting, you would do something like the
|
|
following:
|
|
|
|
<pre class="example"> f.addStep(ShellCommand, command=["make", "test"],
|
|
env={'PYTHONPATH': ["/usr/local/lib/python2.3",
|
|
"/home/buildbot/lib/python"] })
|
|
</pre>
|
|
<br><dt><code>want_stdout</code><dd>if False, stdout from the child process is discarded rather than being
|
|
sent to the buildmaster for inclusion in the step's LogFile.
|
|
|
|
<br><dt><code>want_stderr</code><dd>like <code>want_stdout</code> but for stderr. Note that commands run through
|
|
a PTY do not have separate stdout/stderr streams: both are merged into
|
|
stdout.
|
|
|
|
<br><dt><code>logfiles</code><dd>Sometimes commands will log interesting data to a local file, rather
|
|
than emitting everything to stdout or stderr. For example, Twisted's
|
|
“trial” command (which runs unit tests) only presents summary
|
|
information to stdout, and puts the rest into a file named
|
|
<samp><span class="file">_trial_temp/test.log</span></samp>. It is often useful to watch these files
|
|
as the command runs, rather than using <samp><span class="command">/bin/cat</span></samp> to dump
|
|
their contents afterwards.
|
|
|
|
<p>The <code>logfiles=</code> argument allows you to collect data from these
|
|
secondary logfiles in near-real-time, as the step is running. It
|
|
accepts a dictionary which maps from a local Log name (which is how
|
|
the log data is presented in the build results) to a remote filename
|
|
(interpreted relative to the build's working directory). Each named
|
|
file will be polled on a regular basis (every couple of seconds) as
|
|
the build runs, and any new text will be sent over to the buildmaster.
|
|
|
|
<pre class="example"> f.addStep(ShellCommand, command=["make", "test"],
|
|
logfiles={"triallog": "_trial_temp/test.log"})
|
|
</pre>
|
|
<br><dt><code>timeout</code><dd>if the command fails to produce any output for this many seconds, it
|
|
is assumed to be locked up and will be killed.
|
|
|
|
<br><dt><code>description</code><dd>This will be used to describe the command (on the Waterfall display)
|
|
while the command is still running. It should be a single
|
|
imperfect-tense verb, like “compiling” or “testing”. The preferred
|
|
form is a list of short strings, which allows the HTML Waterfall
|
|
display to create narrower columns by emitting a <br> tag between each
|
|
word. You may also provide a single string.
|
|
|
|
<br><dt><code>descriptionDone</code><dd>This will be used to describe the command once it has finished. A
|
|
simple noun like “compile” or “tests” should be used. Like
|
|
<code>description</code>, this may either be a list of short strings or a
|
|
single string.
|
|
|
|
<p>If neither <code>description</code> nor <code>descriptionDone</code> are set, the
|
|
actual command arguments will be used to construct the description.
|
|
This may be a bit too wide to fit comfortably on the Waterfall
|
|
display.
|
|
|
|
<pre class="example"> f.addStep(ShellCommand, command=["make", "test"],
|
|
description=["testing"],
|
|
descriptionDone=["tests"])
|
|
</pre>
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Simple-ShellCommand-Subclasses"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Python-BuildSteps">Python BuildSteps</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#ShellCommand">ShellCommand</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">6.1.4 Simple ShellCommand Subclasses</h4>
|
|
|
|
<p>Several subclasses of ShellCommand are provided as starting points for
|
|
common build steps. These are all very simple: they just override a few
|
|
parameters so you don't have to specify them yourself, making the master.cfg
|
|
file less verbose.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Configure">Configure</a>
|
|
<li><a accesskey="2" href="#Compile">Compile</a>
|
|
<li><a accesskey="3" href="#Test">Test</a>
|
|
<li><a accesskey="4" href="#Build-Properties">Build Properties</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Configure"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Compile">Compile</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.4.1 Configure</h5>
|
|
|
|
<p><a name="index-buildbot_002esteps_002eshell_002eConfigure-53"></a>
|
|
This is intended to handle the <code>./configure</code> step from
|
|
autoconf-style projects, or the <code>perl Makefile.PL</code> step from perl
|
|
MakeMaker.pm-style modules. The default command is <code>./configure</code>
|
|
but you can change this by providing a <code>command=</code> parameter.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Compile"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Test">Test</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Configure">Configure</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.4.2 Compile</h5>
|
|
|
|
<p><a name="index-buildbot_002esteps_002eshell_002eCompile-54"></a>
|
|
This is meant to handle compiling or building a project written in C. The
|
|
default command is <code>make all</code>. When the compile is finished, the
|
|
log file is scanned for GCC error/warning messages and a summary log is
|
|
created with any problems that were seen (TODO: the summary is not yet
|
|
created).
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Test"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Build-Properties">Build Properties</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Compile">Compile</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.4.3 Test</h5>
|
|
|
|
<p><a name="index-buildbot_002esteps_002eshell_002eTest-55"></a>
|
|
This is meant to handle unit tests. The default command is <code>make
|
|
test</code>, and the <code>warnOnFailure</code> flag is set.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Build-Properties"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Test">Test</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.4.4 Build Properties</h5>
|
|
|
|
<p><a name="index-build-properties-56"></a>
|
|
Each build has a set of “Build Properties”, which can be used by its
|
|
BuildStep to modify their actions. For example, the SVN revision
|
|
number of the source code being built is available as a build
|
|
property, and a ShellCommand step could incorporate this number into a
|
|
command which create a numbered release tarball.
|
|
|
|
<p>Some build properties are set when the build starts, such as the
|
|
SourceStamp information. Other properties can be set by BuildSteps as
|
|
they run, for example the various Source steps will set the
|
|
<code>got_revision</code> property to the source revision that was actually
|
|
checked out (which can be useful when the SourceStamp in use merely
|
|
requested the “latest revision”: <code>got_revision</code> will tell you
|
|
what was actually built).
|
|
|
|
<p>In custom BuildSteps, you can get and set the build properties with
|
|
the <code>getProperty</code>/<code>setProperty</code> methods. Each takes a string
|
|
for the name of the property, and returns or accepts an
|
|
arbitrary<a rel="footnote" href="#fn-7" name="fnd-7"><sup>7</sup></a> object. For example:
|
|
|
|
<pre class="example"> class MakeTarball(ShellCommand):
|
|
def start(self):
|
|
self.setCommand(["tar", "czf",
|
|
"build-%s.tar.gz" % self.getProperty("revision"),
|
|
"source"])
|
|
ShellCommand.start(self)
|
|
</pre>
|
|
<p><a name="index-WithProperties-57"></a>
|
|
You can use build properties in ShellCommands by using the
|
|
<code>WithProperties</code> wrapper when setting the arguments of the
|
|
ShellCommand. This interpolates the named build properties into the
|
|
generated shell command.
|
|
|
|
<pre class="example"> from buildbot.steps.shell import ShellCommand, WithProperties
|
|
|
|
f.addStep(ShellCommand,
|
|
command=["tar", "czf",
|
|
WithProperties("build-%s.tar.gz", "revision"),
|
|
"source"])
|
|
</pre>
|
|
<p>If this BuildStep were used in a tree obtained from Subversion, it
|
|
would create a tarball with a name like <samp><span class="file">build-1234.tar.gz</span></samp>.
|
|
|
|
<p>The <code>WithProperties</code> function does <code>printf</code>-style string
|
|
interpolation, using strings obtained by calling
|
|
<code>build.getProperty(propname)</code>. Note that for every <code>%s</code> (or
|
|
<code>%d</code>, etc), you must have exactly one additional argument to
|
|
indicate which build property you want to insert.
|
|
|
|
<p>You can also use python dictionary-style string interpolation by using
|
|
the <code>%(propname)s</code> syntax. In this form, the property name goes
|
|
in the parentheses, and WithProperties takes <em>no</em> additional
|
|
arguments:
|
|
|
|
<pre class="example"> f.addStep(ShellCommand,
|
|
command=["tar", "czf",
|
|
WithProperties("build-%(revision)s.tar.gz"),
|
|
"source"])
|
|
</pre>
|
|
<p>Don't forget the extra “s” after the closing parenthesis! This is
|
|
the cause of many confusing errors. Also note that you can only use
|
|
WithProperties in the list form of the command= definition. You cannot
|
|
currently use it in the (discouraged) <code>command="stuff"</code>
|
|
single-string form. However, you can use something like
|
|
<code>command=["/bin/sh", "-c", "stuff", WithProperties(stuff)]</code> to
|
|
use both shell expansion and WithProperties interpolation.
|
|
|
|
<p>Note that, like python, you can either do positional-argument
|
|
interpolation <em>or</em> keyword-argument interpolation, not both. Thus
|
|
you cannot use a string like
|
|
<code>WithProperties("foo-%(revision)s-%s", "branch")</code>.
|
|
|
|
<p>At the moment, the only way to set build properties is by writing a
|
|
custom BuildStep.
|
|
|
|
<h3 class="heading">Common Build Properties</h3>
|
|
|
|
<p>The following build properties are set when the build is started, and
|
|
are available to all steps.
|
|
|
|
<dl>
|
|
<dt><code>branch</code><dd>
|
|
This comes from the build's SourceStamp, and describes which branch is
|
|
being checked out. This will be <code>None</code> (which interpolates into
|
|
<code>WithProperties</code> as an empty string) if the build is on the
|
|
default branch, which is generally the trunk. Otherwise it will be a
|
|
string like “branches/beta1.4”. The exact syntax depends upon the VC
|
|
system being used.
|
|
|
|
<br><dt><code>revision</code><dd>
|
|
This also comes from the SourceStamp, and is the revision of the
|
|
source code tree that was requested from the VC system. When a build
|
|
is requested of a specific revision (as is generally the case when the
|
|
build is triggered by Changes), this will contain the revision
|
|
specification. The syntax depends upon the VC system in use: for SVN
|
|
it is an integer, for Mercurial it is a short string, for Darcs it is
|
|
a rather large string, etc.
|
|
|
|
<p>If the “force build” button was pressed, the revision will be
|
|
<code>None</code>, which means to use the most recent revision available.
|
|
This is a “trunk build”. This will be interpolated as an empty
|
|
string.
|
|
|
|
<br><dt><code>got_revision</code><dd>
|
|
This is set when a Source step checks out the source tree, and
|
|
provides the revision that was actually obtained from the VC system.
|
|
In general this should be the same as <code>revision</code>, except for
|
|
trunk builds, where <code>got_revision</code> indicates what revision was
|
|
current when the checkout was performed. This can be used to rebuild
|
|
the same source code later.
|
|
|
|
<p>Note that for some VC systems (Darcs in particular), the revision is a
|
|
large string containing newlines, and is not suitable for
|
|
interpolation into a filename.
|
|
|
|
<br><dt><code>buildername</code><dd>
|
|
This is a string that indicates which Builder the build was a part of.
|
|
The combination of buildername and buildnumber uniquely identify a
|
|
build.
|
|
|
|
<br><dt><code>buildnumber</code><dd>
|
|
Each build gets a number, scoped to the Builder (so the first build
|
|
performed on any given Builder will have a build number of 0). This
|
|
integer property contains the build's number.
|
|
|
|
<br><dt><code>slavename</code><dd>
|
|
This is a string which identifies which buildslave the build is
|
|
running on.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Python-BuildSteps"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Transferring-Files">Transferring Files</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">6.1.5 Python BuildSteps</h4>
|
|
|
|
<p>Here are some BuildSteps that are specifcally useful for projects
|
|
implemented in Python.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#BuildEPYDoc">BuildEPYDoc</a>
|
|
<li><a accesskey="2" href="#PyFlakes">PyFlakes</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="BuildEPYDoc"></a>
|
|
Next: <a rel="next" accesskey="n" href="#PyFlakes">PyFlakes</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Python-BuildSteps">Python BuildSteps</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Python-BuildSteps">Python BuildSteps</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.5.1 BuildEPYDoc</h5>
|
|
|
|
<p><a name="index-buildbot_002esteps_002epython_002eBuildEPYDoc-58"></a>
|
|
<a href="http://epydoc.sourceforge.net/">epydoc</a> is a tool for generating
|
|
API documentation for Python modules from their docstrings. It reads
|
|
all the .py files from your source tree, processes the docstrings
|
|
therein, and creates a large tree of .html files (or a single .pdf
|
|
file).
|
|
|
|
<p>The <code>buildbot.steps.python.BuildEPYDoc</code> step will run
|
|
<samp><span class="command">epydoc</span></samp> to produce this API documentation, and will count the
|
|
errors and warnings from its output.
|
|
|
|
<p>You must supply the command line to be used. The default is
|
|
<samp><span class="command">make epydocs</span></samp>, which assumes that your project has a Makefile
|
|
with an “epydocs” target. You might wish to use something like
|
|
<samp><span class="command">epydoc -o apiref source/PKGNAME</span></samp> instead. You might also want
|
|
to add <samp><span class="command">--pdf</span></samp> to generate a PDF file instead of a large tree
|
|
of HTML files.
|
|
|
|
<p>The API docs are generated in-place in the build tree (under the
|
|
workdir, in the subdirectory controlled by the “-o” argument). To
|
|
make them useful, you will probably have to copy them to somewhere
|
|
they can be read. A command like <samp><span class="command">rsync -ad apiref/
|
|
dev.example.com:~public_html/current-apiref/</span></samp> might be useful. You
|
|
might instead want to bundle them into a tarball and publish it in the
|
|
same place where the generated install tarball is placed.
|
|
|
|
<pre class="example"> from buildbot.steps.python import BuildEPYDoc
|
|
|
|
...
|
|
f.addStep(BuildEPYDoc, command=["epydoc", "-o", "apiref", "source/mypkg"])
|
|
</pre>
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="PyFlakes"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#BuildEPYDoc">BuildEPYDoc</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Python-BuildSteps">Python BuildSteps</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.5.2 PyFlakes</h5>
|
|
|
|
<p><a name="index-buildbot_002esteps_002epython_002ePyFlakes-59"></a>
|
|
<a href="http://divmod.org/trac/wiki/DivmodPyflakes">PyFlakes</a> is a tool
|
|
to perform basic static analysis of Python code to look for simple
|
|
errors, like missing imports and references of undefined names. It is
|
|
like a fast and simple form of the C “lint” program. Other tools
|
|
(like pychecker) provide more detailed results but take longer to run.
|
|
|
|
<p>The <code>buildbot.steps.python.PyFlakes</code> step will run pyflakes and
|
|
count the various kinds of errors and warnings it detects.
|
|
|
|
<p>You must supply the command line to be used. The default is
|
|
<samp><span class="command">make pyflakes</span></samp>, which assumes you have a top-level Makefile
|
|
with a “pyflakes” target. You might want to use something like
|
|
<samp><span class="command">pyflakes .</span></samp> or <samp><span class="command">pyflakes src</span></samp>.
|
|
|
|
<pre class="example"> from buildbot.steps.python import PyFlakes
|
|
|
|
...
|
|
f.addStep(PyFlakes, command=["pyflakes", "src"])
|
|
</pre>
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Transferring-Files"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Python-BuildSteps">Python BuildSteps</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">6.1.6 Transferring Files</h4>
|
|
|
|
<p><a name="index-File-Transfer-60"></a><a name="index-buildbot_002esteps_002etransfer_002eFileUpload-61"></a><a name="index-buildbot_002esteps_002etransfer_002eFileDownload-62"></a>
|
|
Most of the work involved in a build will take place on the
|
|
buildslave. But occasionally it is useful to do some work on the
|
|
buildmaster side. The most basic way to involve the buildmaster is
|
|
simply to move a file from the slave to the master, or vice versa.
|
|
There are a pair of BuildSteps named <code>FileUpload</code> and
|
|
<code>FileDownload</code> to provide this functionality. <code>FileUpload</code>
|
|
moves a file <em>up to</em> the master, while <code>FileDownload</code> moves
|
|
a file <em>down from</em> the master.
|
|
|
|
<p>As an example, let's assume that there is a step which produces an
|
|
HTML file within the source tree that contains some sort of generated
|
|
project documentation. We want to move this file to the buildmaster,
|
|
into a <samp><span class="file">~/public_html</span></samp> directory, so it can be visible to
|
|
developers. This file will wind up in the slave-side working directory
|
|
under the name <samp><span class="file">docs/reference.html</span></samp>. We want to put it into the
|
|
master-side <samp><span class="file">~/public_html/ref.html</span></samp>.
|
|
|
|
<pre class="example"> from buildbot.steps.shell import ShellCommand
|
|
from buildbot.steps.transfer import FileUpload
|
|
|
|
f.addStep(ShellCommand, command=["make", "docs"])
|
|
f.addStep(FileUpload,
|
|
slavesrc="docs/reference.html",
|
|
masterdest="~/public_html/ref.html")
|
|
</pre>
|
|
<p>The <code>masterdest=</code> argument will be passed to os.path.expanduser,
|
|
so things like “~” will be expanded properly. Non-absolute paths
|
|
will be interpreted relative to the buildmaster's base directory.
|
|
Likewise, the <code>slavesrc=</code> argument will be expanded and
|
|
interpreted relative to the builder's working directory.
|
|
|
|
<p>To move a file from the master to the slave, use the
|
|
<code>FileDownload</code> command. For example, let's assume that some step
|
|
requires a configuration file that, for whatever reason, could not be
|
|
recorded in the source code repository or generated on the buildslave
|
|
side:
|
|
|
|
<pre class="example"> from buildbot.steps.shell import ShellCommand
|
|
from buildbot.steps.transfer import FileUpload
|
|
|
|
f.addStep(FileDownload
|
|
mastersrc="~/todays_build_config.txt",
|
|
slavedest="build_config.txt")
|
|
f.addStep(ShellCommand, command=["make", "config"])
|
|
</pre>
|
|
<p>Like <code>FileUpload</code>, the <code>mastersrc=</code> argument is interpreted
|
|
relative to the buildmaster's base directory, and the
|
|
<code>slavedest=</code> argument is relative to the builder's working
|
|
directory. If the buildslave is running in <samp><span class="file">~buildslave</span></samp>, and the
|
|
builder's “builddir” is something like <samp><span class="file">tests-i386</span></samp>, then the
|
|
workdir is going to be <samp><span class="file">~buildslave/tests-i386/build</span></samp>, and a
|
|
<code>slavedest=</code> of <samp><span class="file">foo/bar.html</span></samp> will get put in
|
|
<samp><span class="file">~buildslave/tests-i386/build/foo/bar.html</span></samp>. Remember that
|
|
neither of these commands will create missing directories for you.
|
|
|
|
<h4 class="subheading">Other Parameters</h4>
|
|
|
|
<p>The <code>maxsize=</code> argument lets you set a maximum size for the file
|
|
to be transferred. This may help to avoid surprises: transferring a
|
|
100MB coredump when you were expecting to move a 10kB status file
|
|
might take an awfully long time. The <code>blocksize=</code> argument
|
|
controls how the file is sent over the network: larger blocksizes are
|
|
slightly more efficient but also consume more memory on each end, and
|
|
there is a hard-coded limit of about 640kB.
|
|
|
|
<p>The <code>mode=</code> argument allows you to control the access permissions
|
|
of the target file, traditionally expressed as an octal integer. The
|
|
most common value is probably 0755, which sets the “x” executable
|
|
bit on the file (useful for shell scripts and the like). The default
|
|
value for <code>mode=</code> is None, which means the permission bits will
|
|
default to whatever the umask of the writing process is. The default
|
|
umask tends to be fairly restrictive, but at least on the buildslave
|
|
you can make it less restrictive with a –umask command-line option at
|
|
creation time (see <a href="#Buildslave-Options">Buildslave Options</a>).
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Writing-New-BuildSteps"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Transferring-Files">Transferring Files</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">6.1.7 Writing New BuildSteps</h4>
|
|
|
|
<p>While it is a good idea to keep your build process self-contained in
|
|
the source code tree, sometimes it is convenient to put more
|
|
intelligence into your Buildbot configuration. One was to do this is
|
|
to write a custom BuildStep. Once written, this Step can be used in
|
|
the <samp><span class="file">master.cfg</span></samp> file.
|
|
|
|
<p>The best reason for writing a custom BuildStep is to better parse the
|
|
results of the command being run. For example, a BuildStep that knows
|
|
about JUnit could look at the logfiles to determine which tests had
|
|
been run, how many passed and how many failed, and then report more
|
|
detailed information than a simple <code>rc==0</code> -based “good/bad”
|
|
decision.
|
|
|
|
<p>TODO: add more description of BuildSteps.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#BuildStep-LogFiles">BuildStep LogFiles</a>
|
|
<li><a accesskey="2" href="#Adding-LogObservers">Adding LogObservers</a>
|
|
<li><a accesskey="3" href="#BuildStep-URLs">BuildStep URLs</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="BuildStep-LogFiles"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Adding-LogObservers">Adding LogObservers</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.7.1 BuildStep LogFiles</h5>
|
|
|
|
<p>Each BuildStep has a collection of “logfiles”. Each one has a short
|
|
name, like “stdio” or “warnings”. Each LogFile contains an
|
|
arbitrary amount of text, usually the contents of some output file
|
|
generated during a build or test step, or a record of everything that
|
|
was printed to stdout/stderr during the execution of some command.
|
|
|
|
<p>These LogFiles are stored to disk, so they can be retrieved later.
|
|
|
|
<p>Each can contain multiple “channels”, generally limited to three
|
|
basic ones: stdout, stderr, and “headers”. For example, when a
|
|
ShellCommand runs, it writes a few lines to the “headers” channel to
|
|
indicate the exact argv strings being run, which directory the command
|
|
is being executed in, and the contents of the current environment
|
|
variables. Then, as the command runs, it adds a lot of “stdout” and
|
|
“stderr” messages. When the command finishes, a final “header”
|
|
line is added with the exit code of the process.
|
|
|
|
<p>Status display plugins can format these different channels in
|
|
different ways. For example, the web page shows LogFiles as text/html,
|
|
with header lines in blue text, stdout in black, and stderr in red. A
|
|
different URL is available which provides a text/plain format, in
|
|
which stdout and stderr are collapsed together, and header lines are
|
|
stripped completely. This latter option makes it easy to save the
|
|
results to a file and run <samp><span class="command">grep</span></samp> or whatever against the
|
|
output.
|
|
|
|
<p>Each BuildStep contains a mapping (implemented in a python dictionary)
|
|
from LogFile name to the actual LogFile objects. Status plugins can
|
|
get a list of LogFiles to display, for example, a list of HREF links
|
|
that, when clicked, provide the full contents of the LogFile.
|
|
|
|
<h3 class="heading">Using LogFiles in custom BuildSteps</h3>
|
|
|
|
<p>The most common way for a custom BuildStep to use a LogFile is to
|
|
summarize the results of a ShellCommand (after the command has
|
|
finished running). For example, a compile step with thousands of lines
|
|
of output might want to create a summary of just the warning messages.
|
|
If you were doing this from a shell, you would use something like:
|
|
|
|
<pre class="example"> grep "warning:" output.log >warnings.log
|
|
</pre>
|
|
<p>In a custom BuildStep, you could instead create a “warnings” LogFile
|
|
that contained the same text. To do this, you would add code to your
|
|
<code>createSummary</code> method that pulls lines from the main output log
|
|
and creates a new LogFile with the results:
|
|
|
|
<pre class="example"> def createSummary(self, log):
|
|
warnings = []
|
|
for line in log.readlines():
|
|
if "warning:" in line:
|
|
warnings.append()
|
|
self.addCompleteLog('warnings', "".join(warnings))
|
|
</pre>
|
|
<p>This example uses the <code>addCompleteLog</code> method, which creates a
|
|
new LogFile, puts some text in it, and then “closes” it, meaning
|
|
that no further contents will be added. This LogFile will appear in
|
|
the HTML display under an HREF with the name “warnings”, since that
|
|
is the name of the LogFile.
|
|
|
|
<p>You can also use <code>addHTMLLog</code> to create a complete (closed)
|
|
LogFile that contains HTML instead of plain text. The normal LogFile
|
|
will be HTML-escaped if presented through a web page, but the HTML
|
|
LogFile will not. At the moment this is only used to present a pretty
|
|
HTML representation of an otherwise ugly exception traceback when
|
|
something goes badly wrong during the BuildStep.
|
|
|
|
<p>In contrast, you might want to create a new LogFile at the beginning
|
|
of the step, and add text to it as the command runs. You can create
|
|
the LogFile and attach it to the build by calling <code>addLog</code>, which
|
|
returns the LogFile object. You then add text to this LogFile by
|
|
calling methods like <code>addStdout</code> and <code>addHeader</code>. When you
|
|
are done, you must call the <code>finish</code> method so the LogFile can be
|
|
closed. It may be useful to create and populate a LogFile like this
|
|
from a LogObserver method See <a href="#Adding-LogObservers">Adding LogObservers</a>.
|
|
|
|
<p>The <code>logfiles=</code> argument to <code>ShellCommand</code> (see
|
|
see <a href="#ShellCommand">ShellCommand</a>) creates new LogFiles and fills them in realtime
|
|
by asking the buildslave to watch a actual file on disk. The
|
|
buildslave will look for additions in the target file and report them
|
|
back to the BuildStep. These additions will be added to the LogFile by
|
|
calling <code>addStdout</code>. These secondary LogFiles can be used as the
|
|
source of a LogObserver just like the normal “stdio” LogFile.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Adding-LogObservers"></a>
|
|
Next: <a rel="next" accesskey="n" href="#BuildStep-URLs">BuildStep URLs</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#BuildStep-LogFiles">BuildStep LogFiles</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.7.2 Adding LogObservers</h5>
|
|
|
|
<p><a name="index-LogObserver-63"></a><a name="index-LogLineObserver-64"></a>
|
|
Most shell commands emit messages to stdout or stderr as they operate,
|
|
especially if you ask them nicely with a <code>--verbose</code> flag of some
|
|
sort. They may also write text to a log file while they run. Your
|
|
BuildStep can watch this output as it arrives, to keep track of how
|
|
much progress the command has made. You can get a better measure of
|
|
progress by counting the number of source files compiled or test cases
|
|
run than by merely tracking the number of bytes that have been written
|
|
to stdout. This improves the accuracy and the smoothness of the ETA
|
|
display.
|
|
|
|
<p>To accomplish this, you will need to attach a <code>LogObserver</code> to
|
|
one of the log channels, most commonly to the “stdio” channel but
|
|
perhaps to another one which tracks a log file. This observer is given
|
|
all text as it is emitted from the command, and has the opportunity to
|
|
parse that output incrementally. Once the observer has decided that
|
|
some event has occurred (like a source file being compiled), it can
|
|
use the <code>setProgress</code> method to tell the BuildStep about the
|
|
progress that this event represents.
|
|
|
|
<p>There are a number of pre-built <code>LogObserver</code> classes that you
|
|
can choose from (defined in <code>buildbot.process.buildstep</code>, and of
|
|
course you can subclass them to add further customization. The
|
|
<code>LogLineObserver</code> class handles the grunt work of buffering and
|
|
scanning for end-of-line delimiters, allowing your parser to operate
|
|
on complete stdout/stderr lines.
|
|
|
|
<p>For example, let's take a look at the <code>TrialTestCaseCounter</code>,
|
|
which is used by the Trial step to count test cases as they are run.
|
|
As Trial executes, it emits lines like the following:
|
|
|
|
<pre class="example"> buildbot.test.test_config.ConfigTest.testDebugPassword ... [OK]
|
|
buildbot.test.test_config.ConfigTest.testEmpty ... [OK]
|
|
buildbot.test.test_config.ConfigTest.testIRC ... [FAIL]
|
|
buildbot.test.test_config.ConfigTest.testLocks ... [OK]
|
|
</pre>
|
|
<p>When the tests are finished, trial emits a long line of “======” and
|
|
then some lines which summarize the tests that failed. We want to
|
|
avoid parsing these trailing lines, because their format is less
|
|
well-defined than the “[OK]” lines.
|
|
|
|
<p>The parser class looks like this:
|
|
|
|
<pre class="example"> from buildbot.process.buildstep import LogLineObserver
|
|
|
|
class TrialTestCaseCounter(LogLineObserver):
|
|
_line_re = re.compile(r'^([\w\.]+) \.\.\. \[([^\]]+)\]$')
|
|
numTests = 0
|
|
finished = False
|
|
|
|
def outLineReceived(self, line):
|
|
if self.finished:
|
|
return
|
|
if line.startswith("=" * 40):
|
|
self.finished = True
|
|
return
|
|
|
|
m = self._line_re.search(line.strip())
|
|
if m:
|
|
testname, result = m.groups()
|
|
self.numTests += 1
|
|
self.step.setProgress('tests', self.numTests)
|
|
</pre>
|
|
<p>This parser only pays attention to stdout, since that's where trial
|
|
writes the progress lines. It has a mode flag named <code>finished</code> to
|
|
ignore everything after the “====” marker, and a scary-looking
|
|
regular expression to match each line while hopefully ignoring other
|
|
messages that might get displayed as the test runs.
|
|
|
|
<p>Each time it identifies a test has been completed, it increments its
|
|
counter and delivers the new progress value to the step with
|
|
<code>self.step.setProgress</code>. This class is specifically measuring
|
|
progress along the “tests” metric, in units of test cases (as
|
|
opposed to other kinds of progress like the “output” metric, which
|
|
measures in units of bytes). The Progress-tracking code uses each
|
|
progress metric separately to come up with an overall completion
|
|
percentage and an ETA value.
|
|
|
|
<p>To connect this parser into the <code>Trial</code> BuildStep,
|
|
<code>Trial.__init__</code> ends with the following clause:
|
|
|
|
<pre class="example"> # this counter will feed Progress along the 'test cases' metric
|
|
counter = TrialTestCaseCounter()
|
|
self.addLogObserver('stdio', counter)
|
|
</pre>
|
|
<p>This creates a TrialTestCaseCounter and tells the step that the
|
|
counter wants to watch the “stdio” log. The observer is
|
|
automatically given a reference to the step in its <code>.step</code>
|
|
attribute.
|
|
|
|
<h4 class="subheading">A Somewhat Whimsical Example</h4>
|
|
|
|
<p>Let's say that we've got some snazzy new unit-test framework called
|
|
Framboozle. It's the hottest thing since sliced bread. It slices, it
|
|
dices, it runs unit tests like there's no tomorrow. Plus if your unit
|
|
tests fail, you can use its name for a Web 2.1 startup company, make
|
|
millions of dollars, and hire engineers to fix the bugs for you, while
|
|
you spend your afternoons lazily hang-gliding along a scenic pacific
|
|
beach, blissfully unconcerned about the state of your
|
|
tests.<a rel="footnote" href="#fn-8" name="fnd-8"><sup>8</sup></a>
|
|
|
|
<p>To run a Framboozle-enabled test suite, you just run the 'framboozler'
|
|
command from the top of your source code tree. The 'framboozler'
|
|
command emits a bunch of stuff to stdout, but the most interesting bit
|
|
is that it emits the line "FNURRRGH!" every time it finishes running a
|
|
test case<a rel="footnote" href="#fn-9" name="fnd-9"><sup>9</sup></a>. You'd like to have a test-case counting LogObserver that
|
|
watches for these lines and counts them, because counting them will
|
|
help the buildbot more accurately calculate how long the build will
|
|
take, and this will let you know exactly how long you can sneak out of
|
|
the office for your hang-gliding lessons without anyone noticing that
|
|
you're gone.
|
|
|
|
<p>This will involve writing a new BuildStep (probably named
|
|
"Framboozle") which inherits from ShellCommand. The BuildStep class
|
|
definition itself will look something like this:
|
|
|
|
<pre class="example"> # START
|
|
from buildbot.steps.shell import ShellCommand
|
|
from buildbot.process.buildstep import LogLineObserver
|
|
|
|
class FNURRRGHCounter(LogLineObserver):
|
|
numTests = 0
|
|
def outLineReceived(self, line):
|
|
if "FNURRRGH!" in line:
|
|
self.numTests += 1
|
|
self.step.setProgress('tests', self.numTests)
|
|
|
|
class Framboozle(ShellCommand):
|
|
command = ["framboozler"]
|
|
|
|
def __init__(self, **kwargs):
|
|
ShellCommand.__init__(self, **kwargs) # always upcall!
|
|
counter = FNURRRGHCounter())
|
|
self.addLogObserver(counter)
|
|
# FINISH
|
|
</pre>
|
|
<p>So that's the code that we want to wind up using. How do we actually
|
|
deploy it?
|
|
|
|
<p>You have a couple of different options.
|
|
|
|
<p>Option 1: The simplest technique is to simply put this text
|
|
(everything from START to FINISH) in your master.cfg file, somewhere
|
|
before the BuildFactory definition where you actually use it in a
|
|
clause like:
|
|
|
|
<pre class="example"> f = BuildFactory()
|
|
f.addStep(SVN, svnurl="stuff")
|
|
f.addStep(Framboozle)
|
|
</pre>
|
|
<p>Remember that master.cfg is secretly just a python program with one
|
|
job: populating the BuildmasterConfig dictionary. And python programs
|
|
are allowed to define as many classes as they like. So you can define
|
|
classes and use them in the same file, just as long as the class is
|
|
defined before some other code tries to use it.
|
|
|
|
<p>This is easy, and it keeps the point of definition very close to the
|
|
point of use, and whoever replaces you after that unfortunate
|
|
hang-gliding accident will appreciate being able to easily figure out
|
|
what the heck this stupid "Framboozle" step is doing anyways. The
|
|
downside is that every time you reload the config file, the Framboozle
|
|
class will get redefined, which means that the buildmaster will think
|
|
that you've reconfigured all the Builders that use it, even though
|
|
nothing changed. Bleh.
|
|
|
|
<p>Option 2: Instead, we can put this code in a separate file, and import
|
|
it into the master.cfg file just like we would the normal buildsteps
|
|
like ShellCommand and SVN.
|
|
|
|
<p>Create a directory named ~/lib/python, put everything from START to
|
|
FINISH in ~/lib/python/framboozle.py, and run your buildmaster using:
|
|
|
|
<pre class="example"> PYTHONPATH=~/lib/python buildbot start MASTERDIR
|
|
</pre>
|
|
<p>or use the <samp><span class="file">Makefile.buildbot</span></samp> to control the way
|
|
<samp><span class="command">buildbot start</span></samp> works. Or add something like this to
|
|
something like your ~/.bashrc or ~/.bash_profile or ~/.cshrc:
|
|
|
|
<pre class="example"> export PYTHONPATH=~/lib/python
|
|
</pre>
|
|
<p>Once we've done this, our master.cfg can look like:
|
|
|
|
<pre class="example"> from framboozle import Framboozle
|
|
f = BuildFactory()
|
|
f.addStep(SVN, svnurl="stuff")
|
|
f.addStep(Framboozle)
|
|
</pre>
|
|
<p>or:
|
|
|
|
<pre class="example"> import framboozle
|
|
f = BuildFactory()
|
|
f.addStep(SVN, svnurl="stuff")
|
|
f.addStep(framboozle.Framboozle)
|
|
</pre>
|
|
<p>(check out the python docs for details about how "import" and "from A
|
|
import B" work).
|
|
|
|
<p>What we've done here is to tell python that every time it handles an
|
|
"import" statement for some named module, it should look in our
|
|
~/lib/python/ for that module before it looks anywhere else. After our
|
|
directories, it will try in a bunch of standard directories too
|
|
(including the one where buildbot is installed). By setting the
|
|
PYTHONPATH environment variable, you can add directories to the front
|
|
of this search list.
|
|
|
|
<p>Python knows that once it "import"s a file, it doesn't need to
|
|
re-import it again. This means that reconfiguring the buildmaster
|
|
(with "buildbot reconfig", for example) won't make it think the
|
|
Framboozle class has changed every time, so the Builders that use it
|
|
will not be spuriously restarted. On the other hand, you either have
|
|
to start your buildmaster in a slightly weird way, or you have to
|
|
modify your environment to set the PYTHONPATH variable.
|
|
|
|
<p>Option 3: Install this code into a standard python library directory
|
|
|
|
<p>Find out what your python's standard include path is by asking it:
|
|
|
|
<pre class="example"> 80:warner@luther% python
|
|
Python 2.4.4c0 (#2, Oct 2 2006, 00:57:46)
|
|
[GCC 4.1.2 20060928 (prerelease) (Debian 4.1.1-15)] on linux2
|
|
Type "help", "copyright", "credits" or "license" for more information.
|
|
>>> import sys
|
|
>>> print sys.path
|
|
['', '/usr/lib/python24.zip', '/usr/lib/python2.4', '/usr/lib/python2.4/plat-linux2', '/usr/lib/python2.4/lib-tk', '/usr/lib/python2.4/lib-dynload', '/usr/local/lib/python2.4/site-packages', '/usr/lib/python2.4/site-packages', '/usr/lib/python2.4/site-packages/Numeric', '/var/lib/python-support/python2.4', '/usr/lib/site-python']
|
|
>>>
|
|
</pre>
|
|
<p>In this case, putting the code into
|
|
/usr/local/lib/python2.4/site-packages/framboozle.py would work just
|
|
fine. We can use the same master.cfg "import framboozle" statement as
|
|
in Option 2. By putting it in a standard include directory (instead of
|
|
the decidedly non-standard ~/lib/python), we don't even have to set
|
|
PYTHONPATH to anything special. The downside is that you probably have
|
|
to be root to write to one of those standard include directories.
|
|
|
|
<p>Option 4: Submit the code for inclusion in the Buildbot distribution
|
|
|
|
<p>Contribute the code in an Enhancement Request on SourceForge, via
|
|
http://buildbot.sf.net . Lobby, convince, coerce, bribe, badger,
|
|
harass, threaten, or otherwise encourage the author to accept the
|
|
patch. This lets you do something like:
|
|
|
|
<pre class="example"> from buildbot.steps import framboozle
|
|
f = BuildFactory()
|
|
f.addStep(SVN, svnurl="stuff")
|
|
f.addStep(framboozle.Framboozle)
|
|
</pre>
|
|
<p>And then you don't even have to install framboozle.py anywhere on your
|
|
system, since it will ship with Buildbot. You don't have to be root,
|
|
you don't have to set PYTHONPATH. But you do have to make a good case
|
|
for Framboozle being worth going into the main distribution, you'll
|
|
probably have to provide docs and some unit test cases, you'll need to
|
|
figure out what kind of beer the author likes, and then you'll have to
|
|
wait until the next release. But in some environments, all this is
|
|
easier than getting root on your buildmaster box, so the tradeoffs may
|
|
actually be worth it.
|
|
|
|
<p>Putting the code in master.cfg (1) makes it available to that
|
|
buildmaster instance. Putting it in a file in a personal library
|
|
directory (2) makes it available for any buildmasters you might be
|
|
running. Putting it in a file in a system-wide shared library
|
|
directory (3) makes it available for any buildmasters that anyone on
|
|
that system might be running. Getting it into the buildbot's upstream
|
|
repository (4) makes it available for any buildmasters that anyone in
|
|
the world might be running. It's all a matter of how widely you want
|
|
to deploy that new class.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="BuildStep-URLs"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Adding-LogObservers">Adding LogObservers</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.1.7.3 BuildStep URLs</h5>
|
|
|
|
<p><a name="index-links-65"></a><a name="index-BuildStep-URLs-66"></a><a name="index-addURL-67"></a>
|
|
Each BuildStep has a collection of “links”. Like its collection of
|
|
LogFiles, each link has a name and a target URL. The web status page
|
|
creates HREFs for each link in the same box as it does for LogFiles,
|
|
except that the target of the link is the external URL instead of an
|
|
internal link to a page that shows the contents of the LogFile.
|
|
|
|
<p>These external links can be used to point at build information hosted
|
|
on other servers. For example, the test process might produce an
|
|
intricate description of which tests passed and failed, or some sort
|
|
of code coverage data in HTML form, or a PNG or GIF image with a graph
|
|
of memory usage over time. The external link can provide an easy way
|
|
for users to navigate from the buildbot's status page to these
|
|
external web sites or file servers. Note that the step itself is
|
|
responsible for insuring that there will be a document available at
|
|
the given URL (perhaps by using <samp><span class="command">scp</span></samp> to copy the HTML output
|
|
to a <samp><span class="file">~/public_html/</span></samp> directory on a remote web server). Calling
|
|
<code>addURL</code> does not magically populate a web server.
|
|
|
|
<p>To set one of these links, the BuildStep should call the <code>addURL</code>
|
|
method with the name of the link and the target URL. Multiple URLs can
|
|
be set.
|
|
|
|
<p>In this example, we assume that the <samp><span class="command">make test</span></samp> command causes
|
|
a collection of HTML files to be created and put somewhere on the
|
|
coverage.example.org web server, in a filename that incorporates the
|
|
build number.
|
|
|
|
<pre class="example"> class TestWithCodeCoverage(BuildStep):
|
|
command = ["make", "test",
|
|
WithProperties("buildnum=%s" % "buildnumber")]
|
|
|
|
def createSummary(self, log):
|
|
buildnumber = self.getProperty("buildnumber")
|
|
url = "http://coverage.example.org/builds/%s.html" % buildnumber
|
|
self.addURL("coverage", url)
|
|
</pre>
|
|
<p>You might also want to extract the URL from some special message
|
|
output by the build process itself:
|
|
|
|
<pre class="example"> class TestWithCodeCoverage(BuildStep):
|
|
command = ["make", "test",
|
|
WithProperties("buildnum=%s" % "buildnumber")]
|
|
|
|
def createSummary(self, log):
|
|
output = StringIO(log.getText())
|
|
for line in output.readlines():
|
|
if line.startswith("coverage-url:"):
|
|
url = line[len("coverage-url:"):].strip()
|
|
self.addURL("coverage", url)
|
|
return
|
|
</pre>
|
|
<p>Note that a build process which emits both stdout and stderr might
|
|
cause this line to be split or interleaved between other lines. It
|
|
might be necessary to restrict the getText() call to only stdout with
|
|
something like this:
|
|
|
|
<pre class="example"> output = StringIO("".join([c[1]
|
|
for c in log.getChunks()
|
|
if c[0] == LOG_CHANNEL_STDOUT]))
|
|
</pre>
|
|
<p>Of course if the build is run under a PTY, then stdout and stderr will
|
|
be merged before the buildbot ever sees them, so such interleaving
|
|
will be unavoidable.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Interlocks"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Build-Factories">Build Factories</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Build-Steps">Build Steps</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Process">Build Process</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">6.2 Interlocks</h3>
|
|
|
|
<p><a name="index-locks-68"></a><a name="index-buildbot_002elocks_002eMasterLock-69"></a><a name="index-buildbot_002elocks_002eSlaveLock-70"></a>
|
|
For various reasons, you may want to prevent certain Steps (or perhaps
|
|
entire Builds) from running simultaneously. Limited CPU speed or
|
|
network bandwidth to the VC server, problems with simultaneous access
|
|
to a database server used by unit tests, or multiple Builds which
|
|
access shared state may all require some kind of interlock to prevent
|
|
corruption, confusion, or resource overload. These resources might
|
|
require completely exclusive access, or it might be sufficient to
|
|
establish a limit of two or three simultaneous builds.
|
|
|
|
<p><code>Locks</code> are the mechanism used to express these kinds of
|
|
constraints on when Builds or Steps can be run. There are two kinds of
|
|
<code>Locks</code>, each with their own scope: <code>MasterLock</code> instances
|
|
are scoped to the buildbot as a whole, while <code>SlaveLock</code>s are
|
|
scoped to a single buildslave. This means that each buildslave has a
|
|
separate copy of each <code>SlaveLock</code>, which could enforce a
|
|
one-Build-at-a-time limit for each machine, but still allow as many
|
|
simultaneous builds as there are machines.
|
|
|
|
<p>Each <code>Lock</code> is created with a unique name. Each lock gets a count
|
|
of how many owners it may have: how many processes can claim it at ths
|
|
same time. This limit defaults to one, and is controllable through the
|
|
<code>maxCount</code> argument. On <code>SlaveLock</code>s you can set the owner
|
|
count on a per-slave basis by providing a dictionary (that maps from
|
|
slavename to maximum owner count) to its <code>maxCountForSlave</code>
|
|
argument. Any buildslaves that aren't mentioned in
|
|
<code>maxCountForSlave</code> get their owner count from <code>maxCount</code>.
|
|
|
|
<p>To use a lock, simply include it in the <code>locks=</code> argument of the
|
|
<code>BuildStep</code> object that should obtain the lock before it runs.
|
|
This argument accepts a list of <code>Lock</code> objects: the Step will
|
|
acquire all of them before it runs.
|
|
|
|
<p>To claim a lock for the whole Build, add a <code>'locks'</code> key to the
|
|
builder specification dictionary with the same list of <code>Lock</code>
|
|
objects. (This is the dictionary that has the <code>'name'</code>,
|
|
<code>'slavename'</code>, <code>'builddir'</code>, and <code>'factory'</code> keys). The
|
|
<code>Build</code> object also accepts a <code>locks=</code> argument, but unless
|
|
you are writing your own <code>BuildFactory</code> subclass then it will be
|
|
easier to set the locks in the builder dictionary.
|
|
|
|
<p>Note that there are no partial-acquire or partial-release semantics:
|
|
this prevents deadlocks caused by two Steps each waiting for a lock
|
|
held by the other<a rel="footnote" href="#fn-10" name="fnd-10"><sup>10</sup></a>. This also means
|
|
that waiting to acquire a <code>Lock</code> can take an arbitrarily long
|
|
time: if the buildmaster is very busy, a Step or Build which requires
|
|
only one <code>Lock</code> may starve another that is waiting for that
|
|
<code>Lock</code> plus some others.
|
|
|
|
<p>In the following example, we run the same build on three different
|
|
platforms. The unit-test steps of these builds all use a common
|
|
database server, and would interfere with each other if allowed to run
|
|
simultaneously. The <code>Lock</code> prevents more than one of these builds
|
|
from happening at the same time.
|
|
|
|
<pre class="example"> from buildbot import locks
|
|
from buildbot.steps import source, shell
|
|
from buildbot.process import factory
|
|
|
|
db_lock = locks.MasterLock("database")
|
|
f = factory.BuildFactory()
|
|
f.addStep(source.SVN, svnurl="http://example.org/svn/Trunk")
|
|
f.addStep(shell.ShellCommand, command="make all")
|
|
f.addStep(shell.ShellCommand, command="make test", locks=[db_lock])
|
|
b1 = {'name': 'full1', 'slavename': 'bot-1', builddir='f1', 'factory': f}
|
|
b2 = {'name': 'full2', 'slavename': 'bot-2', builddir='f2', 'factory': f}
|
|
b3 = {'name': 'full3', 'slavename': 'bot-3', builddir='f3', 'factory': f}
|
|
c['builders'] = [b1, b2, b3]
|
|
</pre>
|
|
<p>In the next example, we have one buildslave hosting three separate
|
|
Builders (each running tests against a different version of Python).
|
|
The machine which hosts this buildslave is not particularly fast, so
|
|
we want to prevent all three builds from all happening at the same
|
|
time. (Assume we've experimentally determined that one build leaves
|
|
unused CPU capacity, three builds causes a lot of disk thrashing, but
|
|
two builds at a time is Just Right). We use a <code>SlaveLock</code> because
|
|
the builds happening on this one slow slave should not affect builds
|
|
running on other slaves, and we use the lock on the build as a whole
|
|
because the slave is so slow that even multiple simultaneous SVN
|
|
checkouts would be too taxing. We set <code>maxCount=2</code> to achieve our
|
|
goal of two simultaneous builds per slave.
|
|
|
|
<pre class="example"> from buildbot import locks
|
|
from buildbot.steps import source
|
|
from buildbot.process import s, factory
|
|
|
|
slow_lock = locks.SlaveLock("cpu", maxCount=2)
|
|
source = s(source.SVN, svnurl="http://example.org/svn/Trunk")
|
|
f22 = factory.Trial(source, trialpython=["python2.2"])
|
|
f23 = factory.Trial(source, trialpython=["python2.3"])
|
|
f24 = factory.Trial(source, trialpython=["python2.4"])
|
|
b1 = {'name': 'p22', 'slavename': 'bot-1', builddir='p22', 'factory': f22,
|
|
'locks': [slow_lock] }
|
|
b2 = {'name': 'p23', 'slavename': 'bot-1', builddir='p23', 'factory': f23,
|
|
'locks': [slow_lock] }
|
|
b3 = {'name': 'p24', 'slavename': 'bot-1', builddir='p24', 'factory': f24,
|
|
'locks': [slow_lock] }
|
|
c['builders'] = [b1, b2, b3]
|
|
</pre>
|
|
<p>In the last example, we use two Locks at the same time. In this case,
|
|
we're concerned about both of the previous constraints, but we'll say
|
|
that only the tests are computationally intensive, and that they have
|
|
been split into those which use the database and those which do not.
|
|
In addition, two of the Builds run on a fast machine which does not
|
|
need to worry about the cpu lock, but which still must be prevented
|
|
from simultaneous database access. We use <code>maxCountForSlave</code> to
|
|
limit the slow machine to one simultanous build, but allow practically
|
|
unlimited concurrent builds on the fast machine.
|
|
|
|
<pre class="example"> from buildbot import locks
|
|
from buildbot.steps import source, shell
|
|
from buildbot.process import factory
|
|
|
|
db_lock = locks.MasterLock("database")
|
|
slavecounts = {"bot-slow": 1, "bot-fast": 100}
|
|
cpu_lock = locks.SlaveLock("cpu", maxCountForSlave=slavecounts)
|
|
f = factory.BuildFactory()
|
|
f.addStep(source.SVN, svnurl="http://example.org/svn/Trunk")
|
|
f.addStep(shell.ShellCommand, command="make all", locks=[cpu_lock])
|
|
f.addStep(shell.ShellCommand, command="make test", locks=[cpu_lock])
|
|
f.addStep(shell.ShellCommand, command="make db-test",
|
|
locks=[db_lock, cpu_lock])
|
|
|
|
b1 = {'name': 'full1', 'slavename': 'bot-slow', builddir='full1',
|
|
'factory': f}
|
|
b2 = {'name': 'full2', 'slavename': 'bot-slow', builddir='full2',
|
|
'factory': f}
|
|
b3 = {'name': 'full3', 'slavename': 'bot-fast', builddir='full3',
|
|
'factory': f}
|
|
b4 = {'name': 'full4', 'slavename': 'bot-fast', builddir='full4',
|
|
'factory': f}
|
|
c['builders'] = [b1, b2, b3, b4]
|
|
</pre>
|
|
<p>As a final note, remember that a unit test system which breaks when
|
|
multiple people run it at the same time is fragile and should be
|
|
fixed. Asking your human developers to serialize themselves when
|
|
running unit tests will just discourage them from running the unit
|
|
tests at all. Find a way to fix this: change the database tests to
|
|
create a new (uniquely-named) user or table for each test run, don't
|
|
use fixed listening TCP ports for network tests (instead listen on
|
|
port 0 to let the kernel choose a port for you and then query the
|
|
socket to find out what port was allocated). <code>MasterLock</code>s can be
|
|
used to accomodate broken test systems like this, but are really
|
|
intended for other purposes: build processes that store or retrieve
|
|
products in shared directories, or which do things that human
|
|
developers would not (or which might slow down or break in ways that
|
|
require human attention to deal with).
|
|
|
|
<p><code>SlaveLocks</code>s can be used to keep automated performance tests
|
|
from interfering with each other, when there are multiple Builders all
|
|
using the same buildslave. But they can't prevent other users from
|
|
running CPU-intensive jobs on that host while the tests are running.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Build-Factories"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Interlocks">Interlocks</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Process">Build Process</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">6.3 Build Factories</h3>
|
|
|
|
<p>Each Builder is equipped with a “build factory”, which is
|
|
responsible for producing the actual <code>Build</code> objects that perform
|
|
each build. This factory is created in the configuration file, and
|
|
attached to a Builder through the <code>factory</code> element of its
|
|
dictionary.
|
|
|
|
<p>The standard <code>BuildFactory</code> object creates <code>Build</code> objects
|
|
by default. These Builds will each execute a collection of BuildSteps
|
|
in a fixed sequence. Each step can affect the results of the build,
|
|
but in general there is little intelligence to tie the different steps
|
|
together. You can create subclasses of <code>Build</code> to implement more
|
|
sophisticated build processes, and then use a subclass of
|
|
<code>BuildFactory</code> (or simply set the <code>buildClass</code> attribute) to
|
|
create instances of your new Build subclass.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#BuildStep-Objects">BuildStep Objects</a>
|
|
<li><a accesskey="2" href="#BuildFactory">BuildFactory</a>
|
|
<li><a accesskey="3" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="BuildStep-Objects"></a>
|
|
Next: <a rel="next" accesskey="n" href="#BuildFactory">BuildFactory</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Build-Factories">Build Factories</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">6.3.1 BuildStep Objects</h4>
|
|
|
|
<p>The steps used by these builds are all subclasses of <code>BuildStep</code>.
|
|
The standard ones provided with Buildbot are documented later,
|
|
See <a href="#Build-Steps">Build Steps</a>. You can also write your own subclasses to use in
|
|
builds.
|
|
|
|
<p>The basic behavior for a <code>BuildStep</code> is to:
|
|
|
|
<ul>
|
|
<li>run for a while, then stop
|
|
<li>possibly invoke some RemoteCommands on the attached build slave
|
|
<li>possibly produce a set of log files
|
|
<li>finish with a status described by one of four values defined in
|
|
buildbot.status.builder: SUCCESS, WARNINGS, FAILURE, SKIPPED
|
|
<li>provide a list of short strings to describe the step
|
|
<li>define a color (generally green, orange, or red) with which the
|
|
step should be displayed
|
|
</ul>
|
|
|
|
<p>More sophisticated steps may produce additional information and
|
|
provide it to later build steps, or store it in the factory to provide
|
|
to later builds.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#BuildFactory-Attributes">BuildFactory Attributes</a>
|
|
<li><a accesskey="2" href="#Quick-builds">Quick builds</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="BuildFactory"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#BuildStep-Objects">BuildStep Objects</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">6.3.2 BuildFactory</h4>
|
|
|
|
<p><a name="index-buildbot_002eprocess_002efactory_002eBuildFactory-71"></a><a name="index-buildbot_002eprocess_002efactory_002eBasicBuildFactory-72"></a><!-- TODO: what is BasicSVN anyway? -->
|
|
<a name="index-buildbot_002eprocess_002efactory_002eBasicSVN-73"></a>
|
|
The default <code>BuildFactory</code>, provided in the
|
|
<code>buildbot.process.factory</code> module, contains a list of “BuildStep
|
|
specifications”: a list of <code>(step_class, kwargs)</code> tuples for
|
|
each. When asked to create a Build, it loads the list of steps into
|
|
the new Build object. When the Build is actually started, these step
|
|
specifications are used to create the actual set of BuildSteps, which
|
|
are then executed one at a time. For example, a build which consists
|
|
of a CVS checkout followed by a <code>make build</code> would be constructed
|
|
as follows:
|
|
|
|
<pre class="example"> from buildbot.steps import source, shell
|
|
from buildbot.process import factory
|
|
|
|
f = factory.BuildFactory()
|
|
f.addStep(source.CVS, cvsroot=CVSROOT, cvsmodule="project", mode="update")
|
|
f.addStep(shell.Compile, command=["make", "build"])
|
|
</pre>
|
|
<p>It is also possible to pass a list of step specifications into the
|
|
<code>BuildFactory</code> when it is created. Using <code>addStep</code> is
|
|
usually simpler, but there are cases where is is more convenient to
|
|
create the list of steps ahead of time. To make this approach easier,
|
|
a convenience function named <code>s</code> is available:
|
|
|
|
<pre class="example"> from buildbot.steps import source, shell
|
|
from buildbot.process import factory
|
|
from buildbot.factory import s
|
|
# s is a convenience function, defined with:
|
|
# def s(steptype, **kwargs): return (steptype, kwargs)
|
|
|
|
all_steps = [s(source.CVS, cvsroot=CVSROOT, cvsmodule="project",
|
|
mode="update"),
|
|
s(shell.Compile, command=["make", "build"]),
|
|
]
|
|
f = factory.BuildFactory(all_steps)
|
|
</pre>
|
|
<p>Each step can affect the build process in the following ways:
|
|
|
|
<ul>
|
|
<li>If the step's <code>haltOnFailure</code> attribute is True, then a failure
|
|
in the step (i.e. if it completes with a result of FAILURE) will cause
|
|
the whole build to be terminated immediately: no further steps will be
|
|
executed. This is useful for setup steps upon which the rest of the
|
|
build depends: if the CVS checkout or <code>./configure</code> process
|
|
fails, there is no point in trying to compile or test the resulting
|
|
tree.
|
|
|
|
<li>If the <code>flunkOnFailure</code> or <code>flunkOnWarnings</code> flag is set,
|
|
then a result of FAILURE or WARNINGS will mark the build as a whole as
|
|
FAILED. However, the remaining steps will still be executed. This is
|
|
appropriate for things like multiple testing steps: a failure in any
|
|
one of them will indicate that the build has failed, however it is
|
|
still useful to run them all to completion.
|
|
|
|
<li>Similarly, if the <code>warnOnFailure</code> or <code>warnOnWarnings</code> flag
|
|
is set, then a result of FAILURE or WARNINGS will mark the build as
|
|
having WARNINGS, and the remaining steps will still be executed. This
|
|
may be appropriate for certain kinds of optional build or test steps.
|
|
For example, a failure experienced while building documentation files
|
|
should be made visible with a WARNINGS result but not be serious
|
|
enough to warrant marking the whole build with a FAILURE.
|
|
|
|
</ul>
|
|
|
|
<p>In addition, each Step produces its own results, may create logfiles,
|
|
etc. However only the flags described above have any effect on the
|
|
build as a whole.
|
|
|
|
<p>The pre-defined BuildSteps like <code>CVS</code> and <code>Compile</code> have
|
|
reasonably appropriate flags set on them already. For example, without
|
|
a source tree there is no point in continuing the build, so the
|
|
<code>CVS</code> class has the <code>haltOnFailure</code> flag set to True. Look
|
|
in <samp><span class="file">buildbot/process/step.py</span></samp> to see how the other Steps are
|
|
marked.
|
|
|
|
<p>Each Step is created with an additional <code>workdir</code> argument that
|
|
indicates where its actions should take place. This is specified as a
|
|
subdirectory of the slave builder's base directory, with a default
|
|
value of <code>build</code>. This is only implemented as a step argument (as
|
|
opposed to simply being a part of the base directory) because the
|
|
CVS/SVN steps need to perform their checkouts from the parent
|
|
directory.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#BuildFactory-Attributes">BuildFactory Attributes</a>
|
|
<li><a accesskey="2" href="#Quick-builds">Quick builds</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="BuildFactory-Attributes"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Quick-builds">Quick builds</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#BuildFactory">BuildFactory</a>,
|
|
Up: <a rel="up" accesskey="u" href="#BuildFactory">BuildFactory</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.3.2.1 BuildFactory Attributes</h5>
|
|
|
|
<p>Some attributes from the BuildFactory are copied into each Build.
|
|
|
|
<p><a name="index-treeStableTimer-74"></a>
|
|
<dl>
|
|
<dt><code>useProgress</code><dd>(defaults to True): if True, the buildmaster keeps track of how long
|
|
each step takes, so it can provide estimates of how long future builds
|
|
will take. If builds are not expected to take a consistent amount of
|
|
time (such as incremental builds in which a random set of files are
|
|
recompiled or tested each time), this should be set to False to
|
|
inhibit progress-tracking.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Quick-builds"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#BuildFactory-Attributes">BuildFactory Attributes</a>,
|
|
Up: <a rel="up" accesskey="u" href="#BuildFactory">BuildFactory</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.3.2.2 Quick builds</h5>
|
|
|
|
<p><a name="index-buildbot_002eprocess_002efactory_002eQuickBuildFactory-75"></a>
|
|
The difference between a “full build” and a “quick build” is that
|
|
quick builds are generally done incrementally, starting with the tree
|
|
where the previous build was performed. That simply means that the
|
|
source-checkout step should be given a <code>mode='update'</code> flag, to
|
|
do the source update in-place.
|
|
|
|
<p>In addition to that, the <code>useProgress</code> flag should be set to
|
|
False. Incremental builds will (or at least the ought to) compile as
|
|
few files as necessary, so they will take an unpredictable amount of
|
|
time to run. Therefore it would be misleading to claim to predict how
|
|
long the build will take.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Process-Specific-build-factories"></a>
|
|
<a name="Process_002dSpecific-build-factories"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#BuildFactory">BuildFactory</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">6.3.3 Process-Specific build factories</h4>
|
|
|
|
<p>Many projects use one of a few popular build frameworks to simplify
|
|
the creation and maintenance of Makefiles or other compilation
|
|
structures. Buildbot provides several pre-configured BuildFactory
|
|
subclasses which let you build these projects with a minimum of fuss.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#GNUAutoconf">GNUAutoconf</a>
|
|
<li><a accesskey="2" href="#CPAN">CPAN</a>
|
|
<li><a accesskey="3" href="#Python-distutils">Python distutils</a>
|
|
<li><a accesskey="4" href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="GNUAutoconf"></a>
|
|
Next: <a rel="next" accesskey="n" href="#CPAN">CPAN</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.3.3.1 GNUAutoconf</h5>
|
|
|
|
<p><a name="index-buildbot_002eprocess_002efactory_002eGNUAutoconf-76"></a>
|
|
<a href="http://www.gnu.org/software/autoconf/">GNU Autoconf</a> is a
|
|
software portability tool, intended to make it possible to write
|
|
programs in C (and other languages) which will run on a variety of
|
|
UNIX-like systems. Most GNU software is built using autoconf. It is
|
|
frequently used in combination with GNU automake. These tools both
|
|
encourage a build process which usually looks like this:
|
|
|
|
<pre class="example"> % CONFIG_ENV=foo ./configure --with-flags
|
|
% make all
|
|
% make check
|
|
# make install
|
|
</pre>
|
|
<p>(except of course the Buildbot always skips the <code>make install</code>
|
|
part).
|
|
|
|
<p>The Buildbot's <code>buildbot.process.factory.GNUAutoconf</code> factory is
|
|
designed to build projects which use GNU autoconf and/or automake. The
|
|
configuration environment variables, the configure flags, and command
|
|
lines used for the compile and test are all configurable, in general
|
|
the default values will be suitable.
|
|
|
|
<p>Example:
|
|
|
|
<pre class="example"> # use the s() convenience function defined earlier
|
|
f = factory.GNUAutoconf(source=s(step.SVN, svnurl=URL, mode="copy"),
|
|
flags=["--disable-nls"])
|
|
</pre>
|
|
<p>Required Arguments:
|
|
|
|
<dl>
|
|
<dt><code>source</code><dd>This argument must be a step specification tuple that provides a
|
|
BuildStep to generate the source tree.
|
|
</dl>
|
|
|
|
<p>Optional Arguments:
|
|
|
|
<dl>
|
|
<dt><code>configure</code><dd>The command used to configure the tree. Defaults to
|
|
<code>./configure</code>. Accepts either a string or a list of shell argv
|
|
elements.
|
|
|
|
<br><dt><code>configureEnv</code><dd>The environment used for the initial configuration step. This accepts
|
|
a dictionary which will be merged into the buildslave's normal
|
|
environment. This is commonly used to provide things like
|
|
<code>CFLAGS="-O2 -g"</code> (to turn off debug symbols during the compile).
|
|
Defaults to an empty dictionary.
|
|
|
|
<br><dt><code>configureFlags</code><dd>A list of flags to be appended to the argument list of the configure
|
|
command. This is commonly used to enable or disable specific features
|
|
of the autoconf-controlled package, like <code>["--without-x"]</code> to
|
|
disable windowing support. Defaults to an empty list.
|
|
|
|
<br><dt><code>compile</code><dd>this is a shell command or list of argv values which is used to
|
|
actually compile the tree. It defaults to <code>make all</code>. If set to
|
|
None, the compile step is skipped.
|
|
|
|
<br><dt><code>test</code><dd>this is a shell command or list of argv values which is used to run
|
|
the tree's self-tests. It defaults to <code>make check</code>. If set to
|
|
None, the test step is skipped.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="CPAN"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Python-distutils">Python distutils</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#GNUAutoconf">GNUAutoconf</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.3.3.2 CPAN</h5>
|
|
|
|
<p><a name="index-buildbot_002eprocess_002efactory_002eCPAN-77"></a>
|
|
Most Perl modules available from the <a href="http://www.cpan.org/">CPAN</a>
|
|
archive use the <code>MakeMaker</code> module to provide configuration,
|
|
build, and test services. The standard build routine for these modules
|
|
looks like:
|
|
|
|
<pre class="example"> % perl Makefile.PL
|
|
% make
|
|
% make test
|
|
# make install
|
|
</pre>
|
|
<p>(except again Buildbot skips the install step)
|
|
|
|
<p>Buildbot provides a <code>CPAN</code> factory to compile and test these
|
|
projects.
|
|
|
|
<p>Arguments:
|
|
<dl>
|
|
<dt><code>source</code><dd>(required): A step specification tuple, that that used by GNUAutoconf.
|
|
|
|
<br><dt><code>perl</code><dd>A string which specifies the <code>perl</code> executable to use. Defaults
|
|
to just <code>perl</code>.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Python-distutils"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#CPAN">CPAN</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.3.3.3 Python distutils</h5>
|
|
|
|
<p><a name="index-buildbot_002eprocess_002efactory_002eDistutils-78"></a>
|
|
Most Python modules use the <code>distutils</code> package to provide
|
|
configuration and build services. The standard build process looks
|
|
like:
|
|
|
|
<pre class="example"> % python ./setup.py build
|
|
% python ./setup.py install
|
|
</pre>
|
|
<p>Unfortunately, although Python provides a standard unit-test framework
|
|
named <code>unittest</code>, to the best of my knowledge <code>distutils</code>
|
|
does not provide a standardized target to run such unit tests. (please
|
|
let me know if I'm wrong, and I will update this factory).
|
|
|
|
<p>The <code>Distutils</code> factory provides support for running the build
|
|
part of this process. It accepts the same <code>source=</code> parameter as
|
|
the other build factories.
|
|
|
|
<p>Arguments:
|
|
<dl>
|
|
<dt><code>source</code><dd>(required): A step specification tuple, that that used by GNUAutoconf.
|
|
|
|
<br><dt><code>python</code><dd>A string which specifies the <code>python</code> executable to use. Defaults
|
|
to just <code>python</code>.
|
|
|
|
<br><dt><code>test</code><dd>Provides a shell command which runs unit tests. This accepts either a
|
|
string or a list. The default value is None, which disables the test
|
|
step (since there is no common default command to run unit tests in
|
|
distutils modules).
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Python%2fTwisted%2ftrial-projects"></a>
|
|
<a name="Python_002fTwisted_002ftrial-projects"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Python-distutils">Python distutils</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>
|
|
|
|
</div>
|
|
|
|
<h5 class="subsubsection">6.3.3.4 Python/Twisted/trial projects</h5>
|
|
|
|
<p><a name="index-buildbot_002eprocess_002efactory_002eTrial-79"></a><!-- TODO: document these steps better -->
|
|
<a name="index-buildbot_002esteps_002epython_005ftwisted_002eHLint-80"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eTrial-81"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eProcessDocs-82"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eBuildDebs-83"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eRemovePYCs-84"></a>
|
|
Twisted provides a unit test tool named <code>trial</code> which provides a
|
|
few improvements over Python's built-in <code>unittest</code> module. Many
|
|
python projects which use Twisted for their networking or application
|
|
services also use trial for their unit tests. These modules are
|
|
usually built and tested with something like the following:
|
|
|
|
<pre class="example"> % python ./setup.py build
|
|
% PYTHONPATH=build/lib.linux-i686-2.3 trial -v PROJECTNAME.test
|
|
% python ./setup.py install
|
|
</pre>
|
|
<p>Unfortunately, the <samp><span class="file">build/lib</span></samp> directory into which the
|
|
built/copied .py files are placed is actually architecture-dependent,
|
|
and I do not yet know of a simple way to calculate its value. For many
|
|
projects it is sufficient to import their libraries “in place” from
|
|
the tree's base directory (<code>PYTHONPATH=.</code>).
|
|
|
|
<p>In addition, the <var>PROJECTNAME</var> value where the test files are
|
|
located is project-dependent: it is usually just the project's
|
|
top-level library directory, as common practice suggests the unit test
|
|
files are put in the <code>test</code> sub-module. This value cannot be
|
|
guessed, the <code>Trial</code> class must be told where to find the test
|
|
files.
|
|
|
|
<p>The <code>Trial</code> class provides support for building and testing
|
|
projects which use distutils and trial. If the test module name is
|
|
specified, trial will be invoked. The library path used for testing
|
|
can also be set.
|
|
|
|
<p>One advantage of trial is that the Buildbot happens to know how to
|
|
parse trial output, letting it identify which tests passed and which
|
|
ones failed. The Buildbot can then provide fine-grained reports about
|
|
how many tests have failed, when individual tests fail when they had
|
|
been passing previously, etc.
|
|
|
|
<p>Another feature of trial is that you can give it a series of source
|
|
.py files, and it will search them for special <code>test-case-name</code>
|
|
tags that indicate which test cases provide coverage for that file.
|
|
Trial can then run just the appropriate tests. This is useful for
|
|
quick builds, where you want to only run the test cases that cover the
|
|
changed functionality.
|
|
|
|
<p>Arguments:
|
|
<dl>
|
|
<dt><code>source</code><dd>(required): A step specification tuple, like that used by GNUAutoconf.
|
|
|
|
<br><dt><code>buildpython</code><dd>A list (argv array) of strings which specifies the <code>python</code>
|
|
executable to use when building the package. Defaults to just
|
|
<code>['python']</code>. It may be useful to add flags here, to supress
|
|
warnings during compilation of extension modules. This list is
|
|
extended with <code>['./setup.py', 'build']</code> and then executed in a
|
|
ShellCommand.
|
|
|
|
<br><dt><code>testpath</code><dd>Provides a directory to add to <code>PYTHONPATH</code> when running the unit
|
|
tests, if tests are being run. Defaults to <code>.</code> to include the
|
|
project files in-place. The generated build library is frequently
|
|
architecture-dependent, but may simply be <samp><span class="file">build/lib</span></samp> for
|
|
pure-python modules.
|
|
|
|
<br><dt><code>trialpython</code><dd>Another list of strings used to build the command that actually runs
|
|
trial. This is prepended to the contents of the <code>trial</code> argument
|
|
below. It may be useful to add <code>-W</code> flags here to supress
|
|
warnings that occur while tests are being run. Defaults to an empty
|
|
list, meaning <code>trial</code> will be run without an explicit
|
|
interpreter, which is generally what you want if you're using
|
|
<samp><span class="file">/usr/bin/trial</span></samp> instead of, say, the <samp><span class="file">./bin/trial</span></samp> that
|
|
lives in the Twisted source tree.
|
|
|
|
<br><dt><code>trial</code><dd>provides the name of the <code>trial</code> command. It is occasionally
|
|
useful to use an alternate executable, such as <code>trial2.2</code> which
|
|
might run the tests under an older version of Python. Defaults to
|
|
<code>trial</code>.
|
|
|
|
<br><dt><code>tests</code><dd>Provides a module name or names which contain the unit tests for this
|
|
project. Accepts a string, typically <code>PROJECTNAME.test</code>, or a
|
|
list of strings. Defaults to None, indicating that no tests should be
|
|
run. You must either set this or <code>useTestCaseNames</code> to do anyting
|
|
useful with the Trial factory.
|
|
|
|
<br><dt><code>useTestCaseNames</code><dd>Tells the Step to provide the names of all changed .py files to trial,
|
|
so it can look for test-case-name tags and run just the matching test
|
|
cases. Suitable for use in quick builds. Defaults to False.
|
|
|
|
<br><dt><code>randomly</code><dd>If <code>True</code>, tells Trial (with the <code>--random=0</code> argument) to
|
|
run the test cases in random order, which sometimes catches subtle
|
|
inter-test dependency bugs. Defaults to <code>False</code>.
|
|
|
|
<br><dt><code>recurse</code><dd>If <code>True</code>, tells Trial (with the <code>--recurse</code> argument) to
|
|
look in all subdirectories for additional test cases. It isn't clear
|
|
to me how this works, but it may be useful to deal with the
|
|
unknown-PROJECTNAME problem described above, and is currently used in
|
|
the Twisted buildbot to accomodate the fact that test cases are now
|
|
distributed through multiple twisted.SUBPROJECT.test directories.
|
|
|
|
</dl>
|
|
|
|
<p>Unless one of <code>trialModule</code> or <code>useTestCaseNames</code>
|
|
are set, no tests will be run.
|
|
|
|
<p>Some quick examples follow. Most of these examples assume that the
|
|
target python code (the “code under test”) can be reached directly
|
|
from the root of the target tree, rather than being in a <samp><span class="file">lib/</span></samp>
|
|
subdirectory.
|
|
|
|
<pre class="example"> # Trial(source, tests="toplevel.test") does:
|
|
# python ./setup.py build
|
|
# PYTHONPATH=. trial -to toplevel.test
|
|
|
|
# Trial(source, tests=["toplevel.test", "other.test"]) does:
|
|
# python ./setup.py build
|
|
# PYTHONPATH=. trial -to toplevel.test other.test
|
|
|
|
# Trial(source, useTestCaseNames=True) does:
|
|
# python ./setup.py build
|
|
# PYTHONPATH=. trial -to --testmodule=foo/bar.py.. (from Changes)
|
|
|
|
# Trial(source, buildpython=["python2.3", "-Wall"], tests="foo.tests"):
|
|
# python2.3 -Wall ./setup.py build
|
|
# PYTHONPATH=. trial -to foo.tests
|
|
|
|
# Trial(source, trialpython="python2.3", trial="/usr/bin/trial",
|
|
# tests="foo.tests") does:
|
|
# python2.3 -Wall ./setup.py build
|
|
# PYTHONPATH=. python2.3 /usr/bin/trial -to foo.tests
|
|
|
|
# For running trial out of the tree being tested (only useful when the
|
|
# tree being built is Twisted itself):
|
|
# Trial(source, trialpython=["python2.3", "-Wall"], trial="./bin/trial",
|
|
# tests="foo.tests") does:
|
|
# python2.3 -Wall ./setup.py build
|
|
# PYTHONPATH=. python2.3 -Wall ./bin/trial -to foo.tests
|
|
</pre>
|
|
<p>If the output directory of <code>./setup.py build</code> is known, you can
|
|
pull the python code from the built location instead of the source
|
|
directories. This should be able to handle variations in where the
|
|
source comes from, as well as accomodating binary extension modules:
|
|
|
|
<pre class="example"> # Trial(source,tests="toplevel.test",testpath='build/lib.linux-i686-2.3')
|
|
# does:
|
|
# python ./setup.py build
|
|
# PYTHONPATH=build/lib.linux-i686-2.3 trial -to toplevel.test
|
|
</pre>
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Status-Delivery"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Command_002dline-tool">Command-line tool</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Build-Process">Build Process</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="chapter">7 Status Delivery</h2>
|
|
|
|
<p>More details are available in the docstrings for each class, use
|
|
<code>pydoc buildbot.status.html.Waterfall</code> to see them. Most status
|
|
delivery objects take a <code>categories=</code> argument, which can contain
|
|
a list of “category” names: in this case, it will only show status
|
|
for Builders that are in one of the named categories.
|
|
|
|
<p>(implementor's note: each of these objects should be a
|
|
service.MultiService which will be attached to the BuildMaster object
|
|
when the configuration is processed. They should use
|
|
<code>self.parent.getStatus()</code> to get access to the top-level IStatus
|
|
object, either inside <code>startService</code> or later. They may call
|
|
<code>status.subscribe()</code> in <code>startService</code> to receive
|
|
notifications of builder events, in which case they must define
|
|
<code>builderAdded</code> and related methods. See the docstrings in
|
|
<samp><span class="file">buildbot/interfaces.py</span></samp> for full details.)
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#HTML-Waterfall">HTML Waterfall</a>
|
|
<li><a accesskey="2" href="#IRC-Bot">IRC Bot</a>
|
|
<li><a accesskey="3" href="#PBListener">PBListener</a>
|
|
<li><a accesskey="4" href="#Writing-New-Status-Plugins">Writing New Status Plugins</a>
|
|
</ul>
|
|
|
|
<!-- @node Email Delivery, , Status Delivery, Status Delivery -->
|
|
<!-- @subsection Email Delivery -->
|
|
<!-- DOCUMENT THIS -->
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="HTML-Waterfall"></a>
|
|
Next: <a rel="next" accesskey="n" href="#IRC-Bot">IRC Bot</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Status-Delivery">Status Delivery</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">7.1 HTML Waterfall</h3>
|
|
|
|
<p><a name="index-Waterfall-85"></a><a name="index-buildbot_002estatus_002ehtml_002eWaterfall-86"></a>
|
|
|
|
<pre class="example"> from buildbot.status import html
|
|
w = html.Waterfall(http_port=8080)
|
|
c['status'].append(w)
|
|
</pre>
|
|
<p>The <code>buildbot.status.html.Waterfall</code> status target creates an
|
|
HTML “waterfall display”, which shows a time-based chart of events.
|
|
This display provides detailed information about all steps of all
|
|
recent builds, and provides hyperlinks to look at individual build
|
|
logs and source changes. If the <code>http_port</code> argument is provided,
|
|
it provides a strports specification for the port that the web server
|
|
should listen on. This can be a simple port number, or a string like
|
|
<code>tcp:8080:interface=127.0.0.1</code> (to limit connections to the
|
|
loopback interface, and therefore to clients running on the same
|
|
host)<a rel="footnote" href="#fn-11" name="fnd-11"><sup>11</sup></a>.
|
|
|
|
<p>If instead (or in addition) you provide the <code>distrib_port</code>
|
|
argument, a twisted.web distributed server will be started either on a
|
|
TCP port (if <code>distrib_port</code> is like <code>"tcp:12345"</code>) or more
|
|
likely on a UNIX socket (if <code>distrib_port</code> is like
|
|
<code>"unix:/path/to/socket"</code>).
|
|
|
|
<p>The <code>distrib_port</code> option means that, on a host with a
|
|
suitably-configured twisted-web server, you do not need to consume a
|
|
separate TCP port for the buildmaster's status web page. When the web
|
|
server is constructed with <code>mktap web --user</code>, URLs that point to
|
|
<code>http://host/~username/</code> are dispatched to a sub-server that is
|
|
listening on a UNIX socket at <code>~username/.twisted-web-pb</code>. On
|
|
such a system, it is convenient to create a dedicated <code>buildbot</code>
|
|
user, then set <code>distrib_port</code> to
|
|
<code>"unix:"+os.path.expanduser("~/.twistd-web-pb")</code>. This
|
|
configuration will make the HTML status page available at
|
|
<code>http://host/~buildbot/</code> . Suitable URL remapping can make it
|
|
appear at <code>http://host/buildbot/</code>, and the right virtual host
|
|
setup can even place it at <code>http://buildbot.host/</code> .
|
|
|
|
<p>Other arguments:
|
|
|
|
<dl>
|
|
<dt><code>allowForce</code><dd>If set to True (the default), then the web page will provide a “Force
|
|
Build” button that allows visitors to manually trigger builds. This
|
|
is useful for developers to re-run builds that have failed because of
|
|
intermittent problems in the test suite, or because of libraries that
|
|
were not installed at the time of the previous build. You may not wish
|
|
to allow strangers to cause a build to run: in that case, set this to
|
|
False to remove these buttons.
|
|
|
|
<br><dt><code>favicon</code><dd>If set to a string, this will be interpreted as a filename containing
|
|
a “favicon”: a small image that contains an icon for the web site.
|
|
This is returned to browsers that request the <code>favicon.ico</code> file,
|
|
and should point to a .png or .ico image file. The default value uses
|
|
the buildbot/buildbot.png image (a small hex nut) contained in the
|
|
buildbot distribution. You can set this to None to avoid using a
|
|
favicon at all.
|
|
|
|
<br><dt><code>robots_txt</code><dd>If set to a string, this will be interpreted as a filename containing
|
|
the contents of “robots.txt”. Many search engine spiders request
|
|
this file before indexing the site. Setting it to a file which
|
|
contains:
|
|
<pre class="example"> User-agent: *
|
|
Disallow: /
|
|
</pre>
|
|
<p>will prevent most search engines from trawling the (voluminous)
|
|
generated status pages.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="IRC-Bot"></a>
|
|
Next: <a rel="next" accesskey="n" href="#PBListener">PBListener</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#HTML-Waterfall">HTML Waterfall</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">7.2 IRC Bot</h3>
|
|
|
|
<p><a name="index-IRC-87"></a><a name="index-buildbot_002estatus_002ewords_002eIRC-88"></a>
|
|
|
|
<p>The <code>buildbot.status.words.IRC</code> status target creates an IRC bot
|
|
which will attach to certain channels and be available for status
|
|
queries. It can also be asked to announce builds as they occur, or be
|
|
told to shut up.
|
|
|
|
<pre class="example"> from twisted.status import words
|
|
irc = words.IRC("irc.example.org", "botnickname",
|
|
channels=["channel1", "channel2"],
|
|
password="mysecretpassword")
|
|
c['status'].append(irc)
|
|
</pre>
|
|
<p>Take a look at the docstring for <code>words.IRC</code> for more details on
|
|
configuring this service. The <code>password</code> argument, if provided,
|
|
will be sent to Nickserv to claim the nickname: some IRC servers will
|
|
not allow clients to send private messages until they have logged in
|
|
with a password.
|
|
|
|
<p>To use the service, you address messages at the buildbot, either
|
|
normally (<code>botnickname: status</code>) or with private messages
|
|
(<code>/msg botnickname status</code>). The buildbot will respond in kind.
|
|
|
|
<p>Some of the commands currently available:
|
|
|
|
<dl>
|
|
<dt><code>list builders</code><dd>Emit a list of all configured builders
|
|
<br><dt><code>status BUILDER</code><dd>Announce the status of a specific Builder: what it is doing right now.
|
|
<br><dt><code>status all</code><dd>Announce the status of all Builders
|
|
<br><dt><code>watch BUILDER</code><dd>If the given Builder is currently running, wait until the Build is
|
|
finished and then announce the results.
|
|
<br><dt><code>last BUILDER</code><dd>Return the results of the last build to run on the given Builder.
|
|
|
|
<br><dt><code>help COMMAND</code><dd>Describe a command. Use <code>help commands</code> to get a list of known
|
|
commands.
|
|
<br><dt><code>source</code><dd>Announce the URL of the Buildbot's home page.
|
|
<br><dt><code>version</code><dd>Announce the version of this Buildbot.
|
|
</dl>
|
|
|
|
<p>If the <code>allowForce=True</code> option was used, some addtional commands
|
|
will be available:
|
|
|
|
<dl>
|
|
<dt><code>force build BUILDER REASON</code><dd>Tell the given Builder to start a build of the latest code. The user
|
|
requesting the build and REASON are recorded in the Build status. The
|
|
buildbot will announce the build's status when it finishes.
|
|
|
|
<br><dt><code>stop build BUILDER REASON</code><dd>Terminate any running build in the given Builder. REASON will be added
|
|
to the build status to explain why it was stopped. You might use this
|
|
if you committed a bug, corrected it right away, and don't want to
|
|
wait for the first build (which is destined to fail) to complete
|
|
before starting the second (hopefully fixed) build.
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="PBListener"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Writing-New-Status-Plugins">Writing New Status Plugins</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#IRC-Bot">IRC Bot</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">7.3 PBListener</h3>
|
|
|
|
<p><a name="index-PBListener-89"></a><a name="index-buildbot_002estatus_002eclient_002ePBListener-90"></a>
|
|
|
|
<pre class="example"> import buildbot.status.client
|
|
pbl = buildbot.status.client.PBListener(port=int, user=str,
|
|
passwd=str)
|
|
c['status'].append(pbl)
|
|
</pre>
|
|
<p>This sets up a PB listener on the given TCP port, to which a PB-based
|
|
status client can connect and retrieve status information.
|
|
<code>buildbot statusgui</code> (see <a href="#statusgui">statusgui</a>) is an example of such a
|
|
status client. The <code>port</code> argument can also be a strports
|
|
specification string.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Writing-New-Status-Plugins"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#PBListener">PBListener</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">7.4 Writing New Status Plugins</h3>
|
|
|
|
<p>TODO: this needs a lot more examples
|
|
|
|
<p>Each status plugin is an object which provides the
|
|
<code>twisted.application.service.IService</code> interface, which creates a
|
|
tree of Services with the buildmaster at the top [not strictly true].
|
|
The status plugins are all children of an object which implements
|
|
<code>buildbot.interfaces.IStatus</code>, the main status object. From this
|
|
object, the plugin can retrieve anything it wants about current and
|
|
past builds. It can also subscribe to hear about new and upcoming
|
|
builds.
|
|
|
|
<p>Status plugins which only react to human queries (like the Waterfall
|
|
display) never need to subscribe to anything: they are idle until
|
|
someone asks a question, then wake up and extract the information they
|
|
need to answer it, then they go back to sleep. Plugins which need to
|
|
act spontaneously when builds complete (like the Mail plugin) need to
|
|
subscribe to hear about new builds.
|
|
|
|
<p>If the status plugin needs to run network services (like the HTTP
|
|
server used by the Waterfall plugin), they can be attached as Service
|
|
children of the plugin itself, using the <code>IServiceCollection</code>
|
|
interface.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Command-line-tool"></a>
|
|
<a name="Command_002dline-tool"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Resources">Resources</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Status-Delivery">Status Delivery</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="chapter">8 Command-line tool</h2>
|
|
|
|
<p>The <samp><span class="command">buildbot</span></samp> command-line tool can be used to start or stop a
|
|
buildmaster or buildbot, and to interact with a running buildmaster.
|
|
Some of its subcommands are intended for buildmaster admins, while
|
|
some are for developers who are editing the code that the buildbot is
|
|
monitoring.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#Administrator-Tools">Administrator Tools</a>
|
|
<li><a accesskey="2" href="#Developer-Tools">Developer Tools</a>
|
|
<li><a accesskey="3" href="#Other-Tools">Other Tools</a>
|
|
<li><a accesskey="4" href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Administrator-Tools"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Developer-Tools">Developer Tools</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Command_002dline-tool">Command-line tool</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">8.1 Administrator Tools</h3>
|
|
|
|
<p>The following <samp><span class="command">buildbot</span></samp> sub-commands are intended for
|
|
buildmaster administrators:
|
|
|
|
<h3 class="heading">create-master</h3>
|
|
|
|
<p>This creates a new directory and populates it with files that allow it
|
|
to be used as a buildmaster's base directory.
|
|
|
|
<pre class="example"> buildbot create-master BASEDIR
|
|
</pre>
|
|
<h3 class="heading">create-slave</h3>
|
|
|
|
<p>This creates a new directory and populates it with files that let it
|
|
be used as a buildslave's base directory. You must provide several
|
|
arguments, which are used to create the initial <samp><span class="file">buildbot.tac</span></samp>
|
|
file.
|
|
|
|
<pre class="example"> buildbot create-slave <var>BASEDIR</var> <var>MASTERHOST</var>:<var>PORT</var> <var>SLAVENAME</var> <var>PASSWORD</var>
|
|
</pre>
|
|
<h3 class="heading">start</h3>
|
|
|
|
<p>This starts a buildmaster or buildslave which was already created in
|
|
the given base directory. The daemon is launched in the background,
|
|
with events logged to a file named <samp><span class="file">twistd.log</span></samp>.
|
|
|
|
<pre class="example"> buildbot start BASEDIR
|
|
</pre>
|
|
<h3 class="heading">stop</h3>
|
|
|
|
<p>This terminates the daemon (either buildmaster or buildslave) running
|
|
in the given directory.
|
|
|
|
<pre class="example"> buildbot stop BASEDIR
|
|
</pre>
|
|
<h3 class="heading">sighup</h3>
|
|
|
|
<p>This sends a SIGHUP to the buildmaster running in the given directory,
|
|
which causes it to re-read its <samp><span class="file">master.cfg</span></samp> file.
|
|
|
|
<pre class="example"> buildbot sighup BASEDIR
|
|
</pre>
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Developer-Tools"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Other-Tools">Other Tools</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Administrator-Tools">Administrator Tools</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">8.2 Developer Tools</h3>
|
|
|
|
<p>These tools are provided for use by the developers who are working on
|
|
the code that the buildbot is monitoring.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#statuslog">statuslog</a>
|
|
<li><a accesskey="2" href="#statusgui">statusgui</a>
|
|
<li><a accesskey="3" href="#try">try</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="statuslog"></a>
|
|
Next: <a rel="next" accesskey="n" href="#statusgui">statusgui</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Developer-Tools">Developer Tools</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">8.2.1 statuslog</h4>
|
|
|
|
<pre class="example"> buildbot statuslog --master <var>MASTERHOST</var>:<var>PORT</var>
|
|
</pre>
|
|
<p>This command starts a simple text-based status client, one which just
|
|
prints out a new line each time an event occurs on the buildmaster.
|
|
|
|
<p>The <samp><span class="option">--master</span></samp> option provides the location of the
|
|
<code>buildbot.status.client.PBListener</code> status port, used to deliver
|
|
build information to realtime status clients. The option is always in
|
|
the form of a string, with hostname and port number separated by a
|
|
colon (<code>HOSTNAME:PORTNUM</code>). Note that this port is <em>not</em> the
|
|
same as the slaveport (although a future version may allow the same
|
|
port number to be used for both purposes). If you get an error message
|
|
to the effect of “Failure: twisted.cred.error.UnauthorizedLogin:”,
|
|
this may indicate that you are connecting to the slaveport rather than
|
|
a <code>PBListener</code> port.
|
|
|
|
<p>The <samp><span class="option">--master</span></samp> option can also be provided by the
|
|
<code>masterstatus</code> name in <samp><span class="file">.buildbot/options</span></samp> (see <a href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>).
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="statusgui"></a>
|
|
Next: <a rel="next" accesskey="n" href="#try">try</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#statuslog">statuslog</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">8.2.2 statusgui</h4>
|
|
|
|
<p><a name="index-statusgui-91"></a>
|
|
If you have set up a PBListener (see <a href="#PBListener">PBListener</a>), you will be able
|
|
to monitor your Buildbot using a simple Gtk+ application invoked with
|
|
the <code>buildbot statusgui</code> command:
|
|
|
|
<pre class="example"> buildbot statusgui --master <var>MASTERHOST</var>:<var>PORT</var>
|
|
</pre>
|
|
<p>This command starts a simple Gtk+-based status client, which contains
|
|
a few boxes for each Builder that change color as events occur. It
|
|
uses the same <samp><span class="option">--master</span></samp> argument as the <samp><span class="command">buildbot
|
|
statuslog</span></samp> command (see <a href="#statuslog">statuslog</a>).
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="try"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#statusgui">statusgui</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">8.2.3 try</h4>
|
|
|
|
<p>This lets a developer to ask the question “What would happen if I
|
|
committed this patch right now?”. It runs the unit test suite (across
|
|
multiple build platforms) on the developer's current code, allowing
|
|
them to make sure they will not break the tree when they finally
|
|
commit their changes.
|
|
|
|
<p>The <samp><span class="command">buildbot try</span></samp> command is meant to be run from within a
|
|
developer's local tree, and starts by figuring out the base revision
|
|
of that tree (what revision was current the last time the tree was
|
|
updated), and a patch that can be applied to that revision of the tree
|
|
to make it match the developer's copy. This (revision, patch) pair is
|
|
then sent to the buildmaster, which runs a build with that
|
|
SourceStamp. If you want, the tool will emit status messages as the
|
|
builds run, and will not terminate until the first failure has been
|
|
detected (or the last success).
|
|
|
|
<p>For this command to work, several pieces must be in place:
|
|
|
|
<h3 class="heading">TryScheduler</h3>
|
|
|
|
<p><a name="index-buildbot_002escheduler_002eTry_005fJobdir-92"></a><a name="index-buildbot_002escheduler_002eTry_005fUserpass-93"></a>
|
|
The buildmaster must have a <code>scheduler.Try</code> instance in
|
|
the config file's <code>c['schedulers']</code> list. This lets the
|
|
administrator control who may initiate these “trial” builds, which
|
|
branches are eligible for trial builds, and which Builders should be
|
|
used for them.
|
|
|
|
<p>The <code>TryScheduler</code> has various means to accept build requests:
|
|
all of them enforce more security than the usual buildmaster ports do.
|
|
Any source code being built can be used to compromise the buildslave
|
|
accounts, but in general that code must be checked out from the VC
|
|
repository first, so only people with commit privileges can get
|
|
control of the buildslaves. The usual force-build control channels can
|
|
waste buildslave time but do not allow arbitrary commands to be
|
|
executed by people who don't have those commit privileges. However,
|
|
the source code patch that is provided with the trial build does not
|
|
have to go through the VC system first, so it is important to make
|
|
sure these builds cannot be abused by a non-committer to acquire as
|
|
much control over the buildslaves as a committer has. Ideally, only
|
|
developers who have commit access to the VC repository would be able
|
|
to start trial builds, but unfortunately the buildmaster does not, in
|
|
general, have access to VC system's user list.
|
|
|
|
<p>As a result, the <code>TryScheduler</code> requires a bit more
|
|
configuration. There are currently two ways to set this up:
|
|
|
|
<dl>
|
|
<dt><strong>jobdir (ssh)</strong><dd>
|
|
This approach creates a command queue directory, called the
|
|
“jobdir”, in the buildmaster's working directory. The buildmaster
|
|
admin sets the ownership and permissions of this directory to only
|
|
grant write access to the desired set of developers, all of whom must
|
|
have accounts on the machine. The <code>buildbot try</code> command creates
|
|
a special file containing the source stamp information and drops it in
|
|
the jobdir, just like a standard maildir. When the buildmaster notices
|
|
the new file, it unpacks the information inside and starts the builds.
|
|
|
|
<p>The config file entries used by 'buildbot try' either specify a local
|
|
queuedir (for which write and mv are used) or a remote one (using scp
|
|
and ssh).
|
|
|
|
<p>The advantage of this scheme is that it is quite secure, the
|
|
disadvantage is that it requires fiddling outside the buildmaster
|
|
config (to set the permissions on the jobdir correctly). If the
|
|
buildmaster machine happens to also house the VC repository, then it
|
|
can be fairly easy to keep the VC userlist in sync with the
|
|
trial-build userlist. If they are on different machines, this will be
|
|
much more of a hassle. It may also involve granting developer accounts
|
|
on a machine that would not otherwise require them.
|
|
|
|
<p>To implement this, the buildslave invokes 'ssh -l username host
|
|
buildbot tryserver ARGS', passing the patch contents over stdin. The
|
|
arguments must include the inlet directory and the revision
|
|
information.
|
|
|
|
<br><dt><strong>user+password (PB)</strong><dd>
|
|
In this approach, each developer gets a username/password pair, which
|
|
are all listed in the buildmaster's configuration file. When the
|
|
developer runs <code>buildbot try</code>, their machine connects to the
|
|
buildmaster via PB and authenticates themselves using that username
|
|
and password, then sends a PB command to start the trial build.
|
|
|
|
<p>The advantage of this scheme is that the entire configuration is
|
|
performed inside the buildmaster's config file. The disadvantages are
|
|
that it is less secure (while the “cred” authentication system does
|
|
not expose the password in plaintext over the wire, it does not offer
|
|
most of the other security properties that SSH does). In addition, the
|
|
buildmaster admin is responsible for maintaining the username/password
|
|
list, adding and deleting entries as developers come and go.
|
|
|
|
</dl>
|
|
|
|
<p>For example, to set up the “jobdir” style of trial build, using a
|
|
command queue directory of <samp><span class="file">MASTERDIR/jobdir</span></samp> (and assuming that
|
|
all your project developers were members of the <code>developers</code> unix
|
|
group), you would first create that directory (with <samp><span class="command">mkdir
|
|
MASTERDIR/jobdir MASTERDIR/jobdir/new MASTERDIR/jobdir/cur
|
|
MASTERDIR/jobdir/tmp; chgrp developers MASTERDIR/jobdir
|
|
MASTERDIR/jobdir/*; chmod g+rwx,o-rwx MASTERDIR/jobdir
|
|
MASTERDIR/jobdir/*</span></samp>), and then use the following scheduler in the
|
|
buildmaster's config file:
|
|
|
|
<pre class="example"> from buildbot.scheduler import Try_Jobdir
|
|
s = Try_Jobdir("try1", ["full-linux", "full-netbsd", "full-OSX"],
|
|
jobdir="jobdir")
|
|
c['schedulers'] = [s]
|
|
</pre>
|
|
<p>Note that you must create the jobdir before telling the buildmaster to
|
|
use this configuration, otherwise you will get an error. Also remember
|
|
that the buildmaster must be able to read and write to the jobdir as
|
|
well. Be sure to watch the <samp><span class="file">twistd.log</span></samp> file (see <a href="#Logfiles">Logfiles</a>)
|
|
as you start using the jobdir, to make sure the buildmaster is happy
|
|
with it.
|
|
|
|
<p>To use the username/password form of authentication, create a
|
|
<code>Try_Userpass</code> instance instead. It takes the same
|
|
<code>builderNames</code> argument as the <code>Try_Jobdir</code> form, but
|
|
accepts an addtional <code>port</code> argument (to specify the TCP port to
|
|
listen on) and a <code>userpass</code> list of username/password pairs to
|
|
accept. Remember to use good passwords for this: the security of the
|
|
buildslave accounts depends upon it:
|
|
|
|
<pre class="example"> from buildbot.scheduler import Try_Userpass
|
|
s = Try_Userpass("try2", ["full-linux", "full-netbsd", "full-OSX"],
|
|
port=8031, userpass=[("alice","pw1"), ("bob", "pw2")] )
|
|
c['schedulers'] = [s]
|
|
</pre>
|
|
<p>Like most places in the buildbot, the <code>port</code> argument takes a
|
|
strports specification. See <code>twisted.application.strports</code> for
|
|
details.
|
|
|
|
<h3 class="heading">locating the master</h3>
|
|
|
|
<p>The <samp><span class="command">try</span></samp> command needs to be told how to connect to the
|
|
<code>TryScheduler</code>, and must know which of the authentication
|
|
approaches described above is in use by the buildmaster. You specify
|
|
the approach by using <samp><span class="option">--connect=ssh</span></samp> or <samp><span class="option">--connect=pb</span></samp>
|
|
(or <code>try_connect = 'ssh'</code> or <code>try_connect = 'pb'</code> in
|
|
<samp><span class="file">.buildbot/options</span></samp>).
|
|
|
|
<p>For the PB approach, the command must be given a <samp><span class="option">--master</span></samp>
|
|
argument (in the form HOST:PORT) that points to TCP port that you
|
|
picked in the <code>Try_Userpass</code> scheduler. It also takes a
|
|
<samp><span class="option">--username</span></samp> and <samp><span class="option">--passwd</span></samp> pair of arguments that match
|
|
one of the entries in the buildmaster's <code>userpass</code> list. These
|
|
arguments can also be provided as <code>try_master</code>,
|
|
<code>try_username</code>, and <code>try_password</code> entries in the
|
|
<samp><span class="file">.buildbot/options</span></samp> file.
|
|
|
|
<p>For the SSH approach, the command must be given <samp><span class="option">--tryhost</span></samp>,
|
|
<samp><span class="option">--username</span></samp>, and optionally <samp><span class="option">--password</span></samp> (TODO:
|
|
really?) to get to the buildmaster host. It must also be given
|
|
<samp><span class="option">--trydir</span></samp>, which points to the inlet directory configured
|
|
above. The trydir can be relative to the user's home directory, but
|
|
most of the time you will use an explicit path like
|
|
<samp><span class="file">~buildbot/project/trydir</span></samp>. These arguments can be provided in
|
|
<samp><span class="file">.buildbot/options</span></samp> as <code>try_host</code>, <code>try_username</code>,
|
|
<code>try_password</code>, and <code>try_dir</code>.
|
|
|
|
<p>In addition, the SSH approach needs to connect to a PBListener status
|
|
port, so it can retrieve and report the results of the build (the PB
|
|
approach uses the existing connection to retrieve status information,
|
|
so this step is not necessary). This requires a <samp><span class="option">--master</span></samp>
|
|
argument, or a <code>masterstatus</code> entry in <samp><span class="file">.buildbot/options</span></samp>,
|
|
in the form of a HOSTNAME:PORT string.
|
|
|
|
<h3 class="heading">choosing the Builders</h3>
|
|
|
|
<p>A trial build is performed on multiple Builders at the same time, and
|
|
the developer gets to choose which Builders are used (limited to a set
|
|
selected by the buildmaster admin with the TryScheduler's
|
|
<code>builderNames=</code> argument). The set you choose will depend upon
|
|
what your goals are: if you are concerned about cross-platform
|
|
compatibility, you should use multiple Builders, one from each
|
|
platform of interest. You might use just one builder if that platform
|
|
has libraries or other facilities that allow better test coverage than
|
|
what you can accomplish on your own machine, or faster test runs.
|
|
|
|
<p>The set of Builders to use can be specified with multiple
|
|
<samp><span class="option">--builder</span></samp> arguments on the command line. It can also be
|
|
specified with a single <code>try_builders</code> option in
|
|
<samp><span class="file">.buildbot/options</span></samp> that uses a list of strings to specify all
|
|
the Builder names:
|
|
|
|
<pre class="example"> try_builders = ["full-OSX", "full-win32", "full-linux"]
|
|
</pre>
|
|
<h3 class="heading">specifying the VC system</h3>
|
|
|
|
<p>The <samp><span class="command">try</span></samp> command also needs to know how to take the
|
|
developer's current tree and extract the (revision, patch)
|
|
source-stamp pair. Each VC system uses a different process, so you
|
|
start by telling the <samp><span class="command">try</span></samp> command which VC system you are
|
|
using, with an argument like <samp><span class="option">--vc=cvs</span></samp> or <samp><span class="option">--vc=tla</span></samp>.
|
|
This can also be provided as <code>try_vc</code> in
|
|
<samp><span class="file">.buildbot/options</span></samp>.
|
|
|
|
<p>The following names are recognized: <code>cvs</code> <code>svn</code> <code>baz</code>
|
|
<code>tla</code> <code>hg</code> <code>darcs</code>
|
|
|
|
<h3 class="heading">finding the top of the tree</h3>
|
|
|
|
<p>Some VC systems (notably CVS and SVN) track each directory
|
|
more-or-less independently, which means the <samp><span class="command">try</span></samp> command
|
|
needs to move up to the top of the project tree before it will be able
|
|
to construct a proper full-tree patch. To accomplish this, the
|
|
<samp><span class="command">try</span></samp> command will crawl up through the parent directories
|
|
until it finds a marker file. The default name for this marker file is
|
|
<samp><span class="file">.buildbot-top</span></samp>, so when you are using CVS or SVN you should
|
|
<code>touch .buildbot-top</code> from the top of your tree before running
|
|
<samp><span class="command">buildbot try</span></samp>. Alternatively, you can use a filename like
|
|
<samp><span class="file">ChangeLog</span></samp> or <samp><span class="file">README</span></samp>, since many projects put one of
|
|
these files in their top-most directory (and nowhere else). To set
|
|
this filename, use <samp><span class="option">--try-topfile=ChangeLog</span></samp>, or set it in the
|
|
options file with <code>try_topfile = 'ChangeLog'</code>.
|
|
|
|
<p>You can also manually set the top of the tree with
|
|
<samp><span class="option">--try-topdir=~/trees/mytree</span></samp>, or <code>try_topdir =
|
|
'~/trees/mytree'</code>. If you use <code>try_topdir</code>, in a
|
|
<samp><span class="file">.buildbot/options</span></samp> file, you will need a separate options file
|
|
for each tree you use, so it may be more convenient to use the
|
|
<code>try_topfile</code> approach instead.
|
|
|
|
<p>Other VC systems which work on full projects instead of individual
|
|
directories (tla, baz, darcs, monotone, mercurial) do not require
|
|
<samp><span class="command">try</span></samp> to know the top directory, so the <samp><span class="option">--try-topfile</span></samp>
|
|
and <samp><span class="option">--try-topdir</span></samp> arguments will be ignored.
|
|
<!-- is this true? I think I currently require topdirs all the time. -->
|
|
|
|
<p>If the <samp><span class="command">try</span></samp> command cannot find the top directory, it will
|
|
abort with an error message.
|
|
|
|
<h3 class="heading">determining the branch name</h3>
|
|
|
|
<p>Some VC systems record the branch information in a way that “try”
|
|
can locate it, in particular Arch (both <samp><span class="command">tla</span></samp> and
|
|
<samp><span class="command">baz</span></samp>). For the others, if you are using something other than
|
|
the default branch, you will have to tell the buildbot which branch
|
|
your tree is using. You can do this with either the <samp><span class="option">--branch</span></samp>
|
|
argument, or a <samp><span class="option">try_branch</span></samp> entry in the
|
|
<samp><span class="file">.buildbot/options</span></samp> file.
|
|
|
|
<h3 class="heading">determining the revision and patch</h3>
|
|
|
|
<p>Each VC system has a separate approach for determining the tree's base
|
|
revision and computing a patch.
|
|
|
|
<dl>
|
|
<dt><code>CVS</code><dd>
|
|
<samp><span class="command">try</span></samp> pretends that the tree is up to date. It converts the
|
|
current time into a <code>-D</code> time specification, uses it as the base
|
|
revision, and computes the diff between the upstream tree as of that
|
|
point in time versus the current contents. This works, more or less,
|
|
but requires that the local clock be in reasonably good sync with the
|
|
repository.
|
|
|
|
<br><dt><code>SVN</code><dd><samp><span class="command">try</span></samp> does a <code>svn status -u</code> to find the latest
|
|
repository revision number (emitted on the last line in the “Status
|
|
against revision: NN” message). It then performs an <code>svn diff
|
|
-rNN</code> to find out how your tree differs from the repository version,
|
|
and sends the resulting patch to the buildmaster. If your tree is not
|
|
up to date, this will result in the “try” tree being created with
|
|
the latest revision, then <em>backwards</em> patches applied to bring it
|
|
“back” to the version you actually checked out (plus your actual
|
|
code changes), but this will still result in the correct tree being
|
|
used for the build.
|
|
|
|
<br><dt><code>baz</code><dd><samp><span class="command">try</span></samp> does a <code>baz tree-id</code> to determine the
|
|
fully-qualified version and patch identifier for the tree
|
|
(ARCHIVE/VERSION–patch-NN), and uses the VERSION–patch-NN component
|
|
as the base revision. It then does a <code>baz diff</code> to obtain the
|
|
patch.
|
|
|
|
<br><dt><code>tla</code><dd><samp><span class="command">try</span></samp> does a <code>tla tree-version</code> to get the
|
|
fully-qualified version identifier (ARCHIVE/VERSION), then takes the
|
|
first line of <code>tla logs --reverse</code> to figure out the base
|
|
revision. Then it does <code>tla changes --diffs</code> to obtain the patch.
|
|
|
|
<br><dt><code>Darcs</code><dd><code>darcs changes --context</code> emits a text file that contains a list
|
|
of all patches back to and including the last tag was made. This text
|
|
file (plus the location of a repository that contains all these
|
|
patches) is sufficient to re-create the tree. Therefore the contents
|
|
of this “context” file <em>are</em> the revision stamp for a
|
|
Darcs-controlled source tree.
|
|
|
|
<p>So <samp><span class="command">try</span></samp> does a <code>darcs changes --context</code> to determine
|
|
what your tree's base revision is, and then does a <code>darcs diff
|
|
-u</code> to compute the patch relative to that revision.
|
|
|
|
<br><dt><code>Mercurial</code><dd><code>hg identify</code> emits a short revision ID (basically a truncated
|
|
SHA1 hash of the current revision's contents), which is used as the
|
|
base revision. <code>hg diff</code> then provides the patch relative to that
|
|
revision. For <samp><span class="command">try</span></samp> to work, your working directory must only
|
|
have patches that are available from the same remotely-available
|
|
repository that the build process' <code>step.Mercurial</code> will use.
|
|
|
|
<!-- TODO: monotone, git -->
|
|
</dl>
|
|
|
|
<h3 class="heading">waiting for results</h3>
|
|
|
|
<p>If you provide the <samp><span class="option">--wait</span></samp> option (or <code>try_wait = True</code>
|
|
in <samp><span class="file">.buildbot/options</span></samp>), the <samp><span class="command">buildbot try</span></samp> command will
|
|
wait until your changes have either been proven good or bad before
|
|
exiting. Unless you use the <samp><span class="option">--quiet</span></samp> option (or
|
|
<code>try_quiet=True</code>), it will emit a progress message every 60
|
|
seconds until the builds have completed.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Other-Tools"></a>
|
|
Next: <a rel="next" accesskey="n" href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Developer-Tools">Developer Tools</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">8.3 Other Tools</h3>
|
|
|
|
<p>These tools are generally used by buildmaster administrators.
|
|
|
|
<ul class="menu">
|
|
<li><a accesskey="1" href="#sendchange">sendchange</a>
|
|
<li><a accesskey="2" href="#debugclient">debugclient</a>
|
|
</ul>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="sendchange"></a>
|
|
Next: <a rel="next" accesskey="n" href="#debugclient">debugclient</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Other-Tools">Other Tools</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Other-Tools">Other Tools</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">8.3.1 sendchange</h4>
|
|
|
|
<p>This command is used to tell the buildmaster about source changes. It
|
|
is intended to be used from within a commit script, installed on the
|
|
VC server. It requires that you have a PBChangeSource
|
|
(see <a href="#PBChangeSource">PBChangeSource</a>) running in the buildmaster (by being included
|
|
in the <code>c['sources']</code> list).
|
|
|
|
<pre class="example"> buildbot sendchange --master <var>MASTERHOST</var>:<var>PORT</var> --username <var>USER</var> <var>FILENAMES..</var>
|
|
</pre>
|
|
<p>There are other (optional) arguments which can influence the
|
|
<code>Change</code> that gets submitted:
|
|
|
|
<dl>
|
|
<dt><code>--branch</code><dd>This provides the (string) branch specifier. If omitted, it defaults
|
|
to None, indicating the “default branch”. All files included in this
|
|
Change must be on the same branch.
|
|
|
|
<br><dt><code>--revision_number</code><dd>This provides a (numeric) revision number for the change, used for VC systems
|
|
that use numeric transaction numbers (like Subversion).
|
|
|
|
<br><dt><code>--revision</code><dd>This provides a (string) revision specifier, for VC systems that use
|
|
strings (Arch would use something like patch-42 etc).
|
|
|
|
<br><dt><code>--revision_file</code><dd>This provides a filename which will be opened and the contents used as
|
|
the revision specifier. This is specifically for Darcs, which uses the
|
|
output of <samp><span class="command">darcs changes --context</span></samp> as a revision specifier.
|
|
This context file can be a couple of kilobytes long, spanning a couple
|
|
lines per patch, and would be a hassle to pass as a command-line
|
|
argument.
|
|
|
|
<br><dt><code>--comments</code><dd>This provides the change comments as a single argument. You may want
|
|
to use <samp><span class="option">--logfile</span></samp> instead.
|
|
|
|
<br><dt><code>--logfile</code><dd>This instructs the tool to read the change comments from the given
|
|
file. If you use <code>-</code> as the filename, the tool will read the
|
|
change comments from stdin.
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="debugclient"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#sendchange">sendchange</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Other-Tools">Other Tools</a>
|
|
|
|
</div>
|
|
|
|
<h4 class="subsection">8.3.2 debugclient</h4>
|
|
|
|
<pre class="example"> buildbot debugclient --master <var>MASTERHOST</var>:<var>PORT</var> --passwd <var>DEBUGPW</var>
|
|
</pre>
|
|
<p>This launches a small Gtk+/Glade-based debug tool, connecting to the
|
|
buildmaster's “debug port”. This debug port shares the same port
|
|
number as the slaveport (see <a href="#Setting-the-slaveport">Setting the slaveport</a>), but the
|
|
<code>debugPort</code> is only enabled if you set a debug password in the
|
|
buildmaster's config file (see <a href="#Debug-options">Debug options</a>). The
|
|
<samp><span class="option">--passwd</span></samp> option must match the <code>c['debugPassword']</code>
|
|
value.
|
|
|
|
<p><samp><span class="option">--master</span></samp> can also be provided in <samp><span class="file">.debug/options</span></samp> by the
|
|
<code>master</code> key. <samp><span class="option">--passwd</span></samp> can be provided by the
|
|
<code>debugPassword</code> key.
|
|
|
|
<p>The <code>Connect</code> button must be pressed before any of the other
|
|
buttons will be active. This establishes the connection to the
|
|
buildmaster. The other sections of the tool are as follows:
|
|
|
|
<dl>
|
|
<dt><code>Reload .cfg</code><dd>Forces the buildmaster to reload its <samp><span class="file">master.cfg</span></samp> file. This is
|
|
equivalent to sending a SIGHUP to the buildmaster, but can be done
|
|
remotely through the debug port. Note that it is a good idea to be
|
|
watching the buildmaster's <samp><span class="file">twistd.log</span></samp> as you reload the config
|
|
file, as any errors which are detected in the config file will be
|
|
announced there.
|
|
|
|
<br><dt><code>Rebuild .py</code><dd>(not yet implemented). The idea here is to use Twisted's “rebuild”
|
|
facilities to replace the buildmaster's running code with a new
|
|
version. Even if this worked, it would only be used by buildbot
|
|
developers.
|
|
|
|
<br><dt><code>poke IRC</code><dd>This locates a <code>words.IRC</code> status target and causes it to emit a
|
|
message on all the channels to which it is currently connected. This
|
|
was used to debug a problem in which the buildmaster lost the
|
|
connection to the IRC server and did not attempt to reconnect.
|
|
|
|
<br><dt><code>Commit</code><dd>This allows you to inject a Change, just as if a real one had been
|
|
delivered by whatever VC hook you are using. You can set the name of
|
|
the committed file and the name of the user who is doing the commit.
|
|
Optionally, you can also set a revision for the change. If the
|
|
revision you provide looks like a number, it will be sent as an
|
|
integer, otherwise it will be sent as a string.
|
|
|
|
<br><dt><code>Force Build</code><dd>This lets you force a Builder (selected by name) to start a build of
|
|
the current source tree.
|
|
|
|
<br><dt><code>Currently</code><dd>(obsolete). This was used to manually set the status of the given
|
|
Builder, but the status-assignment code was changed in an incompatible
|
|
way and these buttons are no longer meaningful.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name=".buildbot-config-directory"></a>
|
|
<a name="g_t_002ebuildbot-config-directory"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Other-Tools">Other Tools</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a>
|
|
|
|
</div>
|
|
|
|
<h3 class="section">8.4 .buildbot config directory</h3>
|
|
|
|
<p>Many of the <samp><span class="command">buildbot</span></samp> tools must be told how to contact the
|
|
buildmaster that they interact with. This specification can be
|
|
provided as a command-line argument, but most of the time it will be
|
|
easier to set them in an “options” file. The <samp><span class="command">buildbot</span></samp>
|
|
command will look for a special directory named <samp><span class="file">.buildbot</span></samp>,
|
|
starting from the current directory (where the command was run) and
|
|
crawling upwards, eventually looking in the user's home directory. It
|
|
will look for a file named <samp><span class="file">options</span></samp> in this directory, and will
|
|
evaluate it as a python script, looking for certain names to be set.
|
|
You can just put simple <code>name = 'value'</code> pairs in this file to
|
|
set the options.
|
|
|
|
<p>For a description of the names used in this file, please see the
|
|
documentation for the individual <samp><span class="command">buildbot</span></samp> sub-commands. The
|
|
following is a brief sample of what this file's contents could be.
|
|
|
|
<pre class="example"> # for status-reading tools
|
|
masterstatus = 'buildbot.example.org:12345'
|
|
# for 'sendchange' or the debug port
|
|
master = 'buildbot.example.org:18990'
|
|
debugPassword = 'eiv7Po'
|
|
</pre>
|
|
<dl>
|
|
<dt><code>masterstatus</code><dd>Location of the <code>client.PBListener</code> status port, used by
|
|
<samp><span class="command">statuslog</span></samp> and <samp><span class="command">statusgui</span></samp>.
|
|
|
|
<br><dt><code>master</code><dd>Location of the <code>debugPort</code> (for <samp><span class="command">debugclient</span></samp>). Also the
|
|
location of the <code>pb.PBChangeSource</code> (for <samp><span class="command">sendchange</span></samp>).
|
|
Usually shares the slaveport, but a future version may make it
|
|
possible to have these listen on a separate port number.
|
|
|
|
<br><dt><code>debugPassword</code><dd>Must match the value of <code>c['debugPassword']</code>, used to protect the
|
|
debug port, for the <samp><span class="command">debugclient</span></samp> command.
|
|
|
|
<br><dt><code>username</code><dd>Provides a default username for the <samp><span class="command">sendchange</span></samp> command.
|
|
|
|
</dl>
|
|
|
|
<p>The following options are used by the <code>buildbot try</code> command
|
|
(see <a href="#try">try</a>):
|
|
|
|
<dl>
|
|
<dt><code>try_connect</code><dd>This specifies how the “try” command should deliver its request to
|
|
the buildmaster. The currently accepted values are “ssh” and “pb”.
|
|
<br><dt><code>try_builders</code><dd>Which builders should be used for the “try” build.
|
|
<br><dt><code>try_vc</code><dd>This specifies the version control system being used.
|
|
<br><dt><code>try_branch</code><dd>This indicates that the current tree is on a non-trunk branch.
|
|
<br><dt><code>try_topdir</code><br><dt><code>try_topfile</code><dd>Use <code>try_topdir</code> to explicitly indicate the top of your working
|
|
tree, or <code>try_topfile</code> to name a file that will only be found in
|
|
that top-most directory.
|
|
|
|
<br><dt><code>try_host</code><br><dt><code>try_username</code><br><dt><code>try_dir</code><dd>When try_connect is “ssh”, the command will pay attention to
|
|
<code>try_host</code>, <code>try_username</code>, and <code>try_dir</code>.
|
|
|
|
<br><dt><code>try_username</code><br><dt><code>try_password</code><br><dt><code>try_master</code><dd>Instead, when <code>try_connect</code> is “pb”, the command will pay
|
|
attention to <code>try_username</code>, <code>try_password</code>, and
|
|
<code>try_master</code>.
|
|
|
|
<br><dt><code>try_wait</code><br><dt><code>masterstatus</code><dd><code>try_wait</code> and <code>masterstatus</code> are used to ask the “try”
|
|
command to wait for the requested build to complete.
|
|
|
|
</dl>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Resources"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Developer_0027s-Appendix">Developer's Appendix</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Command_002dline-tool">Command-line tool</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="chapter">9 Resources</h2>
|
|
|
|
<p>The Buildbot's home page is at <a href="http://buildbot.sourceforge.net/">http://buildbot.sourceforge.net/</a>
|
|
|
|
<p>For configuration questions and general discussion, please use the
|
|
<code>buildbot-devel</code> mailing list. The subscription instructions and
|
|
archives are available at
|
|
<a href="http://lists.sourceforge.net/lists/listinfo/buildbot-devel">http://lists.sourceforge.net/lists/listinfo/buildbot-devel</a>
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Developer's-Appendix"></a>
|
|
<a name="Developer_0027s-Appendix"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Index-of-Useful-Classes">Index of Useful Classes</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Resources">Resources</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="unnumbered">Developer's Appendix</h2>
|
|
|
|
<p>This appendix contains random notes about the implementation of the
|
|
Buildbot, and is likely to only be of use to people intending to
|
|
extend the Buildbot's internals.
|
|
|
|
<p>The buildmaster consists of a tree of Service objects, which is shaped
|
|
as follows:
|
|
|
|
<pre class="example"> BuildMaster
|
|
ChangeMaster (in .change_svc)
|
|
[IChangeSource instances]
|
|
[IScheduler instances] (in .schedulers)
|
|
BotMaster (in .botmaster)
|
|
[IStatusTarget instances] (in .statusTargets)
|
|
</pre>
|
|
<p>The BotMaster has a collection of Builder objects as values of its
|
|
<code>.builders</code> dictionary.
|
|
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Index-of-Useful-Classes"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Developer_0027s-Appendix">Developer's Appendix</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="unnumbered">Index of Useful Classes</h2>
|
|
|
|
<p>This is a list of all user-visible classes. There are the ones that
|
|
are useful in <samp><span class="file">master.cfg</span></samp>, the buildmaster's configuration file.
|
|
Classes that are not listed here are generally internal things that
|
|
admins are unlikely to have much use for.
|
|
|
|
<h3 class="heading">Change Sources</h3>
|
|
|
|
<ul class="index-cs" compact>
|
|
<li><a href="#index-buildbot_002echanges_002ebonsaipoller_002eBonsaiPoller-36"><code>buildbot.changes.bonsaipoller.BonsaiPoller</code></a>: <a href="#BonsaiPoller">BonsaiPoller</a></li>
|
|
<li><a href="#index-buildbot_002echanges_002efreshcvs_002eFreshCVSSource-30"><code>buildbot.changes.freshcvs.FreshCVSSource</code></a>: <a href="#CVSToys-_002d-PBService">CVSToys - PBService</a></li>
|
|
<li><a href="#index-buildbot_002echanges_002email_002eBonsaiMaildirSource-33"><code>buildbot.changes.mail.BonsaiMaildirSource</code></a>: <a href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a></li>
|
|
<li><a href="#index-buildbot_002echanges_002email_002eFCMaildirSource-31"><code>buildbot.changes.mail.FCMaildirSource</code></a>: <a href="#CVSToys-_002d-mail-notification">CVSToys - mail notification</a></li>
|
|
<li><a href="#index-buildbot_002echanges_002email_002eSyncmailMaildirSource-32"><code>buildbot.changes.mail.SyncmailMaildirSource</code></a>: <a href="#Other-mail-notification-ChangeSources">Other mail notification ChangeSources</a></li>
|
|
<li><a href="#index-buildbot_002echanges_002ep4poller_002eP4Source-35"><code>buildbot.changes.p4poller.P4Source</code></a>: <a href="#P4Source">P4Source</a></li>
|
|
<li><a href="#index-buildbot_002echanges_002epb_002ePBChangeSource-34"><code>buildbot.changes.pb.PBChangeSource</code></a>: <a href="#PBChangeSource">PBChangeSource</a></li>
|
|
<li><a href="#index-buildbot_002echanges_002esvnpoller_002eSVNPoller-37"><code>buildbot.changes.svnpoller.SVNPoller</code></a>: <a href="#SVNPoller">SVNPoller</a></li>
|
|
</ul><h3 class="heading">Schedulers and Locks</h3>
|
|
|
|
|
|
|
|
<ul class="index-sl" compact>
|
|
<li><a href="#index-buildbot_002elocks_002eMasterLock-69"><code>buildbot.locks.MasterLock</code></a>: <a href="#Interlocks">Interlocks</a></li>
|
|
<li><a href="#index-buildbot_002elocks_002eSlaveLock-70"><code>buildbot.locks.SlaveLock</code></a>: <a href="#Interlocks">Interlocks</a></li>
|
|
<li><a href="#index-buildbot_002escheduler_002eAnyBranchScheduler-18"><code>buildbot.scheduler.AnyBranchScheduler</code></a>: <a href="#Scheduler-Types">Scheduler Types</a></li>
|
|
<li><a href="#index-buildbot_002escheduler_002eDependent-23"><code>buildbot.scheduler.Dependent</code></a>: <a href="#Build-Dependencies">Build Dependencies</a></li>
|
|
<li><a href="#index-buildbot_002escheduler_002eNightly-20"><code>buildbot.scheduler.Nightly</code></a>: <a href="#Scheduler-Types">Scheduler Types</a></li>
|
|
<li><a href="#index-buildbot_002escheduler_002ePeriodic-19"><code>buildbot.scheduler.Periodic</code></a>: <a href="#Scheduler-Types">Scheduler Types</a></li>
|
|
<li><a href="#index-buildbot_002escheduler_002eScheduler-17"><code>buildbot.scheduler.Scheduler</code></a>: <a href="#Scheduler-Types">Scheduler Types</a></li>
|
|
<li><a href="#index-buildbot_002escheduler_002eTry_005fJobdir-92"><code>buildbot.scheduler.Try_Jobdir</code></a>: <a href="#try">try</a></li>
|
|
<li><a href="#index-buildbot_002escheduler_002eTry_005fUserpass-93"><code>buildbot.scheduler.Try_Userpass</code></a>: <a href="#try">try</a></li>
|
|
</ul><h3 class="heading">Build Factories</h3>
|
|
|
|
|
|
|
|
<ul class="index-bf" compact>
|
|
<li><a href="#index-buildbot_002eprocess_002efactory_002eBasicBuildFactory-72"><code>buildbot.process.factory.BasicBuildFactory</code></a>: <a href="#BuildFactory">BuildFactory</a></li>
|
|
<li><a href="#index-buildbot_002eprocess_002efactory_002eBasicSVN-73"><code>buildbot.process.factory.BasicSVN</code></a>: <a href="#BuildFactory">BuildFactory</a></li>
|
|
<li><a href="#index-buildbot_002eprocess_002efactory_002eBuildFactory-71"><code>buildbot.process.factory.BuildFactory</code></a>: <a href="#BuildFactory">BuildFactory</a></li>
|
|
<li><a href="#index-buildbot_002eprocess_002efactory_002eCPAN-77"><code>buildbot.process.factory.CPAN</code></a>: <a href="#CPAN">CPAN</a></li>
|
|
<li><a href="#index-buildbot_002eprocess_002efactory_002eDistutils-78"><code>buildbot.process.factory.Distutils</code></a>: <a href="#Python-distutils">Python distutils</a></li>
|
|
<li><a href="#index-buildbot_002eprocess_002efactory_002eGNUAutoconf-76"><code>buildbot.process.factory.GNUAutoconf</code></a>: <a href="#GNUAutoconf">GNUAutoconf</a></li>
|
|
<li><a href="#index-buildbot_002eprocess_002efactory_002eQuickBuildFactory-75"><code>buildbot.process.factory.QuickBuildFactory</code></a>: <a href="#Quick-builds">Quick builds</a></li>
|
|
<li><a href="#index-buildbot_002eprocess_002efactory_002eTrial-79"><code>buildbot.process.factory.Trial</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
|
|
</ul><h3 class="heading">Build Steps</h3>
|
|
|
|
|
|
|
|
<ul class="index-bs" compact>
|
|
<li><a href="#index-buildbot_002esteps_002emaxq_002eMaxQ-95"><code>buildbot.steps.maxq.MaxQ</code></a>: <a href="#Index-of-Useful-Classes">Index of Useful Classes</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002epython_002eBuildEPYDoc-58"><code>buildbot.steps.python.BuildEPYDoc</code></a>: <a href="#BuildEPYDoc">BuildEPYDoc</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002epython_002ePyFlakes-59"><code>buildbot.steps.python.PyFlakes</code></a>: <a href="#PyFlakes">PyFlakes</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eBuildDebs-83"><code>buildbot.steps.python_twisted.BuildDebs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eHLint-80"><code>buildbot.steps.python_twisted.HLint</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eProcessDocs-82"><code>buildbot.steps.python_twisted.ProcessDocs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eRemovePYCs-84"><code>buildbot.steps.python_twisted.RemovePYCs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eTrial-81"><code>buildbot.steps.python_twisted.Trial</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002eshell_002eCompile-54"><code>buildbot.steps.shell.Compile</code></a>: <a href="#Compile">Compile</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002eshell_002eConfigure-53"><code>buildbot.steps.shell.Configure</code></a>: <a href="#Configure">Configure</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002eshell_002eShellCommand-52"><code>buildbot.steps.shell.ShellCommand</code></a>: <a href="#ShellCommand">ShellCommand</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002eshell_002eTest-55"><code>buildbot.steps.shell.Test</code></a>: <a href="#Test">Test</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002esource_002eArch-47"><code>buildbot.steps.source.Arch</code></a>: <a href="#Arch">Arch</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002esource_002eBazaar-49"><code>buildbot.steps.source.Bazaar</code></a>: <a href="#Bazaar">Bazaar</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002esource_002eCVS-39"><code>buildbot.steps.source.CVS</code></a>: <a href="#CVS">CVS</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002esource_002eDarcs-43"><code>buildbot.steps.source.Darcs</code></a>: <a href="#Darcs">Darcs</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002esource_002eGit-94"><code>buildbot.steps.source.Git</code></a>: <a href="#Index-of-Useful-Classes">Index of Useful Classes</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002esource_002eMercurial-45"><code>buildbot.steps.source.Mercurial</code></a>: <a href="#Mercurial">Mercurial</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002esource_002eP4-51"><code>buildbot.steps.source.P4</code></a>: <a href="#P4">P4</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002esource_002eSVN-41"><code>buildbot.steps.source.SVN</code></a>: <a href="#SVN">SVN</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002etransfer_002eFileDownload-62"><code>buildbot.steps.transfer.FileDownload</code></a>: <a href="#Transferring-Files">Transferring Files</a></li>
|
|
<li><a href="#index-buildbot_002esteps_002etransfer_002eFileUpload-61"><code>buildbot.steps.transfer.FileUpload</code></a>: <a href="#Transferring-Files">Transferring Files</a></li>
|
|
</ul><!-- undocumented steps -->
|
|
<p><a name="index-buildbot_002esteps_002esource_002eGit-94"></a><a name="index-buildbot_002esteps_002emaxq_002eMaxQ-95"></a>
|
|
|
|
<h3 class="heading">Status Targets</h3>
|
|
|
|
|
|
|
|
<ul class="index-st" compact>
|
|
<li><a href="#index-buildbot_002estatus_002eclient_002ePBListener-90"><code>buildbot.status.client.PBListener</code></a>: <a href="#PBListener">PBListener</a></li>
|
|
<li><a href="#index-buildbot_002estatus_002ehtml_002eWaterfall-86"><code>buildbot.status.html.Waterfall</code></a>: <a href="#HTML-Waterfall">HTML Waterfall</a></li>
|
|
<li><a href="#index-buildbot_002estatus_002email_002eMailNotifier-96"><code>buildbot.status.mail.MailNotifier</code></a>: <a href="#Index-of-Useful-Classes">Index of Useful Classes</a></li>
|
|
<li><a href="#index-buildbot_002estatus_002ewords_002eIRC-88"><code>buildbot.status.words.IRC</code></a>: <a href="#IRC-Bot">IRC Bot</a></li>
|
|
</ul><!-- TODO: undocumented targets -->
|
|
<p><a name="index-buildbot_002estatus_002email_002eMailNotifier-96"></a>
|
|
<div class="node">
|
|
<p><hr>
|
|
<a name="Index-of-master.cfg-keys"></a>
|
|
<a name="Index-of-master_002ecfg-keys"></a>
|
|
Next: <a rel="next" accesskey="n" href="#Index">Index</a>,
|
|
Previous: <a rel="previous" accesskey="p" href="#Index-of-Useful-Classes">Index of Useful Classes</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="unnumbered">Index of master.cfg keys</h2>
|
|
|
|
<p>This is a list of all of the significant keys in master.cfg . Recall
|
|
that master.cfg is effectively a small python program one
|
|
responsibility: create a dictionary named <code>BuildmasterConfig</code>.
|
|
The keys of this dictionary are listed here. The beginning of the
|
|
master.cfg file typically starts with something like:
|
|
|
|
<pre class="example"> BuildmasterConfig = c = {}
|
|
</pre>
|
|
<p>Therefore a config key of <code>sources</code> will usually appear in
|
|
master.cfg as <code>c['sources']</code>.
|
|
|
|
|
|
|
|
<ul class="index-bc" compact>
|
|
<li><a href="#index-c_005b_0027bots_0027_005d-25"><code>c['bots']</code></a>: <a href="#Buildslave-Specifiers">Buildslave Specifiers</a></li>
|
|
<li><a href="#index-c_005b_0027buildbotURL_0027_005d-14"><code>c['buildbotURL']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li>
|
|
<li><a href="#index-c_005b_0027builders_0027_005d-26"><code>c['builders']</code></a>: <a href="#Defining-Builders">Defining Builders</a></li>
|
|
<li><a href="#index-c_005b_0027debugPassword_0027_005d-28"><code>c['debugPassword']</code></a>: <a href="#Debug-options">Debug options</a></li>
|
|
<li><a href="#index-c_005b_0027manhole_0027_005d-29"><code>c['manhole']</code></a>: <a href="#Debug-options">Debug options</a></li>
|
|
<li><a href="#index-c_005b_0027projectName_0027_005d-12"><code>c['projectName']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li>
|
|
<li><a href="#index-c_005b_0027projectURL_0027_005d-13"><code>c['projectURL']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li>
|
|
<li><a href="#index-c_005b_0027schedulers_0027_005d-16"><code>c['schedulers']</code></a>: <a href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a></li>
|
|
<li><a href="#index-c_005b_0027slavePortnum_0027_005d-24"><code>c['slavePortnum']</code></a>: <a href="#Setting-the-slaveport">Setting the slaveport</a></li>
|
|
<li><a href="#index-c_005b_0027sources_0027_005d-15"><code>c['sources']</code></a>: <a href="#Listing-Change-Sources-and-Schedulers">Listing Change Sources and Schedulers</a></li>
|
|
<li><a href="#index-c_005b_0027status_0027_005d-27"><code>c['status']</code></a>: <a href="#Defining-Status-Targets">Defining Status Targets</a></li>
|
|
</ul><div class="node">
|
|
<p><hr>
|
|
<a name="Index"></a>
|
|
Previous: <a rel="previous" accesskey="p" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>,
|
|
Up: <a rel="up" accesskey="u" href="#Top">Top</a>
|
|
|
|
</div>
|
|
|
|
<h2 class="unnumbered">Index</h2>
|
|
|
|
|
|
|
|
<ul class="index-cp" compact>
|
|
<li><a href="#index-addURL-67">addURL</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li>
|
|
<li><a href="#index-Arch-Checkout-46">Arch Checkout</a>: <a href="#Arch">Arch</a></li>
|
|
<li><a href="#index-Bazaar-Checkout-48">Bazaar Checkout</a>: <a href="#Bazaar">Bazaar</a></li>
|
|
<li><a href="#index-build-properties-56">build properties</a>: <a href="#Build-Properties">Build Properties</a></li>
|
|
<li><a href="#index-Builder-9">Builder</a>: <a href="#Builder">Builder</a></li>
|
|
<li><a href="#index-BuildRequest-8">BuildRequest</a>: <a href="#BuildRequest">BuildRequest</a></li>
|
|
<li><a href="#index-BuildSet-7">BuildSet</a>: <a href="#BuildSet">BuildSet</a></li>
|
|
<li><a href="#index-BuildStep-URLs-66">BuildStep URLs</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li>
|
|
<li><a href="#index-Configuration-11">Configuration</a>: <a href="#Configuration">Configuration</a></li>
|
|
<li><a href="#index-CVS-Checkout-38">CVS Checkout</a>: <a href="#CVS">CVS</a></li>
|
|
<li><a href="#index-Darcs-Checkout-42">Darcs Checkout</a>: <a href="#Darcs">Darcs</a></li>
|
|
<li><a href="#index-Dependencies-22">Dependencies</a>: <a href="#Build-Dependencies">Build Dependencies</a></li>
|
|
<li><a href="#index-Dependent-21">Dependent</a>: <a href="#Build-Dependencies">Build Dependencies</a></li>
|
|
<li><a href="#index-File-Transfer-60">File Transfer</a>: <a href="#Transferring-Files">Transferring Files</a></li>
|
|
<li><a href="#index-installation-3">installation</a>: <a href="#Installing-the-code">Installing the code</a></li>
|
|
<li><a href="#index-introduction-1">introduction</a>: <a href="#Introduction">Introduction</a></li>
|
|
<li><a href="#index-IRC-87">IRC</a>: <a href="#IRC-Bot">IRC Bot</a></li>
|
|
<li><a href="#index-links-65">links</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li>
|
|
<li><a href="#index-locks-68">locks</a>: <a href="#Interlocks">Interlocks</a></li>
|
|
<li><a href="#index-logfiles-4">logfiles</a>: <a href="#Logfiles">Logfiles</a></li>
|
|
<li><a href="#index-LogLineObserver-64">LogLineObserver</a>: <a href="#Adding-LogObservers">Adding LogObservers</a></li>
|
|
<li><a href="#index-LogObserver-63">LogObserver</a>: <a href="#Adding-LogObservers">Adding LogObservers</a></li>
|
|
<li><a href="#index-Mercurial-Checkout-44">Mercurial Checkout</a>: <a href="#Mercurial">Mercurial</a></li>
|
|
<li><a href="#index-PBListener-89">PBListener</a>: <a href="#PBListener">PBListener</a></li>
|
|
<li><a href="#index-Perforce-Update-50">Perforce Update</a>: <a href="#P4">P4</a></li>
|
|
<li><a href="#index-Philosophy-of-operation-2">Philosophy of operation</a>: <a href="#History-and-Philosophy">History and Philosophy</a></li>
|
|
<li><a href="#index-Scheduler-6">Scheduler</a>: <a href="#Schedulers">Schedulers</a></li>
|
|
<li><a href="#index-statusgui-91">statusgui</a>: <a href="#statusgui">statusgui</a></li>
|
|
<li><a href="#index-SVN-Checkout-40">SVN Checkout</a>: <a href="#SVN">SVN</a></li>
|
|
<li><a href="#index-treeStableTimer-74">treeStableTimer</a>: <a href="#BuildFactory-Attributes">BuildFactory Attributes</a></li>
|
|
<li><a href="#index-Users-10">Users</a>: <a href="#Users">Users</a></li>
|
|
<li><a href="#index-Version-Control-5">Version Control</a>: <a href="#Version-Control-Systems">Version Control Systems</a></li>
|
|
<li><a href="#index-Waterfall-85">Waterfall</a>: <a href="#HTML-Waterfall">HTML Waterfall</a></li>
|
|
<li><a href="#index-WithProperties-57">WithProperties</a>: <a href="#Build-Properties">Build Properties</a></li>
|
|
</ul><div class="footnote">
|
|
<hr>
|
|
<a name="texinfo-footnotes-in-document"></a><h4>Footnotes</h4><p class="footnote"><small>[<a name="fn-1" href="#fnd-1">1</a>]</small> this
|
|
@reboot syntax is understood by Vixie cron, which is the flavor
|
|
usually provided with linux systems. Other unices may have a cron that
|
|
doesn't understand @reboot</p>
|
|
|
|
<p class="footnote"><small>[<a name="fn-2" href="#fnd-2">2</a>]</small> except Darcs, but
|
|
since the Buildbot never modifies its local source tree we can ignore
|
|
the fact that Darcs uses a less centralized model</p>
|
|
|
|
<p class="footnote"><small>[<a name="fn-3" href="#fnd-3">3</a>]</small> many VC systems provide more complexity than
|
|
this: in particular the local views that P4 and ClearCase can assemble
|
|
out of various source directories are more complex than we're prepared
|
|
to take advantage of here</p>
|
|
|
|
<p class="footnote"><small>[<a name="fn-4" href="#fnd-4">4</a>]</small> Monotone's <em>multiple heads</em> feature
|
|
violates this assumption of cumulative Changes, but in most situations
|
|
the changes don't occur frequently enough for this to be a significant
|
|
problem</p>
|
|
|
|
<p class="footnote"><small>[<a name="fn-5" href="#fnd-5">5</a>]</small> this <code>checkoutDelay</code> defaults
|
|
to half the tree-stable timer, but it can be overridden with an
|
|
argument to the Source Step</p>
|
|
|
|
<p class="footnote"><small>[<a name="fn-6" href="#fnd-6">6</a>]</small> To be precise, it is a list of objects which all
|
|
implement the <code>buildbot.interfaces.IChangeSource</code> Interface</p>
|
|
|
|
<p class="footnote"><small>[<a name="fn-7" href="#fnd-7">7</a>]</small> Build properties are serialized along with the
|
|
build results, so they must be serializable. For this reason, the
|
|
value of any build property should be simple inert data: strings,
|
|
numbers, lists, tuples, and dictionaries. They should not contain
|
|
class instances.</p>
|
|
|
|
<p class="footnote"><small>[<a name="fn-8" href="#fnd-8">8</a>]</small> framboozle.com is still available. Remember, I get 10%
|
|
:).</p>
|
|
|
|
<p class="footnote"><small>[<a name="fn-9" href="#fnd-9">9</a>]</small> Framboozle gets very excited about running unit
|
|
tests.</p>
|
|
|
|
<p class="footnote"><small>[<a name="fn-10" href="#fnd-10">10</a>]</small> Also note that a clever buildmaster admin
|
|
could still create the opportunity for deadlock: Build A obtains Lock
|
|
1, inside which Step A.two tries to acquire Lock 2 at the Step level.
|
|
Meanwhile Build B obtains Lock 2, and has a Step B.two which wants to
|
|
acquire Lock 1 at the Step level. Don't Do That.</p>
|
|
|
|
<p class="footnote"><small>[<a name="fn-11" href="#fnd-11">11</a>]</small> It may even be possible to provide SSL access by using
|
|
a specification like
|
|
<code>"ssl:12345:privateKey=mykey.pen:certKey=cert.pem"</code>, but this is
|
|
completely untested</p>
|
|
|
|
<p><hr></div>
|
|
|
|
</body></html>
|
|
|