Also, add count of fds and goroutines to the coordinator's status
page.
Change-Id: I857e609623cfa280716d5d079180d0e4021d0bac
Reviewed-on: https://go-review.googlesource.com/27550
Reviewed-by: Quentin Smith <quentin@golang.org>
ctx was initialized in the wrong place.
Change-Id: I7fb3c56071a3e4d5cc2199e07fcc4b8ecbc7a674
Reviewed-on: https://go-review.googlesource.com/22967
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Previously it wasn't noticing their death until the next health check.
Take advantage of that the revdial is always blocked in a Read, so it
will see a TCP shutdown in the case of normal shutdowns. (health checks
will still catch disappearing machines)
Change-Id: I9a7f60a38b3acaf02057b2da9e0cbc91d328f651
Reviewed-on: https://go-review.googlesource.com/14736
Reviewed-by: Andrew Gerrand <adg@golang.org>
should be rare now.
Change-Id: Icc4bfd13c8dfe8f2e189db819bc0d552f35fb3c9
Reviewed-on: https://go-review.googlesource.com/14731
Reviewed-by: Andrew Gerrand <adg@golang.org>
Use the new Go 1.5 mechanism to abort HTTP requests with a channel.
This is respected by the Go 1.5 http.Transport, which we always use.
This CL shouldn't be necessary (CL 14700 was the real bug), but
doesn't hurt. This still would've probably prevented most of
golang/go#12666
Change-Id: I6890e016ee04183fc0d600baed8046c2f79113d8
Reviewed-on: https://go-review.googlesource.com/14701
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Now Close always means destroy.
The old code & API was a half-baked implementation of a (lack of)
design where there was a difference between being done with a buildlet
(with it possibly being reused by somebody else?) and you wanting to
nuke it completely. Unfortunately this just grew messier and more
broken over time.
This attempts to clean it all up.
We can add the sharing ideas back later when there's actually a design
and implementation. (We're not losing anything with this CL, because
nothing ever shared buildlets)
In fact, this CL fixes a problem where reverse buildlets weren't
getting their underlying net.Conns closed when their healthchecks
failed, leading to 4+ hour (and counting) build hangs due to buildlets
getting killed at inopportune moments (e.g. me testing running a
reverse buildlet on my home mac to help out with the dashboard
backlog)
Change-Id: I07be09f4d5f0f09d35e51e41c48b1296b71bb9b5
Reviewed-on: https://go-review.googlesource.com/14585
Reviewed-by: David Crawshaw <crawshaw@golang.org>
Reviewed-by: Andrew Gerrand <adg@golang.org>
* reverse buildlet rework (multiplexed TCP connections, instead
of a hacky reverse roundtripper)
* scaleway ARM image improvements
* parallel gzip implementation, which makes things ~8x faster on
Scaleway.
* merge watcher into the coordinator, for easier deployments
Change-Id: I55d769f982e6583b261435309faa1f718a15fde1
Reviewed-on: https://go-review.googlesource.com/12665
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This creates a mechanism for clients (such as cmd/release and
cmd/gomote) to obtain buildlets via the coordinator. Previously
cmd/release and cmd/gomote could only create GCE VMs themselves, and
required the GCE project's credentials. In addition to the awkwardness
of needing to hand out the GCE credentials, it also meant ARM and
Darwin buildlets (which use the reverse buildlet pool) weren't usable.
Instead, this creates a new auth mechanism where the coordinator is
contacted over TLS with key pinning (the CA system isn't used) in the
same way that the reverse builders already dialed into the
coordinator, and then a "user build type" and hash are sent as the
username and password. The same master key is used to sign user
builder keys, and they always start with "user-". (which isn't a GOOS).
Then the coordinator provides an API to create and list buildlets.
They auto-expire after a duration and are auto-renewed upon use.
The buildlet library (as used by cmd/release etc) then proxies HTTP
requests via the coordinator to the backend buildlet.
See doc/remote-buildlet.txt for protocol details.
Change-Id: I12e27eae788fdd91927cb182b950893dc759f8e9
Reviewed-on: https://go-review.googlesource.com/11901
Reviewed-by: Andrew Gerrand <adg@golang.org>
Now you have to fail 3 in a row before you're killed.
This helped Plan 9 a bunch for a few days, before the change was
reverted (and then it started failing consistently again).
We still don't know why Plan 9 sucks at replying to heartbeats.
Change-Id: Ic64d2e8fb75f544c7c3e9a62c28ab9e20ff8392d
Reviewed-on: https://go-review.googlesource.com/11132
Reviewed-by: Andrew Gerrand <adg@golang.org>
Reviewed-by: David du Colombier <0intro@gmail.com>
Add background heartbeats to detect dead GCE VMs (the OpenBSD
buildlets seem to hang a lot),
Add timeouts to test executions.
Take helper buildlets out of service once they're marked bad.
Keep the in-order buildlet running forever when sharding tests, in
case all the helpers die. (observed once)
Keep a cache of recently deleted VMs and don't try to delete VMs again
if we've recently deleted them. (they're slow to delete)
More reverse buildlets more paranoid in their health checking and closing
of the connection.
Make status page link to /try set URLs.
Also, better logging (more sometimes, less others0, and misc bug fixes.
Change-Id: I57a5e8e39381234006cac4dd799b655d64be71bb
Reviewed-on: https://go-review.googlesource.com/10981
Reviewed-by: Andrew Gerrand <adg@golang.org>
Also: heartbeat reverse buildlets more often
Also: adjust cmd/go test expected duration.
Change-Id: I75fbb66a0b20ac7357ad7cf78fc545101ac9aa33
Reviewed-on: https://go-review.googlesource.com/10884
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Andrew Gerrand <adg@golang.org>
The reverse buildlets' RoundTrip are hanging, which is its own problem,
but this calling code should be robust and time out anyway.
Change-Id: Id9e3e1d9feb6ffa58cc0995d0623bd90845bb9d6
Reviewed-on: https://go-review.googlesource.com/10847
Reviewed-by: Andrew Gerrand <adg@golang.org>
Also gomote updates which came about during the process of developing
and debugging this.
Change-Id: Ia53d674118a6b99bcdda7062d3b7161279b6ad52
Reviewed-on: https://go-review.googlesource.com/10463
Reviewed-by: Andrew Gerrand <adg@golang.org>
Have the buildlet send build keys to the buildlet in reverse mode.
Rename Info to Status, as it will be used periodically by the
coordinator to check that the buildlet is OK.
And add a simple connection test.
Change-Id: Ic7285636c7818e2584e11555a8f5b5b66be30638
Reviewed-on: https://go-review.googlesource.com/8593
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
In -reverse mode, the buildlet dials the coordinator. The connection
is then turned around so the coordinator can control the buildlet.
This lets us start buildlets on machines without APIs behind
firewalls.
Also add the -mode flag to the coordinator, which defaults to dev
mode when running off GCE. In prod mode the coordinator attempts to
become farmer.golang.org and pick up its real certificates. In the
new mode, builtin certificates are used allowing you to start a
local coordinator and buildlet for testing.
A simple connection test will be in a followup CL, as soon as key
checking is implemented.
Change-Id: I2a7dcdfbb4efda71df31b571788945e9ce1f3365
Reviewed-on: https://go-review.googlesource.com/8490
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This starts the process of making the coordinator about to use the
buildlet on things other than GCE.
Update golang/go#8647 (ARM, reverse proxy or online.net)
Update golang/go#10267 (windows 2003 on AWS)
Update golang/go#9495 (OS X VMWare VMs on racked Mac Minis)
Change-Id: I5df79ea67e0ececba8b880e81bd93d4c80897455
Reviewed-on: https://go-review.googlesource.com/8198
Reviewed-by: David Crawshaw <crawshaw@golang.org>
Add gomote and buildlet support for listing remote files, and
efficiently syncing from the local workstation to the remote buildlet.
Change-Id: Ifab1fb1c208ca4bc66f8d6916c38e1914001a3a5
Reviewed-on: https://go-review.googlesource.com/4270
Reviewed-by: Andrew Gerrand <adg@golang.org>
Also fix the RemoveAll method by setting the request Content-Type.
Change-Id: I87ec29c5c0da06eba5eaebcd00bdbef18e6ae8ad
Reviewed-on: https://go-review.googlesource.com/3880
Reviewed-by: Andrew Gerrand <adg@golang.org>
And use it in gomote and release tools.
Change-Id: I87fa013d6d6729e7305dacd137be1b3d3b02f5f4
Reviewed-on: https://go-review.googlesource.com/3771
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The 'build' repo (unlike the 'tools' repo) is allowed to depend on
anything.
Change-Id: I4caa9fe61bccf05f488152eac53ed5769a848d4d
Reviewed-on: https://go-review.googlesource.com/3113
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Also:
- Move the watcher to cmd/watcher (somehow this was missed earlier).
- Move dashboard package from the repo root to its own directory.
- Update docker build scripts. (Although not yet the version hashes in
the Dockerfiles; this leaves the docker builds broken, but they were
already broken after moving the builder to cmd/builder. They'll be
fixed in a followup CL after this one is submitted.)
Change-Id: I29a9758da1f3c60446e3ce18174c0df26e4d8325
Reviewed-on: https://go-review.googlesource.com/3077
Reviewed-by: Andrew Gerrand <adg@golang.org>
- get tarballs out of buildlets
- gomote can pass arguments to buildlet's exec handler
- buildlet's exec handler can run system-level commands
- hard-code GOROOT_BOOTSTRAP to be "go1.4" under the workdir
- adjust MTU size on GCE
etc
Change-Id: I73e18b7a5e395a889f5ee93ba9850d331ffb7812
Reviewed-on: https://go-review.googlesource.com/3052
Reviewed-by: Andrew Gerrand <adg@golang.org>
Client + server support, and gomote flags.
Change-Id: I91320f47731f8c69b84c4961028bfbbdfc85467a
Reviewed-on: https://go-review.googlesource.com/3029
Reviewed-by: Andrew Gerrand <adg@golang.org>
Operation Packification, step 2 of tons.
Eventually the buildlet client binary will use this stuff now.
Change-Id: I4cf5f3e6beb9e56bdc795ed513ce6daaf61425e3
Reviewed-on: https://go-review.googlesource.com/2921
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Brad Fitzpatrick
Brad Fitzpatrick
Andrew Gerrand
David Crawshaw