* add support for longtest builders for subrepos.
* fix race builders for subrepos (they weren't passing the -race flag).
* adjust the policy for the js-wasm builders to build fewer subrepos
where it'll never work or isn't worth it.
* fix the android emu builders which disappeared because an empty
string was being passed to buildsRepoAtAll. In some places in the
code an empty string for goBranch for the "go" repo meant the same
as branch, but I forgot that in the new code, so an old caller was
confusing the new config hooks. Rather than make all policy funcs be
aware of both ways, the new code in this CL now maps an empty string
to the same as the repo's branch when the repo == "go". Adds a test
too.
* fix some outdated comments.
Change-Id: Icf3fb85e5542a4d314443b59d02517b306ef46b7
Reviewed-on: https://go-review.googlesource.com/c/build/+/166897
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
We previously hard-coded a Linux-only static map of 2015 data in for
the critical path scheduling of cmd/dist tests over N sharded
buildlets. That worked well only for Linux and only in 2015.
Instead, query BigQuery to find out what the recent timing data looks
like for all builders.
I'd started to work on this back in CL 30716 (Oct 2016) but apparently
never finished. Yay me. But skip the writing-to-CSV step. BigQuery is
much faster than I remember (maybe it got faster?), so just query it
directly. The query takes about 2 seconds, and we only do it every
hour (which is still overkill; daily is probably fine).
Change-Id: I498fc09dfaf24fb1f11b2c0ab4b952b2f15f9c32
Reviewed-on: https://go-review.googlesource.com/c/160037
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Document that https.ListenAndServe always returns a non-nil error,
and use that property to stop gerritbot if the HTTPS server fails
to run.
Remove an unneccessary check for ln != nil in https.ListenAndServe.
net.Listen returns a valid listener when err == nil, and the
err != nil case is already handled above.
Change-Id: Ibbadd551aa36b5376bf062011f12e3e3787bd6b3
Reviewed-on: https://go-review.googlesource.com/c/159698
Reviewed-by: Andrew Bonventre <andybons@golang.org>
There was a race condition when the opt.AutocertCacheBucket field was
specified. The handler = http.HandlerFunc(redirectToHTTPS) line could
run before (or after) the value of handler was used in the following
line that did serveAutocertTLS(handler, opt.AutocertCacheBucket).
As a result, it was possible for the HTTPS server to end up using the
redirectToHTTPS handler, instead of the intended original handler
parameter passed to the ListenAndServe function.
Change-Id: I95967eed1176f236a87131d0c054ded680886591
Reviewed-on: https://go-review.googlesource.com/c/159697
Reviewed-by: Andrew Bonventre <andybons@golang.org>
BuildConfig grew way too large and has too many slices & such to be a
value type. Make it a pointer type, which matches HostConfig.
Change-Id: Ie625bece9d6d8c1ec6cff26e77416bd9b1f256d8
Reviewed-on: https://go-review.googlesource.com/c/145077
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
This change updates information related to mikioh to ensure hitless
switching; I'll delete the account mikioh.mikioh soon.
Change-Id: Id29705e59b12ca4633f700bd1303dbb7ac08b823
Reviewed-on: https://go-review.googlesource.com/c/144877
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
SyncBuilds and SyncSpans are invoked by cmd/coordinator/*
without invoking Close for the various bigquery and datastore
clients instantiations. In syncBuildStatsLoop, SyncSpans is
invoked in a [5min, 16min) loop, which will leave a bunch of
underlying resources and HTTP connections unclosed.
Change-Id: I8713c26c31daef3fc1b2c65a2d1bbfc74c8a63ba
Reviewed-on: https://go-review.googlesource.com/138695
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
CL 132995 made a lot of changes to the gophers table based on
the output of gopherstats -mode=find-gerrit-gophers, which is
a new cmd/gopherstats mode added in that very CL.
The current implementation of Person.mergeIDs makes it so that
the first email seen is considered the Gerrit email.
Many of the additions of CL 132995 did not take that into account,
causing the Gerrit field to change for some gophers incorrectly.
Overall, that CL caused the following changes to gophers:
Github field changes:
tal@whatexit.org: GitHub="" -> "TomOnTime"
Carl Henrik Lunde: GitHub="" -> "chlunde"
Dieter Plaetinck: GitHub="" -> "Dieterbe"
Diogo Pinela: GitHub="" -> "dpinela"
Frank Schroeder: GitHub="" -> "magiconair"
Gregory Man: GitHub="" -> "gregory-m"
Jan Berktold: GitHub="" -> "JanBerktold"
Jean de Klerk: GitHub="" -> "jadekler"
Josselin Costanzi: GitHub="" -> "josselin-c"
Martin Garton: GitHub="MartinGarton" -> "mjgarton"
Matt Harden: GitHub="" -> "nerdatmath"
Michael Darakananda: GitHub="" -> "pongad"
Mostyn Bramley-Moore: GitHub="" -> "mostynb"
Nicholas Maniscalco: GitHub="" -> "nicholasmaniscalco"
Roland Illig: GitHub="" -> "rillig"
Yasha Bubnov: GitHub="" -> "ybubnov"
Zheng Xu: GitHub="" -> "Zheng-Xu"
ltnwgl: GitHub="" -> "gengliangwang"
oiooj: GitHub="" -> "oiooj"
thoeni: GitHub="" -> "thoeni"
Gerrit field changes:
Andrew Bonventre: Gerrit="andybons@golang.org" -> "365204+andybons@users.noreply.github.com"
Carl Mastrangelo: Gerrit="notcarl@google.com" -> "carl.mastrangelo@gmail.com"
Chris McGee: Gerrit="sirnewton_01@yahoo.ca" -> "newton688@gmail.com"
Eric Lagergren: Gerrit="ericscottlagergren@gmail.com" -> "eric@ericlagergren.com"
Filippo Valsorda: Gerrit="filippo@golang.org" -> "6195@62eb7196-b449-3ce5-99f1-c037f21e1705"
Guillaume J. Charmes: Gerrit="guillaume@charmes.net" -> "gcharmes@magicleap.com"
Harshavardhana: Gerrit="hrshvardhana@gmail.com" -> "harsha@minio.io"
Jean de Klerk: Gerrit="jadekler@gmail.com" -> "deklerk@google.com"
Joe Tsai: Gerrit="joetsai@google.com" -> "joetsai@digital-static.net"
Martin Möhrmann: Gerrit="moehrmann@google.com" -> "martisch@uos.de"
Matthew Dempsky: Gerrit="mdempsky@google.com" -> "matthew@dempsky.org"
Olivier Poitrey: Gerrit="rs@dailymotion.com" -> "rs@netflix.com"
Paul Jolly: Gerrit="paul@myitcv.org.uk" -> "paul@myitcv.io"
Ralph Corderoy: Gerrit="ralph@inputplus.co.uk" -> "ralph.corderoy@gmail.com"
Raul Silvera: Gerrit="rsilvera@google.com" -> "rauls5382@gmail.com"
Richard Miller: Gerrit="miller.research@gmail.com" -> "millerresearch@gmail.com"
Sebastien Binet: Gerrit="seb.binet@gmail.com" -> "binet@cern.ch"
Tobias Klauser: Gerrit="tobias.klauser@gmail.com" -> "tklauser@distanz.ch"
Vitor De Mario: Gerrit="vitordemario@gmail.com" -> "vitor.demario@mendelics.com.br"
Googler field changes:
Aaron Kemp: Googler=false -> true
Jason Hall: Googler=false -> true
Jean de Klerk: Googler=false -> true
(It also caused many emails to be added, but I'm not considering
those changes since they're not relevant to golang/go#27517 and
aren't causing harm.)
All of the Github and Googler field changes are good,
but not all of the Gerrit field changes are good.
I've manually checked them against the
go-review.googlesource.com Gerrit server,
and classified them as follows:
bad Andrew Bonventre: Gerrit="andybons@golang.org" -> "365204+andybons@users.noreply.github.com"
bad Carl Mastrangelo: Gerrit="notcarl@google.com" -> "carl.mastrangelo@gmail.com"
ok Chris McGee: Gerrit="sirnewton_01@yahoo.ca" -> "newton688@gmail.com"
bad Eric Lagergren: Gerrit="ericscottlagergren@gmail.com" -> "eric@ericlagergren.com"
bad Filippo Valsorda: Gerrit="filippo@golang.org" -> "6195@62eb7196-b449-3ce5-99f1-c037f21e1705"
bad Guillaume J. Charmes: Gerrit="guillaume@charmes.net" -> "gcharmes@magicleap.com"
bad Harshavardhana: Gerrit="hrshvardhana@gmail.com" -> "harsha@minio.io"
ok Jean de Klerk: Gerrit="jadekler@gmail.com" -> "deklerk@google.com"
bad Joe Tsai: Gerrit="joetsai@google.com" -> "joetsai@digital-static.net"
bad Martin Möhrmann: Gerrit="moehrmann@google.com" -> "martisch@uos.de"
bad Matthew Dempsky: Gerrit="mdempsky@google.com" -> "matthew@dempsky.org"
ok Olivier Poitrey: Gerrit="rs@dailymotion.com" -> "rs@netflix.com"
bad Paul Jolly: Gerrit="paul@myitcv.org.uk" -> "paul@myitcv.io"
bad Ralph Corderoy: Gerrit="ralph@inputplus.co.uk" -> "ralph.corderoy@gmail.com"
bad Raul Silvera: Gerrit="rsilvera@google.com" -> "rauls5382@gmail.com"
ok Richard Miller: Gerrit="miller.research@gmail.com" -> "millerresearch@gmail.com"
bad Sebastien Binet: Gerrit="seb.binet@gmail.com" -> "binet@cern.ch"
bad Tobias Klauser: Gerrit="tobias.klauser@gmail.com" -> "tklauser@distanz.ch"
bad Vitor De Mario: Gerrit="vitordemario@gmail.com" -> "vitor.demario@mendelics.com.br"
I considered any @google.com -> non-@google.com changes as bad.
For the rest, it was based on which email was recognized by the
Gerrit server and had more activity overall, as well as recently.
This CL undoes all the bad Gerrit field changes, reverting them
to their original pre-CL 132995 values. It also changes the Gerrit
email for Gopherbot, and cleans up my own entry. That leaves just:
Gerrit field changes:
Chris McGee: Gerrit="sirnewton_01@yahoo.ca" -> "newton688@gmail.com"
Jean de Klerk: Gerrit="jadekler@gmail.com" -> "deklerk@google.com"
Olivier Poitrey: Gerrit="rs@dailymotion.com" -> "rs@netflix.com"
Richard Miller: Gerrit="miller.research@gmail.com" -> "millerresearch@gmail.com"
In future CLs, we'll need to be careful with the order in which
emails are added, until golang/go#27631 is resolved.
Updates golang/go#27631.
Fixesgolang/go#27517.
Change-Id: I6bd289af6ea2c50c293c4576de3873658994b98a
Reviewed-on: https://go-review.googlesource.com/135456
Reviewed-by: Andrew Bonventre <andybons@golang.org>
It's perf time at Google, so it's time to answer the ol' "What Would
You Say You Do Here?" question.
So, time to update gopherstats with more stats & update the gophers
package in the process.
Change-Id: I0e0981f40a1229f943130621bd55acedffc7a583
Reviewed-on: https://go-review.googlesource.com/132995
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
My primary (preferred) email has changed,
and it's also my Gerrit email now.
Change-Id: I82bb5bb48a99ae12a646eec1a2ab08966199d62d
Reviewed-on: https://go-review.googlesource.com/112995
Reviewed-by: Andrew Bonventre <andybons@golang.org>
x/tools/cmd/godoc and x/playground. I add myself as secondary to both
of them since I cannot do +2 votes, but I would like to participate in
reviews with the best of my knowledge.
Change-Id: Iedeb9320b6f76c67a9d15eb8f6f8956e4d2e2957
Reviewed-on: https://go-review.googlesource.com/112695
Reviewed-by: Andrew Bonventre <andybons@golang.org>
* adds new buildgo.Client type to hold primarily a
buildenv.Environment and a google.Credentials
* starts to unify, document, and otherwise clean up
how we look up authentication info.
* use the new buildgo.Client type in the "buildongce"
(poorly named) cluster bootstrap tool.
* moves the "basepin" creation code from buildongce into
a new method on buildgo.Client, for later use in the
coordinator for golang/go#21305
Change-Id: I976b60c992bf67589e7f85fc5a9133f8cbb32e70
Reviewed-on: https://go-review.googlesource.com/112162
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Then use it from gitmirror and maintnerd.
Updates golang/go#23853
Change-Id: I8112f004638667894676c04fa218a7ced10422ac
Reviewed-on: https://go-review.googlesource.com/94836
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Around Oct 20th, we stopped syncing build stats to BigQuery.
Either the client library was updated around Oct 20th, or the server
changed behavior.
In any case, update the deps (to get a newer version of the bigquery
package) and update the code to work with the new SQL syntax.
This also fixes a bogus loop that was harmless but made the sync
slower than it could've been. (The logic was correct in the nearly
identical span sync later in the file.)
Also adds some TODOs for other log spam I see.
Change-Id: I2aa73084aeb19378bf260839555103fb48eac168
Reviewed-on: https://go-review.googlesource.com/78455
Reviewed-by: Andrew Bonventre <andybons@golang.org>
My e-mail address has changed.
Change-Id: I09b1d82dbf2044ab1cd7791a20e24a3e62bf87d0
Reviewed-on: https://go-review.googlesource.com/75030
Reviewed-by: Ian Lance Taylor <iant@golang.org>
it's hard to find consistent usernames with a name like Jeff Johnson
Change-Id: I66852b9266ef4ac77dc6e167dc883a28477312c3
Reviewed-on: https://go-review.googlesource.com/70090
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
Bigquery update-table calls are out-of-date with current API.
Fixesgolang/go#21925
Change-Id: I2a21bb1c53dcdb1da8b2da4329f89273e28b600b
Reviewed-on: https://go-review.googlesource.com/64450
Reviewed-by: Andrew Bonventre <andybons@golang.org>
Added drchase -> dr2chase, for ssh access
Change-Id: Id36cf3cde94f785db0202d0100d528f4f7e017f6
Reviewed-on: https://go-review.googlesource.com/56270
Run-TryBot: David Chase <drchase@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
And some cleanups in the originally-auto-generated tables used by
stats above.
Change-Id: Ie2279e1926fa05cfc124e04ce074e05df4af2a0c
Reviewed-on: https://go-review.googlesource.com/53230
Reviewed-by: Sarah Adams <shadams@google.com>
This CL is almost entirely code movement. There is new code in
cmd/coordinator/gce.go.
This library-itizes the buildstats code and makes it used by both
cmd/buildstats (when run by hand) and cmd/coordinator, which now runs
it in a loop in the background.
Running cmd/buildstats by hand is still pretty useless because it
doesn't yet(?) have any query subcommands. It only has the original
-sync command, which is now automatic in the coordinator. But it could
still be useful for debugging or running manually in the staging
environment.
This will permit running BigQuery queries at any time, without Brad or
somebody first manually running cmd/buildstats -sync.
This also means we'll be able to get rid of a bunch of the static (and
stale) timing information in the coordinator code and use empirical
numbers instead for buildlet creation times and per-test durations.
This will help the test sharding schedule things better and result in
faster builds.
Updates golang/go#12669
Change-Id: If9a3b414f8d458a65305c4916ea16d59e5a3971c
Reviewed-on: https://go-review.googlesource.com/53072
Reviewed-by: Sarah Adams <shadams@google.com>
So Ian can ssh to gomote instances.
Change-Id: If03187f9038bc3753e0cc69479ce71d260409044
Reviewed-on: https://go-review.googlesource.com/53190
Reviewed-by: Ian Lance Taylor <iant@golang.org>
This adds an SSH server to farmer.golang.org on port 2222 that proxies
SSH connections to users' gomote-created buildlet instances.
For example:
$ gomote create openbsd-amd64-60
user-bradfitz-openbsd-amd64-60-1
$ gomote ssh user-bradfitz-openbsd-amd64-60-1
Warning: Permanently added '[localhost]:33351' (ECDSA) to the list of known hosts.
OpenBSD 6.0 (GENERIC.MP) golang/go#2319: Tue Jul 26 13:00:43 MDT 2016
Welcome to OpenBSD: The proactively secure Unix-like operating system.
Please use the sendbug(1) utility to report bugs in the system.
Before reporting a bug, please try to reproduce it with the latest
version of the code. With bug reports, please try to ensure that
enough information to reproduce the problem is enclosed, and if a
known fix for it exists, include that as well.
$
As before, if the coordinator process is restarted (or crashes, is
evicted, etc), all gomote instances die.
Not yet supported:
* scp (help wanted)
* not all host types are configured. most are. some will need slight
config tweaks to the Docker image (e.g. adding openssh-server)
Supports currently:
* linux-amd64 (host type shared by 386, nacl)
* linux-arm
* linux-arm64
* darwin
* freebsd
* openbsd
* plan9-386
* windows
Implementation details:
* the ssh server process listens on port 2222 in the coordinator
(farmer.golang.org), which is behind a GKE TCP load balancer.
* the ssh server library is github.com/gliderlabs/ssh
* authentication is done via Github users' public keys. It's assumed
that gomote user == github user. But there's a mapping in the code
for known exceptions.
* we can't give out access to this too widely. too many things are
accessible from within the host environment if you look in the right
places. Details omitted. But the Go team and other trusted gomote
users can use this.
* the buildlet binary has a new /connect-ssh handler that acts like a
CONNECT request but instead of taking an explicit host:port, just
says "give me your machine's SSH connection". The buildlet can also
start sshd if needed for the environment. The /connect-ssh handler
also installs the coordinator's public key.
* a new buildlet client library method "ConnectSSH" hits the /connect-ssh
handler and returns a net.Conn.
* the coordinator's ssh.Handler is just running the OpenSSH ssh client.
* because the OpenSSH ssh child process can't connect to a net.Conn,
an emphemeral localhost port is created on the coordinator to proxy
between the ssh client and the net.Conn returned by ConnectSSH.
* The /connect-ssh handler requires http.Hijacker, which requires
fully compliant net.Conn implementations as of Go 1.8. So I needed
to flesh out revdial too, testing it with the
golang.org/x/net/nettest package.
* plan9 doesn't have an ssh server, so we use 0intro's new conterm
program (drawterm without GUI support) to connect to plan9 from the
coordinator ssh proxy instead of using the OpenSSH ssh client
binary.
* windows doesn't have an ssh server, so we enable the telnet service
and the coordinator ssh proxy uses telnet instead on the backend
on the private network. (There is a Windows ssh server but only in
new versions.)
Happy debugging over ssh!
Fixesgolang/go#19956
Change-Id: I80a62064c5f85af1f195f980c862ba29af4015f0
Reviewed-on: https://go-review.googlesource.com/50750
Reviewed-by: Herbie Ong <herbie@google.com>
Reviewed-by: Jessie Frazelle <me@jessfraz.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This moves getSourceTgz* to a new package called sourcecache. The
original functions couldn't be left in coordinator.go because that
would cause a single process to have multiple caches.
The code for building and running benchmarks is moved into the buildgo
package, and the public API becomes GoBuilder.EnumerateBenchmarks and
Run on the resulting BenchmarkItem objects.
Updates golang/go#19871
Change-Id: I28b660e1cdaa6d1c6b0378c08de30f5e58316cc6
Reviewed-on: https://go-review.googlesource.com/44211
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This moves RunMake from coordinator.go, allowing other packages to
build Go on a buildlet.
Updates golang/go#19871
Change-Id: Ic0e7c056020ef434f0620ae816f52b9112ea1c8d
Reviewed-on: https://go-review.googlesource.com/44176
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This package contains the BuilderRev type moved from cmd/coordinator.
The rest of the CL is simply updating coordinator with the new
exported names of the type and its fields.
This refactoring is in preparation for moving the benchmark building and running
code into a separate package.
(Most of the diff could have been avoided with a type alias, but I
assume we'd rather not do that.)
Updates golang/go#19871
Change-Id: Ib6ce49431c8529d6b4e72725d3cd652b9d0160db
Reviewed-on: https://go-review.googlesource.com/44175
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Total mess, but a functional mess, and a starting point for the future.
Change-Id: Ifcc72660f30a0ee2692668b4ea53acd1c4015777
Reviewed-on: https://go-review.googlesource.com/41077
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
The new untar package will be shared by the buildlet later.
Change-Id: I7a33bf46e5cb6b7e5abd59613bb943039c84aa08
Reviewed-on: https://go-review.googlesource.com/41131
Reviewed-by: Jeff Johnson <jrjohnson@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
And deal with Preemptible resource exhaustion errors.
And change all-compile to misc-compile and only do the builders
not covered otherwise (Fixes#11073)
And make the watcher serve git source.
And cache and singleflight fetching of git source.
And a million other things.
Fixesgolang/go#11073
Change-Id: I0f45610f0c6a06bd0c8ba9632b8624e00aeb52fc
Reviewed-on: https://go-review.googlesource.com/10750
Reviewed-by: Andrew Gerrand <adg@golang.org>
Brad Fitzpatrick
Elias Naur
Dmitri Shuralyov
Mikio Hara
Emmanuel T Odeke
Dmitri Shuralyov
Filippo Valsorda
Yury Smolsky
Agniva De Sarker
Andrew Bonventre
Emil Hessman
Michael Munday
Jeff Johnson
Sarah Adams
Joe Tsai
David Chase
Quentin Smith
Andrew Gerrand